Parameterized Pseudo-Differential Operators for Graph Convolutional Neural Networks

1 INTRODUCTION . Convolutional neural networks have performed incredibly well on tasks such as image classification , segmentation , and object detection ( Khan et al. , 2020 ) . While there have been diverse architectural design innovations leading to improved accuracies across these tasks , all of these tasks share the common property that they operate on structured Euclidean domain inputs . A growing body of research on how to transfer these successes into non-Euclidean domains , such as manifolds and graphs , has followed . We focus on unstructured graphs which represent discretizations of an underlying metric space . These data types are ubiquitous in computational physics , faceted surface meshes , and ( with superpixel conversion ) images . Previous efforts to extend CNNs to this type of data have involved parameterized function approximations on localized neighborhoods , such as MoNet ( Monti et al. , 2017 ) and SplineCNN ( Fey et al. , 2017 ) . These function approximations ( Gaussian mixture models in the case of MoNet and B-spline kernels in the case of SpineCNN ) are complex and relatively expensive to calculate compared to CNN kernels . Inspired by earlier work in shape correspondence ( Boscaini et al. , 2016 ) , image segmentation on the unit sphere ( Jiang et al. , 2019 ) , and low-dimensional embeddings of computational physics data ( Tencer & Potter , 2020 ) we seek to utilize parameterized differential operators ( PDOs ) to construct convolution kernels . In contrast to MoNet and SpineCNN , parameterized differential operators are cheap to compute and involve only elementary operations . Boscaini et al . ( 2016 ) used anisotropic diffusion kernels while Jiang et al . ( 2019 ) included gradient operators in addition to an isotropic diffusion operator . Tencer & Potter ( 2020 ) performed an ablation study of the differential operators used and demonstrated that the including the gradient operators is broadly beneficial , but that little is gained by including additional terms . Prior work ( Jiang et al. , 2019 ; Tencer & Potter , 2020 ) has used differential operators precomputed for specific meshes . This approach has two drawbacks : ( 1 ) precomputing operators is not practical for datasets for which the connectivity graph varies between sample points , and ( 2 ) differential operators place restrictions on graph connectivity . Differential operators defined for mesh topologies rely on element connectivity information which is unavailable for more general graphs . Superpixel image datasets highlight both of these deficiencies . In contrast to these prior works , we do not precompute any operators and we do not directly use differential operators . Instead , we formulate pseudo-differential operators which are cheaply computed at run-time for a more general class of graphs . While our approach only applies to graphs with relative position information for each node , the set of graphs with the required positional information is large , encompassing nearly all physical systems as well as a significant number of other graphs , such as graph representations derived from image data . Since our method relies on computing approximate spatial derivatives of nodal features , it is also important that these nodal values represent a meaningfully changing field . This criteria is not necessarily met for the node correspondence task on the FAUST dataset or the shape classification task on the ModelNet10 dataset and a corresponding decrease in performance is observed . In contrast , nodal features are critical to superpixel image classification tasks and our method is observed to perform well for these datasets . Superpixel representations are popular for a wide range of tasks , particularly tasks in which large data quantities make the direct application of CNNs to the raw data impractical , such as hyperspectral imaging ( Hong et al. , 2020 ) and medical diagnostics ( Roth et al. , 2015 ) . For these applications , the superpixel representation serves as a sort of context aware lossy compression . Knyazev et al . ( 2019 ) compared GCNs applied to superpixel images to CNNs applied to low resolution images with approximately the same information content . In those cases , the graph methods not only held their own , but pulled ahead compared to the CNN performance . While those datasets were not at all pushing the limitations of image size , they indicate a possibility that superpixel methods might handle high resolution image data more efficiently and the value in developing methods that perform well on superpixel datasets . Our method is especially well-suited for analyzing superpixel image representations in addition to being applicable to the datasets used by Jiang et al . ( 2019 ) and Tencer & Potter ( 2020 ) to demonstrate their PDO-based approaches . For regular meshes , such as the icosahedral spherical mesh used by Jiang et al . ( 2019 ) , our pseudo-differential operators closely approximate the differential operators used in those works . 1.1 OUR CONTRIBUTIONS . We created a novel layer architecture inspired by PDOs . • We improve upon the static matrix approach of Tencer & Potter ( 2020 ) with a dynamic method that enables support for variable graph forms and eliminates the need to precompute matrices . • Our method utilizes pseudo-differential operators in contrast to the differential operators used in prior works . Pseudo-differential operators are cheap to compute and are applicable to a broader class of graphs than differential operators . • Our novel mixing layer is conceptually simple and easy to code ( integrating painlessly with existing graph libraries ) . ( section 3.1 ) • The new approach remains accurate for both sparsely and densely connected graphs , including state-of-the-art results for the MNIST superpixel 75 dataset both with and without reduced edge connection input data . ( section 4.1 ) • The new approach is faster than common approaches for equivalent numbers of features owing to the simpler mathematical functions involved . ( section 4.1.2 ) 2 PROBLEM . Many real world datasets may be treated as attributed graphs with positional information provided for the nodes or relative positional information provided for the edges . In physics and engineer- ing , high-fidelity simulation tools represent continuous spatial functions using unstructured spatial discretizations . In computer vision , triangle surface meshes and super pixel image representations are common . Tencer & Potter ( 2020 ) have recently used convolutional autoencoder architectures with PDOs to perform manifold learning on datasets from computational physics . Their approach required precomputing matrices corresponding to various differential operators on a sequence of progressively coarser meshes . We sought to generalize their approach to support heterogeneous graphs without explicitly defined gradient operators and to avoid the need to precompute matrices or coarsened spatial discretizations . Our ideal solution , would be fast , simple , scale well , make efficient use of information , and operate effectively on sparsely connected graphs . While there are a number of existing approaches that will work for arbitrary graph information , all have some form of limitation that makes them ill suited for our use cases . Whether ignoring graph connectivity information ( Gray et al. , 2020 ) , non-existent use of position information ( Kipf & Welling , 2016 ; Berg et al. , 2017 ) , or complex formulations ( Fey et al. , 2017 ; Monti et al. , 2017 ) , each has some weakness that made its use less than ideal . 3 METHOD . Our layer works by calculating several quantities that are analogous to the sets of differential operators used in Tencer & Potter ( 2020 ) . For each node of a layer , each input channel has multiple values calculated ( 4 items for 2-dimensional and 5 for 3-dimensional graphs ) : • Value of the node in the prior layer ( identity ) • Average values at the 1-ring neighbors of the node • Average gradient components ( i.e . ∂∂x , ∂ ∂y ) of value along each connected edge weighted by inverse edge length These items are then mixed by a neural network yielding the desired number of output channels . 3.1 NOVEL LAYER . In contrast to previous methods utilizing PDOs , the method presented here generalizes to heterogeneous datasets , in which the number of nodes , the connectivity , and positions vary across samples . Additionally , by relaxing the definitions of the gradient operators , our implementation is applicable to overconnected graphs ( such as superpixel image representations ) rather than only meshes suitable for common PDE solution methods ( finite element , finite volume , etc. ) . A consequence of this generalization is slightly more computational overhead from dynamically generating the required operators on the fly rather than precomputing them offline . However , as seen in Figure 1 , this overhead does not result in slower training time relative to other methods . The key components of Tencer & Potter ( 2020 ) ’ s approach were the sparse matrix mixing , pooling , and interpolation . In our implementation of the mixing layer , we used pytorch-geometric ’ s MessagePassing base class Fey & Lenssen ( 2019 ) . For pooling layers , we evaluated two of pytorch-geometric ’ s clustering method implementations : voxel_grid and graclus ( Dhillon et al. , 2007 ; Simonovsky & Komodakis , 2017 ) . Their voxel_grid implementation gave the best accuracy on the MNIST superpixel dataset . The MessagePassing class operates by calculating new values for each node i on layer ( k ) with information from their prior node values x ( k−1 ) i , prior values at connected nodes x ( k−1 ) j , and/or the edge attributes ei , j . x ( k ) i = γ ( k ) ( x ( k−1 ) i , j∈N ( i ) φ ( k ) ( x ( k−1 ) i , x ( k−1 ) j , ei , j ) ) ( 1 ) j∈N ( i ) is an aggregation method ( such as mean ) that aggregates φ ( k ) for each node j connected to node i in layer ( k−1 ) into a specific number of values independent of the number of connections . φ ( k ) and γ ( k ) are arbitrary functions . Convenient and physically motivated choices for φ ( k ) are derived from mesh differential operators , e.g . I , ∇ , or ∆ . The identity operator I simply passes the value forward . For the others , let f : Ω→ R be a smooth scalar function for which only the values f1 , . . . , fn at the nodes are known and fi corresponds to the value of f at node i . The Laplacian operator ∆ may be expressed as the difference between fi and the average value of fj , j ∈ N ( i ) . For triangle meshes , the nodal gradient operator ∇ is often expressed as the average gradient in the adjacent facets which is equivalent to a weighted sum over the connected edge gradients ∇fi ≈ ∑ vj∈N ( i ) wi , j∇f ( ei , j ) . ( 2 ) with ∇f ( ei , j ) = fi−fjri , j êi , j . We denote the Euclidean distance between the two nodes as ri , j , and the unit vector oriented along the edge ei , j as êi , j . For a 2D mesh without intersecting edges wi , j = Ai , j∑ j∈N ( i ) Ai , j , where Ai , j is the total area of the 2 facets connected to ei , j ( Mancinelli et al. , 2019 ) . For an overconnected graph ( like MNIST superpixel ) , these facet areas are not defined . If we weight the contributions of each edge equally , our choice for φ ( k ) becomes ( x ( k−1 ) i − x ( k−1 ) j ) rx , i , j r2i , j , ( x ( k−1 ) i − x ( k−1 ) j ) ry , i , j r2i , j , x ( k−1 ) j , ( 3 ) where rx , i , j and ry , i , j are the differences in positions of nodes i and j in the x- and y-dimensions , respectively . We can easily extend these to 3 dimensions ( which was required for the FAUST dataset ) by adding a z term , in which case , the parameter cost per channel goes up by only a factor of 25 % . φ ( k ) returns a stack of these values for every input channel . The first 2 terms of equation 3 are xand y-components of the gradient . The 3rd term is the average of the neighboring nodes , which , when combined with the identity term x ( k−1 ) i ( which is concatenated after aggregation ) results in the Laplacian . Given that the next step in the layer is to mix these components via a neural network ( our γ ( k ) function ) , the Laplacian can be reconstructed by blending these terms if desirable . Note that none of these values require complex calculations , which contributes to the layer ’ s superior computational performance ( section 4.1.2 ) . We hypothesize that by limiting the representational space for each node such that it knows only about itself and the local gradient the network is forced to find simpler representations that are more likely to generalize .

Summary: The paper defines simple differential operators at nodes in a graph (gradient, Laplacian) and uses them in the proposed graph convolutional layer. The claim is that simple operators will limit the representation power of the layer leading to better generalization. While this might be true at some level, it goes against the trends in deep learning which move away from such predefined constraints on the representation power of models.