Parameterized Pseudo-Differential Operators for Graph Convolutional Neural Networks
Authors: Kevin M. Potter
Steven Richard Sleder
Matthew David Smith
John Tencer
Venue:  (2021)
Keywords: graph convolutional neural network
superpixel
FAUST
differential operators

Abstract: We present a novel graph convolutional layer that is fast, conceptually simple, and provides high accuracy with reduced overfitting. Based on pseudo-differential operators, our layer operates on graphs with relative position information available for each pair of connected nodes. We evaluate our method on a variety of supervised learning tasks, including superpixel image classification using the MNIST, CIFAR10, and CIFAR100 superpixel datasets, node correspondence using the FAUST dataset, and shape classification using the ModelNet10  dataset. The new layer outperforms multiple recent architectures on superpixel image classification tasks using the MNIST and CIFAR100 superpixel datasets and performs comparably with recent results on the CIFAR10 superpixel dataset. We measure test accuracy without bias to the test set by selecting the model with the best training accuracy. The new layer achieves a test error rate of 0.80% on the MNIST superpixel dataset, beating the closest reported rate of 0.95% by a factor of more than 15%. After dropping roughly 70% of the edge connections from the input by performing a Delaunay triangulation, our model still achieves a competitive error rate of 1.04%.

Under review as a conference paper at ICLR 2021
PARAMETERIZED PSEUDO-DIFFERNNIILL OPERA- TORS FOR GRAPH CONVOLUTIONAL NEURAL NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
We present novel graph convolutional layer that is fast, conceptually simple, and provides high accuracy with reduced overfitting. Based on eseudo-different op- erators, our layer operates graphs with relative position information available for each pair of connected nodes. We evaluate our method on a variety super- vised learning tasks, including superpixel image classífication using the MNIST, CIFARIO, and CIFARI06 100 superpixel datasets, node correspondence using the FAUST dataset, and shape classification using the ModelNet10 dataset. The new layer outperforms multiple recent architectures on superpixel image classification tasks using the MNIST and CIFARI 100 superpixel datasets and performs compa- rably with recent results on the CIFARIO superpixel dataset. We measure test accuracy without bias to the ing accuracy. The new layer achieves a test error rate of 0.80% on the MNIST superpixel dataset, beating the closest reported rate of 0.95% by factor of more than 15%. After dropping roughly 70% of the edge connections from the input by performing a Delaunay triangulation, our model still achieves a competitive error rate of 1. 1.04%.
set by selecting the model with the best train-
INTRODUCTION
Convolutional neural networks have performed incredibly well on tasks such as image classification, segmentation, and object detection (Khan et al. 2020). While there have been diverse architectural design innovations leading to improved accuracies across these tasks, all of these tasks share the common property that they operate on structured Euclidean domain inputs. A growing body of research on how to transfer these successes into non-Euclidean domains, such as manifolds and graphs, has followed.
We focus unstructured graphs which represent discretizations of an underlying metric space. These data types are ubiquitous in computational physics, faceted surface meshes, and (with su- perpixel conversion) images. Previous efforts to extend CNNs this type of data have involved parameterized function approximations on localized neighborhoods, such as MoNet (Monti et al.) 2017) and SplineCNN (Fey et al. 2017). These function approximations (Gaussian mixture models in the case of MoNet and B-spline kernels in the case of SpineCNN) are complex and relatively expensive to calculate compared CNN kernels.
Inspired by earlier work in shape correspondence (Boscaini et al. 2016), image segmentation on the unit sphere (Jiang et al., 2019), and dimeesmiiiooal embeddings of computational physics data (Tencer & Potter (2020) we seek to utilize parameterized differential operators (PDOs) to construct convolution kernels. In contrast to MoNet and SpineCNN, parameterized differential operators are cheap to compute and involve only clementary operations. [Boscaini et al. (2016) used anisotropic diffusion kernels while Jiang et al. (2019) included gradient operators in addition to an isotropic diffusion operator. Tencer & Potter (2020) performed an ablation study of the differential operators used and demonstrated that the inc including the gradient operators is broadly beneficial, but that little is gained by including additional terms.
Prior work (Jiang et al. 2019 Tencer & Potter 2020) has used differential operators precomputed for specific meshes. This approach has two drawbacks: (1) precomputing operators is not practical

---

Under review as a conference paper at ICLR 2021
for datasets for which the connectivity graph varies between sample points, and (2) differential operators place restrictions on graph connectivity. Differential operators defined for mesh topologies rely on element connectivity information which is unavailable for more general graphs. Superpixel image datasets highlight both of these deficiencies. In contrast to these prior works, we do not precompute any operators and do not directly differential operators. Instead, we formulate psendo-differeniaal operators which are cheaply computed at run-time for more general class of graphs.
While our approach only applies to graphs with relative position information for each node, the set of graphs with the required positional information large, encompassing nearly physical systems as well as a significant number of other graphs, such as graph representations derived from image data.
Since our method relies on computing approximate spatial derivatives of nodal features, it is also important that these nodal values represent meaningfully changing field. This criteria not neces- sarily met for the node correspondence task the FAUST dataset or the shape classification task on the ModelNet10 dataset and a corresponding decrease in performance is observed. contrast, nodal features are critical to superpixel image classification tasks and our method is observed to perform well for these datasets.
Superpixel representations are popular for wide range of tasks, particularly tasks in which large data quantities make the direct application of CNNs to the raw data impractical, such as hyperspec- tral imaging (Hong et al.) 2020) and medical diagnostics (Roth al., 2015). For these applications, the superpixel representation serves as a sort of context aware lossy compression. [Kyyzzv et al. (2019) compared GCNs applied to superpixel images CNNs applied to low resolution images with approximately the same information content. In those cases, the graph methods not only held their own, but pulled ahead compared to the CNN performance. While those datasets were not at all pushing the limitations of image size, they indicate a possibility that superpixel methods might handle high resolution image data more efficiently and the value in developing methods that perform well on superpixel datasets.
Our method is especially well-suited for analyzing superpixel image representations in addition to being applicable to the datasets used by byjJian et al. (2019) Tnncer & Potter (2020) to demonstrate their PDO-based approaches. For regular meshes, such as the icosahedral spherical mesh used by Jiang et al.) (2019), our pseudo-differential operators closely approximate the differential operators used in those works.
1.1 OUR CONTRIBUTIONS
We created novel layer architecture inspired by PDOs.
We improve upon the static matrix approach of Tencer & Potter (2020) with dynamic method that enables support for variable graph forms and eliminates the need precompute matrices.
Our method utilizes pseudo-differentia operators in contrast to the differential operators used in prior works. Pseudo-differrntial operators are cheap to compute and are applicable to a broader class of graphs than differential operators.
Our novel mixing layer conceptually simple and easy code (integrating painlessly with existing graph libraries). (section 3.1
The new approach remains accurate for both sparsely and densely connected graphs, in- cluding state-of- oftee----heearr ttthee-rt results for the MNIST superpixel 75 dataset both with and without reduced edge connection input data. (section 4.1
The new approach is faster than common approaches for equivalent numbers of features owing to the simpler mathematical functions involved. (section 4.1.2)
2
PROBLEM
Many real world datasets may be treated as attributed graphs with positional information provided for the nodes relative positional information provided for the edges. In physics and engineer-
2

---

Under review as a conference paper at ICLR 2021
MNIST Superpixel 75 Accuracy
0.990
0.985
Test Accuracy
0.980
Ours 64 wide
Ours 256 wide
0.975
Ours 512 wide
Fey et al. (2017)
Knyazev et al. (2019)
0.970
Gray et al. (2020)
Best overall test accuracy
Test accuracy at best training
0.965
25
50
75
100 Time (minutes)
125
150
175
Figure 1: Accuracy compared to training time. While some of the other methods perform slightly better than our 7 layer deep model in time per epoch, we achieve higher accuracies per training minute (in addition to achieving higher overall accuracy). Of note, using a wider network is not penalty in accuracy training time. Plots show the mean test accuracy/time over runs except Gray et al. (2020), which shows the median results from 10 runs due to their higher variability.
ing, high-fidelity simulation tools represent continuous spatial functions using unstructured spatial discretizations. In computer vision, triangle surface meshes and super pixel image representations are common. [eeneer& Potter (2020) have recently used convolutional autoencoder architectures with PDOs to perform manifold learning datasets from computational physics. Their approach required precomputing matrices corresponding to various differential operators on a sequence of progressively coarser meshes. We sought to generalize their approach to support heterogeneous graphs without explicitly defined gradient operators and to avoid the need to precompute matrices coarsened spatial discretizations.
Our ideal solution, would be fast, simple, scale well, make efficient use information, and operate effectively sparsely connected graphs. While there number of existing approaches that will work for arbitrary graph information, all have some form of limitation that makes them ill suited for
cases. Whether ignoring graph connectivity information (Gray et al. 2020), non-existent use of position information (Kipf & Welling 2016 Berg et al. 2017). or complex formulations (Fey
al. 2017 Monti al. 2017). each has some weakness that made use less than ideal.
METHOD
Our layer works by calculating several quantities that are analogous to the sets differential op- erators used in Tencer & Potter (2020). For each node of layer, each input channel has multiple values calculated (4 items for 2-dimensional and for 3-dimensional graphs):
Value of the node in the prior layer (identity)
Average values the 1-ring neighbors of the node
Average gradient components (i.e. oy) by inverse edge length
9 of value along each connected edge weighted
These items then mixed by neural network yielding the desired number of output channels.
3

---

Under review as a conference paper at ICLR 2021
3.1 NOVEL LAYER
In contrast to previous methods utilizing PDOs, the method presented here generalizes to heteroge- neous datasets, in which the number of nodes, the connectivity, and positions vary across samples. Additionally, by relaxing the definitions of the gradient operators, our implementation is applicable to overconnected graphs (such as superpixel image representations) rather than only meshes suit- able for common PDE solution methods (finite element, finite volume, etc.). A consequence of this generalization is slightly more computational overhead from dynamically generating the required operators on the fly rather than precomputing them offline. However, as seen in Figure T this over- head does not result in slower training time relative to other methods.
The key components of Tencer & Potter (2200)'s s approach were the sparse matrix mixing, pool- ing, and interpolation. our implementation of the mixing layer, we used pytorch-geometric MessageP Assing base class [Fey & Lenssen (2019). For pooling layers, evaluated two of 's clustering method implementations: voxel_ ggriddiriiii and graclus (Dhillon et al. 2007; SSimonosky & Komodakis (017). Their voxel_ grrd implementation gave the best accuracy on the MNIST superpixel dataset.
The MessageP Passing ni class operates by calculating new values for each node on layer (k) with information from their prior node values z; (k-1) prior values at connected nodes z.j (k-1) and/or the edge attributes Ctj-
= (xx(
(x((--)) ( Denr (()) (xf(-1) xxf=-1) ...))
(1)
()eA(() is an aggregation method (such mean) that aggregates (f(k) for each node connected to node in layer (k-1) into a specific number of values independent of the number of connections. (00) and (*) are arbitrary functions.
Convenient and physically motivated choices for ((()) derived from mesh differential operators, e.g. 1, V. or A. The identity operator 1 simply passes the value forward. For the others, let f: F:2- + R be a smooth scalar function for which only the values fi, : fn. at the nodes are known and /i corresponds to the value of f at node i. The Laplacian operator A may be expressed as the difference between f; and the average value of fj.j E (i). For triangle meshes, the nodal gradient operator V is often expressed as the average gradient in the adjacent facets which is equivalent to weighted sum over the connected edge gradients
V R
Σ BEN(()
Wi.5 Wis V((e.))
with (e(s)) = fi-fe We denote the Euclidean distance between the two nodes as rigs and the unit vector oriented along the edge Cijj ē.jj- For 2D mesh without intersecting edges Wi, =
EjeNto A., where As, is the total area of the 2 facets connected to e, (Mancinelli et al,
Aij
2.19). For an overconnected graph (like MNIST superpixel), these facet areas are not defied.
If we weight the contributions of each edge equally, our choice for (A)) becomes
(k-1)
(k-1) Tz,io x
(k-1)
x, (k-1)
x
x
Ty.i.
(k-1)
x,
where Ta.i. and Ty. are the differences in positions of nodes and j in the x- and y-dimensions, respectively. We can easily extend these to dimensions (which was required for the FAUST dataset) by adding a z term, in which case, the parameter cost per channel goes up by only factor of 25%.
((k)) returns stack of these values for every input channel. The first 2 terms of equation 3 are x- and y-components of the gradient. The 3rd term is the average of the neighboring nodes, which,
when combined with the identity term z; (k-1) (which is concatenated after aggregation) results in the Laplacian. Given that the next step in the layer is mix these components via neural network (our (A) function), the Laplacian can be reconstructed by blending these terms if desirable.
Note that none of these values require complex calculations, which contributes to the layer's superior computational performance (section 4.1.2.. We hypothesize that by limiting the representational
4

---

Under review as a conference paper at ICLR 2021
space for each node such that it knows only about itself and the local gradient the network is forced to find simpler representations that are more likely to generalize.
4 EXPERIMENTS
We tested against variants of MNIST superpixel 75, both CIFARIO and CIFARI00 as superpixels, FAUST, and ModelNetI0 (Monti et al., 2017 Krizhevsky et al., 2009 Bogo et al., 2014 Wu et al.) 2015). For the remainder of this paper MNIST superpixel 75 will be referred to as MNIST. In the absence of validation sets, we selected test results based on the best training accuracy in order to avoid biasing our model selection to the test set. For our models this is often comparable, but is not necessarily the overall best test accuracy (figure D While we considered using k-fold cross vali- dation, we found that our method was effective in picking out generally applicable models without biasing to the test set. It was both conceptually simpler and easier to implement as well.
SplineCNN was taken from examples in early stopping to avoid overfitting. For fair comparison, their best overall test accuracy was chosen. For MNIST and CIFAR superpixel, we applied Gray et al. (2020) and Knyazev et al. (2019). We reported results using the test accuracies for the best training accuracy and note any deviations.
(Fey & Lenssen 2019) and utilized
The graph attention convolution layer (GATConv) (Velikovi et al. 2018) was taken from pytorch- geometric's s implementation. GATConv networks have been recently applied to the MNIST and CIFAR-10 superpixel benchmarks (Avelar et al.) 2020). For both MNIST and CIFAR-10, 10, the pub- lished GATConv results are inferior the other comparison models. We tested a variety of head and channel hyperparameters against a version of the SplineCNN model with the spline convolutions replaced with GATConv. We replicated their results for MNIST.
There is some variation in results across our model initializations. The trend of high test perfor- mance following high training set accuracy was robust. This allowed a bias-free way of selecting performant models from multiple runs of the same hyperparameters.
For all training runs, our models had training accuracies on par with test accuracies. Even MNIST had training results at or below test accuracy with appropriate levels of edge dropout (section 4.1.1
All models were trained with Adam optimization using a learning rate of 0.0002 and cross entropy loss for our architecture and the published options for comparison models unless otherwise noted. Learning rates were reduced by a factor of 10 on a plateau of 5 epochs without improving training loss up limit of 1 1/1000th the original learning rate.
4.1 MNIST SUPERPIXEL 75
We tested against 3 variants of the MNIST dataset (Monti et al. 2017). The first variant used all of the edges present in the original graph which call the raw dataset. The second, use variant of the hierarchical set used in Knyazev et al. (2019) with sets of approximately 75, 21, and 7 superpixels forming the feature hierarchy for each sample. The number of edges for each node was limited to its 32 closest by Euclidean distance as we are not focused on optimizing performance for minimal input sizes. Lastly, our pruned version was obtained by discarding all of the edge data and applying Delaunay triangulation to the nodes, which reduced the number of edge connections by around 70% on average.
Our architecture remained fairly consistent throughout testing. The model was comprised of be- tween 3 and 7 layers of downsampling modules followed by fully connected layers with an ex- ponential linear unit between the two. In the end, we selected 7 of our layers for all main results because it provided the best accuracy. Results for shallower networks are in appendix A Each downsampling module consisted of our introduced graph convolutional layer followed by voxel grid pooling operation outputting a reduced set of nodes with a selectable count of features. Pooling done with voxel size halved at each step. Initial voxel size is set such that after the 7 downsampling operations, a x 3 set of nodes is present prior to flattening for the fully connected layers.
No normalization methods were included in the downsampling layers (e.g. batch, dropout, edge dropout, etc.). Standard dropout of 0.5 was applied prior to all the fully connected layers. When used, we applied edge dropout to the data prior to model input.
5

---

Under review as a conference paper at ICLR 2021
Table 1: MNIST superpixel 75 image classification results. Our model and Fey al. (017)) were trained for 100 epochs on the MNIST dataset. Knyazev et al. (019) was trained for 30 epochs and Gray et al. (2020) for 400 epochs, keeping to their implementations. To avoid biasing to the test set, we selected the models by best training accuracy for a given set of hyperparameters with the exception of Fey et al. (0017) for which we chose the overall best for each particular run. Average and standard deviation values taken across 32 runs for the raw and hierarchical variants, and against 10 for the pruned variant. Given that Knyazev et al. (0199) was highly dependent on the extra information from their hierarchical representation, we did not run their code against the pruned version. Velikovi et al. (2018) produced graph attention convolutional layer but when applied to MNIST it performed poorly (and was worse without embedding positions into the input vector). |Fey et al. (0117) overfit badly when using the hierarchical dataset. Gray et al. (2020) reports an accuracy of 99. 99%0 on the raw superpixel dataset, which we were only able to replicate by taking the overall maximum test accuracy out of 5 runs, rather than the average.
Ours
Velikovi
Fey
Knyazev
Gray
Raw superpixel
99.12=0000 99.20
94.70=0.23 95.01 [
97.05 22 97.45
97.11=0.22 97.44
98.81=0.04
98.86
Hierarchical
99.04+-0.00 0.05 99.12
98.29- 0.21 98.55
98. 26 0.09
69.752
98.38
Pruned
98.78
96.9510.11
08.82=0.07
98.96
97.1
98.80
Our architecture with the best performance all the MNIST superpixel variants used 7 downsam- pling layers starting with 128 features, doubling each layer, until a maximum width of 512 features. Edges were dropped out on the input to prevent overfitting to the training set (section I.I.
Our architecture achieved a state-of-the the-art test error of 0.80% against the raw MNIST dataset using best training accuracy as selector among 32 runs. The average results for ours and several competitors are shown in T Our architecture achieved these results using an edge dropout rate of 0.45 and learning rate of 0.002, although runs with edge dropout rates between 0.3 and 0.55 were competitive or beating state-of-the-art in large number of the training runs (section 4.1.1, This compares to the prior best reported value of 0.95% from Gray et al. (2020). We were able to reproduce their result but only by selecting the best overall test accuracy out of runs. For the pruned variant, when choosing the model based on best training accuracy, our model's test error outperforms results from Gray et al. (2020) with an error rate of 1.04%.
For the hierarchical variant, we note slight drop in accuracy relative to the raw dataset. The hierarchical dataset adds additional nodes for various scales of abstraction (parent, grandparent, etc) with each level being progressively coarser. Knyazev et al. (2019) explicitly treats each of these levels differently, recognizing that the edge connection is between child-parent, siblings, etc. as part of its multigraph convolution approach. None of the other methods differentiate between these edge relationships. In our case, hypothesize that the extra information can cause issues with the Laplacian term as it does not have a distance weighting. The extra nodes have reduced quality information which believe can act confuser to the networks when the connection type is ignored. For Fey et al. (017)) the extra information causes extreme overfitting to the training set.
4.1.1 EDGE DROPOUT
To understand how reducing the edge connection through pruning impacted accuracy, we trained against the raw MNIST dataset with varied rates of edge dropout applied to the input data. As our method beats the comparison models on the raw dataset for a range of rates.
shown in Figure] 2a
At dropout rates above 0.6, orphaned nodes become much more likely and are significant portion of the nodes by 0.8 dropout. These orphaned nodes negatively impact our performance as the only data left to pass forward at each orphaned node is the identity function. Given that position information is
Test accuracy selected from best training epoch
Best overall test accuracy selected

---

Under review as a conference paper at ICLR 2021
Fey et al. (2017)
(2920)
Time per epoch (seconds)
150
20
Pruned
178
200
512
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Depths
Features
Edge Dropout
Edge dropout rate
(a) MNIST superpixel error vs edge dropout
(b) Hyperparameters effect on performance
Figure 2: (a) MNIST superpixel error rates as graph edges are dropped on input. Our model per- forms significantly better for range of edge dropout rates. Gray et al. (2020) use technique which ignores the input edges which leads to being unaffected by edge dropout. Fey et al. (2017) sees some benefit from low edge dropout rates (probably from reduced overfitting). (b) Effect of hyper- parameters on performance of our architecture. We modify a default set of parameters - 7 layers deep, 256 initial & max features, no pruning, and 0.5 edge dropout. Each was run for epochs n Vidia V100 and average times per epoch are shown.
required by our method, an appropriate and effective edge network can be acquired using Delaunay triangulation for 2D graphs. Pruning the MNIST graphs in this way results approximately 70% reduction in edges. Using the pruned MNIST dataset, prior reported state-of-the-art test accuracy (against models trained and evaluated using the raw superpixel dataset). When comparing test accuracy from the model selected by training accuracy, we outperform other models trained and tested using the raw dataset. We achieved an error rate of 1.04% using 0.1 edge dropout applied the pruned input graph shown in Table Because of this, the performance degradation for overly sparse graphs (high edge dropout rates) should not be practical limitation for method.
code performs competitively with
Moderate levels of input edge dropout seems to provide a data augmentation effect for the raw MNIST dataset, leading greater test accuracy and less overfitting (our train accuracies remain comparable to test when sufficient edge dropout is used). In all cases, our architectures performed better with some level of input edge dropout than with the original graph. In addition, reduced edge counts have a beneficial impact on training times (section 4.1.2).
Gray et al. (2020) uses novel method which ignores incoming edge information. Because of this, its accuracy is the same for all levels of edge dropout and pruning (within their normal variance). While this eliminates any accuracy penalty for overly sparse graphs, also eliminates the data augmentation and performance benefits of input edge dropout.
4.1.2 PERFORMANCE
Our network is faster per epoch of training time per layer basis. After adding layers create a deeper, wider, and more performant network, it remains competitive in training time and superior when looking at test accuracy achieved per training time as shown in Figure Surprisingly, adding extra width to the network did not worsen convergence time until a slight dip is observed going from 256 512 features wide. While each epoch takes longer, it converges faster.
We also show the impact of various hyperparameters on our network performance in Figure 2b Of note, our pruned dataset running with 0.05 edge dropout has similar performance to the 0.8 edge dropout without pruning. The pruned dataset has around 70-75% fewer edge connections than in the original which accounts for the increase in speed.

---

Under review as a conference paper at ICLR 2021
Table 2: (a) CIFARI00 superpixel image classification test accuracy (%). We show the average and maximum test accuracies for the best training epoch over 32 runs. (b) FAUST node correspon- dence results. All results (except ours) as reported by authors. We did not perform an extensive hyperparameter search and our results are mostly to show that the method is generally applicable.
(a) CIFARIOD accuracy
(b) Faust node correspondence
Model
Superpixel
Hierarchical
Model
Reported Accuracy
Ours
40.39-10.27 40.7 8
41.16::0.32 41.41 [
Boscaini et al.
(016
62.4%
Monti al 2017
73.8%
Sun et al. 2018)
96.9%
Knyazev
1.00
32.34:10.8
Verma et al.
2017
98.7%
31.4 |
33.54
Ours
99.20%
Gray
34.28 +4.284400.4
34.57:10.45
Fey al. (0177)
99.20%
34.8
35.19
Gong et al.
2019)
99.8%
Haan al.
(2020)
99.89% %
4.2 CIFAR 100 SUPERPIXELS
We also tested against CIFAR1O and CIFAR100 (Krizhevsky et al. 2009)
converted to superpixels.
The CIFARTOM 100 results are shown table 2a while the CIFARTO results are available in appendix [A] We achieved best accuracy of 40.71 % on the raw variant and 41.41% % on the hierarchical variant.
The CIFARIO0 superpixel dataset represents significantly more challenging learning task than either standard (non-superpixel) CIFARIO0 or MNIST. CIFAR 100 is a set of 32 x 32 color images across 100 categories. Each category has 600 images, broken into 500 for training and 100 for test. The superpixel and hierarchical variants are generated by applying SLIC transformations as described in Knyazev et al. (2019). Each image was constructed with approximately 150 superpixels with node edges restricted to only its 32 nearest neighbors by Fuclidean distance. The nodes were constructed from levels of approximately 150, 75, 21, and 7 superpixels in the same manner as the hierarchical MNIST dataset (section [..]p. The CIFARIO and CIFAR 100 superpixel experiments used the same architecture as used for the MÑIST experiments, with the exception of additional input channels for color superpixels and output channels for the larger number of classes in CIFAR
4.3 FAUST
We also tested our method on shape correspondence task using the FAUST (Bogo al. 2014) dataset, which contains 100 meshes with 6, 890 nodes each depicting 10 scanned human bodies in 10 different poses. Each node corresponds a particular part of each body and the task is identify which node corresponds to what body part. We used the standard 80/20 training/test split.
A modified version of the architecture used in MNIST experiments was used with 3-D gradients, 8 layers, 16 initial features doubling each layer to maximum of 128, and ending with 2 fully connected layers (dropout applied before each). Scaled exponential linear units were used as the ac- tivation function between each layer. No flattening or pooling operations were used. The final output a softmax with 6, 890 channels per node. Training used a batch size of 4, dropout of 0.3, learning rate of 0.01, and cross entropy loss. For this task, input edge dropout caused significant perfor- mance drop and was not used. Although no extensive hyperparameter tuning was performed, we achieved an accuracy performance comparable with recent results of 99.209.. shown in table 2b
4.4 MODELNET
We also test our method the shape classification task using the ModelNet10 (Wu et al.) 2015) dataset, which contains 4, 899 3D CAD models of objects from 10 categories. Our method performs worse this task, achieving middling classification accuracy of 88.6% on the test set (table 5 We attribute this poor performance to two factors. First, the MedelNet10 dataset does not include
Test accuracy selected from hest training epoch
8

---

Under review as a conference paper at ICLR 2021
nodal features. We attempted using the node positions and normal vectors as input features, but suspect that these were insufficient. Second, and perhaps more significantly, many of the meshes in the dataset contain large local variations in edge length. Consequently, approximate derivatives computed via pseudo-differential operators devalue contributions from distant nodes creating infor- mation bottlenecks in the network. This is, perhaps, not insurmountable obstacle, but resolving these issues is beyond the scope of this work.
5 RELATED WORK
Early approaches showed that CNNs could be used on non-Euclidean domains by introducing new intrinsic convolutional methods (Masci et al.] 2015 Boscaini et al. 2016) that operate on input manifolds. More recent approaches like Monti al. 2017) are capable of performing well on both manifolds and general graphs as input. Monti achieves this by creating pseudo-coordinates for either the vertices of graph or points manifold, and then learns kernel in that space.
Graph Convolutional Neural networks form the basis for other approaches that have shown great results (Zhang et al., 2019)) The literature has been split among methods based on spectral graph
theory (Bruna et al.) 2013; Henaff et al., 2015) Defferrard et al. 2016; Kipf & Welling 2016 Levie al. [217]] Wang et 20.11 Tencer Potter 2020) and methods that operate
al.
2017:Mont
with spatial filters (Micheli] |009 Atwood & Towsley 2015 Niepert et al., 2016) Fey et al. 2017:
Gilmer et al. 2017 Gray al. 2020 Velikovi et al. 2018). Our method falls into the latter category
of spatial approaches.
CONCLUSION
We introduced a simple graph convolutional layer that outperformed every published result we are aware of for superpixel image classification on the MNIST and CIFARI00 superpixel datasets. We demonstrated faster performance and reduced overfitting tendencies with the MNIST dataset as well. In addition. we tested our new layer on the CIFARIO superpixel image classification, FAUST shape correspondence, and ModelNet shape classification datasets. On CIFARIO superpixel and FAUST we achieved accuracies comparable to recent results without significant hyperparameter tuning. However, with ModelNet10 our layer only demonstrated moderate performance.
Input edge dropout often provided positive impacts our accuracy except at extreme values (sec- tion 4.1.1). The edge dropout provides a significant data augmentation effect with even moderate levels of edge dropout as the network sees combinatoric scale effect (relative to local node connec- tions). For sparse graphs, this had a minimal impact but was significant for highly connected graphs. A node with 10 connections and 2 of them dropped has 45 different combinations. In reality, the stochastic nature of the dropout process means that drops will not just be (29) in this case, but some combination of the various possibilities orders magnitude seems plausible for this scenario).
The performance drop witnessed for hierarchical data versus the raw suggests another area for re- search. Our current implementation did not take advantage of the additional edge information avail- able within the hierarchical dataset. This resulted in small but meaningful reduction in accuracy MNIST. The additional information introduced by Knyazev et al. (019)) seems particularly useful for large image
where it would be advantageous inform the network of the broader region. Our convolutional layer could be modified to incorporate this information.
While we have performed some hyperparameter tuning, there is still large degree of unexplored space. We believe that even for the nearly saturated MNIST dataset, there are still gains to be acquired (we had least one approximately 10% reduction in relative error late in testing). We will look to follow up on these questions in future work.
REFERENCES
James Atwood and Don Towsley. Diffission-convolutionna neural networks, 2015.
Pedro H. C. Avelar, Anderson R. Tavares, Thiago L. T. da Silveira, Cludio R. Jung, and Lus C. Lamb. Superpixel image classification with graph attention networks, 2020.

---

Under review as a conference paper at ICLR 2021
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv: 1xkiv:10600 706.00263, 2017.
Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794-3801, 2014.
Davide Boscaini, Jonathan Masci, Emanuele Rodol, and Michael M. Bronstein. Learning shape correspondence with anisotropic convolutional neural networks, 2016.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. 2013.
A. Cheraghian and L. Petersson. 3dcapsule: Extending the capsule architecture to classify 3d point clouds. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1194-1202, 2019,
doi: 10.1109/AAC.2019.000.1
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pp. 3844 3852, 2016.
Inderjit S Dhillon, Yuqiang Guan, and Brian Kulis. Weighted graph cuts without eigenvectors a multilevel approach. IEEE transactions on pattern analysis and machine intelligence, 29(11): 1944-1957, 2007.
2007.
M. Dominguez, R. Dhamdhere, A. Petkar, S. Jain, S. Sah, and R. Ptucha. General-purpose deep point cloud feature extractor. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1972-1981, 2018. doi: 10.1 1..1199//2220188000
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometrie. In ICLR Workshop Representation Learning on Graphs and Manifolds, 2019.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Mller. Splineenn: Fast geometric deep learning with continuous b-spline kernels. 2017.
A. Garcia, F. Gomez-Donose J. Carcia-Rodriguue S. Orts-Escolano, M. Cazorla, and Azorin-Lopez Pointnet: A 3d convolutional neural network for real-time object class recogni- tion. In 2016 International Joint Conference Neural Networks (IICNN), pp. 1578-15844 2016.
doi: 10. N0.T119/JCCNN2201..7777777
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, abs/1704.01212, 2017. URL http://axxx
erg/abs/1700 / 01212
Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. Spiralnet++:: A fast and highly efficient mesh convolution operator, 2019.
Lindsey Gray, Thomas Klijnsma, and Shamik Ghosh. A reduccion 2020.
reduction network for point clouds,
Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs, 2020.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. 2015.
D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot. Graph convolutional networks for hyperspectral image classification. IEEE Transactions Geoscience and Remote Sensing, pp. 1-13, 2020. doi: 10.1109/TGRS 2..11097TGSS2222000115555 2020.3015157.
Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Niessner. Spherical CNNs on unstructured grids. In International Conference on Learning Representations, 2019. URL httphhttaaaa//peeeeeiieeee het.ereeeewe..eeeeooooooeoaa///-
10

---

Under review as a conference paper at ICLR 2021
Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saced Qureshi. A survey of the recent architectures deep convolutional neural networks. Artificial Intelligence Review, 2020, doi:
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works, 2016.
Boris Knyazev, Xiao Lin, Mohamed R Amer, and Graham W Taylor. Image classification with hierarchical multigraph networks. arXiv preprint 1rXiv:1907.09000 009000, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Ron Levie, Federico Monti, Xavier Bresson, and Michael M. Bronstein. Cayleynets: Graph convo- lutional neural networks with complex rational spectral filters, 2017.
Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sequence: Learning the shape representation of 3d point clouds with an attention-basee sequence to sequence network, 2018.
Claudio Mancinelli. Marco Livesu, and Enrico Puppo. A comparison of methods for gradient field estimation on simplicial meshes. Computers and Graphics, 80, 03 2019. doi: 10. 101 wow.//..a...00000 03.005.
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pp. 37-45, 2015.
A. Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions Neural Networks, 20(3):498-55) 2009.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural net- works for graphs, 2016.
Holger R. Roth, Amal Farag, Le Lu, Evrim B. Turkbey, and Ronald M. Summers. Deep con- volutional networks for pancreas segmentation in CT imaging. In Sbastien Ourselin and Mar- tin A. Styner (eds.), Medical Imaging 2015: Image Processing, volume 9413, pp. 378 385. International Society for Opties and Photonics, SPIE, 2015. doi: 10.1117/12.2081420 URL
https: wwt.oorggg.loi..o. 1117/12 2081420
Martin Simonovsky and Nikos Komodakis. Dynamic adge-conditioned filters in convolutional neu- ral networks on graphs. CoRR, abs/1704.02201,, 2017. URL
.rg. abs/
7704. 02901
Zhiyu Sun, Ethan Rooke, Jerome Charton, Yusen He, Jia Lu, and Stephen Baek. Zemet: Convolu- tional neural networks on arbitrary surfaces zernike local tangent space estimation, 2018.
John Tencer and Kevin Potter. Enabling nonlinear manifold projection reduced-order models by extending convolutional neural networks to unstructured data. 2020.
Petar Velikovi, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li, and Yoshua Ben- gio. Graph attention networks, 2018.
Nitika Verma, Edmond Boyer, and Jakob Verbeek. Feastnet: Feature-steered graph convolutions for 3d shape analysis, 2017.
Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spectral graph convolution for point set fea- ture learning. In Proceedings of the European Conference on Computer Vision (ECCV), Septem- ber 2018.

---

Under review as a conference paper at ICLR 2021
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1912-1920, 2015.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Networks, 2019. doi: https://doi.or// 186/ 140649-019-0069.1
A APPENDIX
Table 3: MNIST image classification eπror rates. Models were trained for 100 epochs and results for test eиror are shown for both raw and Delaunay triangulated MNIST superpixel 75. To avoid biasing to the test set, we selected our model's 's test values by best training accuracy for a given set of hyperparameters. Our model training also included edge dropout on input with 0.05-0.1 for pruned and an optimized dropout rate for raw (higher depth networks needed more to avoid overfitting). Comparison models are MNIST examples from Fry Lenssen (2019) showing the performance of their b-spline based method with two pooling strategies. Gray et al. (0220) result is their reported value.
Ours
SplineCNN
Gray
Layers
3
w/ graclus
w/ voxel grid
Raw
4.88
2,89
0.80
2.98
2.74
0.95
Pruned
3.29
1.87
1.04
3.66
2.88
We show error rates for smaller depth networks of our model against MNIST in Table 3 Results for layer depth and used considerably narrower models than used in the layer version (initial width of 8). At 4 layers deep, our network was considerably faster per epoch than the 2 or 3 layer deep SplineCNN (Fey et al., 2017) Gray et al. (2020).
After further experiments showed the clear win for deeper networks, we did not go back and attempt train wide shallow networks. It's quite possible that wider, shallower networks would be able to perform comparably with recent results when properly tuned.
The CIFAR10 superpixel classification results shown in Table Of note, our method's training accuracy stays considerably closer to test results which is consistent with our experience on the other datasets. Training accuracies remaining (or even below) test accuracies was common for our models when sufficient edge dropout was used. The >> 10% difference here is considerably out of the norm when compared to MNIST.
In contrast to MNIST, the hierarchical variant for CIFARIO offered a considerable performance improvement for our network. Gray al.] (2020) manages beat our model's accuracy on the raw CIFAR10 10 superpixel dataset, but we win out on the hierarchical variant. Note that we had not adapted the network to properly use the hierarchical information, so is particularly surprising to see this result.
For the ModelNet10 results, we used a point cloud representation. Compared to other methods which used the same input data type, our method is competitive, but not exceptional. Minimal hyperparameter tuning was performed so the results should be considered a lower bound on possible performance for this dataset.
12

---

Under review as a conference paper at ICLR 2021
Table 4: CIFARIO image classification results. We were unable to replicate the reported results of Knyazev et al. (2019) using the code provided (they reported 69-73% accuracies depending on the variant). We believe that because we only applied these models naively without special tuning or modifications, they reasonable results and are included for comparison sake. Also, given the fact that all the models are performing at roughly the same level, it is possible that our recreation of the superpixel dataset suffers from some unknown flaw. The fact that the hierarchical variant shows improved performance for our model (in contrast to other results section suggests that there something odd with either the raw the hierarchical datasets.
Superpixel
Ours
Knyazev et al. 2019
Gray al. 2020
Train Accuracy (%)
90.02+0.35
91.85+0.09 60.24+13.18
99.89=#0.01
Test Accuracy (%)
6..0
67.65+667.99
Hierarchical
Train Accuracy
86. 16.18:10.41
94.28:10.13
99.85+10.01
Test Accuracy
68.94:10.42
63.74
63.96:1011
Table 5: ModelNet10 shape classification results.
Model
Reported Accuracy
Garcia- et al. (0016)
77.6%
Ours
86.
Dominguez et al. 2018
93.1%
Cheraghian Petersson 2019
94.7%
Liu et al. (2018)
95.3%
13


Reviews:
[{"review_id": "ARtPk3J8jC0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel graph convolutional layer inspired by the pseudo-differential operator. The proposed conv layer is based on Eqs.(1-3), which is  inspired by the differential operators such as gradient and laplacian operator. The paper claims that this proposed convolutional layer enables to achieve state-of-the-art results on super-pixel image classification. \n\nMy major concerns on the work are the limited novelties, insufficient comparisons, and limited significance on the application of super-pixel image classification. \n\n1.  There are several different designs in local convolution over graph nodes in previous works,  such as graph attention convolution, etc. What are the major advantages and novelties of the proposed convolution operators, compared with these previous local convolution designs?\n\n2.  The paper applied the proposed GCN to super-pixel 2D image classification. I have concerns on this application, because the super-pixel-based image recognition using GCN is not a dominant approach in 2D image recognition,  compared with other recognition networks based on regular pixel grids. I suggest that more comparisons on benchmark should be compared, e.g., the 3D shape recognition datasets, or point cloud datasets.\n\n3.  In table I and  II, on the results comparisons on superpixel image recognition, the paper mainly compared with the methods of Knyazev and Gray. I am curious on more comparisons with other GCNs on these datasets if the codes are available. \n\n4. The experimental results show that the proposed network works good on super-pixel image recognition with reduced over-fitting. What are the fundamental reasons that the proposed graph conv. layer is more effective? \n\n\n---\nPost rebuttal comments：\nThanks for the responses. After reading these responses and other reviews, I still has concerns on the justification of proposed convolution compared with other popular graph convolutions, and also the limited experiments on superpixel image recognition.", "extra_scores": {}, "extra_text": {}}, {"review_id": "24n_on_iHDQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new graph convolutional layer for graphs with relative position encoding. These types of graphs occur in applications such as meshes, point clouds, and super-pixel neighborhoods. The method extremely simple and generic.\n\nStrengths:\n+ The method is simple and can be easily integrated into existing frameworks. \n+ The method is more general than approaches designed directly for mesh processing\n+ Despite the simplicity, the paper performs well on the Faust node correspondence task. The results are close to that of recent approaches (Haan et al. 2020, Gong et al. 2019) which where directly designed for meshes, while this approach is more general. \n\nWeaknesses:\n- Most of the experiments were devoted to super-pixel image classification, MNIST and CIfFAR which isn't a very compelling use case for this type of method. The paper would be stronger with more experiments on datasets like Faust where the data is not grid structured. It would be interesting to see results on point cloud data, where graphs could be constructed using KNN or other methods. \n- Limited ablations. The features for message passing are derived from spatial gradients, it would be useful to know which operators are necessary. To my knowledge, without the gradient operators, the network reduces to something more like a generic graph convolutional network. Other parts of the paper say things like \"\n- The term psuedo differentiable operators isn't defined in the introduction which makes it difficult to understand how this paper relates to other work. It is not clear from to me the novelty compared to (Tencer & Potter, 2020)\n\nMinor Points:\n* The paper mentions trying voxel_grid and graclus but does not discuss what theses operations are for\n* The organization of the paper could be improved. Experimental details are scattered throughout the paper. Implementation details should be moved from the method sections to the experiments section.\n\nConclusion\nThe proposed method is simple and seems to work well on super-pixel and mesh processing tasks. However, limited experiments make it difficult to assess the generality of the method to other tasks\n\nPost-Rebuttal Update: The response from the authors addressed several of my concerns and several clarity issues where fixed in the update paper. However, I don't think the results on ModelNet10 provide strong support for this method. While I don't think it is reasonable to expect this method to outperform other works which are specifically designed for mesh/point cloud inputs (given that this method is more general), I think there needs to be some application outside of super-pixel classification where the proposed method shows an clear advantage.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-HhBuJyL-rt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper defines simple differential operators at nodes in a graph (gradient, Laplacian) and uses them in the proposed graph convolutional layer. The claim is that simple operators will limit the representation power of the layer leading to better generalization. While this might be true at some level, it goes against the trends in deep learning which move away from such predefined constraints on the representation power of models. \n\nThe paper is motivated by inherently non-Euclidean domains but mostly uses super-pixel representations of images for experiments which forces inherently Euclidean data into a non-Euclidean representation. Perhaps not surprisingly, while beating the state of the art in graph convolutional neural nets (CNN) on MNIST/CIFAR, the accuracy is still far from regular CNNs for image classification. I wonder if the choice to use the image data comes from the need to have positional information for the nodes of the graph and how well this requirement is met by other applications where the data is inherently non-Euclidean? The introduction only mentions image data as fitting this requirement. This should be further discussed. \n\nPerhaps a better way to demonstrate the claim that constraining the network to simple representations at each node leads to better generalization would be in the context of limited training data (semi-supervised learning, few shot learning etc). On the other hand, the accuracy of the proposed method is behind the state-of-the-art on the only inherently non-Euclidean dataset used (FAUST) which can be considered limited data (100 examples in total).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers expressed consistent concerns on this submission in their reviews. In addition, none of them enthusiastically supported this work during discussion. It is clear this submission does not make the bar of ICLR. Thus a reject is recommended.
Decision: Reject