Title: LEARNING GRAPH CONVOLUTION FILTERS FROM DATA MANIFOLD

Abstract: Convolution Neural Network (CNN) has gained tremendous success in computer vision tasks with its outstanding ability to capture the local latent features. Recently, there has been an increasing interest in extending CNNs to the general spatial domain. Although various types of graph convolution and geometric convolution methods have been proposed, their connections to traditional 2D-convolution are not well-understood. In this paper, we show that depthwise separable convolution is a path to unify the two kinds of convolution methods in one mathematical view, based on which we derive a novel Depthwise Separable Graph Convolution that subsumes existing graph convolution methods as special cases of our formulation. Experiments show that the proposed approach consistently outperforms other graph convolution and geometric convolution baselines on benchmark datasets in multiple domains.

1. INTRODUCTION:

Convolution Neural Network (CNN) (LeCun et al., 1995) has been proven to be an efficient model family in extracting hierarchical local patterns from grid-structured data, which has significantly advanced the state-of-the-art performance of a wide range of machine learning tasks, including image classification, object detection and audio recognition (LeCun et al., 2015) . Recently, growing attention has been paid to dealing with data with an underlying graph/non-Euclidean structure, such as prediction tasks in sensor networks (Xingjian et al., 2015) , transportation systems (Li et al., 2017) , and 3D shape correspondence application in the computation graphics (Bronstein et al., 2017) . How to replicate the success of CNNs for manifold-structured data remains an open challenge.

Many graph convolution and geometric convolution methods have been proposed recently. The spectral convolution methods (Bruna et al., 2013; Defferrard et al., 2016; Kipf & Welling, 2016) are the mainstream algorithm developed as the graph convolution methods. Because their theory is based on the graph Fourier analysis (Shuman et al., 2013) , one of their major limitations is that in this model the knowledge learned from different graphs is not transferrable (Monti et al., 2016) . Other group of approaches is geometric convolution methods, which focuses on various ways to leverage spatial information about nodes (Masci et al., 2015; Boscaini et al., 2016; Monti et al., 2016) . Existing models mentioned above are either not capable of capturing spatial-wise local information as in the standard convolution, or tend to have very large parameter space and hence, are prone to overfitting. As a result, both the spectral and the geometric convolution methods have not produced the results comparable to CNNs on related tasks. Such a misalignment makes it harder to leverage the rapidly developing 2D-convolution techniques in the generic spatial domain. We note graph convolution methods are also widely used in the pure graph structure data, like citation networks and social networks (Kipf & Welling, 2016) . Our paper will only focus on the data with the spatial information.

In this paper, we provide a unified view of the graph convolution and traditional 2D-convolution methods with the label propagation process (Zhu et al., 2003) . It helps us better understand and compare the difference between them. Based on it, we propose a novel Depthwise Separable Graph Convolution (DSGC), which inherits the strength of depthwise separable convolution that has been extensively used in different state-of-the-art image classification frameworks including Inception Network (Szegedy et al., 2016 ), Xception Network (Chollet, 2016) and MobileNet (Howard et al., 2017) . Compared with previous graph and geometric methods, the DSGC is more expressive and aligns closer to the depthwise separable convolution network, and shares the desirable characteristic of small parameter size as in the depthwise separable convolution. In experiments section, we evaluate the DSGC and baselines in three different machine learning tasks. The experiment results show that the performance of the proposed method is close to the standard convolution network in the image classification task on CIFAR dataset. And it outperforms previous graph convolution and geometric convolution methods in all tasks. Furthermore, we demonstrate that the proposed method can easily leverage the advanced technique developed for the standard convolution network to enhance the model performance, such as the Inception module (Szegedy et al., 2016) , the DenseNet architecture (Huang et al., 2016) and the Squeeze-and-Excitation block (Hu et al., 2017) .

The main contribution of this paper is threefold:

• A unified view of traditional 2D-convolution and graph convolution methods by introducing depthwise separable convolution.

• A novel Depthwise Separable Graph Convolution (DSGC) for spatial domain data.

• We demonstrate the efficiency of the DSGC with extensive experiments and show that it can facilitate the advanced technique of the standard convolution network to improve the model performance.

2. A GRAPH PERSPECTIVE OF CONVOLUTION:

We provide a unified view of label propagation and graph convolution by showing that they are different ways to aggregate local information over the graphs or data manifolds. We then discuss connections between graph convolution and depthwise separable convolution over the 2D-grid graph, which motivates us to propose a new formulation that subsumes both methods as special cases.

Unless otherwise specified, we denote a matrix by X, the i-th row in the matrix by x i , and (i, j)-th element in the matrix by x ij . Superscripts are used to distinguish different matrices when necessary. All the operations being discussed below can be viewed as a function that transforms input feature maps X ∈ R N ×P to output feature maps Y ∈ R N ×Q , where N is the number of nodes in the graph and P, Q are the number of input and features (channels) associated with each node respectively. We use N (i) to denote the set of neighbors for i-th node.

2.1. LABEL PROPAGATION:

Label propagation (LP) (Zhu et al., 2003) is a classic approach to aggregate local information over a graph. The basic version of LP can be written as

y iq = j∈N (i) w ij x jq (1)

where W is a normalized adjacency matrix that summarizes the graph structure. The intuition is that the value of node i is updated via a weighted combination of its neighbors.

2.2. GRAPH CONVOLUTION:

Graph convolution (Kipf & Welling, 2016) (GC) is a recently proposed graph convolution operator that can be viewed as an extension of LP, formulated as

y iq = j∈N (i) w ij z jq where z j = U x j (2)

where W is a symmetrically normalized adjacency matrix with a ridge on its diagonal, which is a deterministic matrix given the input data, and U ∈ R P ×Q represents a linear transformation. Following the Chollet (2016), W is named as the spatial filter and U is named as the channel filter. The original form of graph convolution, such as the Spectral Network (Bruna et al., 2013) , is derived from graph signal processing (Shuman et al., 2013) as a generalization of Fourier analysis to the domain of graphs. Several limitations of the Spectral Network, such as its high computation To Compare LP with GC, the former only utilizes the graphical information, while the latter has an additional linear transformation of x j to into the intermediate representation z j via matrix U . This additional step makes GC capable of capturing the dependencies among features (channels), which yields performance improvement.

2.3. DEPTHWISE SEPARABLE CONVOLUTION:

For a full 2d-convolution layer, the convolution filters encode channel correlation and spatial correlation simultaneously (Chollet, 2016) . Then depthwise separable convolution (DSC) is proposed under the intuition that the channel correlation and spatial correlation could be decoupled, and has been found successful in several modern architectures for image classification (Chollet, 2016) . We choose to focus on DSC (instead of full convolution) because of its strong empirical performance with a small number of parameters, and its intimate connections to GC which will be revealed in the following. And we discuss the full convolution formulation with the label propagation process in Section 5.

By viewing each pixel in the image as a node, DSC can be formulated in a graph-based fashion

EQUATION

where ∆ ij denotes the relative position of pixel i and pixel j on the image, and w (q) can be viewed as a lookup table with the pixel-pixel offset ∆ ij as the key, according to the stationarity (weightsharing) assumption of convolution. In the context of images, N (i) denotes the index set of surrounding pixels for i-th pixel, which is equivalent to the k-nearest neighbor set under the Euclidean distant metric. For example, the size of N (i), or k, is 9 for a 3 × 3 convolution filter (considering self-loop).

3. PROPOSED METHOD:



3.1. DEPTHWISE SEPARABLE GRAPH CONVOLUTION:

We notice that the formulation of GC and DSC is similar except that 1. Spatial filters in DSC are channel-specific, while GC uses a global spatial filter.

2. Spatial filters in DSC are learned from the data (under the stationarity constraints), while the filter in GC is a constant matrix with the given input.

On the one hand, DSC does not apply to the domain of generic spatial data lying on the manifold where the space of ∆ ij (defined as the difference of the spatial coordinates between node i and node j) can be infinite. On the other hand, GC suffers from the restriction that all channels have to share the same given spatial filter. This heavily constrains the model capacity, which would be more severe when the deeper network structure is used. In the context of graphs, it would be desirable to have multiple spatial filters-to capture a diverse set of diffusion patterns over the graph or data manifold, which is the same as the convolution filters in the image domain.

To address these limitations, we propose Depthwise Separable Graph Convolution (DSGC) which naturally generalizes both GC and DSC

y iq = j∈N (i) w (q) (∆ ij )z jq where z j = U x j (4)

where we slightly abuse the notation by overloading w (q) (•) as a function, which maps ∆ ij to a real number, and N (i) still represents the k-nearest neighbor sets. To understand the proposed formulation, notice 1. Different from DSC, the stationarity requirement is implemented in a "soft" manner by defining a function instead of by the set of equality constraints. In our experiment, each w (q) (•) is a function parameterized by a two-layer MLP.

2. Different from GC, channel-specific convolution is enabled by learning multiple spatial convolution filters. This amounts to simultaneously constructing multiple graphs under the different node-node similarity metrices, where the metrices are implicitly defined by neural networks and hence, are jointly optimized during the training.

Overfitting is a common issue in graph-based applications, due to limited data available. To alleviate this issue, we propose an option to group the channels into C groups, where D = Q/C channels in the same group would share the same filter.

w (q) (•) = w (q ) (•) if q D = q D (5) 3.2 NORMALIZATION

The context of each node in any given generic graph, namely its connection pattern with neighbors, can be non-stationary over different parts of the graph, while it is constant in the 2d-grid graphs.

It is, therefore, a common practice to normalize the adjacency matrix in order to make the nodes adaptive to their own contexts (Eq.1). A natural way to carry out normalization for DSGC is to apply a softmax function over the predicted spatial filter weights at each node, which can be written as wi = sof tmax(w i ), where w i stands for the i-th row of spatial filter W learned by a neural network. We empirically find normalization leads to better performance and significantly speeds up the convergence.

In the following experiments, we use the proposed depthwise separable graph convolution with a linear highway bypass as the basic convolution component and imitate the rest setting of the standard convolution neural network to solve different machine learning tasks.

4. EXPERIMENTS:



4.1. EXPERIMENT SETTING:

We evaluate the proposed Depthwise Separable Graph Convolution (DSGC) method with representative baselines in the prediction tasks of image classification, time series forecasting, and document categorization. The algorithms are implemented in PyTorch; all the data and the code are made publicly accessible 1 . For controlled experiments, all the graph convolution methods share the same empirical settings unless otherwise specified, including network structures, the dimension of latent factors, and so on. The optimization algorithm is applied to all models. The neural network used to model the spatial convolution filter (w (q) (•)) in Eq.4 is a two-layers MLP with 256 hidden dimension and tanh activation function. We have conducted ablation tests with the two-layer MLP by changing the number of layers and activation function of each hidden layer, and by trying several weight sharing strategies. The results are very similar; the two-layer MLP provides a reasonable performance with the shortest running time. Appendix A contains more details, such as the network architecture and model hyper-parameters.

4.2. EVALUATION ON IMAGE CLASSIFICATION:

We conduct experiments on CIFAR10 and CIFAR100 (Krizhevsky & Hinton, 2009) , which are popular benchmark datasets in image classification. Both sets contain 60000 images with 32 × 32 pixels but CIFAR10 has 10 category labels and CIFAR100 has 100 category labels. Each image is typically treated as a 32 × 32 grid structure for standard image-based convolution. To enable the comparison on generic graphs, we create the modified versions of CIFAR10 and CIFAR100, respectively, by subsampling only 25% of the pixels from each graph. As illustrated in Figure 2 , the subsampling results in irregularly scattered nodes for each image. For comparison we include the traditional 2d convolution and graph convolution networks as baselines, including standard CNN; Xception network (Chollet, 2016) which uses the depthwise separable convolution; DCNN (Atwood & Towsley, 2016) , the method using multi-hops random walk as the graph filters; ChebyNet (Defferrard et al., 2016) , the method using Chebyshev polynomial to approximate the Fourier transformation of (irregular) graphs; GCN (Kipf & Welling, 2016) which is described in Section 2; MoNet (Monti et al., 2016) , the method using Gaussian function to define the propagation weights over (irregular) graphs. For a fair comparison, we use the VGG13 architecture (Simonyan & Zisserman, 2014) in all the methods above as the basic platform, and replace the convolution layers according to the methods. The pooling layer is performed by the kmean clustering. The centroid of each clusters is regarded as the new node after pooling, and its hidden vector is the mean or max over the nodes in the cluster, based on the pooling method. Notice that, we only normalize the input signals to [0,1] and do not have other preprocessing or data augmentation.

The experiment results are summarized in Table 1 . Firstly, we observe that Xception and CNN have the best results; this is not surprising because both methods use grid-based convolution which is naturally suitable for image recognition. Secondly, DSGC outperforms all the other graph-based convolution methods, and its performance is very close to that of the grid-based convolution methods. Furthermore, contributed by the depthwise separable convolution and sharing graph technique, our model can achieve the competitive performance without increasing the number of parameters as GCN, the one with the smallest number of parameters among the graph convolution approaches. In appendix A.4, we further report the variance of DSGC model, which shows the improvement is significant and stable.

4.3. EVALUATION ON TIME SERIES FORECASTING:

As another important application domain, here we are interested in how to effectively utilize the locality information about sensor networks in time series forecasting. For example, how to incorporate the longitudes/latitudes of sensors w.r.t. temporal cloud movement is an important question in spatiotemporal modeling for predicting the output of solar energy farms in the United States. Appendix A provides the formal definition of this task. We use the daily maximum temperature data and precipitation data, and refer them as the USHCN-TMAX and USHCN-PRCP sets, respectively.

• The solar power production records in the year of 20063 has the data with the production rate of every 10 minutes from 1,082 solar power stations in the west of the U.S. The sequence length is 52,560. We refer this set of data as Solar.

All the datasets have been split into the training set (60%), the validation set (20%) and the test set (20%) in chronological order.

All the graph convolution methods (DCNN, ChebyNet, GCN and MoNet) in the previous section (Section 4.2) are included to form the baselines for comparison. We also add traditional methods for time series forecasting, such as (1) Autoregressive model (AR) which predicts future signal using a window of historical data based on a linear assumption about temporal dependencies, (2) Vector autoregressive model (VAR) which extends AR to the multivariate version, namely, the input is the signals from all sensors in the history window, and (3) the LSTNet deep neural network model (Lai et al., 2017) which combines the strengths of CNN, RNN and AR. None of those methods is capable of leveraging locational dependencies via graph convolution. We exclude the CNN and Xception methods, the 2D-grid based convolution, which could not be generalized to irregular graphs which we focus here.

Table 2 summarizes the evaluation results of all the methods, where the performance is measured using the Root Square Mean Error (RMSE). The best result on each dataset is highlighted in boldface. The first chunk of three methods does not leverage the spatial or locational information in data. The second chuck consists of the neural network models which leverage the spatial information about sensor networks. The graph convolution methods in the second chunk clearly outperforms the methods in the first chunk, which does not explicitly model the spacial correlation within sensor networks. Overall, our proposed method (DSGC) has the best performance on all the datasets, demonstrating its strength in capturing informative local propagation patterns temporally and specially.

4.4. DOCUMENT CATEGORIZATION:

For the application to text categorization we use the 20NEWS dataset (Joachims (1996) ) for our experiments. It consists of 18,845 text documents associated with 20 topic labels. Individual words in the document vocabulary are the nodes in the graph for convolution. Each node also has its word embedding vector which is learned by running the Word2Vec algorithm (Mikolov et al. (2013) ) on this corpus. Following the experiment settings in Defferrard et al. (2016) we select the top 1000 most frequent words as the nodes. 64.64% FC2500-FC500 † 65.76% DCNN (Atwood & Towsley, 2016) 70.35% ChebyNet (Defferrard et al., 2016) 70.92% GCN (Kipf & Welling, 2016) 71.01% MoNet (Monti et al., 2016) 70.60% DSGC 71.88%

Table 3 : Accuracy on the validation set. The results with † come from Defferrard et al. (2016) .

4.5. DSGC VARIANTS WITH ADVANCED CONVOLUTION ARCHITECTURES:

The proposed convolution method (DSGC) can be considered as an equivalent component to the depthwise separable convolution method. Naturally, we can leverage the technique developed for the standard convolution network to improve the DSGC framework. Hence we examine DSGC with the following techniques which are popular in recent years for standard convolution over images:

(1) Inception module (Szegedy et al., 2016) , (2) DenseNet framework (Huang et al., 2016) and (3) Squeeze-and-Excitation block (Hu et al., 2017) . The details of those architectures are included in the Appendix A. The results are presented in 

5. RELATED WORK:

In this section, we will summarize the graph convolution methods proposed in recent years with the label propagation process, which reveals the difference between traditional 2D-convolution and them. Firstly, we provide the formulation of the full convolution (LeCun et al., 1995) ,

EQUATION

different from the depthwise separable convolution, it captures the channel correlation and spatial correlation simultaneously by W (pq) , which leads to the larger number of parameters.

In Spectral Network (Bruna et al., 2013) , the authors try to leverage the graph Fourier transformation as the basic convolution operation in the graph domain, which can be written as,

EQUATION

where Φ ∈ R n×n contains the eigenvectors of Laplacian matrix of the graph, and Λ is a diagonal matrix and learned by the supervision data. The Spectral Network can be matched with the full convolution, but with the different filter subspace, in other words, with different basic filters. However, it suffers from several limitations. (1) It needs to conduct eigenvector decomposition over the Laplacian Matrix, which is a very expensive operation.

(2) The filters are not localized in the spatial domain.

(3) The number of parameters grows linearly with the number of nodes in the graph. In order to address the previous problems, researchers try to use the Chebyshev polynomial to approximate the non-parameter filter Λ, which is referred to as ChebyNet (Defferrard et al., 2016) . It can be written as,

y iq = K k=1 j∈N (i) T k (L) ij z (k) iq where z (k) i = U (k) x j (8)

where T k (L) is the k-th order Chebyshev polynomial term. The ChebyNet can be considered as the integration of K depthwise separable convolution components in a layer. But still, it suffers from the similar limitation as the GCN, which is using one graph filter over all channels and the graph filter is constant given the input. So its model capacity still cannot compare with depthwise separable convolution. With larger K, the ChebyNet can approximate the non-parameter filers in the Spectral Network. However, it would require large number of parameters and face the similar limitation as the Spectral Network.

Besides the graph convolution methods, researchers propose another type of models, geometric convolution methods (Masci et al., 2015; Boscaini et al., 2016; Monti et al., 2016) , to deal with data in the general spatial domain. Here, we introduce the most advanced one, MoNet (Monti et al., 2016) framework, which is also the most related one to our paper. The updating formula of MoNet in the label propagation process is,

EQUATION

where

w k (v) = exp(-1 2 (v -µ k ) T Σ -1 k (v -µ k ))

, and v(i, j) is a mapping from a node pair to a embedding vector, similar to ∆ ij in our model. µ k , Σ k are both model parameters, and Σ k is constrained as the diagonal matrix. MoNet can be viewed as an extension of the ChebyNet by letting the graph filters learn from the data. But it still has two limitations compared with the depthwise separable convolution and proposed method: (1) It uses a simple Gaussian function, which is weaker than non-parametric filter in the depthwise separable convolution, and neural network function in the proposed method. (2) It uses a graph filter for all channels. In order to capture complex propagation patterns in a layer, the model requires a larger K, which leads to much larger number of parameters. And finally the experiment results show that the proposed method (DSGC) consistently outperforms the MoNet with less parameters in multiple tasks. Intuitively, different sensors may have node-level hidden features to influence its propagation patterns and final outputs. Then for each node, the model learns a node embedding vector and concatenate it with the input signals. By using this trick, each node has limited freedom to interface with its propagation patterns. This trick is proven to be useful in this task, USHCN-PRCP and Solar specifically. We set the embedding size as 10 for these two datasets.

One thing readers may notice is that there are 10% data in USHCN dataset missing. To deal with that, we add an additional feature channel to indicate which point is missing. For the time series models, we tune the historical window p according to the validation set. For the rest of models, we set the window size p = 18 for Solar dataset and p = 6 for USHCN datasets. The network architecture used in this task is 7 convolution layers followed by a regression layer. The ij setting is the same as the previous one. We use the Adam optimizer (Kingma & Ba, 2014) for this task, and train each model 200 epochs with learning rate 0.001.

A.3 IMPLEMENTATION DETAILS OF DOCUMENT CATEGORIZATION:

The data preprocessing follows the experiment details in Defferrard et al. (2016) . And the network architecture for all models is 5 convolution layers followed by two MLP layers as the classifier. After each convolution layer, a dropout layer is performed with dropout rate of 0.5. The nodes' coordinate is the word embedding, and the method to calculate ij is similar to the previous ones. The optimizer used in this task is the same as the CIFAR experiment.

A.4 VARIANCE OF DSGC PERFORMANCE:

In this section, we report the variance of DSGC method in all 3 tasks. We run the DSGC model for 10 times and report the mean±std: CIFAR 7.39 ± 0.136, USHCN-TMAX 5.211 ± 0.0498, 20news 71.70 ± 0.285. Obviously, the variance is significantly smaller than the performance gap between the DSGC model and best baseline results (CIFAR 8.34, 20news 71.01 ).

:

Code: Data: the links are anonymous due to the double-blind policy.

http://cdiac.ornl.gov/epubs/ndp/ushcn/daily_doc.html

http://www.nrel.gov/grid/solar-power-data.html

CONCLUSIONIn this paper, we propose a novel Depthwise Separable Graph Convolution (DSGC) Network which is explicitly generalized from the depthwise separable convolution, and goes beyond to the general graph space. The extensive experiments on multi-field benchmark datasets demonstrate that our method can outperform strong baseline methods with a relatively small number of model parameters, and that it can be easily extended to leverage the advanced techniques/architectures in standard convolution networks for further improvement of the performance. In future work, we want to explore its impact on a broader range of applications, such as social networks and molecular structures by leveraging technical improvements about node/edge embedding based on graph structure information.

Figures and Tables:

Figure 1: Visualization of different convolution operations with three output channels. We use different colors to represent different filters (weight configurations of the links). (a) DSC defined on 2D grid graphs. (b) GC defined on generic graphs. (c) DSGC defined on generic graphs.

Figure 2: How to construct subsampled CIFAR datasets: (a) is an example image from CIFAR dataset. (b) is the subsampled pixels map. The blue points indicate which points are sampled. (c) is the image after sampling, where the black points are those being sampled out.

Figure 3: Inception Module

Time series prediction: Experiment result in terms of RMSE.the feature set of the top 1000 words, which is the same setting as in the graph convolution models. If all words are used, traditional classifiers would have higher performance.

Table content: <table><tr><td>Dataset</td><td colspan="2">USHCN-TMAX USHCN-PRCP</td><td>Solar</td></tr><tr><td>AR</td><td>8.2354</td><td>30.3825</td><td>0.03195</td></tr><tr><td>VAR</td><td>17.9743</td><td>29.2597</td><td>0.03296</td></tr><tr><td>LSTNet (Lai et al., 2017)</td><td>10.1973</td><td>29.0624</td><td>0.02865</td></tr><tr><td>DCNN (Atwood &amp; Towsley, 2016)</td><td>6.5188</td><td>29.0424</td><td>0.02652</td></tr><tr><td>ChebyNet (Defferrard et al., 2016)</td><td>5.5823</td><td>27.1298</td><td>0.02531</td></tr><tr><td>GCN (Kipf &amp; Welling, 2016)</td><td>5.4671</td><td>27.1172</td><td>0.02512</td></tr><tr><td>MoNet (Monti et al., 2016)</td><td>5.8263</td><td>26.8076</td><td>0.02564</td></tr><tr><td>DSGC</td><td>5.1738</td><td>25.8228</td><td>0.02453</td></tr><tr><td>Method</td><td/><td>Accuracy</td><td/></tr><tr><td>Linear SVM  †</td><td/><td>65.90%</td><td/></tr><tr><td colspan="2">Multinomial Naive Bayes  †</td><td>68.51%</td><td/></tr><tr><td>Softmax  †</td><td/><td>66.28%</td><td/></tr><tr><td>FC2500  †</td><td/><td/><td/></tr></table>

Clearly, combined with the advantageous techniques/architectures, the performance of DSGC in image classificationcan can be further improved. It demonstrates that the DSGC can easily enjoy the benefit of the traditional 2d-convolution network development.

Table content: <table><tr><td/><td colspan="2">Subsampled Graphs</td><td/><td colspan="2">Original Graphs</td><td/></tr><tr><td>Dataset</td><td colspan="2">CIFAR10 CIFAR100</td><td>P</td><td colspan="2">CIFAR10 CIFAR100</td><td>P</td></tr><tr><td>DSGC-VGG13</td><td>18.72%</td><td>44.33%</td><td>5.7M</td><td>7.31%</td><td>27.29%</td><td>9.9M</td></tr><tr><td>DSGC-INCEPTION</td><td>18.27%</td><td>43.41%</td><td>9.9M</td><td>6.44%</td><td>28.55%</td><td>12M</td></tr><tr><td>DSGC-DenseNet</td><td>17.17%</td><td>43.34%</td><td>2.7M</td><td>7.14%</td><td>26.50%</td><td>2.9M</td></tr><tr><td>DSGC-SE</td><td>18.71%</td><td>44.15%</td><td>6.1M</td><td>7.00%</td><td>27.26%</td><td>10M</td></tr><tr><td>CNN</td><td>18.03%</td><td>43.42%</td><td>18M</td><td>6.86%</td><td>26.86%</td><td>18M</td></tr><tr><td>Xception</td><td>17.07%</td><td>41.54%</td><td>3.1M</td><td>7.08%</td><td>26.84%</td><td>3.1M</td></tr></table>

Summary of error rate on the test set in different settings.4.6 TRAINING TIME COMPARISONIn table 5, we report the mean training time per epoch for DSGC and GCN, the fastest graph convolution baseline. In DSGC, our model computes the convolution weight for each edge of the graph, which requires more computation resources. However, we always perform the graph convolution on the sparse graph, which the number of edges grows only linearly in the graph size. Therefore the training is fairly efficient. Notably, learning the convolution filters as in DSGC leads to consistently better performance over all previous methods, with around 0.5x-3x running time.

Table content: <table><tr><td colspan="4">Dataset CIFAR USHCN-TMAX 20news</td></tr><tr><td>GCN</td><td>1.75</td><td>0.465</td><td>0.207</td></tr><tr><td>DSGC</td><td>3.81</td><td>1.73</td><td>0.280</td></tr></table>

Training time per epoch for GCN and DSGC methods. The unit is minute.

Table content: <table/>

Peer review (iclr2018):
 Paper Summary: This work proposes a new geometric CNN model to process spatially sparse data. Like several existing geometric CNNs, convolutions are performed on each point using nearest neighbors. Instead of using a fixed or Gaussian parametric filters, this work proposes to predict filter weights using a multi-layer perception. Experiments on 3 different tasks showcase the potential of the proposed method. Paper Strengths: - An incremental yet interesting advance in geometric CNNs. - Experiments on three different tasks indicating the potential of the proposed technique. Major Weaknesses: - Some important technical details about the proposed technique and networks is missing in the paper. It is not clear whether a different MLP is used for different channels and for different layers, to predict the filter weights. Also, it is not clear how the graph nodes and connectivity changes after the max-pooling operation. - Since filter weight prediction forms the central contribution of this work, I would expect some ablation studies on the MLP (network architecture, placement, weight sharing etc.) that predicts filter weights. But, this is clearly missing in the paper. - If one needs to run an MLP for each edge in a graph, for each channel and for each layer, the computation complexity seems quite high for the proposed network. Also, finding nearest neighbors takes time on large graphs. How does the proposed technique compare to existing methods in terms of runtime? Minor Weaknesses: - Since this paper is closely related to Monti et al., it would be good if authors used one or two same benchmarks as in Monti et al. for the comparisons. Why authors choose different set of benchmarks? Because of different benchmarks, it is not clear whether the performance improvements are due to technical improvements or sub-optimal parameters/training for the baseline methods. - I am not an expert in this area. But, the chosen benchmarks and datasets seem to be not very standard for evaluating geometric CNNs. - The technical novelty seems incremental (but interesting) with respect to existing methods. Clarifications: - See the above mentioned clarification issues in *major weaknesses*. Those clarification issues are important to address. - *Non-parametric filter* may not be right word as this work also uses a parametric neural network to estimate filter weights? Suggestions: - It would be great if authors can add more details of the multi-layer perceptron, used for predicting weights, in the paper. It seems some of the details are in Appendix-A. It would be better if authors move the important details of the technique and also some important experimental details to the main paper. Review Summary: The proposed technique is interesting and the experiments indicate its superior performance over existing techniques. Some incomplete technical details and non-standard benchmarks makes this not completely ready for publication.