Title: Convolutional Neural Networks Generalization Utilizing the Data Graph Structure
Authors: Yotam Hechtlinger
Purvasha Chakravarti
Jining Qin
Venue: ICLR 2017
Keywords: Supervised Learning
Deep learning

Abstract: Convolutional Neural Networks have proved to be very efficient in image and audio processing. Their success is mostly attributed to the convolutions which utilize the geometric properties of a low - dimensional grid structure. This paper suggests a generalization of CNNs to graph-structured data with varying graph structure, that can be applied to standard regression or classification problems by learning the graph structure of the data. We propose a novel convolution framework approach on graphs which utilizes a random walk to select relevant nodes. The convolution shares weights on all features, providing the desired parameter efficiency.  Furthermore, the additional computations in the training process are only executed once in the pre-processing step.  We empirically demonstrate the performance of the proposed CNN on MNIST data set, and challenge the state-of-the-art on Merck molecular activity data set.

TL;DR: A generalization of CNNs to standard regression and classification problems by using random walk on the data graph structure.

Review discussion:
1. p. 8, 4.2, How does the proposed method compare to regular CNNs with the same number of parameters?
2. p. 8, 4.2, How one would ensure consistent ordering of pixels for all the spatial locations? For some pixels, the expected number of visits is the same and it's essential that the ties are broken consistently, otherwise, for some nodes, the top pixel would go before the bottom one, while for the others the order would be swapped.
3. For some applications/datasets, the spatial structure of the graph matters a lot (e.g. relative locations of pixels w.r.t. the central pixel). How would one approach incorporating that information into the computation of the graph convolution?
1&2. Please see the revision we've just submitted. This is a very important insight. Thank you for catching it. Following your remark, we noticed ties has been broken arbitrarily.
Rather than making it consistent, we think that the fact that the model actually get 1.1 error rate that way makes the example more compelling.
3. We agree that the spatial structure of the graph matters a lot. We actually think it what matters the most in this method. Notice this are two independent questions.
Constructing an effective graph structure to a dataset is a graph learning problem heavily researched in some applications. This paper steps in once you have the graph structure. It provides a tool incorporating the information provided/learned from the graph structure into a standard supervised learning framework.
You can apply the graph convolution on any graph structure.
But unlike the regular convolution, the structure is no longer trivial, and as you mention, it is very important. In that sense, you can think of our usage of the naive correlation matrix as a lower bound on the potential of the method.
We apologize for the delayed respond, some of us have been presenting at the NIPS workshops.
This work proposes a convolutional architecture for any graph-like input data (where the structure is example-dependent), or more generally, any data where the input dimensions that are related by a similarity matrix. If instead each input example is associated with a transition matrix, then a random walk algorithm is used generate a similarity matrix.
Developing convolutional or recurrent architectures for graph-like data is an important problem because we would like to develop neural networks that can handle inputs such as molecule structures or social networks.
However, I don't think this work contributes anything significant to the work that has already been done in this area.
The two main proposals I see in this paper are:
1) For data associated with a transition matrix, this paper proposes that the transition matrix be converted to a similarity matrix. This seems obvious.
2) For data associated with a similarity matrix, the k nearest neighbors of each node are computed and supply the context information for that node. This also seems obvious.
Perhaps I have misunderstood the contribution, but the presentation also lacks clarity, and I cannot recommend this paper for publication.
Specific Comments:
1) On page 4: "An interesting attribute of this convolution, as compared to other convolutions on graphs is that, it preserves locality while still being applicable over different graphs with different structures."  This is false; the other proposed architectures can be applied to inputs with different structures (e.g. Duvenaud et. al., Lusci et. al. for NN architectures on molecules specifically).
We wish to thank you for reviewing our paper.
This paper’s main contribution is a novel way to apply convolutions on data which lacks a grid structure. This is being done in a similar way to what you've described in (2), only we keep the weights shared across all the variables according to their order (this is (3) - the decision to fix the order is important).
In addition (1) explains how to do it for a general graph which lacks the similarity matrix. We combine (1), (2) and (3) together, then explain how to implement this in an efficient way, and demonstrate through empirical experiments that this works.
With regards to the opinion that the ideas in the the paper are obvious: we disagree. We instead believe they are natural given the nature of the problem and also easy to understand, both of which are arguably good qualities. We also think this is a straightforward, immediate generalization of CNN.
We would like to challenge the "Clear rejection" conclusion the reviewer draws from the fact that our solutions seem obvious to her/him.
In particular, we would love to hear about any reference in which a methodology similar to the one we propose is used for similar purpose, in a similar manner and with comparable performance, rendering our contribution not novel and possibly redundant.
Regarding the clarity - If you would elaborate on what troubled you with our writing style we will be happy to address it and revise the paper.
Regarding Specific Comments - Thank you for the remark. Lusci et. al. do Recursive Neural Networks on the graph, and Duvenaud et. al. offer a specific solution for molecules.
But we agree this can be written better. We were mostly referring to methods using the graph Laplacian. We will be more accurate in the next revision.
Update: I thank the authors for their comments! After reading them, I decided to increase the rating.
This paper proposes a variant of the convolution operation suitable for a broad class of graph structures. For each node in the graph, a set of neighbours is devised by means of random walk (the neighbours are ordered by the expected number of visits).
As a result, the graph is transformed into a feature matrix resembling MATLAB’s/Caffe’s im2col output. The convolution itself becomes a matrix multiplication.
Although the proposed convolution variant seems reasonable, I’m not convinced by the empirical evaluation. The MNIST experiment looks especially suspicious. I don’t think that this dataset is appropriate for the demonstration purposes in this case. In order to make their method applicable to the data, the authors remove important structural information (relative locations of pixels) thus artificially increasing the difficulty of the task.
At the same time, they are comparing their approach with regular CNNs and conclude that the former performs poorly (and does not even reach an acceptable accuracy for the particular dataset).
I guess, to justify the presence of MNIST (or similar datasets) in the experimental section, the authors should modify their method to incorporate additional graph structure (e.g. relative locations of nodes) in cases when the relation between nodes cannot be fully described by a similarity matrix.
I believe, in its current form, the paper is not yet ready for publication but may be later resubmitted to a workshop or another conference after the concern above is addressed.
We thank you for your review.
The MNIST experiment was done to show how the graph convolution can generalize regular convolution when specific graph structure is used, and to show that this method works also when the spatial structure is not present and CNNs are not applicable.
We think that CNN is a great tool, which should be used whenever possible. We do not try to compete with CNNs on images or other data sets with grid structure, but rather offer an alternative to the other methods when the grid structure is not present and CNNs are not applicable.
For that reason we do not expect this method to break the 0.75% of CNN on MNIST (even with more innovative graph structures).
With that said, following your review we have redone the experiment, increasing the epochs from 40->100, and the number of convolutions from 20->40 and 50->80. This resulted with 0.88% error rate.
Still not CNNs, but better. Our intuition on this result is that there is finite number of ways to break ties, and the larger the number of convolutions, the closer we get to the regular CNNs.
We've also done a sainty check to see where we stand in terms of the required publication benchmark when using the MNIST dataset, by checking  NIPS 2016 published papers with MNIST in the abstract (https://nips.cc/Conferences/2016/Schedule?q=mnist). There are 13 different papers total. 3 papers use CNN and report excellent results. 5 papers address different type of problems and doesn't report accuracy. The remaining 5 use MNIST to demonstrate other methods than CNN. The papers (and their best MNIST result) are:
-Supervised Learning Tensor Networks (0.97% error)
-Binarized Neural Networks (0.96% error)
-Direct Feedback Alignment Provides Learning in Deep Neural Networks (1.01% error)
-Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (0.86% error)
-Dense Associative Memory for Pattern Recognition (around 1.4% - not directly reported)
So we think this is an acceptable result, especially given the fact that grid structure data is not the main purpose of the method.
It should be noted that in spite of the impression we might have made - we consider the Merck experiment to be the main experiment, in which we tackle a non trivial regression problem that has been tackled quite thoroughly in Kaggle, and achieve almost state of the art with a shallow application of the method. The MNIST usage is mostly due to the intuition it provides, and if you still think it is the paper major problem, we will be happy to explore other alternatives.
Thank you again for taking the time to review our work.
Previous literature uses data-derived adjacency matrix A to obtain neighbors to use as foundation of graph convolution. They propose extending the set of neighbors by additionally including nodes reachable by i<=k steps in this graph. This introduces an extra tunable parameter k, so it needs some justification over the previous k=1 solution. In one experiment provided (Merk), using k=1 worked better. They don't specify which k that used, just that it was big enough for their to be p=5 nodes obtained as neighbors. In the second experiment (MNIST), they used k=1 for their experiments, which is what previous work (Coats & Ng 2011) proposed as well. A compelling experiment would compare to k=1 and show that using k>1 gives improvement strong enough to justify an extra hyper-parameter.
Please notice that the paper by Coates & Ng (2011) construct locally connected receptive fields in a feed forward neural network style. They connect a given feature with it's 200 most correlated neighbors to create a single output unit. We share the weights and convolve the same weights on all the variables according to the order, generalizing convolutional neural networks.
The two papers address different neural networks architectures and have different goals - regardless of the hyper-parameter k.
Regarding the hyper-parameter - in 3.3 we explain why lower values of k are preferred over larger values of k.
Indeed we have used k=1 when it makes sense. There are situations when this is infeasible, particularly when the graph is sparse.

Meta-review:
role: PC
data: comment: This work studies the problem of generalizing a convolutional neural network to data lacking grid-structure. 
 
 The authors consider the Random Walk Normalized Laplacian and its finite powers to define a convolutional layer in a general graph. Experiments in Merck molecular discovery and mnist are reported. 
 
 The reviewers all agreed that this paper, while presenting an interesting and important problem, lacks novelty relative to existing approaches. In particular, the AC would like to point out that important references seem to be missing from the current version. 
 
 The proposed approach is closely related to 'Convolutional neural networks on graphs with fast localized spectral filtering', Defferrand et al. NIPS'16 , which considers Chevyshev polynomials of the Laplacian and learns the polynomial coefficients in an efficient manner. Since the Laplacian and the Random Walk Normalized Laplacian are similar operators (have same eigenvectors), the resulting model is essentially equivalent. Another related model that precedes all the cited works and is deeply related to the current submission is the Graph Neural Network from Scarselli et al.; see 'Geometric Deep Learning: going beyond Euclidean Data', Bronstein et al, https://arxiv.org/pdf/1611.08097v1.pdf' and references therein for more detailed comparisons between the models.

Decision: Reject