Geometric Random Walk Graph Neural Networks via Implicit Layers

Abstract: Graph neural networks have recently attracted a lot of attention and have been applied with great success to several important graph problems. The Random Walk Graph Neural Network model was recently proposed as a more intuitive alternative to the well-studied family of message passing neural networks. This model compares each input graph against a set of latent ``hidden graphs'' using a kernel that counts common random walks up to some length. In this paper, we propose a new architecture, called Geometric Random Walk Graph Neural Network (GRWNN), that generalizes the above model such that it can count common walks of infinite length in two graphs. The proposed model retains the transparency of Random Walk Graph Neural Networks since its first layer also consists of a number of trainable ``hidden graphs'' which are compared against the input graphs using the geometric random walk kernel. To compute the kernel, we employ a fixed-point iteration approach involving implicitly defined operations. Then, we capitalize on implicit differentiation to derive an efficient training scheme which requires only constant memory, regardless of the number of fixed-point iterations. The employed random walk kernel is differentiable, and therefore, the proposed model is end-to-end trainable. Experiments on standard graph classification datasets demonstrate the effectiveness of the proposed approach in comparison with state-of-the-art methods.

Reviews and author responses:

 Having read the other reviews and the authors' responses, I am inclined to keep my score, even though I do think this paper is very much on the border of accept and reject. As I mentioned in my original review: I think the paper is well written and the idea is sound.

However, I am still not convinced that the idea is significantly novel (see below). Moreover, some of the other reviewers have raised valid questions about the experimental protocol.

> Furthermore, while most models employ implicit layers to achieve infinite depth (infinitely many layers), we just solve the fixed point equation to compute the geometric random walk kernel and use implicit differentiation in order to backpropagate the error through the fixed point solver in the backward pass. Thus, our setting is different from that of related works.

> In contrast to other models that employ implicit layers to achieve infinite depth, in our setting, we solve the fixed point equation just to compute the geometric random walk kernel. Thus, the GRWNN model is different from other works which employ implicit layers to define infinite depth models.

I am not convinced by this distinction. In both cases a fixed point is calculated, which is the solution to a contractive mapping when the number of steps goes to infinity. Whether or not to view each step as a layer (and hence having "infinitely many layers") or whether to view it as calculating a kernel (which is defined as an infinite sum, equation 2) seems arbitrary.

> Many of the benchmark datasets were introduced several years ago, and are now very well studied.

> First of all, we believe that it is too optimistic to expect a method to outperform all baselines on all those benchmark datasets since most of them were introduced several years ago, and are now very well studied.

I agree that a work shouldn't be expected to set a state-of-the-art result on each task in old benchmarks. However, this becomes harder to accept when the work lacks significant novelty, and it does seem to me that this work is a relatively straightforward application of implicit differentiation.

The paper discusses random walk graph kernels for graph neural network feature generation in other works. They work by generating features for graphs by comparing random walks present in each graph compared to a set of trainable hidden graphs. It is stated in this paper that an advantage of random walk kernels is that they are end to end differentiable, but current implementations have the weakness of only being efficient for walks of very short length. They introduce a way to efficiently extend walks to ‘infinite length’, theoretically making features generated by the kernel more informative. They compare their results with top GNNs and with other graph kernel approaches such as the mentioned random walk kernels on graph classification datasets. Out of 10 datasets they achieve best performance on 2 (1 of the 2 is tied for best), second best on 3, and third best on 1, It should be noted that many placements are well within the margin of error.  strengths: 

The main contribution, which I believe is their solution to allowing an ‘infinite step’ geometric random walk to be efficiently differentiated and thus the hidden graphs trained, is thoroughly detailed in section 4. Also, they thoroughly justify and demonstrate settings for their main hyper-parameter lambda. Assuming correctness of the implementation itself, which I am not very confident in my ability to critique, I believe this is an idea worth exploring regardless of the outcome. 

Although the results aren’t clearly superior, it seems that none of the competitors listed are in a significant lead, and instead each has strengths and weaknesses in the diverse pool of graph datasets. I would say this method proved itself to be a good choice for some kinds of graph classification problems. 


weaknesses: 

A major problem of this work is numbers reported the experimental study. In table 1, many results are far lower than that in their original papers. For example, GIN achieves 75.1 and 52.3 on IMDB-B and IMDB-M but was reported 71.2 and 48.5 (https://arxiv.org/pdf/1810.00826.pdf). The state-of-the-art performances on these datasets have been greatly improved and far better than the performances achieved by GRWNN. The authors need to address this issue. Otherwise, I would have concern that the reported numbers are misleading the audiences and reviewers.

The conclusion seems rushed and doesn’t analyze the results very well, such as the fact that allowing longer random walks is, in some cases but not others, an advantage over the 1-step random walks on all datasets. I’m naive to the true implications in this case, but I assume there is knowledge to be gleaned from the results even if they aren’t a large improvement over competitors.  Although the authors make great contribution to the technical part, the experimental results have big flaw, which needs to be addressed.

 I read the rebuttal and increased my score to 5. I am still concerned with the results that are not reported from the original paper since different experimental settings can have significant impact on model performances. Also, no additional new state-of-the-art models are compared.

 I read the rebuttal and will keep my score. 

 Thank you very much for your review and the constructive feedback. Below, we respond to your main concerns:

> You state that "Such local patterns...

An example of such a property is connectivity. Take for example the following two graphs: a graph $G_1$ that consists of 2 cycles of length 5 (2 connected components), and a graph $G_2=C_{10}$ (a cycle of length 10), and let $\mathbf{A}_1, \mathbf{A}_2$ denote the respective adjacency matrices. The matrices $\mathbf{A}_1^2$ and $\mathbf{A}_2^2$ that count walks of length 2 from each node to any other node are similar to each other, i.e., the two graphs exhibit similar patterns of walks of length 2. However, the matrices $(\mathbf{I}-\lambda \mathbf{A}_1)^{-1}$ and $(\mathbf{I}-\lambda \mathbf{A}_2)^{-1}$ that count walks of infinite length are different from each other since in $G_1$ no paths exist from nodes of one connected component to nodes of the other connected component. We need to mention though that since the employed kernel computes the sum of all paths to produce a single kernel value, it is not always guaranteed that it will capture such properties of graphs. 

> Why did you not evaluate the fixed-length RW models on the OBG datasets?

For the fixed-length RW models, we just used the results reported in the original paper. And those models were not evaluated on the OGB datasets. However, since the code of the fixed-length RW models is publicly available, we will evaluate these models on those datasets and report the results in the final version.

> Does your architecture offer any insights on how to design deep GNNs?

In contrast to other models that employ implicit layers to achieve infinite depth, in our setting, we solve the fixed point equation just to compute the geometric random walk kernel. Thus, the GRWNN model is different from other works which employ implicit layers to define infinite depth models.


> Suggestions To show a clear performance gain...

Although the datasets from Table 1 are relatively small in terms of the number of graphs, they are quite diverse with regards to the size of the graphs contained in them. There are datasets that contain small-sized graphs (IMDB-MULTI, MUTAG), medium-sized graphs (D&D), but also graphs of larger size (REDDIT-BINARY, REDDIT-MULTI-5K). Therefore, we believe that an evaluation on those datasets could be very useful since it can give insights into how effective each method is on different types of graphs.

 We thank you for your review. We proceed, next, into answering your main concerns:

> Although the authors make efforts...

Although this is generally true, since both the number of $\mathbf{A}_i$'s and the number of nodes of $\mathbf{A}_i$'s are hyperparamters, we can set them to specific values such that their product is roughly equal to the number of parameters of an MPNN. The GRWNN model will then only be slightly more complex than an MPNN model due to the fixed point method (see Appendix G). We also need to mention that we have experimentally found (see Appendix H) that even for a small product of the number of $\mathbf{A}_i$'s and the number of nodes of $\mathbf{A}_i$'s, the GRWNN model can yield high levels of performance.

> The experimental results in Table 1...

First of all, we believe that it is too optimistic to expect a method to outperform all baselines on all those benchmark datasets since most of them were introduced several years ago, and are now very well studied. Still, GRWNN achieves high levels of performance on some of those datasets (e.g., ENZYMES, IMDB-BINARY). We believe that the main advantage of the proposed model is its novelty and the fact that it can capture  properties that models that consider only the local neighborhood of nodes cannot (e.g., long-range dependencies).

> Besides RWNN, the factorization-based methods...

We thank the reviewer for pointing us to these papers. The setting in those papers is a bit different from ours since they learn graph representations in an unsupervised manner. Furthermore, those approaches are not directly related to our approach. For instance, in [b] each input graph is not compared against a set of trainable graphs but is assumed to be a combination of graph atoms, i.e., dictionary elements which the authors learn. We will discuss the similarities and differences between [a, b] and our work in the final version.

> Although the authors provide...

$\lambda$ can indeed be a parameter of the neural network. We have empirically observed (in preliminary experiments) that the value of $\lambda$ depends a lot on its initialization, and is not affected much during training.


 We would like to thank you for your evaluation. We next proceed into responding to the mentioned weaknesses:

> A major problem of this work is numbers reported...

We did not evaluate GIN and the other graph neural network baselines (DGCNN, DiffPool, ECC, GraphSAGE) ourselves. We instead use the numbers reported in [1] which proposes a standardized and reproducible experimental protocol for evaluating graph neural networks. Unfortunately, the experimental evaluation employed by many papers is not correct. For instance, many of them use the class labels of the test samples for model selection. Thus, for a fair comparison, we employed the widely accepted assessment protocol proposed in [1].

[1] Errica, F., Podda, M., Bacciu, D. and Micheli, A., A fair comparison of graph neural networks for graph classification. In ICLR'20


> The conclusion seems rushed...

This is an interesting point. We observed that GRWNN outperforms the models that use p-step random walk kernels on specific types of datasets (those that contain unlabeled graphs and those that contain node-attributed graphs). It is known that the GRWNN can better capture the overall shape of graphs compared to p-step random walk based models. However, it is not clear to us what properties a dataset should exhibit such that GRWNN outperforms the simpler models on that dataset, and how those properties are related to the graph's overall structure. This is something we would like to investigate in the future.


 Thank you very much for your review and the constructive comments. Below, we respond to your main concerns:

> The only real question...

Such implicit differentiation schemes are not very well-studied in the field of graph neural networks so far. If the reviewer is aware of any further papers, we would appreciate if he/she could share those references with us. Furthermore, while most models employ implicit layers to achieve infinite depth (infinitely many layers), we just solve the fixed point equation to compute the geometric random walk kernel and use implicit differentiation in order to backpropagate the error through the fixed point solver in the backward pass. Thus, our setting is different from that of related works.


> Moreover, the experimental results...

Many of the benchmark datasets were introduced several years ago, and are now very well studied. Therefore, one cannot expect significant improvements. These datasets are also quite diverse (e.g., some contain small graphs, while others large graphs - some contain k-hop graphs extracted from larger graphs, while others contain entire graphs). Thus, one cannot expect a single approach to be universally optimal, e.g., to outperform all the other approaches on all datasets.

> The authors refer repeatedly to Bai et al. (2019)...

We thank the reviewer for pointing us to this paper. 

This paper shows that the geometric random walk kernel is differentiable using implicit differentiation and can hence be used in a graph neural network to match an input graph to a set of learned graphs. The experimental results show that this method is competitive with a variety of other methods. This is a well-written and well-executed paper with encouraging and thorough experimental results.

The only real question in my mind is whether it meets the bar for novelty and significance: It falls under the large number of papers that take an iterative process and use implicit differentiation to efficiently include it in a neural network for training. With this formula in mind, the derivations are relatively straightforward. Moreover, the experimental results are not particularly surprising and don't show very significant improvements over other methods. With that in mind, perhaps this paper is more appropriate for a workshop? Or perhaps there is a way that this work could be extended?

The authors refer repeatedly to Bai et al. (2019) for the implicit differentiation. Although this might be the correct reference for the recent rediscovery of implicit differentiation in deep learning, I believe that "Automatic Differentiation and Iterative Processes" (Gilbert, 1992) should probably be cited as one of the papers that originally introduced the idea.

 A well-written and decent paper that struggles to meet the bar for novelty and significance.

The paper deals with learning with graphs, specifically graph-level classification. Given a graph G, the main idea of the paper is to compare the graph G to a set of N graphs using the (infinite-length) random walk kernel (Vishwanathan et al. 2010), resulting in a vector with N components. The resulting vector is fed into an MLP performing the classification tasks. Crucially, the N graphs are learned using end-to-end learning.

Further, the authors show how to compute the RW kernel efficiently, building on previous work  (Vishwanathan et al. 2010), and how to do differentiate through the architecture efficiently, building on (Bai et al. 2016). The paper can be seen as an extension of the RWNN model (Nikolentzos et al. 2020), which uses a finite-length random kernel. 

The methodology is complemented with an experimental study showing promising performance on standard benchmarks datasets. 




 **Strengths**
1. Simple architecture that give good empirical results 
2. Well-written and easy to follow

**Weaknesses**
1. Incremental work, building on known work 
2. No clear performance gain over fixed-length RW models, see Table 1 
3. Comparision to fixed-length RW models only done on small-scale datasets, compare Table 1 and Figure 1(a)


**Questions**
1. You state that "Such local patterns may fail to capture the overall large-scale shape of the graphs, while several interesting
properties of graphs depend on the graph’s global structure."

Can you quantify exactly for which kind of graph structures this is the case? 
E.g., by quantifiying the graph structures captured by the infinite-length RW models but not by finite-length RW models.

2. Why did you not evaluate the fixed-length RW models on the OBG datasets?

3. Does your architecture offer any insights on how to design deep GNNs?

**Suggestions**
To show a clear performance gain, especially over the fixed-length RW models, one should compare the model on more large-scale datasets, e.g., from OGB and graphlearning.io. The small-scale datasets from Table 1 exhibit large standard deviations which make comparisons in terms of accuracy often not meaningful. 

**Minor comments**
1. You seem to use \citet{} when citep{} would be more appropriate
2. First page, second paragraph:  "graphs, So far, ..." -> "graphs. So far, ...
 The paper is well-written and easy to follow. However, the idea is incremental, building largely on previous work. Moreover, the gain in expressiveness is not quantified, e.g., by showing that the infinite-length RW models can expressive more permutation-invariant functions compared to the fixed-length RW models. Morever, the emperical results are not convincing, i.e., they do not clearly win over  the fixed-length RW models.

In this paper, the authors proposed an interesting geometric random walk neural network for graph representation. The proposed method can be viewed as an improvement of the random walk neural network in [Nikolentzos & Vazirgiannis, 2020], which learns graph representations by modeling the geometric series of the “product” between observed graphs and some hidden and learnable graphs. 
As a result, the graph representations are learned by finding fixed points of linear systems.  Pros:
1. The idea of this work is interesting and has sufficient novelty. An efficient computational method is designed to simplify the computation associated with the “graph product”.

2. The experiments are sufficient and representative graph datasets and baselines are considered.

3. The paper is well-written and thus easy to follow. 


Cons:
1. Although the authors make efforts to reduce the complexity of the proposed method. Compared with message passing neural networks (MPNN), the proposed method involves inevitable dense matrix multiplication — in Eq.(6) and Proposition 1, A_i is a dense learnable matrix. As a result, when there are many large hidden graph A_i’s, the proposed method will be much worse than MPNN.

2. The experimental results in Table 1 and Figure 1 do not show the superiority of the proposed method. Considering the potential high complexity of the method, what is the main advantage of the proposed GRWNN method?

3. Besides RWNN, the factorization-based methods in [a, b] also leverage several learnable hidden graphs to learn graph representations, and they are applicable in unsupervised learning cases. The authors should consider these works and the corresponding learning strategy in the related work and discuss them in-depth.

[a] Xu, Hongteng. "Gromov-Wasserstein factorization models for graph clustering." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.
[b] Vincent-Cuaz, Cédric, et al. "Online Graph Dictionary Learning." arXiv preprint arXiv:2102.06555 (2021).

Minors:
1. Although the authors provide an empirical method to determine “\lambda”, I wonder if it is possible to learn lambda automatically. 
 The idea proposed in this paper is interesting indeed, which provides a new way to learn graph representations. However, the superiority of the proposed method to existing work is verified neither on classification accuracy nor runtime. Additionally, some related graph representation strategies are not considered or discussed.

Meta-review: This paper shows how to back-propagate through a kernel between graphs that counts common random walks of infinite length between the graphs. Reviewers tend to agree that the paper is well-written and the technical contributions are sound. However, there are concerns about the significance and novelty of the method relative to related work, alongside mixed experimental results. Overall that puts it as a very borderline paper. In the rebuttals, the authors argued for the significance of the contribution, but reviewers were generally unconvinced.

Decision: Reject