Title: How Powerful are Graph Neural Networks?
Authors: Keyulu Xu*
Weihua Hu*
Jure Leskovec
Stefanie Jegelka
Venue: ICLR 2019
Keywords: graph neural networks
theory
deep learning
representational power
graph isomorphism
deep multisets

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.

TL;DR: We develop theoretical foundations for the expressive power of GNNs and design a provably most powerful GNN.

Review discussion:
This paper presents a very interesting investigation of the expressive capabilities of graph neural networks, in particular focusing on the discriminative power of such GNN models, i.e. the ability to tell that two inputs are different when they are actually different.  The analysis is based on the study of injective representation functions on multisets.  This perspective in particular allows the authors to distinguish different aggregation methods, sum, mean and max, as well as to distinguish one layer linear transformations from multi-layer MLPs.  Based on the analysis the authors proposed a variant of the GNN called Graph Isomorphism Networks (GINs) that use MLPs instead of linear transformations on each layer, and sum instead of mean or max as the aggregation method, which has the most discriminative power following the analysis.  Experiments were done on node classification benchmarks to support the claims.
Overall I quite liked this paper.  The study of the expressive capabilities of GNNs is a very important problem.  Given the popularity of this class of models recently, theoretical analysis for these models is largely missing.  Previous attempts at studying the capability of GNNs focus on the function approximation perspective (e.g. Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction by Hertiz et al. which is worth discussing).  This paper presents a very different angle focusing on discriminative capabilities.  Being able to tell two inputs apart when they are different is obviously just one aspect of representation power, but this paper showed that studying this aspect can already give us some interesting insights.
I do feel however that the authors should make it clear that discriminative power is not the only thing we care, and in most applications we are not doing graph isomorphism tests.  The ability to tell, for example, how far two inputs are, when they are not the same is also very (and maybe more) important, which such isomorphism / injective map based analysis does not capture at all.
In fact the assumption that each feature vector can be mapped to a unique label in {a, b, c, ...} (Section 3 first paragraph) is overly simplistic and only makes sense for analyzing injective maps.  If we want to reason anything about the continuity of the features and representations, this assumption does not apply, and the real set is not countable so such a mapping cannot exist.
In equation 4.1 describes the GIN update, which is proposed as “the most powerful GNN”.
However, such architecture is not really new, for example the Interaction Networks (Battaglia et al. 2016) already uses sum aggregation and MLP as the building blocks.
Also, it is said that in the first iteration a simple sum is enough to implement injective map, this is true for sum, but replacing that with mean and max can lose information very early on.
Another MLP on the input features at least for mean or max aggregation for the first iteration is therefore necessary.  This isn’t made very clear in the paper.
The training set results presented in section 6.1 is not very clear.  The plots show only one run for each model variant, which run was it?  As the purpose is to show that some variants fit well, and some others overfit, these runs should be chosen to optimize training set performance, rather than generalization.
Also the restrictions should be made clear that all models are given the same (small) amount of hidden units per node.  I imagine if the amount of hidden units are allowed to be much bigger, mean and max aggregators should also catch up.
As mentioned earlier I quite liked the paper despite some restrictions anc things to clarify.  I would vote for accepting this paper for publication at ICLR.
Considering the counter-example given above, I'm lowering my scores a bit.  The proof of theorem 3 is less than clear.  The proof for the first half of theorem 3 (a) is quite obvious, but the proof for the second half is a bit hand-wavy.
In the worst case, the second half of theorem 3 (a) will be invalid.  The most general GNN will then have to use an update function in the form of the first half of 3(a), and all the other analysis still holds.  The experiments will need to be rerun.
Update: the new revision resolved the counter-example issue and I'm mostly happy with it, so my rating was adjusted again.
We thank the reviewer for the positive review and constructive feedback! We are glad that the reviewer likes our paper.
First, we completely agree that the ability of GNNs to capture structural similarity of graphs is very important besides their discriminative power, and we believe this is one of the most important benefits of using GNNs over WL kernel. We have now made this point clearer in Section 4.
Furthermore, we emphasized that we do consider node features to lie in R^d so that they can capture the similarity. The subtlety is that (as R1 nicely pointed out), we need a common assumption that node features at each layer are from countable set in R^d (not from the entire R^d). This is satisfied if the input node features are from a countable set, because for a graph neural network, countability propagates across all layers in a GNN. We leave uncountable node input features for future work and add a more detailed discussion in Section 4 of the revised paper.
In the following, we respond to R3’s other helpful comments and suggestions:
1. RE: Architecture is similar to, e.g., Interaction Networks
Thank you for the pointers. Some of our GIN’s building blocks, e.g. sum and MLP indeed appeared in other architectures. We emphasize that while previous work tend to be somewhat ad-hoc in designing GNN architectures, our main emphasis is on deriving our GIN architecture based on the theoretical motivation. In Section 6 of the revised version, we mention related GNN architectures and discuss the differences.
2. RE: Using MLP for mean or max in the initial step is more fair?
We think there might be a slight misunderstanding here: as we discussed with concrete examples in Section 5.2, mean or max pooling are inherently incapable of capturing the multiset information regardless of the use of MLP. Especially, in our experiments, we use one-hot encodings as input node features, so the use of MLP on top of them does not increase the discriminative power of mean/max pooling.
3. RE: Training set results optimized for test performance?
The results were not actually optimized for test performance.
Instead, we used exactly the same configurations for all the datasets: For all the GNNs, the same configurations were used across datasets: 5 GNN layers (including the input layer), hidden units of size 64, minibatch of size 128, and 0.5 dropout ratio. For the WL subtree kernel, we set the number of iterations to 4, which is comparable to the 5 GNN layers. We clarified this in Figure 6 of the revised paper.
I thank the authors for the revision of the paper and the response.   I have readjusted my rating.
The solution to the question raised by the counter example in the new equation (4.1) is a technical one, I would rather prefer not to simplify the function g(c, X) which uses two functions phi and f in this form, as it really doesn't buy us much.
W.r.t. related work, the statement "Not surprisingly, some building blocks of GIN, e.g. sum aggregation
and MLP encoding, also appeared in other models" (section 6) is not fair and misleading.  As it is not the case that "some building blocks" also appear in other models, but rather some other models, like interaction networks, already contains "all" the essential building blocks (sum, MLP, etc.) presented in this paper.  This doesn't undermine the theoretical contribution of this paper, but the authors should be fair to previous work.
Thank you for the encouraging review! We respond to your further comments below.
1) We probably do not fully understand your comment regarding Eqn (4.1) and g(c,X). Especially, could you please clarify your meaning of “simplify g(c, X)”? In our GIN in Eqn (4.1), we compose phi and f in Corollary 6.
2) We will further edit related work according to your suggestions. Interaction Networks is a great work and we like it.
What I meant was, in g(c, X) you have two functions phi and f, which is the form required by Theorem 3.  The problem of the counter-example comes in when you used a single function instead of 2 functions, which ignores the difference between the node at the center and all its neighbors.
Introducing an epsilon is a technical solution to this problem (in my opinion), I think you actually don't need this because the original form of g(c, X) is enough, and using a single function rather than 2 does not save you much.
Note: I think of phi and f as MLPs, "," as concat, and "{}" as some aggregation operator, like sum.
Thank you for the clarification. We would like to first clarify that the letters (phi, f) in Corollary 6 and Theorem 3 do not have direct correspondence;
but we can easily rearrange Eqn (4.1) to obtain the corresponding (phi, f) in the form of Theorem 3. Intuitively, what Theorem 3 asks for is to injectively represent a pair of a node and its neighbors, so the injective function, g(c, X), corresponds to (phi, f) in Theorem 3.
Furthermore, our motivation for designing Eqn (4.1), i.e. GIN-0 and GIN-eps, rather than simply applying concatenation, is for better empirical performance. In our preliminary experiments, we found such concatenation was harder to train compared to our simple GINs (both GIN-0 and GIN-eps) and achieved lower test accuracy than GINs. The simplicity of GINs brings better performance in practice. We leave the extensive investigation and comparison to our future work.
The author study the expressive power of neighborhood aggregation mechanisms used in Graph Neural Networks and relates them to the 1-dimensional Weisfeiler-Lehman heuristic (1-WL) for graph isomorphism testing. The authors show that GCNs with injections acting on the neighborhood features can distinguish the same graphs that can be distinguished by 1-WL.
Moreover, they propose a simple GNN layer, namely GIN, that satisfies this property.
Moreover, less powerful GNN layers are studied, such as GCN or GraphSage. Their advantages and disadvantages are discussed and it is shown which graph structures they can distinguish.
Finally, the paper shows that the GIN layer beats SOTA GNN layers on well-known benchmark datasets from the graph kernel literature.
Studying the expressive power of neighborhood aggregation mechanisms is an important contribution to the further development of GCNs. The paper is well-written and easy to follow. The experimental results are well explained and the evaluation is convincing.
However, I have some concerns regarding the main result in Theorem 3. A consequence of the theorem is that it makes no differences (w.r.t. expressive power) whether one distinguishes the features of the node itself from those of its neighbors. This is remarkable and counterintuitive, but not discussed in the article.
However, it is discussed in the proof of Theorem 3 (Appendix) which suggests that the number of iterations must be increased for some graphs in order to obtain the same expressive power. Unfortunately, at this point, the proof is a bit vague. I would like to see a discussion of this differences in the article. This should be clarified in a revised version.
Edit:
The counter example posted in a comment ( https://openreview.net/forum?id=ryGs6iA5Km&noteId=rkl2Q1Qi6X&noteId=rkl2Q1Qi6X ) actually shows that my concerns regarding Theorem 3 and its proof were perfectly justified. I agree that the two graphs provide a counterexample to the main result of the paper.
Therefore, I have adjusted my rating. I will increase my rating again when the problem can be resolved.
However, this appears to be non-trivial.
Moreover, the novelty of the results compared to the related work, e.g., mentioned in the comments, should be pointed out.
Some further questions and remarks:
(Q1) Did you use a validation set for evaluation? If not, what kind of stopping criteria did was use?
(Q2) You use the universal approximation theorem to prove Theorem 3. Could you please say something about the needed width of the networks?
(R1) Could you please provide standard deviations for all experiments. I suspect that the accuracies on the these small datasets fluctuates quite a bit.
(R2) In the comments it was already mentioned, that some important related work, e.g., [1], [2], are not mentioned. You should address how your work is different from theirs.
Minor remarks:
- The colors in Figure 1 are difficult to distinguish
[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4703190
[2] https://people.csail.mit.edu/taolei/papers/icml17.pdf
Update:
Most of the weak points were appropriately addressed by the authors and I have increased my rating accordingly.
Thank you for the detailed reviews. In the general post, we have addressed your chief concern regarding our original Eqn (4.1) and part of Theorem 3a). We sincerely hope R2 can revisit the rating in light of our revision and response.
Answers to R2’s other questions:
1. RE: Standard deviations
We added the standard deviations in Table 1. Note that on many datasets, standard deviations are fairly high for the previous methods as well as our methods due to the small training datasets. Our GINs achieved statistically significant improvement on the two REDDIT datasets where the number of graphs are fairly large. We leave the empirical evaluation on larger datasets to future work, but we believe that more expressive GNN models like our GINs can benefit more from larger training data by better capturing important discriminative structural features.
2. RE: Discussion on related work
Following the suggestion, in Section 6 of the revised paper, we discuss the difference of our work to e.g., [1][2].
In short, the important difference is that  [1][2] both focus on the specific GNN architectures, while we provide a general framework for analyzing and characterizing the expressive power of a broad class of GNNs in the literature.
3. RE: Experimental setup and stopping criteria
We selected an epoch with the highest cross-validation accuracy (averaged over 10 folds) following what previous deep learning papers do, e.g., [3][4]. This is for fair comparison as most previous papers on graph classification only report cross-validation accuracy.
4. RE: Network width
Our proofs focus on existential analysis, i.e., there exists a way we can represent multisets with unique representations.
Thus, the network width necessary for the functions provided in our proofs may only serve as an upper bound. For practical purposes, in our experiments, we found 32 or 64 hidden units are usually sufficient to perfectly fit the training set.
[3] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International Conference on Machine Learning (ICML), pp. 2014–2023, 2016.
[4] Sergey Ivanov and Evgeny Burnaev. Anonymous walk embeddings. In International Conference on Machine Learning (ICML), pp. 2191–2200, 2018.
Thanks for your detailed reply. The mentioned weak points 1, 2 and 4 were appropriately addressed by the authors and I have increased my rating accordingly.
Regarding point 3.
>> We selected an epoch with the highest cross-validation accuracy (averaged over 10 folds) following what previous deep learning papers do, e.g., [3][4].
I think there is no common approach to this and the experimental setup in previous papers differs. Many papers use nested cross-validation, others use cross-validation with a fixed validation set, e.g., [5].
Also in [4] a validation seems to be used.
If I understand your method correctly, you report the best accuracy value obtained for any combination of hyperparameters -- instead of applying the classifier with the hyperparameters that work best for a validation set to the test set. In my opinion the approach is problematic.
In particular, comparing to accuracy results obtained with a different experimental setup is not meaningful.
[5] Hierarchical Graph Representation Learning with Differentiable Pooling
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, Jure Leskovec
NeurIPS 2019
Thank you for the response. We address your question regarding experimental setup.
First, past work in graph classification report the best cross-validation accuracy as what we did in our experiments [3]. The graph classification dataset sizes are often small, and therefore using a (single) validation dataset to select hyper-parameters is very unstable (for instance, MUTAG only has 180 data points, so each validation set only contains 18 data points. Compare this to standard deep learning benchmark sets like MNIST that has 70000 data points)**.
Therefore, in our paper, we reported cross-validation accuracy for fair comparison to the previous methods.
Moreover, our GNN variants and the WL kernel all follow the same experimental setups, so the comparison among them is definitely meaningful;
consequently, our conclusion regarding the expressive power is also meaningful. We are planning for future work to evaluate our method on larger datasets, e.g. those mentioned in the post by one of our readers Mr. Christopher Morris, in https://openreview.net/forum?id=ryGs6iA5Km&noteId=B1xLcPaKpQ.
We have thoroughly addressed all the concerns of R2. If Reviewer2 still has other questions or concerns regarding our work, we are happy to answer them.
**[5] uses a test set, but its experiments focus on the larger datasets.
Even though the same approach was used in a previous paper, it is not convincing. Typically the results vary greatly between the epochs. Picking the one with the best validation accuracy leads to unrealistic results.
Also the comparison to the results of the WL kernel is not meaningful since it was obtained with an SVM, where the number of hyperparameters is less.
Therefore, you cannot pick the best value from such a large set of values. It is questionable to speak of "generalization" in the discussion of your results.
I would like to propose to state the method you used more clearly in the paper and check the experimental setup used to obtain the results you have copied from other papers.
Since the main contribution of the paper is theoretical, I will keep my rating, although I think that the experimental setup is a clear weak point.
Regarding experimental setup, we emphasize again that the graph classification datasets are extremely small compared to standard benchmarks in computer vision and NLP, e.g. ImageNet.
Therefore, using a (single) validation dataset to select hyper-parameters is very unstable (for instance, MUTAG only has 180 data points, so each validation set only contains 18 data points).
Therefore, following some of the previous deep learning work, we reported the ordinary cross-validation accuracy (the same hyper-parameters, such as number of epochs and minibatch size, were used for the entire folds). That being said, we understand the existing benchmarks and evaluation for graph classification are limited and we should all move on to large datasets as an anonymous reader pointed out in https://openreview.net/forum?id=ryGs6iA5Km&noteId=ryGs6iA5Km&noteId=H1gkUYX76Q. In the final version, we will also state our experimental setup more clearly. Thank you for your nice suggestion.
This papers presents an interesting take on Weisfeiler-Lehman-type GNNs, where it shows that a WL-GNNs classification power is related to its ability to represent multisets. The authors show a few exemplar networks where the mean and the max aggregators are unable to distinguish different multisets, thus losing classification power. The paper also proposes averaging the node representation with its neighbors (foregoing the “concatenate” function) and using sum pooling rather than mean pooling as aggregator. All these observations are wrapped up in a GNN, called GIN. The experiments on Table 1 are inconclusive, unfortunately, as the average accuracies of the different methods are often close and there are no confidence intervals and statistical tests to help guide the reader to understand the significance of the results.
My chief concern is equating the Weisfeiler-Lehman test (WL-test) with Weisfeiler-Lehman-type GNNs (WL-GNNs). The WL-test relies on countable set inputs and injective hash functions. Here, the paper is oversimplifying the WL-GNN problem. After the first layer, a WL-GNN is operating on uncountable sets. On uncountable sets, saying that a function is injective does not tells us much about it; we need a measure of how closely packed we find the points in the function’s image (a measure in measure theory, a density in probability). On countable sets, saying a function is injective tells us much about the function.
Moreover, the WL-test hash function does not even need to operate over sets with total or even partial orders. As a neural network, the WL-GNN “hash” ($f$ in the paper) must operate over a totally ordered set (\mathbb{R}^n, n > 0). Porting the WL-test argument of “convergence to unique isomorphic fingerprints” to a WL-GNN requires a measure-theoretic analysis of the output of the WL-GNN layers, and careful analysis if the total order of the set does not create attractors when they are applied recursively.
To illustrate the above *attractor* point, let’s consider the construct of Theorem 1 of (Xu et al., 2018), where the WL-GNN “hash” ($f$) is (roughly) described as the transition probability matrix of a random walk on the input graph. Under well-known conditions, the successive application of this operator ("hash" or transition probability matrix P in this case) can go towards an attractor (the steady state). Here, we need a measure-theoretic analysis of the “hash” even if it is bijective: random walk mixing. The random walk transition operator can be invertible (bijective), but we still say the random walker will mix, i.e., the walker forgets where it started, even if the transition operation can be perfectly undone by inversion (P^{-1}). In a WL-GNN that only uses the last layer for classification, this would manifest itself as poor performance in a WL-GNN with a large number of layers, and vanishing gradients.
Of course, since (Xu et al., 2018) argued to revert back to the framework of (Duvenaud et al., 2015) of using the embeddings of all layers, one can argue that this mixing problem is just a problem of “wasted computation”.
The matrix analysis of the last paragraph also points to another potential problem with the sum aggregator. GIN needs to be shallow. With ReLU activations the reason is simple: for an adjacency matrix $A$, the value of $A^j$ grows very quickly with $j$ (diverges). With sigmoid activations, GIN would experience vanishing gradients in graphs with high variance in node degrees.
The paper should be careful with oversimplifications. Simplifications are useful for insight but can be dangerous if not prefaced by clear warnings and a good understanding of their limitations. I am not asking for a measure-theoretic analysis revision of the paper (it could be left to a follow-up paper). I am asking for a *relatively long* discussion of the limitations of the analysis.
Suggestions to strengthen the paper:
•	Please address the above concerns.
•	Table 1 should have confidence intervals (a statistical analysis of significance would be a welcome bonus).
•	Please mention the classes of graphs where the WL-test cannot distinguish two non-isomorphic graphs. See (Douglas, 2011), (Cai et al., 1992) and (Evdokimov and Ponomarenko, 1999) for the examples. It is important for the WL-GNN literature to keep track of the more fundamental limitations of the method.
•	(Hamilton et al, 2017) also uses the LSTM aggregator, besides max aggregator and mean aggregator, which outperforms both max and mean in some tasks. Does the LSTM aggregator also outperforms the sum aggregator in the tasks of Table 1? It is important for the community to know if unusual aggregators (such as the asymmetric LSTM) have some yet-to-be-discovered class-distinguishing power.
--------- Update -------
The counter-example in
https://openreview.net/forum?id=ryGs6iA5Km&noteId=rkl2Q1Qi6X
is indeed a problem for Theorem 3 if  \{h_v^{(k-1)}, h_u^{(k-1)} : u \in \mathcal{N}_v\} is not a typo for a set of tuples \{(h_v^{(k-1)}, h_u^{(k-1)}) : u \in \mathcal{N}_v\}. Unfortunately, in their proof, the submission states "difficulty in proving this form of aggregation mainly lies in the fact that it does not immediately distinguish the root or central node from its neighbors", which means \{h_v^{(k-1)}, h_u^{(k-1)} : u \in \mathcal{N}_v\} is actually \{h_v^{(k-1)}\} \cup \{ h_u^{(k-1)} : u \in \mathcal{N}_v\}, which is not as powerful as WL. Concatenating is more powerful than the summing the node's own embedding, but it results in a  simpler model and could be easier to learn in practice. And I am still concerned about the countable x uncountable domain/image issue I raised in my review.
Still, the reviewers seem to be doing all the discussion among themselves, with no input from the authors. I am now following Reviewer 2.
Reverting my score to my original score. The authors have addressed most of my concerns, thank you. The restricted theorems and propositions better describe the contribution.
I would like to note that while the proof of (Xu et al., 2018) is limited that does not mean it is not applicable to GIN or GraphSAGE or similar models. The paper uses 5 GNN layers, which in my experience is the maximum I could ever use with GNNs without seeing a degradation in performance. I don't think this should be a topic for this paper, though.
Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K., & Jegelka, S. (2018). Representation Learning on Graphs with Jumping Knowledge Networks. In ICML.
Cai, J. Y., Fürer, M., & Immerman, N. (1992). An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4), 389-410.
Douglas, B. L. (2011). The Weisfeiler-Lehman method and graph isomorphism testing. arXiv preprint arXiv:1101.5211.
Evdokimov, S., & Ponomarenko, I. (1999). Isomorphism of coloured graphs with slowly increasing multiplicity of Jordan blocks. Combinatorica, 19(3), 321-333.
Theorem 3 states (as a sidline) that it makes no difference whether we consider a) (label(v), {label(u) : uv in E}) or b) just the set {label(v)} \cup  {label(u) : uv in E}. The set notation used for b) in the paper is a bit unclear, but this appears to be the intended meaning (from the proof and the approach used in section 4.1). For this set, Equation (4.1) yields an injection as claimed.
Therefore the error actually affects Theorem 3, the main result of the paper. Clearly, WL is not perfect (otherwise it would solve the graph isomorphism problem), but that does not make the flaw any less serious. In my opinion, a revision of the authors is absolutely necessary.
We are now working hard for the thorough response and revision to fully address the concern of Reviewer2 and the anonymous reader. Thanks for your patience.
I understood \{h_v^{(k-1)}, h_u^{(k-1)} : u \in \mathcal{N}_v\} as a typo for a set of tuples \{(h_v^{(k-1)}, h_u^{(k-1)}) : u \in \mathcal{N}_v\}.  Which would have been fine.
But you are right that looking at the proof in the appendix, it states "difficulty in proving this form of aggregation mainly lies in the fact that it does not immediately distinguish the root or central node from its neighbors" ... which is not how WL is supposed to work. Thanks!
On top of these issues, WL requires a countable space while their approach operates over uncountable spaces (which remains my main concern). Even reverting to aggregation will not fix this mismatch.
Thank you for the detailed reviews and constructive feedback! We are glad that the reviewer finds our paper interesting. We apologize for the somewhat delayed response; it took us time to run additional experiments and add more careful analysis so that we can present an improved and more polished paper to everyone. We appreciate your understanding.
In the following, we first address the main concern on equating the WL test and the WL-GNNs by showing its validity under a mild practical assumption.
Then, we clarify the misunderstanding regarding the random walk mixing behavior of the WL-GNNs, showing that our GIN architecture does not suffer from such behavior.
Finally, we discuss confidence intervals of our experimental results and also address other concerns of the reviewer.
1. RE: Validity of equating the WL test operating on countable sets to the WL-GNN operating on uncountable sets.
The reviewer makes a great observation that countability of node features is essential and necessary for our theory, and we acknowledge that our current Theorem 3 and Lemma 5 are built on the common assumption that input node features are from a countable universe. We have now made this clear in our paper. We also filled in a technical gap/detail to address R1’s concern that after the first iteration, we are in an uncountable universe: this actually does not happen. We can show that for a fixed aggregation function, hidden node features also form a countable universe, because the countability of input node features recursively propagates into deeper layers.
We also added a rigorous proof for this (Lemma 4 in our revised paper). As the reviewer nicely suggests, for the uncountable setting, it would be useful to have measure-theoretic analysis, which we leave for future work. Often input node features in graph classification applications (e. g. , chemistry, bioinformatics, social) come from a countable (in fact, finite) universe, so our assumption is realistic. In the revised version, we clearly stated our assumptions at the beginning of Section 3 and have added further discussion on the relation between the WL test and WL-GNN after Theorem 3.
2. RE: Random walk mixing behavior of the GIN architecture.
We think there might be a slight misunderstanding here: (1) Theorem 1 of (Xu et al., 2018) relates the random walk to the influence distribution in Definition 3.1 of (Xu et al., 2018), rather than the precise node representation, and (2) the analysis of Theorem 1 is specific to the GCN architecture (Kipf & Welling, 2017), where 1-layer perceptrons with mean pooling are used for neighbor aggregation. The GIN architecture does not suffer from the problem of random walk mixing because (1) Theorem 1 in (Xu et al., 2018) shows the influence distribution converges to a random walk limit distribution, however, it does not yet tell whether the node representations converge to the random walk limit distribution.
Thus, “the walker forgetting where it started” may not happen.  (2) The GIN architecture uses MLPs rather than the 1-layer perceptron in (Kipf & Welling, 2017). The analysis in (Xu et al., 2018) specifically applies to models using 1-layer perceptrons, and therefore, it is not clear whether this analysis still holds for GIN.
Furthermore, the reviewer is concerned with a possibly exploding value due to the sum aggregation, but this can be avoided because we have different learnable neural networks at each layer that can scale down the summed output (also, in practice, we did not observe such explosion).
3. RE: Confidence interval in experiments
Following the suggestion, we added the standard deviations in Table 1. Because of space limit, we only added standard deviation in Table1, and confidence interval can be obtained via the standard deviation. The confidence interval of 95% is mean 0.754*std, and confidence interval of 90% is mean 0.611*std. Note that on many datasets, standard deviations are fairly high for the previous methods as well as our methods due to the small training datasets. Our GINs achieved statistically significant improvement on the two REDDIT datasets where the number of graphs are fairly large. We leave the empirical evaluation on larger datasets to future work, but we believe that more expressive GNN models like our GINs can benefit more from larger training data by better capturing important discriminative structural features.
4. Other comments:
We also thank the reviewer for many other comments to strengthen our paper. In the revised paper, we clarified that WL-test cannot distinguish e.g., regular graphs. We discussed in Section 5.5 that the expressive power of other poolings such as LSTM and attention pooling can be analyzed under our framework, but we leave the empirical investigation to future work.
For 10-fold cross validation, it is important to highlight that it tends to underestimate the confidence interval range (see (Bengio and Grandvalet, 2004)). Important to let readers know that there is more uncertainty in the results, which was not quantified.
I also find the use of boldface confusing. Summing and subtracting the confidence intervals and a lot more models overlap.
Bengio, Yoshua, and Yves Grandvalet. "No unbiased estimator of the variance of k-fold cross-validation." Journal of machine learning research 5, no. Sep (2004): 1089-1105.
Thank you for the detailed response.
Regarding the depth of the networks, GIN does not suffer from the curse of depth, i.e. we can use many layers, because we apply architectures similar to JK-Nets (specifically, JK-Concat) in Xu et al. 2018 for readout as described in Section 4.2. We conducted graph classification experiments using 5-layers GNNs (with JK-net) and they work nicely in our experiments.
Moreover, as R1 nicely suggested, the influence distribution expansion phenomenon in Xu et al. 2018 indeed would apply to GraphSAGE, GIN etc, though the transition probabilities may not follow canonical random walks when MLP is applied. That being said, Xu et al. 2018 is a great work and we like it. We just wanted to clarify that Theorem 1 was about influence distribution rather than node features, thus, there would be no issue for GIN in terms of invertibility. We hope you are happy with our clarification.
Regarding cross-validation, thanks for letting us know the work. We will mention it in the final version. To clarify, we use the boldface to indicate the best performance in terms of mean accuracy. As we mentioned in the rebuttal, the graph classification benchmark datasets are extremely small compared to other standard deep learning benchmarks for computer vision or NLP, e.g. ImageNet. That’s why standard deviations are high for all the methods (including the previous methods). We do believe we all should move beyond the conventional evaluation on these small datasets, but that is beyond the scope of this paper.
Thank you again for your nice suggestions and detailed reviews. We hope our clarification regarding the analysis addresses your concerns.
Thank you for the detailed response.
Regarding the depth of the networks, GIN does not suffer from the curse of depth, i.e. we can use many layers, because we apply architectures similar to JK-Nets (specifically, JK-Concat) in Xu et al. 2018 for readout as described in Section 4.2. We conducted graph classification experiments using 5-layers GNNs (with JK-net) and they work nicely in our experiments.
Moreover, as R1 nicely suggested, the influence distribution expansion phenomenon in Xu et al. 2018 indeed would apply to GraphSAGE, GIN etc, though the transition probabilities may not follow canonical random walks when MLP is applied. That being said, Xu et al. 2018 is a great work and we like it. We just wanted to clarify that Theorem 1 was about influence distribution rather than node features, thus, there would be no issue for GIN in terms of invertibility. We hope you are happy with our clarification.
Regarding cross-validation, thanks for letting us know the work. We will mention it in the final version. To clarify, we use the boldface to indicate the best performance in terms of mean accuracy. As we mentioned in the rebuttal, the graph classification benchmark datasets are extremely small compared to other standard deep learning benchmarks for computer vision or NLP, e.g. ImageNet. That’s why standard deviations are high for all the methods (including the previous methods). We do believe we all should move beyond the conventional evaluation on these small datasets, but that is beyond the scope of this paper.
Thank you again for your nice suggestions and detailed reviews. We hope our clarification regarding the analysis addresses your concerns.
We also conducted extensive experiments on the modified GIN architecture with Eq. (**), where we learn epsilon by gradient descent. We included the additional results in Section 7 of our revised paper.
In terms of training accuracy, which is the main focus of our paper, we observed from our new Figure 4 (in the revised paper) that the modified GIN (we call it GIN-eps in our paper) gives the same results as our original GIN (GIN-0) does, showing no improvement on the training accuracy. This is because the original GIN already fits the training data very well, achieving nearly 100% training accuracy on almost all of our datasets.
Consequently, the explicit learning of epsilon in the modified GIN (GIN-eps) does not help much. Interestingly, in terms of the test accuracy, we observed from Table 1 (in the revised paper) that for GIN-eps (modified GIN) there is a slight drop in test accuracy  (0.5% on average) compared to GIN-0 (original GIN). Since GIN-0 and GIN-eps showed almost no difference in training accuracy, both have sufficient discriminative power on this data, and the slight drop in test accuracy should be explained by generalization rather than expressiveness. We leave the investigation of the effectiveness of GIN-0 for future work. We want to emphasize that the pooling scheme (sum vs. average vs. max) and mapping scheme (MLP vs. linear) does affect the performance w.r.t. training accuracy, and consequently also affects the test accuracy.
Thus, our main findings distinguishing the sum-MLP architecture from other aggregation schemes for maximally expressive GNNs is still valid.
As a final remark, as R1 nicely commented, instead of Eq. (**), a node and neighbors can be concatenated, rather than summed, to achieve the same power as the WL test. Interestingly, as R1 cleverly predicted, in our preliminary experiments, we found such concatenation was harder to train compared to our simple GINs (both GIN-0 and GIN-eps) and achieved lower test accuracy. We leave the extensive investigation and comparison to our future work.
We sincerely appreciate the reviewer and commenter for the great suggestions and insights, which enabled us to further strengthen our work and make our paper stronger. We hope our new version resolves the reviewers’ main concerns.
We begin by acknowledging that Eqn (4.1) and Theorem 3a-Eqn.2) in our initial submission (which does not distinguish the center nodes from their neighbors) were indeed insufficient to be as powerful as the WL test. The example provided by the anonymous reader makes a great point about the corner case.
That said, we agree that in order to realize the most powerful GNN, its aggregation scheme needs to distinguish the center node from its neighbors.
The good news is that we can resolve this corner case by making a very simple modification to our GIN aggregation scheme in Eq. (4.1) of the initial submission, so that the modified GIN can provably distinguish the root/center node from its neighbors during the aggregation. This implies that our modified GIN handles the counter-example raised by the anonymous reader, and, more importantly, we can prove that the modified GIN is as powerful as the WL test under the common assumption that the input node features are from a countable universe. In the following, we will explain these points in more detail.
First, we present a simple update to our current GIN aggregation scheme, and show that it now handles the counter-example provided by the anonymous reader. Our simple modification to the original GIN aggregation in Eq. (4.1) of the initial submission is:
h_v^{(k)} = MLP ( (1 + \epsilon) h_v^(k-1) + \sum_{u \in neighbor} h_u^(k-1) ), (**), Eq. (4.1) of the revised paper.
where \epsilon is a fixed or learnable scalar parameter. We will show that there exist infinitely many \epsilon where the modified GIN (as defined above) is as powerful as WL. Note that setting \epsilon = 0 reduces to our original GIN aggregation in Eq. (4.1) of the initial submission.
Thus, the above equation (Eq. (**)) smoothly “extrapolates” the original GIN architecture, and with the epsilon term, the modified GIN can now distinguish the center node from its neighbors. Before moving to the formal proof, let us first illustrate how modified GIN handles the counter-example by the anonymous reader:
R - R                 R - G
|      |     v.s.    |      |
G - G                G - R
Assume we use the one-hot encodings for the input node features, i.e., R = [1, 0] and G = [0, 1]. After 1 iteration of aggregation defined by Eq. (**), our modified GIN obtains the following node representations (before applying MLP in (**));
thus, it successfully distinguishes the two graphs with small non-zero eps=\epsilon.
[2+eps, 1]  -- [2+eps, 1]                  [1+eps, 2] -- [2, 1+eps]
|                            |           vs.         |                            |
[1, 2+eps] -- [1, 2+eps]                   [2, 1+eps] -- [1+eps, 2]
The key here is that with non-zero (small) eps, [2+eps, 1] and [2, 1+eps] are now different.
In other words, adding \epsilon term in Eq. (**) enables the modified GIN to “identify” the center nodes and distinguish them from neighboring nodes.
With the intuition above, we now give a formal proof for the modified GIN architecture. We start with Lemma 5 (universal multiset functions) in our revised paper, and extend it to Corollary 6 in the revised paper that can distinguish center node from the neighboring nodes.
Crucially, the function h(c, X) in Corollary 6 is now the injective mapping over the *pair* of a center node c and its neighbor multiset X. This implies that h(c, X) in Corollary 6 can distinguish center nodes from their neighboring nodes.
Corollary 6
Assume \mathcalcal{X} is countable. There exists a function f: \mathcal{X} → R^n so that for infinitely many choices of \epsilon, including all irrational numbers, h(c, X) \equiv (1 + \epsilon) f(c) + \sum_{x \in X} f(x) is unique for each pair (c, X), where c \in \mathcal{X}, and X \subset \mathcal{X} is a finite multiset.
---Proof sketch (details are provided in Appendix of the revised paper, see Proof of Corollary 6)
The proof builds on Lemma 5 that constructs the function f that maps each finite multiset uniquely to a rational scalar with N-digit-expansion representation. With the same choice of f from Lemma 5, the irrationality of \epsilon enables us to distinguish the center node representation c from any combination of multiset representation, which is always rational.
That is, h(c,X) is unique for each unique pair (c,X).
Using h(c, X) for the aggregation, we can straightforwardly derive our modified GIN aggregation in Eq. (**) (similarly to the MLP-sharing-across-layer trick described after Lemma 5.) We included a detailed derivation in Section 4.1 of the revised paper.
We sincerely appreciate all the reviews, they give positive and high-quality comments on our paper with a lot of constructive feedback. We also thank the many anonymous commenters for their interest and helpful discussion. In the revised paper, we did our best to address the concerns and suggestions to strengthen our paper. We sincerely hope reviewers revisit the rating in light of our revision and response. The following summarizes our revisions. Please see our rebuttal for the detailed discussion.
Major revisions:
1. An anonymous reader and Reviewer2 made a clever observation that our original GIN aggregation in Eq. (4.1) and Theorem 3a-Eqn.2) of the initial submission and cannot distinguish certain corner case graphs that the WL test can distinguish. We fixed this issue by 1) making a slight modification to GIN’s aggregation in Eq. (4.1), and 2) adding Corollary 6 to show Eqn. (4.1) in the revised paper is as powerful as WL, 3) removed Theorem 3a-Eqn.2). The modified GIN aggregation smoothly extrapolates the original one, avoids the corner case, and can be shown to be as powerful as the WL test. We conducted extensive experiments on the modified GIN to further validate our model. (see below https://openreview.net/forum?id=ryGs6iA5Km&noteId=ryGs6iA5Km&noteId=SyeZ3MU4AX for our detailed response.)
2. Based on the helpful comments of Reviewer1 on countability of node features, we have now made our setting much clearer: We clarified the common assumption that input node features are from a countable set, and we further added Lemma 4 in the revised paper to prove that the hidden node features are also always from a countable set under this assumption. With the countability assumption, it is meaningful to discuss injectiveness in Theorem 3, and our countability assumption used in Lemma 5 (universal multiset functions) always holds. We also provided detailed discussion on the correspondence between the WL test and WL-GNN under the countability assumption, validating our theory to equate those two.
Minor revisions:
1. R3 makes a great point that beyond distinguishing different graphs, it is equally important for GNNs to capture their structural similarity. We have already mentioned this point after Theorem 3. We now made this clearer and added a more detailed discussion in Section 4.
2. In response to R3 and R2, we added Section 6 for detailed discussion of related work.
3. Following the suggestions by R1 and R2, we added standard deviations in the experiments.
4. Based on the great insight by an anonymous reader, we added discussion on the expressive power of Sum-Linear when the bias term is included.

Meta-review:
role: AC
data: metareview: Graph neural networks are an increasingly popular topic of research in machine learning, and this paper does a good job of studying the representational power of some newly proposed variants. The framing of the problem in terms of the WL test, and the proposal of the GIN architecture is a valuable contribution. Through the reviews and subsequent discussion, it looks like the issues surrounding Theorem 3 have been resolved, and therefore all of the reviewers now agree that this paper should be accepted. There may be some interesting followup work based on studying depth, as pointed out by reviewer 1, but this may not be an issue in GIN and is regardless a topic for future research.

Decision: Accept (Oral)