Decoupling the Depth and Scope of Graph Neural Networks

Abstract: State-of-the-art Graph Neural Networks (GNNs) have limited scalability with respect to the graph and model sizes. On large graphs, increasing the model depth often means exponential expansion of the scope (i.e., receptive field). Beyond just a few layers, two fundamental challenges emerge:  1. degraded expressivity due to oversmoothing, and 2. expensive computation due to neighborhood explosion. We propose a design principle to decouple the depth and scope of GNNs – to generate representation of a target entity (i.e., a node or an edge), we first extract a localized subgraph as the bounded-size scope, and then apply a GNN of arbitrary depth on top of the subgraph. A properly extracted subgraph consists of a small number of critical neighbors, while excluding irrelevant ones. The GNN, no matter how deep it is, smooths the local neighborhood into informative representation rather than oversmoothing the global graph into “white noise”. Theoretically, decoupling improves the GNN expressive power from the perspectives of graph signal processing (GCN), function approximation (GraphSAGE) and topological learning (GIN). Empirically, on seven graphs (with up to 110M nodes) and six backbone GNN architectures, our design achieves state-of-the-art accuracy with orders of magnitude reduction in computation and hardware cost.


Reviews and author responses:

 Dear reviewer k7tB,


First of all, we thank you again for the valuable feedback on our paper! As there is only about 1 week left for the discussion phase, we would greatly appreciate it if you could confirm whether our response has addressed your concerns. 

Please allow us to briefly summarize our previous response. In our posts on Aug 10, we have
* (Weakness 1) Clarified the significance and correctness of our theoretical results from various perspectives. In particular, we have
  * (Limitation 1) Explained why our oversmoothing analysis, even though intuitive, leads to powerful / useful insights;
  * (Limitation 2) Provided tighter error bounds for approximating the $\tau$ function by shaDow-SAGE;
  * (Limitation 3) Explained why the expressive power comparison between shaDow-GIN and 1-WL is correct and fair. 
* (Weakness 2) Explained the differences between the sampling based GNN and our decoupling based GNN, from various perspectives. 
* (Weakness 3) Pointed out existing experiments evaluating “non-oversmoothing” baselines (DropEdge and JK-Net); Provided additional results on PairNorm and DAGNN according to your valuable suggestions.

If you think there is still any other issue, we are more than happy to follow up with you before the end of the discussion phase. Please kindly let us know. 


Thanks,

Authors

 We thank the reviewer again for the positive feedback. Yes, we will make sure to integrate the suggested changes from *all* the reviewers in our next revision. 

 We thank the reviewer for the insightful and constructive feedback. We address each of the concerns in the following.

> Most of the theoretical analysis is straightforward

Please see below for detailed responses to each of the reviewer’s concerns on the theorems. 

> The idea of decoupling depth and scope of GNNs is not new, especially from the perspective of sampling neighborhoods such as in FastGCN [6].

Our shaDow-GNN is fundamentally different from sampling approaches. We have described their differences in Sections 1 and 4. We restate the key points here:
1. Despite both changing the receptive field, sampling approaches aim to efficiently approximate some quantities related to the full graph (such as approximating the aggregation of the full $L$-hop neighborhood), while shaDow-GNN inherently does not aim at estimating any full-graph quantities because it is designed to look at only locally related subgraphs. This conceptually distinguishes shaDow-GNN from sampling approaches.
2. Sampling approaches can only accelerate the training computation, while shaDow-GNN naturally accelerates both the training and inference. Training (multiple epochs) is more tolerant to the information loss from sampling, while inference generates the predictions only once. That’s why practical implementations of sampling approaches apply sampling only during training, but not during inference (in order to avoid significant accuracy loss). In contrast, shaDow-GNN applies the same EXTRACT function on both the training and test data. This explains why shaDow-GNN achieves significant reduction in inference computation cost, even compared with sampling based models such as GraphSAINT (see Table 1). 
3. **Sampling approaches do not decouple the depth and scope of GNNs** -- the receptive field of an $L$-layer FastGCN is still (on expectation) the full $L$-hop neighborhood. In contrast, the key insight of shaDow-GNN is to decouple the depth and scope, which allows using $L’$-layer GNNs on the $L$-hop neighborhood (where $L’>L$).

> If the performance gain is because the proposed approach addresses the oversmoothing problem, a comparison should be made with recent approaches for the same purpose

Our current experiments have already covered some recent models / techniques to address oversmoothing.
* The Table 1 baselines have all been tuned with DropEdge [38], a popular regularization technique to address oversmoothing. The tuning procedure is detailed in Appendix E.4. 
* In Appendix F.2, Table 12 shows a comparison with the JK-Net [50] backbone. Jumping Knowledge is a popular skip-connection architecture which can address oversmoothing (see lines 932 to 934). 

Nevertheless, we thank the reviewer for mentioning other related works. In our original submission, we have cited PairNorm and “Towards Deeper Graph Neural Networks” (i.e. DAGNN). We will cite “Scattering GCN” in our next revision. In the following table, we present additional experimental results for PairNorm and DAGNN (for DAGNN, we tune its order in the range of {1, 3, 5, 10}). Due to the time constraint, we cannot finish the evaluation on Scattering GCN. 


|          | Flickr | Reddit | ogbn-arxiv | 
| ------- | ------- | -------- |  ------------- |
| GCN (PairNorm) L = 3 | 0.5165 | 0.9540 | 0.7159 |
| GCN (PairNorm) L = 5 | 0.5210 | 0.9529 | 0.7167 | 
| GCN (DropEdge) L = 3 | 0.5159 | 0.9532 | 0.7170 |
| GCN (DropEdge) L = 5 | 0.5217 | 0.9495 | 0.7186 |
| DAGNN | 0.4973 | 0.9302 | 0.7051 |


*Conclusion from the above table*: Compared with the normal GCN (non-shaDow version), applying PairNorm or using DAGNN does not bring additional accuracy gains. As our original GCN baselines have already been fine-tuned with DropEdge, oversmoothing has been addressed to a large extent even without PairNorm and DAGNN. We also find that the DAGNN accuracy is lower: Even though DAGNN addresses oversmoothing, its expressive power may be limited due to the lack of trainable parameters in the message passing steps. For shaDow-GCN, its performance gains w.r.t. accuracy and efficiency have already been evaluated and explained in detail in Section 5. 

Note that Figure 2 directly shows the empirical benefits brought by non-oversmoothing of shaDow-GNN. 

> I’m not sure whether Theorem 3.2 is useful. If we use a small subgraph around each node for inference, the prediction is made by the features of nodes contained in the subgraph. Then, it is straightforward that two nodes in different subgraphs make different representations, as they combine different sets of node features. 

The reviewer’s summary is consistent with the intuition we have provided in Section 3.1 (lines 157 to 161). While the intuition is simple, it still takes a considerable amount of careful thinking to rigorously formulate and prove it (see Appendix A.1 for details). 

Despite conveying a simple intuition,Theorem 3.2 is still useful. Firstly, it directly proves that shaDow-GNN does not oversmooth in the general setting. The condition on EXTRACT is also mild (see our response to the next point). Secondly, from Theorem 3.2 and the analysis in Section 3.1, we see how structure and feature information each play their role in the non-oversmoothed embeddings. From Proposition 3.1 and its analysis (lines 162 to 167), we see how a deep shaDow-GCN preserves rich structure information. Then, since Theorem 3.2 holds for any structure-based $\phi_G(v)$ function, we conclude that preserving feature information is indeed another key reason for the final embeddings being distinguishable. 

> The theorem is useful only if the authors provide the exact condition that makes the two representations the same or different.

The exact condition has been specified in Theorem 3.2 and its corollaries -- EXTRACT needs to return non-identical subgraphs for different target nodes. Such a condition is mild and can be empirically satisfied by practical EXTRACT functions. To support our claim, we additionally perform the following experiment. Given a graph, we randomly pick a target node $v$ and obtain its $G_{[v]}$. We then compare $G_{[v]}$ with $G_{[u]}$ for all $u \neq v$ in the graph. We report on expectation, how many $G_{[u]}$ are identical to $G_{[v]}$. For the PPR based EXTRACT following Table 8’s hyperparameters, the expected number of $u$ such that $G_{[u]} = G_{[v]}$ is: 0.0155 (Reddit), 0.219 (ogbn-arxiv) and 0.1235 (ogbn-products). This means, on ogbn-products of millions of nodes, only less than 1 (0.1235 in expectation) pair of nodes will have the same subgraph, which is negligible.   

(Please proceed to [Part 2] ...)

 Thank you for your response.
I expect that  you can implement the suggestions provided by the reviewers.

 We appreciate the valuable feedback from the reviewer. 

> it is running a GNN on a new altered graph...
> I do think a better comparison to graph-altering work could be interesting

We agree that shaDow-GNN is related to works performing graph alterations / reconstruction (e.g., APPNP [22], GDC [23], AM-GCN [46], SIGN [9] and PPRGo [4] as discussed in our submission). Yet, we believe that the key idea of shaDow-GNN is quite different and original. In Section 4 (lines 297 to 303), we have summarized the similarities and differences between the two types of works. In Section 3.5 (lines 270 to 277), the case study shows how shaDow-GNN can be more expressive than PPRGo [4] by not altering the original edge structure. Specifically, 
* **Similarity**: Both graph reconstruction works and shaDow-GNN generalize the concept of “neighborhood”. 
* **Differences**: 
    * ShaDow-GNN decouples the depth and receptive field of the GNN, while graph reconstruction works do not perform decoupling. From our theoretical and empirical analysis, making the layers deeper than the receptive field can improve the expressive power and accuracy. 
    * ShaDow-GNN does not alter the connections between nodes. EXTRACT filters out subgraph nodes, and then preserves the original graph structure by returning “induced subgraph” (see line 761 and lines 771 to 773 in Appendix C). Keeping the original edge connections help preserve more structural information (lines 270 to 277). 
* **Connections**: graph altering work can be further decoupled by our technique. By lines 301 to 302, “shaDow-GNN can also work with the reconstructed graph $G’$ by simply applying EXTRACT on $G’$”. 

We will clarify the above discussion in our revised paper.

> I suppose EXTRACT_1 and EXTRACT_2 are 1 and 2 hops respectively, but I don’t think that’s defined. 

EXTRACT_1 and EXTRACT_2 are two abstract functions, defined in Corollary 3.2.1 and 3.2.2. They do not necessarily return the 1 and 2 hop neighborhoods. 

There are many EXTRACT algorithms that can satisfy the conditions specified by Corollary 3.2.1 and 3.2.2. For example, consider the PPR based EXTRACT (used in our experiments in Section 5):
* Since we take the top-$p$ nodes with the highest PPR score (lines 763 to 770), then PPR based EXTRACT belongs to EXTRACT_1.
* Empirically, the PPR based EXTRACT returns non-identical $V_{[v]}$ and $V_{[u]}$ for different $u$ and $v$. So in practice, we can also regard it as belonging to EXTRACT_2. 

> To some extent, being more powerful is somewhat misleading in the sense that we’re restricting (altering) the graph via EXTRACT, so we’re giving the NN more (conditional on human choices and adding additional bias) information.

We agree with the reviewer that EXTRACT is an important inductive bias in shaDow-GNN. On the positive side, the proposed decoupling principle provides a way to efficiently utilize such human knowledge to enhance any GNN architecture. In addition, we can treat EXTRACT as a hyperparameter and search for the best EXTRACT algorithm through cross validation.


 (Continue from [Part 1])

> It is not clear when GraphSAGE fails at expressing the $\tau$ function

According to the reasoning in Section 3.2, GraphSAGE can express $\tau$ only if the following two conditions are satisfied:
1. $\tau$’s input $G_{[v]}$ is the $L$-hop neighborhood of $v$
2. There exists a linear combination of the row vectors $A_{v,:}^{\ell}$ (where $1\leq \ell \leq L$) generating the coefficients of the linear function $\tau$. 

The above conditions are quite restrictive -- only very special neighborhood structures may satisfy condition 2. In our example in Appendix A.3, even when $G_{[v]}$ is a regular subgraph (i.e., $A$ has identical entries), the linear combination of $A_{v,:}^{\ell}$ still cannot generate a uniform coefficient vector. In general, condition 2 is not satisfied and thus GraphSAGE cannot express $\tau$. 

(Note that there is a typo in the equations below line 696, Appendix A.3. The three rows should be as follows (i.e., a permuted version of the original writing)

$1, 0, 0, 0, 0, 0, 0, 0$

$0, \frac{1}{3}, \frac{1}{3}, 0, \frac{1}{3}, 0, 0, \frac{1}{3}$ 

$\frac{1}{3}, \frac{1}{9}, \frac{1}{9}, \frac{1}{9}, 0, \frac{1}{9}, \frac{1}{9}, \frac{1}{9}$
)

> The authors say in Section 3.2 that “SHADOW-SAGE can approximate $\tau$ with error decaying exponentially with the model depth,” but the error bound is not given...
>It is not clear … how exactly SHADOW-SAGE performs better.

The error bound can be straightforwardly obtained by applying the theory of Markov chain. For the sake of reader’s interest, we provide a brief proof sketch in the following.

Due to the way GraphSAGE (or shaDow-SAGE) normalizes the adjacency matrix $A$, we can regard $A$ as the transition matrix of a Markov chain. The coefficients of the linear function $\tau$ equal the stationary distribution of such a Markov chain. Therefore, we can use the convergence theorem of Markov chain to characterize the convergence of $A^{L’}$ towards $\tau$’s coefficients. We can also use the mixing time of the Markov chain to derive the error bound of shaDow-SAGE. All detailed mathematical derivations can be found in [26]. We now make a brief summary. 
1. Directly applying Theorem 4.9 of [26], we know that the error of shaDow-SAGE approximating $\tau$ reduces exponentially with the model depth (where model depth corresponds to transition step in the Markov chain). As the convergence theorem of Markov chain does not depend on the initial state, the exponential decaying of the approximation error also holds for any node feature $X_{[v]}$. 
2. From Equation 4.36 of [26], we can directly relate the mixing time of Markov chain with the required shaDow-SAGE depth to reach any $\epsilon$ error. 
3. Now we only need to characterize the mixing time to derive the final error bound of shaDow-SAGE. By Theorem 12.3 of [26], the mixing time can be bounded by the “absolute spectral gap” of the transition matrix (i.e., the normalized subgraph adjacency matrix). Note that 1. the absolute spectral gap is calculated from the eigenvalues of the transition matrix. 2. Theorem 12.3 applies when the Markov chain is reversible. Such a condition is satisfied by the GraphSAGE style of normalization. 

Putting the above three points together, it is straightforward to derive the mathematical expression for the error bound of shaDow-SAGE. Since the OpenReview platform has limited support for math equations, we omit the step-by-step equations here. 

> The authors should rewrite Section 3.2 and provide formal theorems and proofs.

Thanks for the suggestion. We will do that in the next revision. 

> I’m not sure Theorem 3.3 is a fair comparison between SHADOW-GNN and the WL test, because making the subgraph for each node makes a severe limitation on the expressiveness. I think there is no single setting that SHADOW-GNN generalizes 1-WL in any cases, unlike the conclusion that the paper implies.

*Theorem 3.3 is correct and fair.* In statistical learning theory, when we compare the capacity or discriminating power of two classifiers, we are comparing two *function classes*, instead of two function realizations. A function class contains a set of functions with tunable hyperparameters and learnable parameters. Here, shaDow-GNN is a function class with $L$, $L’$ being its hyperparameters, and 1-WL (as a classifier) is also a function class with the number of iterations being its hyperparameter. When we state shaDow-GNN is more expressive than 1-WL, we mean the following: 
1. For all target nodes and graphs, if there exists some function in the 1-WL function class capable of discriminating the target nodes, then we can always find the corresponding shaDow-GNN model (i.e. find the corresponding $L$ and $L’$) in the shaDow-GNN function class to discriminate such target nodes. 
2. There exists some target nodes in some graph, where no function in the 1-WL function class can discriminate the target nodes, while some function in the shaDow-GNN function class can discriminate the target nodes. 

Now, it should be clear from Appendix A.2 that shaDow-GNN is more discriminative than 1-WL. The proof exactly follows the two points above. 

> For instance, consider two graphs with the same node features having the following edge sets: ■ E1 = {(a, c), (b, c), (c, d)} ■ E2 = {(e, f), (f, g)}. Then, SHADOW-GNN described in the paper cannot distinguish between d and g, as it looks at only the direct neighborhood of each node. 

Based on the above clarification, $L$ of shaDow-GNN is a hyperparameter and is not restricted to 1. In the reviewer’s example, although an $L=1$ shaDow-GNN cannot discriminate $d$ and $g$, a shaDow-GNN with $L=2$ (and $L’>L$) can. Similarly, a 1-WL with 1 iteration also cannot discriminate $d$ and $g$, but a 1-WL with more than 2 iterations can do so.

 We appreciate the positive feedback from the reviewer. We also believe that shaDow-GNN has the potential to significantly impact realistic, large scale graph learning tasks. 

> while a rigorous theoretical evaluation of cost is provided, it would have been interesting to show also empirical training/inference times

Thanks for the suggestion. We will include the measured computation time in the next revision. In the original version, we have included only the theoretical computation cost in the main experiment table (Table 1), since this metric is independent of hardware and implementation strategies. As shown in Appendix E.2, we have run experiments on three machines ranging from a low-end desktop to a mid-end server. The actual computation time differs significantly from machine to machine. 

In Figure 8 of Appendix F.5, we have also included the actual inference time for shaDow-GNN and the normal versions, using Machine 3. 

This paper proposes a design principle to decouple the depth and scope of graph neural networks (GNN). The proposed approach first extracts a localized subgraph for each target node as a bounded-size receptive field and then applies a GNN of arbitrary depth on top of the subgraph. Such decoupling improves the expressive power of popular GNN architectures including GCN, GraphSAGE and GIN. On seven graphs and six backbone GNN architectures, the design achieves state-of-the-art accuracy with a reduction in computation cost.  Strengths
- The performance of the proposed approach is significantly better than the baselines for various backbone GNNs.
- The authors present theoretical analysis on the expressive power of the proposed design principle.
- Experiments are presented for both node classification and link prediction, which are representative problems in the graph domain.

Weaknesses
- Most of the theoretical analysis is straightforward (details in the next section).
- The idea of decoupling depth and scope of GNNs is not new, especially from the perspective of sampling neighborhoods such as in FastGCN [6].
- There are experiments only with base GNN models. If the performance gain is because the proposed approach addresses the oversmoothing problem, a comparison should be made with recent approaches for the same purpose:
--- PairNorm: Tackling Oversmoothing in GNNs [ICLR 2020]
--- Scattering GCN: Overcoming Oversmoothness in Graph Convolutional Networks [NeurIPS 2020]
--- Towards Deeper Graph Neural Networks [KDD 2020]
 I’m not sure whether Theorem 3.2 is useful. If we use a small subgraph around each node for inference, the prediction is made by the features of nodes contained in the subgraph. Then, it is straightforward that two nodes in different subgraphs make different representations, as they combine different sets of node features. The theorem is useful only if the authors provide the exact condition that makes the two representations the same or different.

The authors say in Section 3.2 that “SHADOW-SAGE can approximate \tau with error decaying exponentially with the model depth,” but the error bound is not given. It is not clear when GraphSAGE fails at expressing the tau function, and how exactly SHADOW-SAGE performs better. The authors should rewrite Section 3.2 and provide formal theorems and proofs.

I’m not sure Theorem 3.3 is a fair comparison between SHADOW-GNN and the WL test, because making the subgraph for each node makes a severe limitation on the expressiveness. For instance, consider two graphs with the same node features having the following edge sets:
■	E1 = {(a, c), (b, c), (c, d)}
■	E2 = {(e, f), (f, g)}

Then, SHADOW-GNN described in the paper cannot distinguish between d and g, as it looks at only the direct neighborhood of each node. I think there is no single setting that SHADOW-GNN generalizes 1-WL in any cases, unlike the conclusion that the paper implies.


The authors propose a new approach to graph neural models that decouples layer depth and vertex receptive field: a subgraph is extracted for each vertex, and vertex representations are produced for each vertex via a multi-layer graph convolutional network. Authors theoretically prove better expressive capabilities of their approach, and reduced computational cost on very large-scale datasets. Experimental results show consistent improvement on graph classification tasks. 

  While other approaches to graph sampling in order to reduce computational cost on large-scale graphs have been proposed, this paper appears to be the first to frame it as a strategy to decouple GNN layer depth and vertex receptive field. Furthermore, their approach allows for more representative models with respect to graph features, since graph extraction strategies can produce non 1-WL isomorphic subgraphs.
The paper is clearly written, with design choices justified by theoretical proofs; experiments are rigorously conducted on well-established vertex classification tasks, showing overall improvements both regarding accuracy and computational cost with respect to common graph models (while a rigorous theoretical evaluation of cost is provided, it would have been interesting to show also empirical training/inference times).
Due to the significant advantage in relation to computational cost and memory requirements, the proposed methodology is likely to impact significantly on large-scale (i.e. >millions vertices) graph tasks.
 No negative societal impact to report.

This paper explores on approach of decoupling depth and scope of GNNs. To mitigate degraded expressivity due to over smoothing and expensive computation due to neighborhood explosion, their shadow-GNN approach first extracts a localized subgraph around each target entity (node or edge) and apply a GNN of arbitrary depth on top of the extracted subgraph. They do some theoretical analysis to show this combats over smoothing and increases expressivity. They also performs experiments and show state-of-the-art performance on benchmarks.  Originality: This is a simple and not completely novel approach (effectively it is running a GNN on a new altered graph) but the level of analysis and the experiments are novel and contributed original insights. I haven’t seen this detailed work around this approach and analysis before.

Quality: The work is thorough and of high quality. The analysis is useful and careful. I do think a better comparison to graph-altering work could be interesting. 

Clarity: This paper is clear but also a little bit unnecessarily complicated. For example, I suppose EXTRACT_1 and EXTRACT_2 are 1 and 2 hops respectively, but I don’t think that’s defined. But overall I think they’re doing a good job.

Significance: I believe this is significant work that analyses a core aspect of graph learning in a helpful and insightful way. It provides clear take-aways and good experiments. However, I believe it should include a little bit of discussion around the results. To some extent, being more powerful is somewhat misleading in the sense that we’re restricting (altering) the graph via EXTRACT, so we’re giving the NN more (conditional on human choices and adding additional bias) information. 
 Not addressed. 

Meta-review: This paper proposes a new way to perform graph sampling by using small neighborhoods of points for training and testing.  The resulting method enables very deep networks to be used on a range of graph problems.  The reviewers agree that the method is simple, and that this is a merit.  While there’s some discrepancy over the usefulness of the theory, there seems to be consensus around the point of view that the presence of theoretical results adds to the paper.



Decision: accept