Title: On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph neural networks

vanishing gradients

long-range propagation

oversmoothing

oversquashing
Abstract: Graph Neural Networks (GNNs) are models that leverage the graph structure to transmit information between nodes, typically through the message-passing operation. While widely successful, this approach is well-known to suffer from representational collapse as the number of layers increases and insensitivity to the information contained at distant and poorly connected nodes. In this paper, we present a unified view of on the appearance of these issues through the lens of vanishing gradients, using ideas from linear control theory for our analysis. We propose an interpretation of GNNs as recurrent models and empirically demonstrate that a simple state-space formulation of an GNN effectively alleviates these issues at no extra trainable parameter cost. Further, we show theoretically and empirically that (i) Traditional GNNs are by design prone to extreme gradient vanishing even after few layers; (ii) Feature collapse is directly related to the mechanism causing vanishing gradients; (iii) Long-range modeling is most easily achieved by a combination of graph rewiring and vanishing gradient mitigation. We believe our work will help bridge the gap between the recurrent and graph neural network literature and will unlock the design of new deep and performant GNNs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of vanishing gradients in Message-Passing Neural Networks (MPNNs) through a theoretical analysis grounded in signal processing. The authors highlight that many MPNNs experience what they term \"extreme gradient vanishing\", a phenomenon closely tied to over-smoothing. To address this, they introduce a state-space formulation for GNNs, referred to as kGNN-SSM, and demonstrate that in certain settings, this model avoids vanishing gradients and performs well on some long-range tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The paper is very well-written.\n2. While prior work has suggested a connection between vanishing gradients and issues like over-squashing and over-smoothing, this paper does an excellent job of formalizing those intuitions. The authors present a rigorous theoretical framework that makes these relationships explicit and clarifies a set of long-standing assumptions in the community.\n3. The paper also effectively links over-smoothing to the limited long-range capabilities of MPNNs, adding further clarity to how these phenomena affect their performance.\n4. The authors offer several insightful observations on why standard approaches for handling long-range dependencies or representation collapse, especially those borrowed from sequence modeling, may not translate effectively to GNNs. For instance, the comment from lines 164–165 offers a particularly clear example.\n5.  I find the empirical analysis of the different GNN models to be insightful (e.g., Figs. 2 - 3, Figs. 5 - 13).\n\n**Weaknesses**:\n\n1. Minor typo: line 108: GNNs are known.\n2. Minor typo: I believe that the authors wanted to refer to peptides-func on Line 823 - \"Conversely, on peptides-func (...)\". \n3. The font for some figures should be increased. Figure 2 is particularly hard to read.\n4. The authors might consider adding a related work discussion regarding the connection between their analysis and recently proposed spectral methods for modelling long-range interactions, such as [[4]] or [[5]].\n5. While the theoretical analysis is strong and the authors connect vanishing gradients to over-smoothing and over-squashing in a compelling way, the proposed GNN-SSM model is very similar to prior work, such as GRED [[1]], with the main difference being the use of a custom SSM formulation instead of an LRU. As noted by the authors in Line 215, one could easily incorporate mechanisms like gating or other constraints on the state transition matrix, but the paper does not explore such variants. This makes the model feel more like a proof of concept rather than a technically novel contribution. The paper misses an opportunity to explore these minimal architectural variations of the GNN-SSM. Even a small set of experiments testing whether design choices like gating improve performance would have helped assess the proposed approach better.\n6. The empirical evaluation of GNN-SSM is limited in both scope and depth. The model is only tested on the Peptides datasets from the LRGB, while other relevant datasets like PCQM-Contact are excluded. Moreover, the analysis focuses mainly on long-range tasks and does not consider standard graph benchmarks such as QM9, ZINC, ogbn-arxiv, or ogbn-proteins. Including these would help determine whether GNN-SSM generalizes beyond specialized cases.\n7. It is unclear to me whether the peptides-func dataset is a truly reliable indicator of long-range modeling capabilities. Prior work has shown that adding positional encodings to standard MPNNs (such as a GCN) can already lead to strong performance [[3]], which calls into question how much the gains on peptides-func are due to actual long-range modelling. A more targeted evaluation, such as the mechanistic in-context learning task from S2GNN [[4]], would better demonstrate the model’s ability to handle long-range dependencies in practice.\n8. The evaluation of over-smoothing effects would be more convincing if it included better heterophilic datasets. Although the authors report results on the Texas dataset from the WebKB collection, this collection has been heavily scrutinized in recent work, and more reliable alternatives have been proposed [[2]]. I recommend including at least one of these newer datasets to provide a more solid quantitative analysis.\n\n**Conclusions**:\n\nOverall, I appreciate the theoretical framework developed by the authors. Formalizing the link between vanishing gradients, over-smoothing, and over-squashing adds valuable clarity to ongoing discussions around the limitations of MPNNs, while also building meaningful connections to the sequential modeling literature. The idea of permutation-equivariant GNN-SSMs is compelling, and the theoretical setup provides a strong justification for the model's design.\n\nHowever, I found the empirical evaluation underwhelming. The proposed kGNN-SSM shares conceptual similarities with prior works such as Drew, GRED, and to some extent, with more recent models like S2GNN [[4]], which also draws parallels between long-range graph learning and state-space models. The current evaluation does little to differentiate the proposed model or explore its capabilities in depth. As it stands, the model functions more as a theoretical proof-of-concept than a technically novel or well-tested contribution. While this framing is valid due to the more theoretical nature of the work, it feels like a missed opportunity to more fully explore the empirical potential of the proposed model. I would have preferred either a more concise theoretical discussion paired with deeper empirical analysis or a more extended version of the paper, which would maybe be better suited as a journal submission.\n\nScoring this paper was difficult. It is very well-written, and the theoretical contributions are sound and interesting. The authors demonstrate that both vanishing gradients and over-smoothing affect the long-range capabilities of MPNNs, and that they are, in fact, connected. However, the GNN-SSM model lacks technical novelty, and its empirical evaluation is limited. Despite the comprehensive comparisons between various GNN models in the Supplementary, the evaluation of the proposed GNN-SSM itself feels disconnected from the strong theoretical motivation.\n\nMy initial score is a 4 (Borderline Accept). That said, I am genuinely inclined toward accepting the paper and will raise my score to a 5 (Accept) or 6 (Strong Accept) if the authors address some of my concerns. The most valuable and \"low-hanging\" additions, in my opinion, would be evaluating GNN-SSM on the heterophilic datasets proposed in [[2]] and on more standard graph or node property prediction tasks (see Weakness 5). These datasets are relatively small, and the experiments should be feasible to run during the rebuttal period. Please note that I will not lower my score if GNN-SSM underperforms on these datasets; my concern is about completeness, not performance.\n\nI would also be excited to see variants of GNN-SSM that incorporate gating or related architectural changes, and I believe that running the mechanistic in-context learning experiment from S2GNN [[4]] would provide especially strong evidence of long-range modeling capabilities. While I understand these additions may be difficult to implement within the rebuttal timeline, I highly recommend that the authors consider them for an eventual camera-ready version. In my view, they would significantly strengthen the paper.\n\n\n[1]: https://arxiv.org/pdf/2312.01538v1\n[2]: https://arxiv.org/pdf/2312.01538v1\n[3]: https://arxiv.org/abs/2309.00367\n[4]: https://arxiv.org/pdf/2405.19121\n[5]: https://arxiv.org/pdf/2506.07624\n\nQUESTIONS:\n- Please see the \"Weaknesses\" section above and the conclusions.\n\n- It is unclear to me if any kind of positional encodings were used. Mentioning this in Table 3 would be useful.\n\nLIMITATIONS:\nThe authors should discuss their limited empirical evaluation for the kGNN-SSM model if additional experiments will not be performed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to study problems associated with graph neural networks such as oversmoothing and oversquashing through the lens of vanishing gradients. Vanishing gradients were originally studied in recurrent neural networks, and this paper argues that vanishing gradients can also be used to study oversmoothing and oversquashing in GNNs. Gradients are already commonly used as a tool for quantifying oversquashing, but this paper uses vanishing gradients to unify the study of oversmoothing and oversquashing. The contributions of this paper are as follows:\n- Theorem 3.2 proves that if the weight matrix is normally distributed with variance < 1, then the Jacobian of a GCN has eigenvalues < 1, so GCNs would experience vanishing gradients in this setting. Figures 2, 5, and 6 analyses the eigenvalues of real GNNs and show that Theorem 3.2 is true in practice.\n- The paper proposes GNN-SSM as a remedy for the vanishing gradient problem posed by Theorem 3.2. GNN-SSM combined elements of classical GNNs with state-space models. \n- Proposition 3.3 shows that the eigenvalues of the Jacobian of GNN-SSM are controlled both by a GNN update (which by Theorem 3.2 has small eigenvalues) and a state transition matrix. Thus, by controlling the eigenvalues of the state transition matrix, the eigenvalues of the entire Jacobian can be controlled. This is validated in Figure 2.\n- The paper use vanishing gradient to prove why may GNNs experience oversmoothing. Proposition 4.4 shows that GNNs that experience vanishing gradients will also experience oversmoothing. It proves this by showing that the feature of GNNs with vanishing gradients will necessarilly converge to the zero vector as their norms shrink with each layer. This also implies that the Dirichlet energy of the node features will decrease after each layer. Figures 7 and 8 demonstrate that a GNN-SSM has constant Dirichlet energy across layers.\n- The paper argue GNN-SSMs can also help alleviate oversquashing. It was previously argued that oversquashing was caused by small gradients between the features of different nodes. The authors argue that GNN-SSMs could increase the gradient between nodes, thereby alleviating oversquashing. One of the most common remedies for oversquashing is rewiring. Figure 4 shows that using rewiring in conjunction with GNN-SSMs improves over using either of them separately.\n- The authors test GNN-SSMs on various real world datasets and show that GNN-SSMs can improve over vanilla GNNs in these settings. Tables 1, 2, and 4 show that GNN-SSMs typically outperform classical GNN baselines on these tasks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- The authors do an excellent job of validating their claims both theoretically and experimentally. Every theorem in the paper has an accompanying experiment showing this holds in practice and an experiment demonstrating how properly designed GNN-SSMs can alleviate the observed bad behavior. Thus, the authors demonstrate conclusively that GNNs face several problems that GNN-SSMs can help fix.\n- The authors empirically show that GNN-SSMs can achieve strong performance on real datasets compared to many baselines.\n- In this weaknesses, I mention that some of the theoretical results only prove sufficient but not necessary conditions for many of the claims in the paper. However, in these instances, the paper does a good job of augmenting the theoretical results with empirical evidence to show that fixing vanishing gradients can help fix the problem in question.\n\n# Weaknesses\n\n- Many of the theoretical results are one-sided. e.g., only proving a sufficient but not necessary condition for the expected behavior. For example, Proposition 3.3 only proves an upper bound on the Jacobian by the norm of the state transition matrix, so increasing the norm of the state transition matrix will not necessarily increase the norm of the Jacobian. Likewise, Proposition 4.4 show that vanishing gradients can lead to decreasing Dirichlet energy. Thus, Proposition 4.4 does not prove that decreasing Dirichlet energy only happens when there are vanishing gradients, and alleviating vanishing gradients is necessary but not necessarily sufficient to fix these oversmoothing.  \nLikewise, Theorem 3.2 only holds for GCNs, while ideally it would hold for more general message-passing networks like GINs or GATs.\n\nQUESTIONS:\n# Questions\n- How are the state transition matrices $\\Lambda$ computed, and how do you control their eigenvalues? Are these matrices learnable or fixed?\n- In Figure 3, middle, what is model is this plotting the features for?\n\n# Suggestions\n- Figure 1 is a little hard to understand, even after reading the paper. What are the blue lines? Which circles correspond to features of the same node? What circle are at the same layer, and which are at different layers? I think this figure could be improved if each element was more clearly labelled.\n- The boxes around the theorem statements look awkward as the text inside the box is too close to the edge. I would recommend using larger margins on the border of the box. Borders on the boxes and rounded corners may also improve the look of the boxes. \n\n# Typos\nLine 109: \"over-smoothing [15, 68] describes the tendency\". I am not sure what you intended to say here, but there must have been a typo as this sentence doesn't make sense.\nLine 777: \"were run in\" -> \"were run on\"\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a GNN as convolutional neural network with recurrent state updates. This connection is then exploited to create a state-space formulation of the GNN gradient descent updates. The nature of gradient update across the shared weights is shown as an evolution dynamics in state space. The problems of vanishing gradient and homogenization are explained in terms of this state-space dynamics. The issue of diminishing effect of distant nodes are also shown to exist.\n\nNext, authors propose changes in the learning and update mechanism which arrest these problems. The state-space formulation allows a control theoretic analysis of the strategies and synthesis of effective update rules. The method is elegant.\n\nSTRENGTHS AND WEAKNESSES:\nThe strength of the paper is the elegant state-space formalization of the GNN learning problem. The effects of vanishing gradient and homogenization can be easily observed in this analysis framework. Mitigation of these challenges using control theoretic methods in the state-space is found to be effective. Experimental results demonstrate effectiveness of the proposed technique.\n\nThe theoretical formulation is nice. However, there are some concerns about practical applicability in large scale GNN. The resulting state-space may explode and many of the existing control theoretic methods my break-down in large state-spaces. The typical state-space sizes common in control theoretic problems are much smaller.\n\nComparison with state-of-art GNN training schemes are not presented in the experimental results section. particularly the ones which rely on modifications of the message passing techniques.\n\nQUESTIONS:\n1. Does the proposed method scale to very large scale GNN where the challenges of vanishing gradient and homogenization are even more pronounced.\n\n2. How does the proposed method compare with state of art GNN training techniques which are not based on state space formulations but use sophisticated optimizer and message passing heuristics.\n\n3. Provide a discussion between the functional differences in the proposed method to avoid vanishing gradient in comparison with the known approaches in literature.\n\nLIMITATIONS:\nThe method is elegant. More results on large scale GNNs are required to consider the preent work as a state of art in GNN training.\n\nFurther experimental results comparing with state of art in GNN with respect to the identified challenges should be presented.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes depth‐related performance drops in GNNs (e.g., over-smoothing, over-squashing) through the lens of vanishing gradients. By vectorizing node features and treating layer-to-layer propagation as a dynamical system, this paper shows that standard graph convolution and attention layers are susceptible to gradient vanishing, whose Jacobians are strongly contractive because of the normalized adjacency operators. This paper introduces GNN-SSM, a state-space model-inspired architecture that explicitly controls the eigenvalues of the state transition matrix. This preserves gradient norms, prevents feature collapse, and, when combined with k-hop aggregation, achieves strong performance on synthetic long-range benchmarks and the LRGB tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros: \n- This paper provides a unified perspective that links vanishing gradients, over-smoothing, and over-squashing via spectral bounds of Jacobians\n- To validate the theoretical analysis, this paper designs a simple remedy to the gradient-vanishing problem by making the eigenvalues of the state transition matrix close to 1, inspired by state-space models, and further combines k-hop aggregation to address over-squashing. The model's good empirical performance validates the theory\n\nCons:\n\nMy major concern with this paper is that its contribution/novelty compared to existing works is unclear.\n\nTheory side, the theoretical analysis of this paper has a lot of overlaps with previous studies. E.g., [1] analyzed over-smoothing from the dynamical system's point of view and reached similar conclusions about convergence to fixed points. [2] explored the connection between gradient-vanishing and over-squashing, similarly to this paper.\n\nModel design side, this paper proposes to connect SSMs and GNNs. This idea has already been explored in the graph SSM literature. Especially, the GNN-SSM architecture proposed by this paper is very similar to GRED [3]. GRED has already used LRU to bound the eigenvalues of the state transition matrix for stable signal propagation, and combined k hops to address over-squashing (similar to kGNN-SSM). However, the relation to GRED seems to be downplayed in this paper, lacking proper comparison/discussion\n\n[1] Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. ICLR 2020.  \n[2] On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology. ICML 2023.  \n[3] Recurrent Distance Filtering for Graph Representation Learning. ICML 2024\n\nQUESTIONS:\nPlease see Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}