A Survey on Over-smoothing and Over-squashing: Unified Propagation Perspectives on Graph Neural Networks and Transformers
Authors: Alvaro Arroyo
Federico Barbero
Hugh Blayney
Michael M. Bronstein
Xiaowen Dong
Pietro Lio
Razvan Pascanu
Pierre Vandergheynst
Venue: Accepted by TMLR ()

Abstract: Decoder-Transformers have achieved remarkable success and have laid the groundwork for the development of Large Language Models (LLMs). At the core of these models is the self-attention matrix, which allows different tokens to interact with each other. This process is remarkably similar to the message-passing mechanism used in Graph Neural Networks (GNNs), and as such decoder-Transformers suffer many of the optimization difficulties studied extensively in the GNN literature. In this paper, we present a unified graph perspective that bridges the theoretical understanding of decoder-Transformers and GNNs. We systematically examine how well-known phenomena in GNNs, such as over-smoothing and over-squashing, directly manifest as analogous issues like rank collapse and representational collapse in deep Transformer architectures. By interpreting Transformers' self-attention as a learned adjacency operator, we reveal shared underlying principles governing signal propagation and demonstrate how insights from one field can illuminate challenges and solutions in the other. We analyze the role of architectural components like residual connections, normalization, and causal masking in these issues. We aim to provide a framework for understanding how information flows through deep learning models that perform sequence mixing through an adjacency operator, and to highlight areas for cross-pollination of research, as well as to provide a comprehensive reference for researchers interested in the underpinnings of these architectures.

Published in Transactions on Machine Learning Research (03/2026)
A Survey on Over-smoothing and Over-squashing: Unified Propagation Perspectives on Graph Neural Networks and Transformers
Álvaro Arroyo University of Orford
wetro.arroyooeng.ooo on..eengg.............
Federico Barbero University of Onford
Hugh Blayney University of Oaford
Michael Bronstein University of Ouford, ALTHYRA
Xiaowen Dong University of Orford
Pietro Liô University of Cambridge
Razvan Pascanu Mila
Pierre Vandergheynst EPFL
Reviewed on OpenReview: https: // openreview. net/ forum? id= H92hC5p VnH
Abstract
Prooder--raasformers have achieved remarkable success and have laid the groundwork for the development of Large Language Models (LLMs). At the core of these models is the self-allention malris, which allows different tokens to interact with each other. This process is remarkably similar to the message messagg-passing mechanism used in Graph Neural Networks (GNNs), and as such decodee-Transfommers suffer many of the optimization difficulties stud- ied extensively in the GNN literature. In this paper, we present a unified graph perspective that bridges the theoretical understanding of tematically examine how well-known phenomena in GNNs, such over-smoothing and over-squashing, directly manifest analogous issues like rank collapse and represen- tational collapse in deep Transformer architectures. By interpreting Transformers' self- attention as a learned adjacency operator, we reveal shared underlying principles governing signal propagation and demonstrate how insights from one field can illuminate challenges and solutions in the other. We analyze the role of architectural components like residual connections, normalization, and causal masking these issues. We aim to provide frame- work for understanding how information flows through deep learning models that perform sequence mézing through adjacency operator, and to highlight areas for cross pross-pollination of research, well as to provide comprehensive reference for researchers interested in the underpinnings of these architectures.
and GNNs. We sys-
1 Introduction
The remarkable success of large language models (LLMs) in natural language procossing, code generation, and multimodal reasoning has spurred intensive research into understanding their internal mechanisms and inductive biases. At the heart of modern LLMs lies the Transformer architecture (Vaswani et al. 2017). which leverages self-attention to capture both short and long-range dependencics in sequences. Despite their empirical success, little is understood about their learning dynamics, how their expressive power scales with depth and width (sequence length), and how to diagnose and mitigate different failure modes. In parallel, the

---

Published in Transactions on Machine Learning Research (03/2026)
field of graph neural networks (GNNs) has matured into rich intersection of spectral graph theory, dynamical systems, and message-passing paradigms (Gilmer et al.
2017 Defferrard et al. 2016 Kipf & Welling) 2017).
GNNs process relational data by propagating node features along graph edges. While these two architectures
soem distinct, it has been argued (Joshi 2025 Pappone 2025 that Transformers are an instance of a GNN, which due to the analogous way that lokens (or nodes) are mixed and iteratively updated through successive layers of the network. This similarity in information processing suggests that many insights and optimization-relatedd problems from GNN theory, such as over-smoothing (Oono & Suzuki 2020 and over- squashing (Topping et al. 2021), may shed light analogous phenomena in decoder- Transformers, and hence, LLMs.
In this perspective and survey paper, we advocate for a graph approach to understanding LLMs, one that unifies sequence and graph modeling under the common umbrella of mixing architectures. We organize our exposition around the core idea of information propagation. In particular, we elaborate on how the normalized adjacency matrix the key component of mixing architectures affects information flow in depth (in the number of layers) and width (in the diameter of the graph sequence). We examine how concepts like over-smoothing and over-squashing, well-studied in GNNs, have manifested as analogous challenges named rank collapse and representational collapse in Transformer literature. We focus specifically on decoder-style cansal Transformers as these are both widely used in practice and studied in recent literature, and the cansal masking makes propagation effects particularly relevant. However, we highlight that the core concept of studying mixing via the adjacency operator extends to other Transformer families and modalities such as images (Dosovitskiy al. 2021), audio (Gong et al., 2021) and point clouds (Zhao all 2021); refer readers Islam et al. (2024) for a comprehensive survey of applications. By drawing these connections and citing the most relevant work in both bodies of literature, we aim to provide the reader with a cohesive perspective on the shared underlying principles governing the behavior of both GNNs and decoder- Toaddr--TTafffomerr. highlighting how advancements in one domain can inform theoretical understanding and practical improvements in the other.
Contributions and Scope This paper makes several contributions:
(64.1) We formalize decoder self-attention as a Iearned adjacency (mixing) operator and relate it muessage-psssigg GNN updates, making explicit the shared structure of a mixing step followed by a feature update.
2.
($3-$4.2) We organize depth- and width-related pathologics under a single information lens, connecting over-smoothing/ vaaigg/iiiiiinngggg gradients and over-squashing (present in the GNN liter- ature) to mank/repreennationa collapse and long-context degradation mechanisms (present: in the Transformer literature).
3. (64.3) We systematize mitigation strategies as mechanisms that preserve representation diversity and stable gradient flow, and compare how these themes appear across both GNNs and Transformer variants.
4. We present inoragation-centicc framework intended to encourage cross-pollination between the GNN and Transformer communities, highlighting transferable tools from graph signal processing and network science for attention-based models, and vice versa.
2 Background and Foundations
In this section, we aim to provide the reader with the relevant background in both sequence modoling and GNNs. We begin by introducing foundational sequence-modeling architectures - starting with Recurrent Neural Networks and their gradient-flow challenges, progressing to the attention mechanism, and culminating in the Transformer architecture. We then provide an overview of graph neural networks and their message- passing foundations, as well as their associated challenges.
2.1 Background on Sequence Modeling
Sequence modeling a foundational concept in machine learning that deals with causally ordered data, such as text (Achiam et al., 2023), speech (Goel al.] (022),, or time series (Lim & Zohren 2021). While such sequential tasks have often relied on handcrafted priors or simple linear models with closed-form solutions,
2

---

Published in Transactions on Machine Learning Research (03/2026)
the increased availability of compute power and data has led to the advent of deep learning approaches. Here, we cover some of the most common frameworks, providing a foundation for gradient-based phenomena that underpin later sections.
2.1.1 Fundamentals of Sequence Modeling
Deep neural networks build intermediate representations by applying a scries of iterated nonlinear transfor- mations. This process enables the progressive abstraction of raw input data into higher-level features. Many mappings,
architectures in this domain can be viewed
go (0:f) + y (0:t))
(1)
where the input and output sequences up to time step L are defined
(0it) =
(x (1)
())
x )
(2)
y (0:t) (y y(u) (()).
Here, each input vector (*)) E Rd represents the data at time step k while each output vector y(h) E R" corresponds to the model's prediction at that time step. The parameter set corresponds to all learnable weights of the model.
In sequence modeling, the inherent temporal (or causal) structure of the data is captured by having the hidden state h(() be function of all previous information from the input sequence:
h (4) = ene (0:t)
(3)
where (l::t) denotes the collection of all inputs from time 0 to t, and gă learnable encoder function. The output at time step t is then generated by the decoder,
(())
go dec
h (Otr))
(4)
with dec representing the learnable decoder function.
2.1.2 The Vanishing and Exploding Gradient Problem
A particularly important instantiation of the encoder gete is obtained by imposing a recurrent structure on the hidden state. In Recurrent Neural Networks (RNNs), the hidden representation is updated recursively
as
(1) = Fe (o(-)
(5)
for some parametric transition map Fo. When unfolded over T time steps, this recursion defines a depth- T computational graph with shared parameters across layers. Training such models via backpropagation through time (BPTT) therefore requires propagating gradients through many repeated applications of the same state-transition Jacobian.
shariigg structure, which first exposed the vanishing and exploding gradient problem in sequence modeling (Bengio 1994 Hochreiter & Schmidhhuerr 1997 Pascanu et al., 2013). This occurs because, during BPTT, the gradients of the hidden states are computed by repeatedly applying the chain rule across many time steps. As result, the process involves multiplying a long sequence of Jacobians that are correlated to each other due to being the Jacobian of the same update rule. When this sequence is long, the accumulated product can either shrink rapidly toward sero or blow up uncontrollably, which leads to uncontrolled gradient propagation. In particular, following Pascanu et al. (2013), we let ((t) denote the loss at time step t. The total loss over a sequence of length is then given by
This recursive
T
=
(1)
(6)
3

---

Published in Transactions on Machine Learning Research (03/2026)
Accordingly, the gradient of the total loss with respect to the parameters 0 can be expressed as
ac
T
aC
Ohlt)
ah(h)
(7)
00
Oh(t)
(h(k)
00
As identified by Pascanu et al. (013)) major issue arises from the product Jacobian,
(h())
Oh
J
II
II J, imk+1
(8)
(h
Oh
k+++1
If [|xl|2 Re for all layers, then
2 S X
(9)
Thus, it is necessary that À for gradients to neither vanish nor explode condition often referred to as the edge of chaos. Specifically, if A < 1 for certain terms in the gradient sum, these terms will disappear and will be absent from the gradient calculation. Conversely, if X > 1, these gradients will explode and the gradient norm will increase greatly. RNN architectures such as LSTMs (Hochreiter & Schmidhuber 1997) or GRUs (Cho et al. 2214)) mitigate vanishing gradients via gated additive recurrence, but remain susceptible exploding gradients and also struggle with long sequences. As such, very relevant line of research has been dedicated to fixing this issue, examples of which include norn-preserving weight constraints (Arjovsky
et al. 2016: Henaff et al.) 2016 Chang al. 2018 Cezcano-Casale & Martmez-Rubio, 2019). initialization Keller et. al.
schemes (Tallec & Ollivier 2018 and ihysics-inspired inductive biases (Erichson et al. 2020
2023).
2.1.3 Transformers
The introduction of the attention mechanism has led to the Transformer architecture (Vaswani et al. 2017) This model relies the self-attention mechanism: in multi-head Transformer, each self-attention head applies scaled dot-product attention to calculate similarity scores between different inputs in the sequence. Let X E f denote a sequence of n feature vectors, each with dimension f. For the ith head we compute query, koy and value matrices using projection matrices W E R/xdy, W E Rixd, and W E fxd,
as
Q: XW Q
K, = Ki
V = XW:
(10)
The scaled dot-product attention is then defined as
Q.K? Jh
h; Attention (Q., K, V.) softmax
V
(11)
Each head learns an attention function that captures complex dependencies within the input data. With multiple heads, the model jointly attends different subspaces of the input sequence. The outputs from each head are concatenated to obtain joint representation:
MultiHead (Q, K, V) = Concat(h, ba,...,,y! Wo.
(12)
for output matrix Wo E Hd, Merging all heads' outputs through a linear function in this way produces latent representation that encodes the most relevant information from the input features.
In oregressive lasks, such next-token prediction, causal mask is essential to prevent token from accessing future information. To achieve this, the standard attention computation is modified by adding a mask M to the scaled doo-puosssts, as shown in
QKT Vol
Attention Q2 K, V) = softmax
+ M V,
(13)
where the mask is defined by
0,
j
Mig
(14)
-00,
j>b i
4

---

Published in Transactions on Machine Learning Research (03/2026)
thereby ensuring that positions corresponding to future tokens obtain zero attention after the softmax. This mechanism is critical for maintaining the autoregressive property by ensuring that each token is computed solely on its preceding context, which preserves the sequential nature of the data.
Despite its strengths, self-attention runs in O(n'd + nd?) time and requires O(n² + nd) memory, for di- mension (assuming d = f = d2 = d,) and sequence length n. This arises since forming the score matrix QK" costs ((ndd) and the Q, K, V projections cost ((dd). To cope with the computational challenges, these operalions are mapped to huge, parallel batched matrix multiplications that GPUs handle with high arithmetic intensity, predictable memory access, and static tensor shapes that allow aggressive kernel fusion,
This marriage between algorithm and hardware (also termed the 'hardware lottery' (Hooker 2021 has led to the effective parameter scaling of altention-based models in practice (Achiam et al. 2023
2.2 Background on Graph Neural Networks
Many real-world systems, from molecular structures and protein protein interaction networks in biology to citation graphs and social networks in information science, naturally manifest as graphs, where nodes represent entities and edges encode relationships. Learning predictive models on such graph-structured data is crucial for tasks like node classification, link prediction, and graph-level regression, yet traditional machine learning methods that assume independent and identically distributed (i.i.d)) samples cannot exploit the rich relational inductive biases inherent in these domains.
Graph Neural Networks (GNNs) (e.g. Sperduti 1993 Gori et al. 2005 Scarselli et al. 2008 Bruna et al, 2013 Defferrard et al. 2016) have emerged as a powerful framework for learning on graphs by combining spectral and spatial insights. In the spectral view, graph convolutions are defined via eigendecomposition of a graph shift operator (e.g., the graph Laplacian), enabling filtering in the frequency domain (Bruna ct al.] 2013 Defferrard ct al.) 2016);; in the spatial view, methods dircctly aggregate information from each node's local neighborhood. Most contemporary architectures adopt a message-passing paradigm, wherein nodes iteratively exchange and transform "messages" with their fiighbors-culminatne
in the Message-Passing Neural Network (MPNN) framework (Gilmer et al. 2017). Although these two lines of work have been developed independently, it is reasonable to use MPNNs as an umbrella framework to describe both. In this subsection, we will provide the necessary background on MPNNs and discuss related challenges.
2.2.1 Message Passing Neural Networks
Let a graph G be a tuple (V, E), where V is the set of nodes and is the set of edges. An edge from node u V to node e E V is denoted by (u, v) E E. The graph's connectivity captured by an adjacency matrix A E Raxx, where n is the number of nodes. We assume that G is undirected and that each node v is associated with feature vector h. Rd. A GNN can be broadly defined as function
fo : (G, (br))
(15)
with parameters 0 learned via gradient descent, and representing a prediction at the node or graph level. These functions typically adopt the message- paassgggg paradigm, which in computing the latent representation at the ltyer can be formulated as:
h = ((()) ( (k-1) ()()) (fh!- (w,2) E E)))
(16)
for - 10:=,..... 1,.... K. Here, (b)) a permutation invariant aggregation function, and (x*a) combines the incoming messages from a node's neighbors with its previous embedding to produce an updated representation. A common aggregation function is given by:
(i(k) ((h (u, E EE} EAu Euuth!hh! h!c-1),
(17)
where Ā AD-AAD and D E Raxn is diagonal matrix with Dj: = E3 Ag.
5

---

Published in Transactions on Machine Learning Research (03/2026)
A widely used instance of GNNs is the Graph Convolutional Network (GCN) (Kipf & Welling 2017 which can be interpreted as an MPNN. Specifically, one can express the node features in matrix form H Rexd; and the GCN update equation is
H kk) =
w(b-1),,
(18)
with
A - (A+ D(+
(19)
and (-) denoting nonlinearity. Another popular MPNN instance is that of Graph Attention Networks (GATs) (Veličković et al. 2018). where a learned adjacency matrix replaces the fixed normalized adjacency to dynamically modulate connectivity while preserving key spectral properties. In particular, the aggregation is computed as
(
h ((+)))
Eg
Σ
1 oi w
(20)
JEN(1)
where the attention coefficients are given by
exp(LeakyReLU(a (a T [h
( [hhf))
o =
(21)
EmEN()) exp(LeakyReLU(a [Wh, (1) Wh""))
Multi-head attention is then introduced by computing K independent sets of coefficients {al (). (with corresponding weights W(h, (*)) and either concatenating averaging their outputs:
K
(1+1) h
K k=1
a
Σ
Misk ag
w())
Or
a
Σ
() ag
w(0
(22)
K
JEN(()
k=
(EN(()
Other notable MPNNs include GraphSAGE (Hamilton et al., 2017), Residual Gated Graph Convolutional Networks (Gated GCNs) [Bresson & Laurent 2017), and Graph Isomorphism Networks (GLN) (Xu et al. 2018).
Graph Transformers We note that a large body of work studies Graph Transformers, ie., Transformer- style architectures designed for graph-struetured inputs. These methods typically adapt self-attention to op- crate over nodes by (i) treating nodes as tokens, (ii) restricting or biasing attention using the graph structure (e.g,, via adjacency-based sparsification or structural biases), and (iii) injecting graph positional/siu pintiionllsstuctuure encodings (e.g., shortest-path distances, centrality, Taplacian features) so that attention is aware of topol- ogy rather than purely content- based. Many Graph Transformers are motivated by limitations of classical message- passegggaasngggggg GNNs and by the desire to combine local neighborhood aggregation with global token mixing in rollabee and scalable way.
Representative architectures include Graphormer, which augments attention with graph structural biases such as shortest-path and centrality encodings (Ying et al. 2021), GraphGPS, which proposes practi- cal "recipe" combining local message passing with global attention while retaining linear complexity in nodes/edges (Rampášck et al. 2022), and approaches that study graph inductive biases in Transformers without explicit message passing (Ma et al. (023). We also highlight recent surveys that provide a system- atic overview of Graph Transformer design choices (Shehzad et al. 2026
3 Signal Propagation in GNNs
In this section, we turn our attention to the mechanisms that govern information propagation in common GNN architectures. In Section 3.1 we introduce well-known issues around stacking GNN layers which lead to the notion of over-smoothing used to explain the degradation in performance for common GNN architectures, and in Section 3.2 we introduce the issues of over-squashing and long-range communication between distant
We emphasize that (graph) Transformers be viewed family of CNNs operating with attention-bascd message passing; our goal
contrast "distinct" model classes, but use this shared mixing/ peaessgg--ssssing viewpoint to build common language for depth/widh widdhh pathologies and mitigation strategies across settings.

---

Published in Transactions on Machine Learning Research (03/2026)
Depth
Width
The computational tree for a given root node (in red) for GNNs and Transformers. For GNNs, the children of each node are typically its immediate neighbors in the underlying graph at the previous layer. For Transformers, the children of each node are typically the hidden representations of each (previous, in the case of a causal mask) sequence position at the previous layer. Depth refers to the depth of this tree (number of layers). Width loosely refers the width of the tree: in GNNs this relates to the node receptive field, and in Transformers sequence length.
Figure 1:
nodes. We make use of the original definitions given to these terms, but also note recent work that highlights the role of vanishing gradients in these concepts (Arroyo et al. 2025) and discuss the interplay between them (Armaiz- Rodiii-z & Errica 2025p. To help unify nomenclature between GNN and Transformer literature, we will separate our discussion into considerations of depth (primarily addressed as the issue of over-smoothing in GNN literature) and width (addressed as over-squashing in GNN literature). These terms broadly relate to the computational tree, which follows very similar structure between GNNs and Transformers, thus affording this unification of issues. We visualize highlight here a potential nomenclature clash: in this work, "width" does not refer to hidden dimension size but refers approximately to the width of the computational tree, which relates to node receptive field and sequence length in GNNs and Transformers respectively.
example of such computational tree in Figure | We
3.1 The Depth Regime in GNNs: Over-Smoothing and Vanishing Gradients
Early Approaches: Over-Smoothing. GNNs are known to severely struggle and break down after the application of few messago-passing layers. Until very recently, this phenomenon has been attributed the idea of over-smoolhing. Over-smoothing refers to the phenomenon whereby the repeated application of
cerssage-passing operations in GNNs causes the node features to become increasingly similar, visualized in Figure 2 Some carly works that analyze this theoretically are Oono & Suzuki (0220) Cai & Wang (2020). A common way to quantify this effect is via the unnormalized Dirichlel eneryy of the node representations, which has been used in a number of works in the area Rusch et al., 2022). Given a graph G = (V, E) with n nodes and node features E Baxd, the unnormalized Dirichlet energy defined as
E(H)
Σ [b, hell
(23)
(M,D)E
where h2 and h, denote the feature vectors for nodes u and v, respectively This energy measures the variability between features adjacent nodes; a lower energy indicates smoother (i.e., more similar) node reprosentations. In many GNN architectures, as more layers are stacked, the iterative aggregation tends minimize (H) to the extent. that all node features converge toward common value, as shown in Figure 3 (left)). The common consensus in the literature has been that while some degree of smoothing is beneficial. excessive smoothing leads to a loss of discrininative power, thereby hindering performance in downstream tasks, Several works have aimed to prevent over-smoothing in GNNs, from simpler approaches (Rong et al.
2019 Zhao & Akoglu 2019) to physics-inspired methods (Chamberlain et al. 2021 Bodnar et. al. 2022
Giovanni et al.) 2022: Bamberger et al. 2024 Other approaches that have emerged over the years
"Note that other uode similarity measures exist, including the normalized Dirichlet energy, used in the original works of Cai
& Wang 2020) or other measures such the employed Wi al. 2024
Note that the above assumes unitary edge weights.

---

Published in Transactions on Machine Learning Research (03/2026)
include Chen et al. (2000) Fang et al. (2023 Lu et al. (2024): Maskey et al. (2024)
Roth et al. (2024)
Figure 2: Illustration of the effect of feature collapse on a graph, commonly referred to as over-smoothing, From left to right more GNN layers are applied and node signals, here depicted by color, become more uniform over the graph. Nodes are sized by degree for illustration purposes.
Issues with Information Propagation and Optimization at Large Depth. Recently, the degrada- tion in performance of GNNs with large number of layers has also been associated with fundamental issues of signal propagation and optimization. In particular, Arroyo et al. (2025) highlight the importance of van- ishing gradients and the zero-collapse phenomenon in GCNs and GATs. The central argument of this work is that while feature collapse is an issue, it should be possible for MPNNs to learn weights that counteract this smoothing effect: however, due to issues of vanishing (and exploding gradients), this is often not the case. This work shows that the same phenomena that gives risc the over-smoothing effect (the product Jacobian) also gives rise to vanishing gradients. Similar notions were explored in Trmaiz-Rodriguee & Errica (0255, who highlight both that over-smoothing is not issue in all GNNs, and that over wver-smoothing does not necessarily cause performance degradation. We note that some of these topics were later explored in
Keriven 2025))
Mathematically, the main reason why GCNs and GATs are in expectation more vulnerable to vanishing gradients than standard MLPs is the presence of an additional message-paassing step using the normalized adjacency matrix: Å in the simplest case of GCN shown in Equation (18). From dynamical systems perspective, this extra step induces a contraction in the spectrum of the Jacobian, which makes the chain of Jacobians in the backward pass dissipate information at much higher rate in the backward pass of the network. This is illustrated in Figure 3 (right), together with an example of zero-collapsing of 2-dimensional random projection of node fcatures on the Cora datasct evolved with a GCN (left). To this end, we highlight a number of works that have employed ideas from vanishing gradients or dynamical isometry (Rusch el al., 2022 2023 Epping al., 2024 Scholkemper et al. 2024 Wang et al. 2025). Several of these works motivate their methods through comparison physical systems, which themselves encode stability and favorable optimization properties by design. We also highlight that the divergence effect reported in Arnaiz- Rodriguez & Errica (2025) can be understood from the perspective of optimization. This is due to the fact that the lyse-wiie Jacobians of some frequently used GNNs (e.g. GIN (Xu et al. 2018) or Gated-GCN (Bresson & Laurent 2017 are extremely unstable and will lead to gradient explosion al even moderate depths (see
Arroyo et al. 2025).
3.2 The Width Regime in GNNs: Over-squashing in Graph Neural Networks
A complementary and highly explored topic in the GNN literature is that of over-squashing. O Over-squashing originally introduced in Alon & Yahav (2021) as the compression of information from exponentially growing receplive field of nodes into a fixed-size vector during the message-passee was later linked to a problem of sensitivity and mathematically quantified in Topping et al. (2021): Di Gio- vanni et al. (2023) through the Jacobian. In particular, this was done by considering an MPNN whose node representations updated layer- Let (K) denote the representation of node after K layers, and
process. This phenomenon
8

---

Published in Transactions on Machine Learning Research (03/2026)
2500
Linear
GCN-ReLU
2000
Count
1500
1000
500
0.0
0.2
0.4
Figure Left: 2-Dimensional random projection showing node feature evolution for GCN on the Cora dataset. Right: Histogram of eigenvalue modulus of the Jacobian for linear, linear convolutional, and nonlinear GCN layers. Figures adapted from Arroyo et al. (2225).
h the initial representation of a distant node u. In Di Giovanni et al. 2023), the following bound on the sensitivity was derived:
Consider stendard MPNN with & layers, where Co is the Lipschitz conslant of the activalion a, is the maximal entry-value over all weight matrices, and d is the embedding dimension. For v E V we have
Theorem 3.2.1 (Sensitivity bounds, (Di Giovanni et al. 2023
Thg
(k)
S (co wd) (O*)eus
(24)
Th (0)
model
topology
where O= O.cI+ R=xn x is the pessage-passsing matriz adopted by the MPNN, and where or and co are the contributions of the self-connection and aggregation term.
Notice that the above formulation is composed of both a model term and a lopology term. Many of the earlier works in the space explored rewiring, which explicitly engineered the graph's connectivity to enhance message propagation and mitigate bottlenecks (Gasteiger et al. 2019 Topping et al. 2021 Arnaiz-Rodríguez et al.
2022 Barbero et al. 2023 Gutteridge et al.] 20023 Finkelshtein et al. 2024). On the other hand, a parallel strand of research concentrated almost exclusively on the model term itself, crafting non non-dissipative update rules or otherwise taming the dynamics without touching the underlying graph (Gravina et al., 2023 2025
Heilig et al., 2025). However, as pointed out in AArryo al. (2025), some the most robust solutions the literature combine both perspectives, jointly accounting for the graph topology for information propagation and the non-vanishing node-update dynamics. In particular, performing graph rewiring in isolation does not remove the zero collapse phenomenon and general optimization issues that characterize GNNs, while simply controlling node stability evolution will not enable long-range communication due to the modulation by the inverse square-root of the node degree typically performed when carrying out a messago-passingg step. We highlight that some these problems are confined to the use MPNN architectures, as recent work (Hariri et al. 2025) has also demonstrated the effectiveness of early spectral GNN designs in long-range information propagation. Although the depth terms in this bound mean that over-squashing can be accentuated in deep GNNs, note too that shallow bottlenecks long contexts in LLMs (Barbero et al.) 2024a). We refer readers to Arnaiz-Rodriguee & Errica (2025) for discussion on over-squashing and depth, and to Blayney et al. 2025) who investigate over-squashing in
issucs; this is of particular relevance when considering
shallow C
4 Intersections Between Transformers and GNNs
Having introduced the classical tools and problems in sequence and graph modeling, we are now ready to make connections between the two. Our analysis is based the interpretation of sequence and Transformer
We note that the literature sometimes treats over-squashing problem of topological bottlenecks, and sometimes more generally problem bottlenecks in the computational graph: we refer the reader to Arnaiz- Rodriguez & Errica 2025 for discussion this and other aspects of over-squashing.

---

Published in Transactions on Machine Learning Research (03/2026)
models as message-passing architectures. This intersection can be articulated at several levels. First, at conceptiull level, the core operations of Transformers can be cast as form of Graph Neural Network (GNN) operating on a fully-connected graph, a perspective originally popularized in Joshi (2025). Second, as a direct consequence of this conceptual similarity, both models suffer from analogous issucs in information flow, such as signal degradation in deep models and information bottlenecks. Third, and most importantly for this discussion, the strategics developed to mitigate these issues share common philosophy: preserving signal integrity and improving information flow through architectural changes like residual connections, normalization, and graph rewiring.
This section will unpack these points. We will first formalize the interpretation of Transformers as GNNs. We will then discuss the shared pathologies related to model depth and input width. Finally, we will dedicate a subsection to a direct comparison of the mitigation strategies that have emerged both fields, highlighting their parallels.
4.1 Interpreting Transformers as GNNs
Even though Transformers and GNNs have traditionally been viewed as distinct architecture 1 they both interleave MLP blocks (building rich representations) with a "mixing" step that can be interpreted as a message-passingg operation. Concretely, in Transformer layer, we have
QKT
H k)) softmax
V
(25)
dd
H 1) We (H (1 Wx)
1
softmax
H H(k-1) Wv
(26)
( Wv.
(27)
where À is the corresponding attention matrix. From this, it is clear that Transformers and GNNs share a similar updating procedure where a learned adjacency matrix, A, is used to mix node (token) features. Therefore, MPNNs (especially GCNs and GATs) can be seen as a microcosm for how Transformers operate, and we should expect them to share similar pitfalls and learning dynamics. In particular, the spectral properties of the attention matrix will determine the behavior of the system in depth, while the characteristics of the softmax function will determine the properties of the graphs defined by this adjacency matrix. We will elaborate on both of these considerations in the next subsections.
4.2 Shared Pathologies: Information Flow Bottlenecks
The structural similarity between GNNs and Transformers leads to analogous failure modes related to infor- mation propagation, both in depth of the model and width of the input.
The Depth Regime: Over-smoothing and Rank Collapse Just as deep GNNs suffer from over- smoothing, where node representations converge to a uniform vector, naively stacking Transformer layers leads to rank collapse (Dong et al. 2021). In deep Transformers, the matrix of token embeddings tends to become low-rank, meaning the representations for all tokens collapse into a common subspace and lose their discriminative power. This phenomenon direct analogue of sver-smoothing in both cases, the iterative application of a mixing operator (A) is theoretically expected to yield increasingly similar node (or token) representations.
The root cause of this shared pathology is the mixing operator Å itself, which is the very element that distinguishes these architectures from simple MLPs. This is not merely a representational issue; it has a critical effect on trainability. The same mechanism that drives over-smoothing (in the sense of zero collapse) is also respousible for vanishing gradients, making deep GNNs and Transformers notoriously difficult to train Noci et al. 2022 Arroyo et al. 2025). The mathematical link is the contructive nature of the mixing
"Transformer-basc"" architectures graphs (often termed Graph Transformers) have been extensively studied and can themselves be viewed GNNs with learned adjacency operators. Our statement concerns the historical separation of the sequence Transformer and GNN research communities, which this work seeks bridge via unified propagation-centric view.
10

---

Published in Transactions on Machine Learning Research (03/2026)
30
100
140
148
70
150
130
40
Fingurac
Poopueay
20
-0
Figure 4: Attention matrices (top) and associated eigenvalue modulus histograms (bottom) for several heads in the early layers of the Gemma 7B model. These histograms demonstrate that the action of the attention matrix is often highly contractive
matrix's s spectrum, as shown in Arroyo et al. (2025 In MPNNs, the use of the normalized adjacency matrix is common, whose eigenspectrum satisfies Axg S 1. In turn, modern LLMs, such as the Llama (Tourron et al. 2023a by Graltafiori et al. 2024) and Gemma (Team et al. 2024a b 2025) families, impose a triangular (causal) mask for next-token generation. The subsequent application of the softmax function additionally mcans the resulting matrix is row-stochastic. This implics it can be analyzed as the transition matrix of a Markov chain, whose eigenvalues are similarly bounded by |Ag] S Recent work by Wu et al.) 2024h has formally extended the notion of rank collapse to these causally masked attention matrices, with complementary candom-matrin results in Naderi et al. (2024) employing spectral approach to analyze their asymptotic behavior.
This creates a fundamental trade-off. On the one hand, contractive mixing causes training instability. On the other hand, empirical evidence from models like Gemma 7B shows that many learned attention heads are, in fact, highly contractive (Figure 4. This suggests the model finds these structures useful for its tasks. This presents a dilemma: solve its task the model must learn contractive patterns, but these inherently inhibit the gradient flow required for deep, stable training. In practice, this solved using the architectural solutions discussed in Section 4.3 To make this link precise, recall two standard consequences of block-wise contraction: repeated composition drives representations to stable fixed point and monotonically suppresses graph variation (Dirichlet energy), at rate controlled by the product of layer Lipschitz constants.
Lemma 4.2.1. (Arryo et al. 2025) Consider a GNN layer fK as in Equation 18 with non-linearity o such that (0) = 0 (e.g. ReLU or tanh). Then, f(0) = 0, i.c. O is fized point of f.
Proposition 4.2.2 (Convergence to a unique fixed point. (Arroyo et al. 2025 ()). Let [| Lip < - for for all k L. Then, for H U Rnd, have that:
|S(H]] <1 (1-6) -)) |
H|]
(28)
In porticular, as K 0c, f(H) 0.
11

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition 4.2.3 (Contractions decrease Dirichlet energy. (Arroyo et al. 2025 Let f be GNN or Allention-based layer, Fb] be the number of edges in G, and H E Rndd We have the following bound:
E((H)) S F
f 1 Lip
[
(29)
In particular, if l 1 for some e for all ===1 1 K, then as K - 00 we have that ((f(H) +
In particular, these results formalize how contractive mixing-update block simultancously (i) collapses node states toward a common fixed point and (ii) decreases Dirichlet energy; in the sell-attention setting the same contraction intuition applies in token space, where repeated (masked, row-stochastic) attention mixing can drive the token-embeddingg matrix toward a low-rank subspace (rank collapse), motivating the need for stabilization mechanisms such as residual pathways and normalization.
The Width Regime: Over-squashing and Represen- tational Collapse Beyond the depth-wise issue, decoder- Transformers face 'width-wise' issues, which manifest partic- ularly in long-context scenarios. Some of these challenges are analogous to over-squashing in GNNs. In GNNs, the origi- nal definition of werrsquashing (Alon & Yahav) 2021) stated that this phenomenon occurs when information from an expo- nentially growing number of nodes at a distance must be com- pressed into a fixed-size node embedding vector. A key nuanced point needed to understand information propagation across the soquence in Transformers is the fact they operate over causal graph (or Markov chain), as mentioned before.
MLP Block
Norm
Attoution Mossage Paseing
Norm
One of the main consequences emerging from the causal struc- ture of the attention matrix is a tendency of carlier tokens to overwhelm later ones, a problem which was formally introduced in Barbero et al. (2024a). This can be interpreted as a form of the over-squashing problem, where the limit in which the context size grows, only the first token will have impact the output of the model. Beyond this path-counting effect, Barbero et al. (2024a) also prove the appearance of represen- lalional collapse, where for some distinet sequences (e.g., a string that ends with a certain token versus the same string with that token repeated at the end), the last-token representations become arbitrarily close as length grows, rendering the model effectively unable to distinguish them under finite precision. This collapse is exacerbated by quantization/ weatiiionnoww-eeeeee -oo-ppeciiionn arithmetic and helps explain systematic failures on copying and counting in long, repetitive contexts. We present the formalization of these below.
Figure 5: Illustration of "Translormer-stylee blocks in both Transformers and GNNs, where either Attention or message-pasinng acts as a 'mixing" step.
Theorem 4.2.4 (Representational Collapse (Barbero et al. 2024a)). Let (()) Rand be sequence and (0) E [a++1)xd be another sequence equal to v(U) with the last token of v(1) repeated. Assume that the positional encoding information decoys to with the distance. Then, their representations become arbitrarily close as n increases.
Theorem 4.2.5 (Over-squashing in Transformers (0) Let C> be some constant and a
Barbero et al. 2024a Consider an imput sequence
di.j, then:
32
Oyn e0 (0)
Σ
(L-1) (t-1) (0)
(30)
24 kL AL2RL-1
2=2
Besides the issue of representational collapse, the fact that each token must intograte information from all previous tokens in the sequence leads additional problems. In particular, we highlight the phenomenon of over-mizing outlined in Barbero et. al. (2025), where the model thus loses the ability to distinguish
12

---

Published in Transactions on Machine Learning Research (03/2026)
Ag
A2
A
A
A
Ag
A
A
A2
A
A
A4
Figure 6: Illustration of information over-squashing in Transformers, adapted from similar figures in Barbero et al. (2024a) 2025). Left: Information in Seccode--Trasformnne 'overmixes', which results in issues such high sensitivity to perturbations early in the sequence. Right: The causal graph topology associated with a Tranddrr-Tanformmer causes earlier tokens in the sequence (whose impact on later representations are depicted in red) to representationally 'overwhelm' later ones (whose impact on later representations are depicted in blue)
individual tokens in the sequence and becomes more exposed to input perturbations. In the next subsection, we will present a number of attention patterns which large-scale models naturally learn through the gradient descent process to prevent this phenomenon. Illustrations of information over-squashing and over-mixing can be found in Figure B where the impact of given token on later tokens is visualized by color propagating throughout the model.
Finally, we highlight that complementing these characterizations, a growing literature documents additional long-context failure modes and practical mitigations in TransorrTransssrmeess in relation to attention sinks. For instance, training-free free attention interventions can reduce extreme-token dominance and improve long- context stability (Han et al.j 2025). while geometric and optimization viewpoints analyze how sink-like behavior emerges in attention dynamics (Ruscio et al. [2025 Alcalde et al. (025). Related work also connects attention-sin dynamics to head collapse and proposes sink-aware training mcchanisms to restore stable long- context routing (Fu et al.] 2026), and other architectural proposals explicitly modulate the attention branch to mitigate extreme-token phenomena (Bu et al. 2025 Fu et al. 2026).
4.3 Shared Mitigation Strategies: Preventing Collapse in Depth and Width
Previously, we have introduced the over-smoothing and over-squashing issues, and how they manifest in both decoder-Trrnnsforeee and GNNs. In this subsection, we will emphasize how and why different architectural decisions can prevent or mitigate these phenomena, and what shortcuts models naturally learn during training to protect themselves against these issues.
The Importance of the Transformer Block. As we have discussed, naively stacking musssag-passing or self-attention layers typically leads to trainability issues and problems with diversity of representations, even after few layers. In the case of Transformers, self-atlention layers are typically not stacked one after the other, but interleaved with normalization and residual layers to form Transformer block, which is depicted in Figure 5 On the other hand, GNN layers were originally naively stacked one after the other without placing a similar emphasis on architectural optimization. While this choice was likely due to the locality present in some of the early GNN benchmarks, such as citation networks (McCallum et al. 2000 Sen et al. 2008), it could have been one of the culprits of the well-reported issue of weer-smothhin in the GNN community.
Preventing Collapse in Depth:
6This Figure is intended as general schematic of Transforner--ttlee stabilization block, rather than claim that there is single canonical block used across architectures. For Transformers, canonical variants include post-LN (Vaswani et al. 2017) and the now-standard pre-LN formulation (Xiong al. 2020 In GNNs, deep architectures similarly adopt residual connections and normalization, but implementation details differ materially; particular, dropout and normalization are often applied after message passing (e-g., in the implementations Luo et 2024).
13

---

Published in Transactions on Machine Learning Research (03/2026)
Figure 7: Example of sharp attention matrices in the Gemma 7B model. Left: An identity head. Middle Left: A BoS head. Middle Right: A previous token head. Right: An apostrophe head.
We highlight that the specific design of the Transformer block is critical for mitigating the undesirable con- traction properties that can characterize the self-attention matrix, and facilitates signal propagation through the network, and hence the overall training process. In the original Transformer architecture (Vaswani et al., 2017), the LN block was placed after the residual connection (also known as post-LN). However, in this orig- inal post-LN sctup, large gradicnts at initialization force the use of warm-up phase to prevent instability. As such, Xiong et al. (2020) proposed to reposition the LN inside the residual branch, (pre-LN), which led to well- gradients that neither explode nor vanish even with large initial learning rate, thus removing the necessity for warm-up stage.
Despite the success of the block structure in Transformers, we highlight that it is also possible to achieve good signal propagation without relying on this specific structure. An example of this is He et al. (2023). where the use of bias matrices, location-dependent rescaling, and preseelpreserving initializations that let vanilla Transformers (without skip connections or normalization) maintain vnii-variance signal all the way through hundreds of layers, enabling effective signal propagation. Their follow-up study (He & Hofmann 2023) goes further by pruning value projections, merging the attention and MLP sequence into a single operation, and discarding LayerNorm, and they obtain blocks that are 15% faster and 15% smaller yet match the per-update training speed and final accuracy of standard pre-LN stacks. We also highlight the recent work of Zhu et al. (2025), who replace layer normalization with the Dynamic Tanh (DyT) activation function in the transformer block while achieving comparable performance.
The extent to which individual elements of the Transformer block prevent rank collapse has been studied in a growing theoretical literature, including in some of the papers already mentioned. At the level of the mixing operator itself, Dong et al. (202I) show that attention-only stacks exhibit rapid (doubly-exponenntal) rank decay with depth, motivating the noed for architcctural "stabilizers" beyond pure attention. From a training and signal-propagation perspective, Noci et al. (2022) argue that appropriate depth--ependent scaling of the residual branches can restore stable propagation, perspective recently echoed in Joseph et al. (2024). Complementarily, Wu et al.] (2024) analyze LayerNorm and attention masking change the collapse dynamic, and show that although masking alone does not remove collapse, sparsity searsity/localle /locality in masks can slow it down, and that LayerNorm can materially alter the dynamical system, admitting richer non-collapsed behaviors for suitable parameter regimes. On the other hand, Naderi et al. (2024) attribute collapse and propagation pathologics to outlier spectral structure of attention operators, and propose simple mitigation principles that effectively temper these dominating modes. Finally, recent work further sharpens what it means to "fix" collapse via the block: Alman & Song (2025) argue that skip connections alone are not sufficient under small-weight regimes, introducing a related notion of layer collapse and concluding that sufficiently large weights are necessary for expressive depth even in the presence of residual pathways.
Analogously, the GNN literature has converged on canonical "deep GNN block" that mirrors the Trans- former block's intent of preserving stable gradient propagation across many mixing steps while preventing feature collapse. A number of modern GNNs (i et al. 2019: 2020) typically wrap message passing inside residual/skip pathways, and augment it with normalization and regularization that explicitly control the contraction induced by repeated adjacency mixing (Rong et al. 2019).. Concretely, methods like PairNorm (Zhao & Akoglu 2019) (and related feature-normalization schemes) directly counteract over-smoothing by
14

---

Published in Transactions on Machine Learning Research (03/2026)
maintaining representation variance across nodes. Importantly, many of these mechanisms instantiate the same two primitives that are central to the Transformer block, an identity (skip) pathway plus normalization suggesting a shared "trainability template" for deep mixing architectures, see also (Scholkemper et al. 2024). Interostingly, the application of similar design principles in GNNs has yielded impressive, even state-of-the. art, performance (Tuo al.) 2024 p025). We note, however, that the specific instantiation of these blocks in GNNs is not identical the standard Transformer block: particular, GNN implementations often place normalization and dropout at slightly different locations. Nevertheless, the underlying design princi- ple (combining identity pathways with normalization to stabilize repeated mixing) remains closely aligned across the two settings. See Figure X for an illustrative comparison of how adding residual connections and vormalization/ssaling reshapes the layer-wise Jacobian eigenspectrum, shifting it toward the identity and suppressing unstable modes.
At the same time, it is worth noting that, until very recently, the GNN community often framed depth limitations primarily through a representational lens (over-smoothing), with optimization and gradient- flow considerations receiving comparatively less emn phasis. Arroyo et al. (2025) revisited this idea by ex- plicitly linking over-smoothingg/ haapooooooiinneeeooolllapssss zeroocollaape to van- ishing gradients and trainability, thereby placing GNN block design on the same mechanistic level as signal-propagation analyses in deep Transform- ers. Overall, we believe this alignment shows that Transformer-style block design is not unique to at- tention; rather, it is general recipe for making mix- ing and pointwise update models trainable depth while limiting collapse.
Imaginary
Imaginary
-1
Real
Real
Figure 8: Illustration of the stabilizing effect of resid- ual connections and normalization on GNN dynamics. Left: Figenspectrum of the layer-wise Jacobian of vanilla GCN, where repeated application of the nor- malized adjacency induces strong contraction and con- centrates eigenvalues near the origin, leading to van- ishing gradients and fcature collapse at depth. Right: Eigenspectrun a GCN augmented with a residual connection and output scaling. The residual pathway introduces an identity component that shifts eigenval- controls
Other research has identified methods es- tablished in the GNN and Graph Signal Processing literature that to mitigate depth collapse in Transform- ers. Choi et al. (2023) introduce modi- fied attention mechanism inspired by graph filters, to encourage retention of higher fre- quencies and avoid over-smoothing. Ko (2024) similarly introduce graph propa- gation concepts in order to more faithfully capture local dependencies in vision Transform- ers.
be adapted
ues toward (1, while normalization/scal spectral spread and suppresses unstable (exploding) modes. Together, these mechanisms stabilize signal and gradient propagation and mitigate depth-induced collapse. Adapted form Arroyo el. al. (2025).
Kim &
Preventing collapse in width: Sharpness and gating help with over-squashing. One of the most effective strategies Transformers can learn to prevent. this representational bottleneck by developing specialized, sharp attention heads. Instead of forming densely connected graph, these heads learn sparse patterns that route information in more structured way, effectively preventing over-mixing. To this end, language models empirically tend to construct attention heads that either attend to very specific patterns in the sequence (previous token, spaces, or apostrophes), or attend only to the BoS (Beginning of Sequence) token despite not having any semantic meaning. This can be clearly seen in the example heads from the Gemma 7B model shown in Figure 7 which have also boen reported empirically in [Barbrro et al. (2025): Queipo-de Llano el. al. (2025). This "altention sink' phenomenon
originally reported in Xiao el. al.) 2003)) and analyzed in more detail in Gu et al. 2024 Barbero et al. 2025). We note that recent work
commonly instantiated using the block-level stabilization template sequence Transformers while injecting graph inductive hias via structural/posit /positiooonlllllooo encodings and, in hybrid variants, additional local cessage-passingg modules (Ying et al. 2021 Rampášek et 2022 Ma all 2023).
We also note that Craph Transformers
15

---

Published in Transactions on Machine Learning Research (03/2026)
QQu et al. 2025 also prevented the appearance of attention sink by incorporating gating into the attention mechanism, which can be seen as an alternative (and more direct) way of preventing over-mixing. While these ideas have been heavily explored in the context of recurrent architectures (Hochreiter & Schmidhuber
Gu & Dao 2023 Beck et al. 2024 De et al.) 2024), it appears that similar approaches can mitigate the undesirable effects over-mixing in decodde--Transtommeen We highlight. also other works that identify other "width" issues arising from long contexts. BBrbber et al. (2024b) examine RoPE and discover that it introduces issues when generalizing to long contexts. Xiong et al. (2025) examine attention "biases" in long contexts and design KV caching technique to reduce the effect of these biases. Further, related empirical studies also document systematic degradation in long-context use, such as retrieval failures where evidence placed in the middle of the prompt is underutilized <Liu et al. 2024).
1997
A useful way to make this precise is to view influence propagation through a Prrooder-rranfommmer as sum over causal paths in the attention graph. The following bound (Barbero et al., 2025) makes explicit how attention weights determine the sensitivity of a late token to an early token through products of mixing coefficients:
Theorem 4.3.1 (More detailed over-squashing bounds. (Barbero et al., 2025)): Let Cmaz > 0 be the greatest
Lipschitz constant of any layer of the Trunsformer, H be the number of heads, and 8f be 1 iff i = j and O otherwise. Let k E P,4s be a path from i to j of length L.. Set a =
- En a (e.N)
+ &
H Then:
| /0v
C
max
E
ajkzz ake 11ke ... a
(L) (1-1))
(1)
(31)
KEP
This bound exposes the mechanism behind width failures: the influence | /0v is controlled by a path-sum of products of attention weights. When attention is diffuse (many moderate aue), the number of contributing causal paths can grow rapidly and the model can overmix: many tokens become entangled through many comparable routes, amplifying sensitivity to small perturbations and pushing representations toward common subspace. Conversely, when attention is sharp, most paths have near-zero weight, the path-sum is dominated by small set of routes, which reduces mixing. Likewise, gating effectively downweights the attention branch in the path products, which reduces cumulative mixing as well.
The ability of Geroder-Transformeen to reliably learn and maintain these desirable sharp patterns is also fundamentally constrained by the choice of attention function. To this end, recent work (Veličković et al. 2024) argue that the standard softmax will fail in scttings that require generalization to sequences longer than those seen during training. In these cases, the nature of softmax can cause allention scores to bleed and diffuse, making it difficult for a head to remain sharply focused on a single token. We highlight other works which have further analyzed the softmax function (Masarczyk et al. 2025) as well as other variants (Saratchandran et al. 2024 Ramapuram et al. 2024), including ones that prevent the formation of attention sinks (Zuhri et al.] 2025). We note that Over-squashing in GNNs is also sometimes addressed through methods that reduce bottlenecks in the computational tree. One recent method to do this has been adaptive message-passingg (Errica et al.) 2023; Finkelshtein et al. 2024), which can also be seen as a more explicit way to enforce "sharpness'.
5 Conclusion
In this survey, we have explored the conceptual and mathematical parallels between Transformers and GNNs, advocating for unified graph-based perspective information propagation in these seemingly disparate architectures. We have shown how fundamental challenges in GNNs, such as over-smoothing and over-squashing. find direct analogues in Transformers as rank collapse and representational col-
lapse.
Our analysis revealed that the iterative resssge-passing operations in GNNs, which can lead to node feature homogenization, mirrors the dist rank collapse observed in Transformers where token representations lose their distinctiveness. The role of residual connections and normalization techniques, erucial for mit- igating these issues in both architectures, underscores shared need for mechanisms that preserve signal
16

---

Published in Transactions on Machine Learning Research (03/2026)
diversity and prevent information dissipation concentration. Furthermore, the sensitivity bounds derived for GNNs provide lens through which to understand how information from distant parts of sequence can be "squashed" in Transformers, highlighting the importance of managing receptive field and information flow.
By viewing inlFaatennioonmmesnse passing as the repeated application of mixing-update operator, our perspective makes explicit a mechanistic link between propagation properties and architectural choices. In depth, when the operator is contractive, iterating it drives representations toward fixed points and sup- presses variation, aligning with rank /feattueeee collapse; residual pathways, normalization, and scaling act by injectingmmantaaiinnn maintaining an identity component and controlling contraction, improving trainability. In width, influence between distant ookens/nodes is governed by path-sums of products of mixing coefficients; sharp attention patterns and gating reduce the mumber/weight of active paths, mitigating over wver-mixing and sen- sitivity bottlenecks. Overall, the contribution is a unifying propagation lens that organizes existing failure modes and clarifics how common stabilizers intervene, while also suggesting concrete directions for more principled miligation design.
Ultimately, this unified view not only enhances our theoretical understanding of these powerful models but also paves the way for pros--oolinaaton of ideas. Solutions developed for performant GNNs, particularly those addressing vanishing gradients and information bottlenecks, could inspire novel architect designs or training strategies for Transformers, and vice-versa. As the fields of graph representation learning and large language models continue to advance, a shared vocabulary and conceptual framework will be valuable for developing more robust, efficient, and interpretable deep learning architectures.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXie:2303.087 08774, 2023.
Albert Alcalde, Borjan Geshkovski, and Domènec Ruiz-Balet. Attention's forward pass and frank-wolfe. arXiv preprint ar arXit:2508.06638 2025.
Josh Alman and Zhao Song. Only large weights (and not skip connections) can prevent the perils of rank collapse. arviv preprint arXie:2505.16284 2025.
Cri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations, 2021.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Inter- national conference on machine learning, pp. 1120-1128. PMLR, 2016.
Adrian Aamaiz-Rodrigee and Federico Errica. Oversmoothing, oversquashing, heterophily, long-range, and more: Demystifying common beliefs in graph machine learning. arXiv preprint drXiv:2505.1554' 15547, 2025.
Adríán Rodriguz, Ahmed Begga, Francisco Escolano, and Nuria Oliver. Diffwire: Inductive graph rewiring via the lovász bound. arXiv preprint Sll:22060003669 2022.
Álvaro Arroyo, Alessio Gravina, Benjamin Gutteridge, Federico Barbero, Claudio Gallicchio, Xiaowen Dong,
Michael Bronstein, and Pierre Vandergheynst. On vanishing gradients, over-smoothing, and over-squashing in gnns: Bridging recurrent and graph learning. arXio preprint urXiw:250..10818 2025.
Jacob Bamberger, Foderico Barbero, Xiaowen Dong, and Michael M Bronstein. Bundle neural networks for message diffusion on graphs. arXiv preprint arXiv:2405115540, 2024.
Federico Barbero, Amcya Velingker, Amin Saberi, Michael Bronstein, and Francesco Di Giovanni. Locality- aware graph-rewiring in gnns. Xr preprint urXiw:2310.00 211.066300 2023.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João GM Araújo, Alex Vitvit- skyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! information over-squashing in language tasks. Xix preprint or Xiv:244406.0 04267, 2024a.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličkovié. Round and round we go! what makes rotary positional encodings useful? arv preprinl. orXin:2410006200 2024b.
Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do Ilms attend to the first token? arXiv preprint arXiv:204.00279. 2025.
Maximilian Bock, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Xin preprint or 2rXi:22455.44517, 2024.
Yoshua Bengio. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neurll networks, 5(2):157-166, 1994.
Hugh Blayney, Álvaro Arroyo, Xiaowen Dong, and Michael M Bronstein. glstm: Miligating over-squashing by increasing storage capacity. arXiv preprint arXiv:2510.08450 2025.
Cristian Bodnar, Francesco Di Giovanni, Benjamin P. Chamberlain, Pictro Liò, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs, 2022.
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07555 2017.
Joan Bruna, Wojciech Zaremba, Arthur Sdlam, and Yann LeCun. Spectral networks and locally connected networks graphs. arXiv preprint orXiv:1312.6200 6203, 2013.
Rui Bu, Haofeng Zhong, Wenzheng Chen, and Yangyan Li. Value-state gated attention for mitigating extreme-token phenomena in transformers. arXiv preprint arXi::251000001 2025.
Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2066. 13318, 2020.
Benjamin Paul Chamberlain, James Rowbottom, Maria Gorinova, Stefan Webb, Emanuele Rossi, and Michael M Bronstein. GRAND: Graph neural diffusion. In International Conference on Machine Learning (ICML), pp. 1407-1418. PMLR, 2021.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrnn: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, 2018.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725-1735. PMLR, 2020.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase ropresentations using rnn encoder-decoder for statisti- cal machine translation, 2014.
Jeongwhan Choi, Hyowon Wi, Jayoung Kim, Yehjin Shin, Kookjin Lee, Nathaniel Trask, and Noseong Park. Graph convolutions enrich the self-attention transformers! arXiv preprint arXiv:2312.04.2 042444 2023.
Soham De, Samuel L Smith, Annshan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint 4xX4::2402.1922, 2024.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the 30th International Conference on Neural Infor- malion Processing Systems, pp. 3844 3852, 2016.
Francesco Di Giovanni, James Rowbottom, Benjamin P Chamberlain, Thomas Markovich, and Michael M Bronstein. Graph neural networks as gradient flows: understanding graph convolutions via energy. arXiv preprint ar Xix::2206. 10991, 2022.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pictro Lio, and Michael M Bronstein. On over-squashing in message passing neural networks: The impact of width, depth, and topology- In International Conference on Machine Learning, pp. 7865 7885. PMLR, 2023.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pp. 2793 2803. PMLR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Bastian Epping, Alexandre René, Moritz Helias, and Michael T Schaub. Graph neural networks do not always oversmooth. arXiv preprint arXiv:2406. 02269, 2024.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney, Lipschitz recurrent neural networks. In International Conference on Learning Representations, 2020.
Federico Errica, Henrik Christiansen, Viktor Zaverkin, Takashi Maruyama, Mathias Niepert, and Francesco Alesiani. Adaptive message passing: A general framework to mitigate oversmoothing, oversquashing, and underreaching. arXiv preprint orXiv:2312. 16560, 2023.
Taoran Fang, Zhiqing Xiao, Chunping Wang, Jiarong Xu, Xuan Yang, and Yang Yang. Dropmessage: Unifying random dropping for graph neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 4267-4275, 2023.
Ben Finkelshteinn, Xingyue Huang, Michael M Bronstein, and Ismail Ilkan Ceylan. Cooperative graph neural networks. In forty-first International Conference on Machine Learning, 2024.
Zizhuo Fu, Wenxuan Zeng, Runsheng Wang, and Meng Li. Attention sink forges native moe in attention layers: Sink-aware training to address head collapse. arXiv preprint arXiu:200.01200, 2026.
Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning.
Advances in neurol information processing systems, 32, 2019.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263 1272 PMLR, 2017.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International conference machine learning, pp. 1616-7633. PMLR, 2022.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXio preprint arXiv:2104.01770 2021.
M Gori, G Monfardini, and F Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference Neural Networks, 2005., volume 2, pp. 729 734. IEEE, 2005.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The laaaa 3 herd of models. arXiv preprint 2xXx:2407.21788 2024.
Alessio Gravina, Davide Bacciu, and Claudio Gallicchio. Anti-Symmetric DGN: stable architecture for Deep Graph Networks. In The Eleventh International Conference on Learning Representations, 2023.
Alessio Gravina, Moshe Eliasof, Claudio Gallicchio, Davide Bacciu, and Carola-Bibiane Schönlich. On oversquashing in graph neural networks through the lens of dynamical systems. In The 39th Annual AAAI Conference Artificial Intelligence, 2025.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 0rXiv:2312.00722, 2023.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXie preprint arXiv:2410. 10781, 2024.
Benjamin Gutteridge, Xiaowen Dong, Michael M Bronstein, and Francesco Di Giovanni. DRew: dynamically rewired message passing with delay. In International Conference on Machine Learning, pp. 12252-12261 PMLR, 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Feijiang Han, Xiaodong Yu, Jianheng Tang, Delip Rao, Weihua Du, and Lyle Ungar. Zerotuning: Un- locking the initial token's power to enhance large language models without training. arXiv preprint arXiv:2505.11 11055.11721 11799, 2025.
Ali Hariri, Álvaro Arroyo, Alessio Gravina, Moshe Eliasof, Carola-Bibiane Schönlieb, Davide Bacciu, Kam- yar Arizzadenssheli, Xiaowen Dong, and Pierre Vandergheynst. Return of chebnet: Understanding and improving an overlooked gnn long range tasks. arXiv preprint SKKt:2506.07622 2025.
Bobby He and Thomas Hofmann. Simplifying transformer blocks. arXiv preprint orXiv:2311.0190 2023.
Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, and Yee Whye Teh. Deep transformers without shortcuts: Modifying self-attention for faithful signal propa- gation. Xixi preprint arXiv:2302. 103222 2023.
Simon Heilig, Alessio Gravina, Alessandro Trenta, Claudio Gallicchio, and Davide Bacciu. Port-hamiltonian architectural bias for long-range propagation in deep graph networks. In The Thirteenth International Conference on Learning Representations, 2025.
Mikael Henaff, Arthur Sulam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. Tn International Conference on Machine Learning, pp. 2034 2042. PMLR, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 988):1735-1780, 1997.
Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58 65, 2021.
Saidul Islam, Hanac Elmekki, Ahmed Elscbai, Jamal Bentahar, Nagat Drawel, Gaith Rjoub, and Witold Pedrycz. A comprehensive survey on applications of transformers for deep learning tasks. Expert Systems with Applications, 241:122666, 2024.
Federico Arangath Joseph, Jerome Sieber, Melanie Zeilinger, and Carmen Amo Alonso. Lambda-skip connections: the architectural component that prevents rank collapsc. arXiv preprint arXiv:2410. 100.000....000 2024.
Chaitanya K Joshi. Transformers are graph neural networks. arXiv preprint 25l::2500.2288,, 2025.
T Anderson Keller, Lyle Muller, Terrence Sejnowski, and Max Welling, Traveling waves encode the recent past and enhance sequence learning. In The Twelfth International Conference on Learning Representations, 2023.
Nicolas Keriven. Backward oversmoothing: why is it hard to train deep graph neural networks? ar preprint arXiv:2505.1675 2025.
Hycongjin Kim and Byoung Chul Ko. Rethinking attention mechanisms in vision transformers with graph structures. Sensors, 24(4):1111, 2024.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Represenlations, 2017.
Mario Lezcano- Casado and David Maattnez-Rubiae Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. International Conference on Machine Learning, pp. 3794-3803. PMLR, 2019.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep cnns? In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9267-9276, 2019.
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. Xixiv preprint arXiv:2006.07739 2020.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey, Philosophical Transac- tions of the Royal Society A, 279021944:202002009 2021.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the associalion for computational linguisties, 12:157- 173,-177,111 2024.
Weigang Lu, Yibing Zhan, Binbin Lin, Ziyu Guan, Liu Liu, Baosheng Yu, Wei Zhao, Yaming Yang, and Dacheng Tao. Skipnode: On alleviating performance degradation for deep graph convolutional networks. IEEE Transactions on Knowledge and Dala Engineering, 2024.
Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic gnns are strong baselines: Reassessing gnns for node classification. arXiv preprint or Xrx:n22406. 089999 2024.
Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Unlocking the potential of classic gnns for graph-level tasks: Simple architectures meet excellence. arXiv preprint arXiv:2502.0996. 2025.
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puncet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph inductive biases in transformers without message passing. In International Conference on Machine Learning, pp. 23321 -23337. PMTRR, 2023.
Wojciech Masarezyk, Mateusz Ostaszewski, Tin Sum Cheng, Aurelien Lucchi, Razvan Pascamu, et al. Un- packing softmax: How temperature drives representation collapse, compression, and generalization. arXiv preprint Xrx::22006. 01562, 2025.
Sohir Maskey, Raffaele Paolino, Aras Bacho, and Gitta Kutyniok. A fractional graph laplacian approach to oversmoothing. Advances in Neural Information Processing Systems, 36, 2024.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construc- tion of internet portals with machine learning. Information Retrieval, 3(2):127-163, 2000.
Alireza Naderi, Thiziri Nait Saada, and Jared Tanner. Mind the gap: a spectral analysis of rank collapse and signal propagation in transformers. arXiv preprint (tlo:2410.079900 2024.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvicto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Newral Information Processing Systems, 35:27195 1821211, 2022.
Kenta Oono and Taiji Suzuki. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. In International Conference on Learning Representations, 2020.
Francesco Pappone. Beyond attention as a graph. Blogpost, 2025.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning, volume 28, pp. III-1310, 2013.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention- sink-free. arXic preprint arXiv:2505.067 067000 2025.
Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in Ilms are two sides of the same coin. arXiv preprinl or Xrx::225100064 77, 2025.
Jason Ramapuram, Federico Danieli, Eeshan Dhekanc, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint ar arXiu:2409. Ki::2409.04431, 2024.
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501-14515, 2022.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arxiv Stt::1977.1000 2019.
Andreas Roth, Franka Bause, Nils M Kriege, and Thomas Liebig. Preventing representational rank collapse in mpnns by splitting the computational graph. arXiv preprint orXic:2409.11500 2024.
T Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning, pp. 18888-18909. PMLR, 2022.
T Konstantin Rusch, Benjamin Paul Chamberlain, Michael W Mahoncy, Michael M Bronstein, and Sid- dhartha Mishra. Gradient gating for deep multi-rate learning on graphs. In The Eleventh International Conference on Learning Representations, 2023.
Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. What are you sinking? geometric approach on attention sink. arXiv preprint or arX::22508.005 2025.
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. Rethinking softmax: Self-attention with polynomial activations. arXiv preprint arXiv:2410.18613, 2024.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Michael Scholkemper Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normal- ization can provably prevent oversmoothing in gnns. arXio preprint 0xXi:2206.02297, 2024.
Prithwiraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collec- tive classification in network data. Al magazine, 29(3): 93, 2008.
Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. Graph transformers: A survey. IEFE Trunaactons on Neural Networks and Learning Systems, 2026.
Alessandro Sperduti. Encoding labeled graphs by labeling raam, Advonces in Newral Information Processing Systems, 6, 1993.
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? arxio preprinl vrXiv:1804.11188 2018
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. arXie preprint or NKv:2403.08220 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giusoppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. Xiu preprint ar arXiv:2408.000 00118, 2024b.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint Xre:225003. 19786, 2025.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXi::2111.1422, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinct, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprinl ar Xiv:20002 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiu:2307.09280 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp aut-of-distribution)) arXiv preprint arXiv:2410.01100 2024.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph atlention networks. ArXiu, 2018.
Keqin Wang, Yulong Yang, Ishan Saha, and Christine Allen-Blacchette. Understanding oversmoothing in gnns as consensus in opinion dynamics. arXiv preprint arXiv:2501. 19089, 2025.
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. arXiv preprint arXiv:2405.1878. 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient strcaming language models with attention sinks. ar preprinl ar arXe:22309. 1 17453, 2023.
Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, and Ngai Wong. Parallelcomp: Parallel long-contens compressor for length extrapolation. arXiv preprint Nrv:2200 14317, 2025.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference on machine learning, pp. 10524-10533. PMLR, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka, How powerful are graph neural networks? arXiv preprint ar Xiv: 1810:18000000 00.00086, 2018.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Yan Liu. Do transformers really perform badly for graph representation? Advances in Newral Information Processing Systems, 34:28877-28888, 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEF/CVF international conference compuler vision, pp. -6259, 2021.
Lingxiao Zhao and Leman Akoglu. Pairnorm: arXiv:1909.1900 2019.
Tackling oversmoothing in gnns. arXiv preprint
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14901-14911 2025.
Zayd MK Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. Softpick: No attention sink, no massive activations with rectified softmax. ar Xiu preprint 2rXin:2504.20966 2025.
23


Reviews:
[{"review_id": "M6V8v1Mr22", "reviewer": "Reviewer_per9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** The paper surveys the literature on optimization problems in graph neural networks and relates it to the attention mechanism in decoder-only transformer models, arguing that the architectural choices of LLMs may be explainable as attempting to prevent 'over-squashing' and 'over-smoothing' problems known to plague GNNs. \n\n**Strengths:**\n- The paper written in a relatively accessible manner and has a complete background for sequence models, self-attention, vanishing gradients, and graph neural networks. \n- The paper serves as a strong review/survey of literature related to optimization and information propagation in graph-neural networks and sequence models.\n- It introduces a potentially valuable connection between known issues with transformer architectures and known issues in the graph neural network and sequence modeling communities. \n\n**Weaknesses:**\n- The major weakness with the paper is the limited depth with which the new perspective is put to use to support the major claim that it provides a new 'understanding' of transformers.\n\t- The majority of the paper (up to page 9 & 1/2) is background or discussion that surveys prior work. \n\t- The main new contributions of the paper are presented in Section 4, but the formalism introduced early in the paper is not leveraged to make the claims convincing. Specifically:\n\t\t- Sentences such as: \"We note that recent work (Qiu et al., 2025) also prevented the appearance of attention sink by incorporating gating into the attention mechanism, which can be seen as an alternative (and more direct) way of preventing over-mixing\". This claims that their overmixing perspective applies to explain why the gating of (Qiu et al., 2025) is successful, but gives nothing to the reader in terms of how this perspective achieves this explanation. \n\t\t- Similarly, the idea of why sharp attention heads fix collapse in width is not explained at a mechanistic level, despite the introduction of the formal background earlier in the paper that would make this possible. \n\t\t- Ultimately, the quote in the conclusion is not fully justified: \"This perspective offers a powerful framework for understanding how the spectral properties of attention matrices influence signal propagation and why architectural choices, like the Transformer block, are essential for robust learning.\" While this is likely true, the authors do not make a clear link between their perspective and the understanding they purport to provide. \n\n- I do not agree with the sentence \"While some of these works have attributed the success of their models to a superior inductive bias (e.g. through \"physically-inspired methods\"), it is more fundamentally tied to other optimization-related phenomena\". First,  'other optimization-related phenomena' is non-specific enough to be of value to readers. Secondly, the distinction between physically inspired inductive biases and optimization related properties is an invalid distinction in my opinion. The physical systems these methods are based on have known stability properties that the authors are knowingly employing. I would argue it is disingenuous to claim that they are ignorant of the 'optimization-related' benefits of the physical systems they choose. \n\n- On page 10: \"Just as deep GNNs suffer from over-smoothing, where node representations converge to a uniform vector, naively stacking Transformer layers leads to rank collapse (Dong et al., 2021)\". However, this is not what Dong 2021 proves, and does not align with the argument in section 4.3 -- Dong argues that stacking pure attention leads to rank loss, but skip connections and MLPs prevent this (just as the authors also argue later). \n\n- Figure 5 is not intuitive, the meanings of the different colors are not explained, so it is not clear what is being represented. \n\n\n**Minor:**\n- Typo in footnote on page 4: \"choice on nonlinearities\"\n- The Sutskever et al., 2013 citation on Page 4 seems out of place -- it is not about reservoir computing .\n- \"width\" on page 6 and  \"shallow” on page 9 have mixed quotation styles.\n- The text on page 8 mislabels left vs. right for Figure 3: \"This is illustrated in Figure 3 (right),\" should say (left), and similarly for the next sentence.\n- Page 12, you cite the wrong Xiong et al. paper. You cite 2021, but I believe you meant to cite the 2020 paper on Layer norm (https://arxiv.org/abs/2002.04745).\n- A number of bibliography errors: \n\t- \"The llama 3 herd of models\" has an incorrect arxiv identifier. \n\t- \"Diffwire\" paper has a broken accent. \n\t-  Oversmoothing,\" oversquashing\", -- extra quotes in title.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Since this is mainly a survey paper, the majority of the claims made are simply taken from prior work and thus true (except where mis-cited, as noted above e.g. for Dong et al.) -- justifying the answer 'yes'. The 'new' claims of the paper are also likely true, but the evidence for supporting them is left up to the reader to infer and therefore not clear or convincing (see major weakness above). I would therefore answer 'partial yes' if available.", "audience_yn": "Yes", "audience_explanation": "Rank collapse in decoder-only transformers is an active relevant topic in the literature that the current paper purports to provide a new lens and toolkit through which to study this phenomenon. The paper further offers a good review of the literature on oversquashing, oversmoothing, and vanishing gradients in recurrent neural networks.", "requested_changes": "- Addition of a concrete (mathematical) formalization of how the paper's perspective can explain the 'mitigation strategies' (or at least some of them).\n- The colors in Figure 5 should be explained in more detail to make it interpretable. \n- The noted minor issues (citations, typos, etc.).", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "wwZ1tBU3vq", "reviewer": "Reviewer_JEi5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is motivated by the identification of common properties between message-passing Graph Neural Networks and decoder-Transformer architectures. \nThe manuscript is organized into three parts: \n- **Section 2** provides a foundational overview of sequence modeling (using RNNs and Transformers) and message-passing GNNs;\n- **Section 3** describes in-depth the concepts of over-smoothing, vanishing gradients and over-squashing, overviewing the existing literature within GNN field;\n- **Section 4** proposes a unified representation of Transformers and MPGNNs. It specifically addresses how Transformers mitigate rank and representation collapse.\n\n## Strengths\n\n1. Section 3 provides an in-depth overview of over-smoothing, vanishing gradients, and over-squashing within the GNN domain.\n2. Section 4.3 lists current methods to prevent rank and representation collapse in the Transformer domain.\n\n\n## Weaknesses\n(see more details in the Requested Changes section)\n\n1. The manuscript lacks valuable insights, which can be demonstrated by either:\n    * Comprehensive literature review *OR*\n    * Extensive experiments showing practical mitigation of over-smoothing or over-squashing concepts in transformers *OR*\n    * Theory for presented concepts when applied to transformers\n2. While the paper defines a connection between message passing GNNs and Transformers, this unified view is not leveraged to generate new insights or drive the subsequent discussion.\n3. The paper fails to address the existing body of literature on Graph Transformers. This literature must be addressed when graph and transformer concepts are discussed together.\n4. The authors dedicate more than one page out of thirteen to the mathematical foundations of RNNs in Section 2, yet these are never addressed in the remainder of the paper.\n6. The motivation and discussion are overly focused on Large Language Models (LLMs). The proposed \"unified view\" is not specific to LLMs. Thus, the paper lacks a connection to vision transformers, time-series transformers, graph transformers, etc.", "claims_and_evidence_yn": "No", "claims_explanation": "**NOTE:** The paper is not fully clear regarding its primary claims (see requested changes). Here, I evaluate claims extracted from the abstract, introduction, and conclusion.\n\n1. &#x2705; \"we present a unified graph perspective that bridges the theoretical understanding of decoder-Transformers and GNNs\" and \"interpreting the self-attention mechanism as a learned adjacency matrix, we bridge the gap between spectral graph theory and the dynamics of attention.\"\nThis claim is largely met. Equations (26)-(28) in Section 4 establish a unified mathematical view of GNNs and Transformers. \n2. &#x274C; \"We systematically examine how well-known phenomena in GNNs, such as over-smoothing and over-squashing, directly manifest as analogous issues like rank collapse and representational collapse in deep Transformer architectures.\"\nThe paper provides no systematic experimental evidence to support this claim. The lack of empirical validation is a weakness.\n3. &#x2753; \"we reveal shared underlying principles governing signal propagation\"\nThe formulation of this claim is ambiguous. On the one hand, section 4.1 shows that a self-attention block can be viewed as a Graph Attention Transformer layer (GAT). At the same time, GAT is founded on the same self-attention math that is later used in self-attention block of the transformer. Veličković et al. (2018) explicitly define GAT as self-attention.\nTherefore, the paper does not \"reveal\" a new principle, but rather restates an existing one.\n4. &#x274C; \"[we] demonstrate how insights from one field can illuminate challenges and solutions in the other\" and \"[we aim] to highlight areas for cross-pollination of research\"\nThis claim is vaguely formulated and arguably is not met. For example, let us look at the \"The Depth Regime\" subsection of Section 4.2. While the text notes conceptual similarities between rank collapse and over-smoothing based on the eigenspectrum, it fails to demonstrate how GNN research provides actionable solutions for Transformer rank collapse. The Gemma 7B experiment in Section 4.2 illustrates the rank collapse problem but offers no cross-disciplinary solution.\n5. &#x2753; \"We analyze the role of architectural components like residual connections, normalization, and causal masking in these issues.\"\nThis claim is somewhat true. Although the paper discusses components that prevent these issues and cites relevant literature, it lacks empirical or theoretical analysis.\n6. &#x274C; \"We aim to provide a framework for understanding how information flows through deep learning models that perform sequence mixing through an adjacency operator\"\nThe manuscript overviews existing works on information flow in message-passing GNNs, but it fails to synthesize these into a distinct, cohesive framework.\n8. &#x2705; \"we elaborate on how the normalized adjacency matrix – the key component of mixing architectures – affects information flow in depth (in the number of layers) and width (in the diameter of the graph or sequence)\"\nThis is sufficiently discussed in Section 3.\n9. &#x2705; \"we have explored the conceptual and mathematical parallels between decoder-Transformers and GNNs, advocating for a unified graph-based perspective on information propagation in these seemingly disparate architectures.\"\nWhile I disagree with the statement that \"Transformers and GNNs [are]...seemingly disparate architectures\" (see below), this claim is technically fully met in Section 4.1 and Section 4.2.\n10. &#x274C; \"Our analysis revealed that the iterative message-passing operations in GNNs, which can lead to node feature homogenization, mirrors the depth-wise rank collapse observed in Transformers where token representations\"\nThe analysis in Section 4.2 is superficial.\nI must highlight that a convincing link between concepts requires more than semantic similarity (e.g., both having \"contractive\" components).\nIt requires a rigorous demonstration of empirical or theoretical similarities.\nThe fact that GNNs have \"the *contractive nature* of the mixing matrix's spectrum\" and for Transformers \"many learned attention heads are, in fact, highly *contractive*\" does not fit such a demonstration of similarity.\n11. &#x274C; \"this unified view ... enhances our theoretical understanding of [transformers]\"\nWhile a unified view has the potential to enhance theoretical understanding of transformers, this manuscript presents no new theoretical results to achieve this goal.", "audience_yn": "No", "audience_explanation": "In its current state, the paper is unlikely to be of interest for the TMLR's audience.\nIt lacks either a comprehensive literature review, *OR* theoretical contributions, *OR* rigorous empirical results demonstrating how GNN-based insights can be substantively applied to Transformer architectures.\n\nThe manuscript is inadequate as a cohesive survey.\nWhile Section 3 provides a comprehensive literature overview of over-smoothing, vanishing gradients, and over-squashing in GNNs, it lacks an analysis of how these issues are measured, the specific tasks where they occur, or a taxonomic approach to mitigation strategies.\n\nThe paper presents no theoretical analysis.\nThe purported connection between over-smoothing and rank collapse is not established in a technically sound manner. \nFurthermore, the discussion regarding over-squashing and representational collapse is merely a brief summary of theoretical findings from Barbero et al. (2024), omitting details and leading to no new insights.\n\nThere are no empirical experiments to support the authors' claims. \nWhile Section 4.1 outlines mathematical similarities between message passing GNNs and Transformers, this \"unified view\" does not result in a novel architectural design or a comparative demonstration of shared properties.\nThe manuscript fails to discuss how this unified view can be leveraged to improve Transformer.", "requested_changes": "**IMPORTANT:** I see three ways how this work can be strengthened: survey route, theory route, and experimentation route.\nTo secure my positive recommendation, the authors should address my comments from a \"common\" critical section and *ONE* route of their choice (Survey, Theory, or Experimentation).\nThis will provide the depth currently missing from the manuscript.\n\n# Common Critical Changes (Required)\n\nThe paper relies on the premise that there are \"structural similarity between GNNs and Transformers.\" However, **Transformers are GNNs**. The paper completely ignores the field of Graph Transformers, which studies how to process a graph with a transformer. To demonstrate how extensive this field is, I encourage authors to check out a survey on Graph Transformer by Shehzad, et al. (2024). Authors must incorporate some literature on Graph Transformers, at the very least mentioning the known ones, like Rampášek et al. (2022), Ma et al. (2023), and Wu et al. (2022). \n\nWhy is the paper so focused on LLMs? It seems arbitrary. Not a single concept in Section 2-4 is exclusive to LLMs. Yes, you may mention that LLMs had \"the remarkable success\" in recent years. But you also need to mention that there are the other types of transformers with different specificity (Vision, Time-Series, and Graph. For other types, you may consult the survey by Islam et al. (2024). The authors must either explicitly justify the exclusive focus on LLMs *OR* make the motivation more general.\n\nThe current claims are formulated vaguely and are scattered over the text. Perhaps, the authors can summarize the contributions in a list *OR* write an explicit problem statement section. This will make it easier for a reader and increase the quality of your work.\nFor example, \"In this paper, we present a unified graph perspective that **bridges the theoretical understanding** of decoder-Transformers and GNNs.\" (from the Abstract).  Which part of the paper presents said bridging theory? Such a formulation makes the reader expect a specific theoretical result.\n\nSections 4.1–4.2 identify similarities between problems in GNN domain and problems in Transformer domain. Can authors discuss in depth how analysis from GNN field listed in Section 3 can be used to improve transformer's performance, training stability, etc.?\nFor example, the most direct application of GNNs can be by taking a Transformer block (say, in Gemma 7B) and replacing MHA with the GNN layer and a causal graph. As a matter of fact, Choi et al. (2024) and Kim & Byoung (2024) present updates to a general Transformer structure with exactly this approach.\n\nSection 4.3 is called \"Shared Mitigation Strategies\" and promises to introduce how \"they manifest in both decoder-Transformers and GNN\". Despite its title, this section focuses almost entirely on collapse-preventing strategies for Transformers. The author must include the discussion of how the similar mitigation strategies manifest in GNNs. The authors should discuss what are the similarities and differences of these strategies.\n\nWhy is RNN background literature covered? In this work, no similarities between RNN and GNN are leveraged. The inclusion of RNN mathematical foundations is currently a distraction for the reader. The RNNs are only briefly mentioned in Section 4.3 in the context of gated attention, which does not require such an extensive background. The authors have to either integrate RNNs into the analysis *OR*  remove this subsection.\n\nPage 10. The histogram of its associated eigenvalue modulus shown in Figure 4 is a trivial consequence of the attention matrix being row-stochastic. The row-stochastic matrix always has eigenvalue modulus upper-bounded by one. This fact is directly acknowledged on Page 11 of the text. Why is this fully expected result significant and relevant? Could you provide a more nuanced analysis of this figure?\n\nPage 12. While the block on Figure 6 may be a valid block, it is important to cite the paper this block is taken from (both Transformer literature and GNN literature). There is a huge variation in the design of the transformer block. \nFor example, your block is different from the Transformer block by Vaswani et al. (2017).\nThis block design is also not very typical for message-passing GNNs, which rely on Dropouts and layerwise attention. Beside that, dropout and normalization in GNNs are commonly applied after the message passing (as an example, see the GNN implementation in Luo et al. (2024)).\n\n\n# Critical Routes (Choose One)\n\n## Survey route\n**ROUTE NOTE:** Currently, some sections focus either exclusively on GNNs or exclusively on Transformers. The main idea is to survey the literature to further establish similarities between them.\n\nSystematize the prevention methods for over-smoothing (GNN) vs. rank collapse (Transformer) and over-squashing (GNN) vs. representational collapse (Transformer). Section 3 contains a great overview of over-smoothing, vanishing gradients, and over-squashing concepts in GNNs. It lists a number of works that are analysing these concepts. Can you summarize what are the nature of the main methods for preventing them? Can you **survey Transformer papers** that are analysing and preventing the respective concepts?\n\nFor example, let us consider over-smoothing (GNN) vs. rank collapse (Transformer).\nWhat are the top methods for preventing over-smoothing? \nWhat are the top methods for preventing rank collapse?\nDo such methods have anything in common?\nWhat are the differences?\nIf any, why and what can be borrowed?\nThe text answering these questions will constitute a survey that bridges GNN and Transformers.\nMake the same analysis for over-squashing in GNN and representation collapse in Transformers\n\nCurrently, your survey on over-squashing (GNN) vs. representational collapse (Transformer) consists on only two papers. Are there no works analysing over-squashing in GNNs? Barbero et al. (2024) and Barbero et al. (2025) have more than 50 citations in total. Does none of those works propose an additional analysis for representational collapse or over-mixing?\n\nSection 4.3 discusses the design of a Transformer block. You need to add a discussion of a GNN block design. How GNN block is structured? What are the similarities in design between these blocks? Are the designs guided by the similar issues? Do the similarities in design result in the similar properties? Can the transformer block be enhanced with methods from Luo et al. (2024) (similarly to a classic GNN)?\n\nIn addition to a regular Transformer, you have to discuss the design of a graph transformer block, since it is bridging both fields. What are the design choices of Graph Transformer that are different from other Transformer blocks? Are there similarities between Graph transformer blocks and GNN blocks? How over-smoothing, rank collapse, over-squashing, and representational collapse manifest in Graph Transformer literature? What methods from GNN field are adapted by Graph Transformer to mitigate it? \n\n## Theory route\n**ROUTE NOTE:** Sections 4.1-4.2 vaguely point at similar mathematical properties for GNN and transformer, as well as referring to the paper analysing such properties from the theoretical point of view. Currently, this work does not introduce any theory. That makes analysis seem superficial.\n\nFormulate a technically sound connection between over-smoothing and rank collapse. For example, could you show a sensitivity bound for Transformer similar to Theorem 3.2.1? Given the unified view from Section 4.1 and the theoretical connections and GNN literature from Section 3, can you claim something about the transformer? Can you show it with any toy experiment?\n\nLook at how Barbero et al. (2024) formulates the theory connecting over-squashing and representational collapse. It is presented in two theorems with proofs. Can you make a similar connection for over-smoothing and rank collapse?\n\nThe theoretical connection for over-squashing and representational collapse can be cited directly from Barbero et al. (2024) appendix. However, it has to be incorporated into the presented theory.\n\nThe authors must discuss how the specificity of the graph domain and LLM structure affect the presented analysis. One of the main problems with applying spectral graph theory to LLMs is that spectral graph theory is mainly done for **undirected** graphs. The attention matrix of LLM is making a **directed** graph due to causal masking. Therefore, the assumptions of the literature you cite might not hold. You have to be especially careful about assumptions of eigenvalues.\n\nMake sure any lemmas, theorems, or propositions you add have rigorous mathematical proofs.\n\n## Experimentation route\n**ROUTE NOTE:** One way to show a connection between GNN and GNN interpretation of a Transformer is to experimentally demonstrate that similarity between behaviour of GNNs and Transformers.\nUnlike for the theory route above, one does not require bringing theory for this route, yet the experiments have to be convincing.\nAt the moment, this paper does not feature experiments that illustrate similar properties of the unified view of Transformer and GNNs. Figures 4 and 7, which can be considered example results, are only showing the specific properties of Transformer.\n\nDemonstrate side-by-side over-squashing in both Transformer and GNN. Discuss how graph transformers mitigate it.\n\nDemonstrate side-by-side sharp attention in Transformer and adaptive message passing GNN (or GAT).\n\nDemonstrate side-by-side over-smoothing and rank collapse.\n\nDemonstrate that a Transformer block modified with GNN-inspired layers (per Choi et al., 2024) exhibits predictable behaviour.\n\nMake sure you are using two-three GNNs and two-three different transformers in every experiment. Specifically, include GAT. It is based on the same design idea as the transformer and thus can make a relevant point of connection.\n\n# Minor\nPage 7. Figure 1 is a little misleading. First, it shows a tree, which is a very special case of graphs. I recommend presenting an arbitrary graph colouring a root node and 1-hop, 2-hop,... neighbourhoods (see Figure 1 in Zhang, et al. (2022) for the reference). Second, message passing GNNs and graph transformers propagate signal on a graph differently. Figure 1 is suitable for message-passing GNNs. I recommend pairing it side-by-side with a signal propagation picture for the transformer.\n\nPage 8. The colour scheme on Figure 3 is hard to distinguish, especially when histograms are overlapping each other. Consider more contrastive colours.\n\nPage 9. \"Transformers and GNNs have traditionally been viewed as distinct architectures\" is generally wrong, because graph transformers are a thing and have to be mentioned in this paper.\n\nPage 11. \"In turn, modern LLMs ... impose a triangular (causal) mask for next-token generation. This, together with the softmax function, makes it row-stochastic.\" Consider rephrasing this bit for better readability. First, makes what stochastic? Second, if we assume \"it\" is \"attention matrix\", then softmax is solely responsible for row-stochasticity. The presence of causal masking has nothing to do with it. The attention matrix is row-stochastic even without the mask.\n\nPage 11. Causal transformers (like modern LLMs) are using mask to ensure the information flows only from the current input to future hidden states. Despite the masking is being mentioned on the same page, there is a factual error, where a causal graph is implied to be a fully connected graph.\n\nPage 12. Verify that Xiong et al. (2021) is the citation for pre-LN transformers.\nThis paper does not seem to be about transformers.\n\n# Literature\n1. [Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in Neural Information Processing Systems (2017).](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n2. [Barbero, Federico, et al. \"Transformers need glasses! information over-squashing in language tasks.\" Advances in Neural Information Processing Systems (2024).](https://proceedings.neurips.cc/paper_files/paper/2024/file/b1d35561c4a4a0e0b6012b2af531e149-Paper-Conference.pdf)\n3. [Barbero, Federico, et al. \"Why do LLMs attend to the first token?.\" Conference on Language Modeling (2025).](https://arxiv.org/pdf/2504.02732)\n4. [Islam, Saidul, et al. \"A comprehensive survey on applications of transformers for deep learning tasks.\" Expert Systems with Applications (2024).](https://arxiv.org/pdf/2306.07303)\n5. [Rampášek, Ladislav, et al. \"Recipe for a general, powerful, scalable graph transformer.\" Advances in Neural Information Processing Systems (2022)](https://proceedings.neurips.cc/paper_files/paper/2022/file/5d4834a159f1547b267a05a4e2b7cf5e-Paper-Conference.pdf)\n6. [Choi, Jeongwhan, et al. \"Graph convolutions enrich the self-attention in transformers!.\" Advances in Neural Information Processing Systems (2024).](https://arxiv.org/pdf/2312.04234v3)\n7. [Kim, Hyeongjin, and Byoung Chul Ko. \"Rethinking attention mechanisms in vision transformers with graph structures.\" Sensors (2024).](https://www.mdpi.com/1424-8220/24/4/1111)\n8. [Veličković, Petar, et al. \"Graph attention networks.\" International Conference on Learning Representations (2018).](https://openreview.net/forum?id=rJXMpikCZ)\n9. [Zhang, Shichang, et al. \"Graph-less neural networks: Teaching old MLPs new tricks via distillation.\" International Conference on Learning Representations (2022).](https://openreview.net/forum?id=4p6_5HBWPCw)\n10. [Luo, Yuankai, Lei Shi, and Xiao-Ming Wu. \"Classic gnns are strong baselines: Reassessing gnns for node classification.\" Advances in Neural Information Processing Systems (2024).](https://proceedings.neurips.cc/paper_files/paper/2024/file/b10ed15ff1aa864f1be3a75f1ffc021b-Paper-Datasets_and_Benchmarks_Track.pdf)\n11. [Shehzad, Ahsan, et al. \"Graph transformers: A survey.\" arXiv preprint arXiv:2407.09777 (2024).](https://arxiv.org/pdf/2407.09777)\n12. [Ma, Liheng, et al. \"Graph Inductive Biases in Transformers without Message Passing.\" International Conference on Machine Learning (2023).](https://proceedings.mlr.press/v202/ma23c/ma23c.pdf)\n13. [Wu, Qitian et al. \"NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification\" Advances in Neural Information Processing Systems (2022).](https://proceedings.neurips.cc/paper_files/paper/2022/file/af790b7ae573771689438bbcfc5933fe-Paper-Conference.pdf)", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A2Aen3Gs0G", "reviewer": "Reviewer_onyQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey connecting GNNs and decoder-Transformers through a unified graph perspective. The main contribution is showing how problems studied in GNNs (over-smoothing, over-squashing) map onto issues in Transformers (rank collapse, representational collapse). The authors argue that the self-attention mechanism can be viewed as a learned adjacency matrix, which means insights from one field can transfer to the other.\n\n  Key strengths:\n  - The unified framework is intuitive and well-motivated\n  - Good coverage of both GNN and Transformer literature\n  - The figures (especially Fig 1, 4, 7) help explain the concepts clearly\n  - Practical discussion of mitigation strategies (residual connections, normalization, sharp attention)\n\n  Weaknesses:\n  - Mostly a literature review, limited new theoretical results\n  - Some sections feel a bit dense with citations without enough synthesis\n  - Missing discussion of computational trade-offs between different approaches", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is primarily a survey, so the claims are well-supported by existing literature. The authors do a reasonable job citing the relevant work from both GNN and Transformer communities. The visualizations (attention matrices from Gemma 7B, eigenvalue histograms) provide empirical grounding for the theoretical arguments. I didn't spot any obvious mischaracterizations of prior work.", "audience_yn": "Yes", "audience_explanation": "Yes, I think this will be useful for researchers working at the intersection of GNNs and LLMs. The unified perspective could help practitioners diagnose issues in their models and potentially transfer solutions across domains. Given the current interest in understanding Transformer internals and scaling behavior, a principled framework connecting them to better-understood GNN dynamics seems timely.", "requested_changes": "1. (Minor) Section 3 and 4 could benefit from a clearer distinction between what's established theory vs. recent observations. Some parts blend these together.\n  2. (Minor) The paper would be stronger with a concrete example or case study showing how a technique from GNNs was successfully applied to improve a Transformer, or vice versa. Right now it's mostly \"these things are similar\" without demonstrating practical cross-pollination.\n  3. (Minor) Table summarizing the key correspondences (over-smoothing ↔ rank collapse, over-squashing ↔ representational collapse, etc.) would help readers quickly grasp the main mapping.\n  4. (Minor) Some newer relevant work on long-context LLMs and their attention patterns could be added to Section 4.2", "broader_impact_concerns": "None. This is a theoretical/survey paper examining architectural properties of neural networks. I don't see any direct ethical concerns.", "additional_comments": "Overall this is a solid survey paper that provides a nice unified lens for understanding Transformers and GNNs. The writing is clear and the structure is logical. My main suggestion would be to go a bit deeper on the practical implications - what should practitioners actually do differently based on these insights? But I understand space constraints. The figures are well done and help a lot with understanding the concepts.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)