Wavy Transformer
Authors: Satoshi Noguchi
Yoshinobu Kawahara
Venue: NeurIPS 2025 poster ()
Keywords: Transformers
Over-smoothing
Wave dynamics on graphs
Partial differential equations
Primary area: deep_learning

Abstract: Transformers have achieved remarkable success across natural language processing (NLP) and computer vision (CV). However, deep transformer models often suffer from an over-smoothing issue, in which token representations converge to similar values as they pass through successive transformer blocks. In this paper, we establish an equivalence between the hidden-state dynamics induced by stacked attention layers and graph neural diffusion on a complete graph. From this perspective, over-smoothing can be interpreted as a consequence of the dissipative nature of the underlying diffusion dynamics. Motivated by this physical interpretation, we propose Wavy Transformer, which consists of a novel attention layer based on second-order wavy dynamics. We also introduce a feed-forward network and a normalization layer designed to preserve the physical state-velocity relationship under the chain rule, thereby extending the transformer architecture. We further validate our proposed techniques on various transformer models for NLP and CV tasks. The results consistently demonstrate that Wavy Transformer improves performance with minimal additional parameters and no extra hyperparameter tuning.

Wavy Transformer
Satoshi Noguchi
Research Institute for Value-Added Information Generation, JAMSTEC Center for Advanced Intelligence Project, RIKEN satoshin@ jamsteicn@mmmteeeee jo.
Yoshinobu Kawahara
Graduate School of Information Science and Technology, The University of Osaka Center for Advanced Intelligence Project, RIKEN
wwwalaratisttoaakk-u.aa...
Abstract
Transformers have achieved remarkable success across natural language process- ing (NLP) and computer vision (CV). However, deep transformer models often suffer from an over-smoothing issue, in which token representations converge similar values as they pass through successive transformer blocks. In this pa- per, we establish equivalence between the hidden-state dynamics induced by stacked attention layers and graph neural diffusion on a complete graph. From this perspective, over- wer-smotthing can be interpreted as consequence of the dis- sipative nature of underlying diffusion dynamics. Motivated by this physical interpretation, we propose Wavy Transformer, which consists of a novel atten- tion layer based on second-order wavy dynamics. We also introduce feed- forward network and normalization layer designed to preserve the physical statevellocity relationship under the chain rule, thereby extending the transformer architecture. We further validate our proposed techniques on various transformer models for NLP, CV, and sparse-graph tasks. The results consistently demonstrate that Wavy Transformer improves performance with minimal additional parameters and no extra hyperparameter tuning. Source code and models are available at attps://githuue hwt.ah/www...ooggcccaaatooaaaeee Transforeerr
Introduction
Transformers have achieved outstanding success in wide range of machine learning fields such as natural language processing (NLP) [ and computer vision (CV) 33
26
24).
These successes clearly demonstrate the effectiveness and generality of the transformer. Despite this remarkable progress, deep transformer models often suffer from an over-smoothing issue, in which all token representations become identical as more layers are added [1]
29
43 37]. This
crucial obstacle preventing transformers from going deeper. Accordingly, several techniques to mitigate the over-smoothing behavior of transformers have been proposed [7 43 29). However, compared to graph neural networks (GNNs), where over-smoothing was first identified and widely studied over-smoothing in transformers has not been adequately
phenomenon also known as the token uniformity problem 112 and recognized
34 31 28 discussed.
14
35
38
20
30
2
17
Based on the above discussion, in this paper, we examine over-smoothing in transformers from the perspective of physical dynamical systems. In addition, we propose new type of attention layer inspired by physical dynamical systems. Our approach originates from interpreting the dynamics of hidden states induced by stacked attention layers as graph neural diffusion [ on complete
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

graph. From this viewpoint, over-smoothing can be seen as consequence of the dissipative nature of the underlying diffusive dynamics. Motivated by this physical interpretation. we introduce Wavy Transformer, which consists of a novel attention layer based on second-order wavy dynamics. The energy-preserving property and the oscillatory behavior of the wave equation are expected to help mitigate the over-smoothing problems in transformers. We also introduce physically inspired feed-forward network and a normalization layer designed to preserve the state-velocity relationship under the chain rule. By combining these components, we build Wavy Transformer to extend the conventional transformer architecture. Wavy Transformer block is easy to use and can be integrated into various transformer-lik architecture. Furthermore, to demonstrate the effectiveness and generality of Wavy Transformer, we conduct extensive experiments on NLP, CV, and sparse- graph tasks.
The remainder of this paper is organized as follows. In Section 2 we provide background on this work, including an introduction to the attention module and its over-smoothing issue, well as a brief review of the fundamental properties of dynamics on smooth manifolds. In Section [3] we present Wavy Transformer, after discussing rigorous physical interpretation of attention layers as graph neural diffusion on a complete graph. In Section [ we review related works. In Section [ we report experimental results demonstrating the capabilities of Wavy Transformer. Finally, we conclude with our main contributions and some closing remarks in Section 6
2 Background
2.1 Attention and Over-smoothing
The key functional component of transformer architecture is the attention module 41 43 which aggregates information from other token representations with respect to the computed attention values. Let X E [ be the input states to attention layer, where n is the number of input tokens and d is the embedding dimension. The attention calculation is formulated as follows:
XWo X XWy=AXWy
Attn (X= = softmax
(I)
Vd
where WK E Rdxds We Rdxds and Wv E Bdxd are the key, query and value weight matrices, respectively. Also, vi denotes scaling factor, and softmax(-) operation on X row-wisely. Here, A softmax XWa(XW*))
E _= In the case of Multi-Head Attention (MHA) [4I), multiple single-head attention modules operate in parallel, and their outputs are concatenated and linearly projected: MHA(X) = [Attn] (X), ................................ A (X) where the subscripts denote the head indices, the total number of heads, and Wo E R2 ffdd projects the concatenated outputs back to the hidden dimension. Since MHA is a parallelized form of single-head attention and is mathematically equivalent, we do not distinguish between them in this paper. In addition to the attention module, each transformer block is equipped with residual connections as:
va
X AX'W, +X',
(2)
where X denotes the (-th layer hidden states. The attention matrix can be regarded as learning pair-wise self-interactions among all components X. Thus, this attention computation encourages transformers to capture globally direct interactions among all tokens, unlike convolutional operations, which expand their receptive field hicrarchically from local to global. As a consequence, properly trained transformers can effectively model global context and achieve outstanding performance.
Despite their remarkable performance across a wide range of NLP and CV applications, deep transformer models often suffer from an over- smoothhgg problem: as the network deepens, token
representations converge and become indistinguishable from one another [37
43
29]. As a result,
deeper transformers do not always outperform their shallower counterparts 49 From a theoretical
continuously erasing high-frequent information and thus reducing feature expressiveness in deeper layers. Consequently, this tendency toward ovor-smgg of stacked the attention layers deep transformer presents a critical challenge that motivates our study.
viewpoint, the attention layer can be understood as a low-pass filter 43
2

---

2.2 Fundamental Dynamics on Manifolds
Here, we briefly review classical dynamics on a smooth manifold M, described by partial differential equations (PDEs), to motivate alternative dynamics for the transformer block [15]. For simplicity, we omit any specification of initial or boundary conditions whenever it not necessary. Let
MM x [0, T) -> IR denote a scalar field on M, where T > 0 ís the terminal time. Its prototypical example is the diffusion equation:
dr(s,t) t) div (iv(D = D t),
(3)
Ot
where M denotes a point (spatial coordinate) on the manifold, t E [0, TJ denotes time, D > 0 is a diffusivity coefficient, V and div are respectively the gradient and divergence operators induced by the Riemannian metric on M, and A = div o V is the Sapplace-Beltran operator. Here, D is assumed to be constant for simplicity. In physics, Eq.(3) models processes such as heat conduction or particle diffusion. In contrast, the hyperbolic (wave) equation on M takes the form
sx((s t) Dt2
div(v' vr(s,t)) = Ar(s,t),
where e 2 0 a constant pave-propngntion speed. Equation governs wave-like phenomena in continuous media. We highlight the key differences between the dynamics of these PDFs.
2.2.1 Kernel Perspective of Diffusion and Wave Equations
Fundamental solutions, which are often called "kernels". represent the response to an initial impulse (e.g., Dirac delta function). These kernels underscore some of the most striking differences between diffusion and wave phenomena. Thus, we review the kernels of both the diffusion and wave equations to clarify how they differ. For simplicity, we focus on the one-dimensional case in the reminder of this section. We also provide fundamental theorems from an energetic perspective that characterize each dynamic in Appendix
Diffusion Equation diffusion equation is given by
We recall the fundamental solution (Green's function) of the one-dimensional with the initial condition (zs. 0) = zu(s). Its kernel (Green's 's function) on RR
13
s²
G(s,t) V4Tl
exp
t>
(5)
4t
where we ignore the diffusive coefficient D. Accordingly, the solution with initial data Zo can be written in convolution form: xxs,t) t) J G(s v. t) zo (v) dv. Key properties of G in one dimension are as follows: Firstly, G(s,t) 0 for all s E R and decays exponentially as |s| And secondly, the heat kernel smooths out irregularities smoothing effect. The first property shows that, while the diffusive influence extends over all space, it becomes exponentially small far from the origin. The second property implies that diffusion rapidly weakens figh-frreqencyy components, inherently suppressing local structural details.
oo. the initial data, which is referred to as the
Next, we consider the one-dimensional wave equation with initial conditions Oz (s, 0) (() and Ø1 (s, 0) g(s)) By d' Alembert's formula (32), the general solution is
Wave Equation
1 x(s [[ [f(s- ut) + f(s + + + f
1 stut g(r) dr,
(6)
2v S
which represents left- and rraht-traveling waves emanating from the initial data. Key properties of this solution are as follows: Firstly, under wave propagation, strong singularities in the initial data can persist. Thus, the signal profile propagates at finite speed, rather than being smoothed out everywhere. And secondly, the general solution reflects conservation of energy: waves transport energy through the domain without diffusive spreading. Hence, contrast to the diffusion equation, the wave equation exhibits sharp, finite wavefront traveling at speed This difference in how features propagate is closely related to the fact that the wave equation conserves energy, whereas the diffusion equation dissipates it. The energy behaviors of each dynamics shown in Appendix C
3

---

(xl+1, r(t!)
Feed-Forward
Layer
FFN(X') of W' 1b]) W: 1bb
Normalization
A
FYN,(YY) = (x' lb] V' W, W₂
Add
Feed Forward
Laver Normalization
LN(x)), 2 m(x))
(x=(x() + +
Layer
Vi.
Normalization
(vD,) (x²(xx) +c 76
Add
Attention + Add
Multi-Head Attention
Y = T (A 1) + YY
X = - +
(X', Y
Figure 1: Schematic of Wavy Transformer block, combining wavy attention layers and velocity- specific layer- noyyyaroommliintioon and feed-forward layers. Each layer is designed to preserve state-velocity relationship. Post-LN is assumed here; the Pre-LN case is discussed in Appendix D.2 47).
3
Wavy Transformer
The above discussion and the fundamental theorems [ and in Appendix (iindicctt that the dynamics governed by the diffusion equation (3) converge constant uniform state, whereas those governed by the wave equation (4) oscillates while conserving its energy without net dissipation. However, conventional transformers implicitly rely on the dissipative diffusion dynamics, causing their hidden states to converge toward a uniformly constant state. Therefore, before presenting the architecture of Wavy Transformer, we first show that the implicit dynamics of attention layers can be interpreted as graph neural diffusion [ on a complete graph.
3.1 Attention as Graph Neural Diffusion
We consider the graph neural diffusion [5] complete graph whose each node feature is token representations:
0x
-)-IXX I)X,
Ōt
where A is the attention matrix and I is the identity matrix. For simplicity, we temporarily ignore the feature transformation by Wy. By definition, A is a right-stochastie matrix, i.e. Σ, Aij Hence, I A can be regarded as the normalized graph Laplacian of the complete graph in which the edge connecting nodes i and is weighted by the node-pair value Aij. Note that I A, which is often called the random-walk graph Laplacian [21], the negative of the discrete Laplacian used here. We can also discretize this graph diffusion equation in time domain as
X 4+1 T(A 1)x' + x' -AX + (1 (1-T)X',
(8)
for 1 1, 1,... N, where N is the number of layers and 7 0 is fixed time interval parameter. Also, X denotes the hidden states at time lT. Interestingly, we consider T = 7. we can get X (AX + X Usually, since each attention layer is followed by a layer normalization (LN), the scale transformation can be ignored. Therefore, this discrete diffusion equation essentially identical to the attention layer update X AX' + X'. Although we here assume Post-LN 47]. the same way, attention with Pre-LN can be understood as diffusion-reaction equation, which is explained in Appendix [ In this sense, the dynamics of attention layers can be understood as graph diffusion on complete graph. Furthermore, because diffusion decreases the system's 's energy
(as shown in Appendix C this interpretation aligns with the discussion in 44 29

---

As result, the conventional transformer architecture can be viewed as executing a diffusion process on complete graph in which each node represents token embedding. Moreover, the attention matrix A defines anisotropic diffusivity interactions among tokens. This perspective suggests that the over-smoothing observed in transformers results from the dissipative smoothing effect of a diffusion equation, analogous to heat flow. An intuitive explanation of this phenomenon is provided in Appendix E By viewing diffusion as a process that smooths small-scale fluctuation erasing high-frequency components, it becomes clear why the attention layer acts as a low-pass filter 43] It also agrees with the interpretation of GNNs as performing low-pass filtering, as noted in 28
In the field of GNNs, there are some attempts to define networks based on the wave equation for mitigation of their sootshnn issues, focusing on its property of energy conservation [4 35]. Considering the finding that transformer architecture can be regarded as the graph neural diffusion on a complete graph, introducing a wavy dynamics into the transformer also expected to enhance its performance. Therefore, this paper introduces novel type of transformer block based on the wavy dynamics and validates its effectiveness and generality several experiments.
3.2 Wavy Dynamics Based Attention
We introduce novel transformer block based on wavy rather than diffusiie dynamics, motivated by the implicit reliance of conventional transformers on diffusive dynamics and some discussion for GNNs 14, 35 Figure I shows a schematic of Wavy Transformer block, which includes a wavy attention layer, celocity-specife layer normalization, and velocity-oriented feed-forwant network. We begin by considering the wave equation on complete graph, analogous to the graph neural diffusion [5]
x ōt2
(A
X)
(9)
By introducing the artificial velocity variable Y equation as first-order systems:
ax
we can rewrite the second-order wave 8t
OY Ot = IXX,,
X
Y.
(10)
Öt
Here, Y can be considered as a difference of hidden states across adjacent layers. The following update rule can be obtained by time-iisertiization of the system (IO)
Y = (A-
x = 1+1 +X'
(11)
for = 1,..., N, where N is the number of layers and r > 0 a fixed time interval parameter. Also, X and Y' denote the hidden states at time Ir. Also, importantly this discrete dynamical system is symplectic, meaning it conserves its system energy 48). Since this property might contribute to avoid the over-smoothing behavior of the hidden states X, we consider the wave transformer block based on I0) instead of Eq.
To highlight the difference from the basic diffusive attention layer, if we directly discretize Eq. (n..99e in
time, rather than bypassing through 10). we obtain X 72 AX' + (1 7²) 7²)X²² + (X
x Ignoring the difference in the constant coefficient (79), compared to the purely diffusive update rule, this wavy update rule includes change in the hidden states similar to velocity. In other words, this can be understood as incorporating momentum into the update process. By preserving past state changes, this mechanism prevents excessive smoothing of node features across layers, allowing richer information to propagate and mitigating smoothng..
additional term,
(x' X'-1) which represents
Also, combinations of first-order diffusion and second-order wave dynamics can be employed. One option is to blend the outputs of the conventional (diffusive) attention layer and the proposed wavy diffuse: where X and x(+ dilluse are given by Eqs. 11 and 8 respectively. The mixing vector À E [0, 1 is defined as X sigmoid(O), with O E R"
attention layer: X = X X i
wave + (1 - X) (+1
a trainable parameter vector. Alternatively, the velocity update in Eq. IT] can be replaced by yl+1 = X [T((A - r)x + Y +1 - (A - r)x' Although Y is somewhat physically unnatural due to mixing dimension-inconseee
two terms, we can regard it pragmatically as the layer-to-layer state change by combining diffusive and wavy components controlled by X.
5

---

3.3 Physically Consistent Layer Normalization and Feed-forward Network
Beside attention module, each transformer block is equipped with a layer normalization and feed- forward network. We consider extension of each layer for preserving the physical state-velocity relationship Y = 0x at
3.3.1 Layer Normalization for Velocity
Layer normalization is formally defined for each d-dimensional feature vectors x ( 21.2 oR.al included in (-th layer's hidden state X' as
LN(x)), = 2 ((x))
1,,,2....... 1,2,..... .d), n + B3 (i= .., =
(12)
Vo' +
where v(x²) = ش o² - ( - And 7/ and 3, are learnable
parameters, and is a small constant for numerical stability. Considering the state-velocity relationship at we can define the corresponding layer romanization for d-dimensional velocity vector y: [bi v1.21 .... slall included in Y as follows:
Y-
0x
This LN. Vop(x++ 75 1,.......... j = .......... .d..
(13)
Based on the relationship, only the scaling parameters (e.g., o² (x({) and 7/)) are applied, while the shit parameters (e.g.. xxx() and 3,) are ignored. In addition, velocity vectors are normalized using mean and variance of states X as well.
3.3.2 Feed-forward Network for Velocity
A typical feed-forward network used in transformer architectures is two-layer fully connected network. Basically, we can define the FFN output as follows:
FFN(X') ( X ( Wi++bb) W2+1b;
(14)
where Wi E RRdxd and W2 E R are the weight matrices, b E Rd and b E Rd are the bias vectors, and 1 E RD is vector of ones (used here to broadcast the biases across all rows). Also, 6(-) is nonlinear activation function (often ReLU or GELU). Applying this operation transforms each row of X' (the hidden state of each token) through two-layer MLP with nonlinear activation, resulting in an output matrix of the same size (n d).
In the same way the layer normalization, for velocity according to the chain rule as follows:
straightforwardly define a feed-forward network
FY") = o' (x'Wi W (iW+1bb] Y' W1 W2.
(15)
where ø represents the derivative function of the activation function Φ Importantly, the velocity is scaled not only by the weight parameters (e.g., W, and W₂) but also by the derivative function o'.
4 Related Works
4.1 Mitigating Over-Smoothing:
Existing Approaches and This Study
To alleviate the over-smoothing problem, some techniques have been introduced in the fields of GNNs [35] [4 45) 34) and transformer- models [43 29 28). From the viewpoint of dynamical systems, these approaches can be summarized as some attempts inject high-frequency disturbance into hidden states dynamics to prevent wver-smoothing. They can be broadly classified into two main categories: (i) direct injection of frequency signals as an external disturbance to prevent conver- gence toward uniform hidden states (e.g. [3 29)), and (ii) introduction of intrinsic modifications of the governing dynamical system that propagate nigh-frequency modes within the hidden states (e.g.
35 [4] Almost all existing attempts to alleviate the over-smoothing problem in transformer-based models belong to category (i). Therefore, this paper offers the first comprehensive discussion of an intrinsic mechanism prevent ever-smoothing in seafformer-based networks.

---

4.2 Physics-Inspires GNN and Transformers
Our study is complementary prior physics-inspired GNNs and transformers. Graph-CON 351 and
PDE-GCN [14] introduce oscillatory or PDE-motivated updates on sparse graphs; in contrast, we target complete-graph attention in transformers. Relatedly, Gravina et al. [8] analyze over-squashing in aparse-graph GNNs, whereas we focus on over-smoothing in all-to-all attention. Deng et al. [9] introduce Hamiltonian structure into transformers through a loss, while we directly replace the residual dynamics with second-order wave update without auxiliary losses (see also HNNS [1]).
4.3 Baselines for Comparative Study
In our comparative study, we include BERT, DeiT, and DIFFormer as representative baselines for NLP, CV, and sparse-graph applications, respectively.
BERT: Bidirectional Encoder Representations from Transformers BERT, introduced by Devlin et al. [ is deep bidirectional transformer encoder trained with masked language modeling and next sentence prediction objectives. By conditioning on both left and right context in every layer, BERT learns rich contextual embeddings that serve as a standard benchmark for wide range of NLP tasks. Thus, we chose BERT as the canonical NLP model for comparison.
DeiT: Data-efficient Image Transformers DeiT, proposed by Touvron et al. [39), adapts the trans- former architecture vision tasks without massive pre-training, DeiT exemplifies a state-of-the-ar CV transformer, which we use as our baseline model for CV task. Also. we review an ex- isting over-smoothing mitigation technique 43 : FeatScale, which re-weights features to boost high-frequency signals. We then show Wavy Transformer can seamlessly integrate with it to enhance their performance.
DIFFormer: Diffusion-based Graph Transformers As a representative graph transformer, we adopt DIFFormer [4] our diffusion baseline on sparse-graph benchmarks.
Experiments
In this section, we present a comprehensive experimental evaluation of Wavy Transformer block for NLP tasks under BERT-like pretraining framework [11], measuring (i) pretraining perplexity
(PPI.) and ML. accuracy, (ii) fine-tuning performance GLUE tasks [42] and (iii) over-smoothing behavior. Moreover, to demonstrate the generality and effectiveness of our approach with other types of transformer-basel backbones DeiT [39], we validate Wavy Transformer on representative computer vision tasks, specifically ImageNet classification [10). Also, we evaluated the performance of Wavy Transformer on servers with eight V100 GPUs, four L40 GPUs, or four H100 GPUs, selected according to task size.
benchmark based on DIFFormer 44 Experiments used
Model Variants We evaluate two variants the Wavy Transformer. Both share the same attention backbone; they differ only in how the velocity residual is realized. Full Wave. A second-order residual realized with an explicit velocity branch that includes a FFN and a LN on the velocity path as shown in Figures [ yielding stronger physical constraints at additional computational costs. Light Wave. A momentum-only realization that keeps the second-order effect while removing FFN/LN for velocity. The update is X 7 'AX + (1 x' + X(X' X where X [0, 1]d controls the mix between the diffusion part and the wave (momentum) part. Ignoring the coefficient mismatch between the diffusion term (T) and the momentum term (r²), setting A 0 recovers diffusion-only.
whereas X = 1 corresponds to the wave-only residual. Formal derivations appear in Appendix.
5.1
Experiments: NLP Tasks
Here, we adopt smaller-scale pretraining setup than standard BERT configurations facilitate faster experimentation. The implementation details are given in Appendix A.I.I
5.1.1 PPL and MLL Accuracy
First, we present the pretraining results to illustrate the fundamental capacity of each model with its respective residual connection language model. Table summarizes the validation PPL and
7

---

Table Validation PPL and MLM accuracy (MLM Acc) after 10k steps. Arrow symbols denote that lower PPL and higher MLM Acc are better.
Residual
PPL ())
MLM Acc (f)
Diffusion
31.76
44.39 %
Full Wave
31.99
44.52 %
Mix (+Full)
29.00
45,56 %
Mix (+Liight)
32.29
44.53 %
Table 2: GLUE results for various models.
The highest score for each task is highlighted in bold.
Residual
CoI.A
SST-2
MRPC
QQP
MNI.I-m/-mm
QNI.
RTE
STS-B WNL.I
Avg.
Diffusion
10.67
83.64
76.36
83.96
73.51/74.03
81.63
52.95
51.11
52.11
64.13
Full Wave
10.64
83.10
76.11
84.17
72.10/72.94
80.74
53.67
56,34
32.91
62.27
Mix (+Full)
12.07
84.25
76.32
83.81
73.04/73.53
81.19
55.35
55.40
29.40
62.44
Mix (+I (+Liight)
12.77
82.76
76.18
83.89
73.71/73.77
81.98
55.96
55.40
64.76
66.12
MLM accuracy after 10k steps of batch size 64 for each model. Interestingly, while the wavy residual connection alone just achieves nearly the same-level performance as the diffusive residual connection, the mixed residual connections outperform both. This suggests that the wavy residual connection capture fundamentally different or complementary dynamics compared to the usual diffusive residual connection, and that gating both types of updates together with trainable À enables the model to leverage a broader range of interactions than a purely diffusive attention update.
5.1.2 GLUE Fine-tuning Performance
The GLUE 42 benchmark provides comprehensive suite of natural language understanding tasks that serve as critical tests for evaluating the performance of pretrained language models. In experiment,
fine-tuned each pretrained BERT-like model each GLUE task dataset for demonstrating advantages of Wavy Transformer over baseline methods. As summarized in Table 2 the mixed residual with the momentum-one average over the 10 metries (+1.99 over Diffusion). Gains are concentrated on CoLA, QNLI, RTE, WNLI, and especially STS-B (+12.65 Diffusion), while remaining competitive on MNLI-m; minor regressions appear on SST-2, MRPC, QQP, and MNLI-mm. The Mix with Full Wave variant reaches the top SST-2 but underperforms overall due to large STS-B drop with the corrected evaluator. Together with [able these results strongly suggest (i) that Wavy Transformer can capture different features unattainable by conventional diffusive connections, and (ii) that combining these different types of features can enhance BERT's performance on NLP tasks.
variant (Mix with Light Wave) attains the best macro
5.1.3 Analysis of Over-smoothing
We further conduct the analysis of the over-smoothing behavior for each residual connections. Especially, we computed the cosine similarity as measure to investigate the over-smoothing behavior
following 37 CosSim n(n-ij
Eiv; T[x: Ix,, I For computation the cosine similarity, we use
data randomly sampled from WIkipedia and BookCorpus as input to the BERT-based pretraining models fine-tuned on the SQuAD dataset [33] Figures 2 compare layerwise cosine similarity:
Figure (0aure22ccon contrasts the difference between diffusive and wavy, and Figure 3 shows the mixed residual with Full Wave (with la shading). As seen in Figure 2 diffusive and wavy residuals exhibit similar over-smoothing at the 24th layer (means and variances are comparable), but their trajectories markedly differ: diffusion increases monotonically with depth, whereas wave oscillates and drops near the final layer, consistent with wave-equation updates. This suggests that diffusive and wavy residual connections extract fundamentally different types of features from the data, which may be related to the frequency of hidden states across layers. Figure 3 shows the mixed residual; together with Tables [ and 2 this supports that wavy features complement diffusive ones, and that gating them broadens the feature range and improves performance.

---

1.0
1.0
wave diffusion
mix
Cosine Similarity
0.0-
Cosine Sinilarity
0.0
0.2
0.2
0.0
0
2
10
12
14 Index
16
18
20
22
24
0.0
10
12
14
16
18
20
22
24
Layer Index
Figure 2: Comparison of cosine similarity across
Figure 3: Cosine similarity across layers for a mix (+ Full Wave) with I-sigma interval shading.
layers for diffuse and wave residual connections with I-sigma interval shading.
5.2 Experiments: CV Tasks
In this section, we demonstrate the capacity of the proposed approach for CV tasks. All of our experiments are conducted on the ImageNet dataset [I0] including around Ik classes 1.3M images in the training dataset and 50k images in the validation set. Also, we choose 12-layer DeiT [39 as our backbone. The implementation details is given in Appendix A.2.I
5.2.1 Results of ImageNet Classification
S experimental results. The two rows labeled "Diffusion" in the residual column are baseline scores taken from [29). The other rows with "+ Wave" are our proposed extensions. These results demonstrate the effectiveness and generality of Wavy Transformer: with almost ignorable minimal additional parameters, it also consistently improves baseline models CV tasks.
Table 3: Experimental evaluation of Wavy Transformer block plugged into DeiT [39. The number
inside the (f -) represents the performance gain compared with the model without Wavy Transformer block. The two rows labeled "Diffusion" are baseline results taken from (2).
Method
Residual
Input Size
#Layer
#Param
Top-1 Acc (%)
DeiT-Ti
Diffusion
224
12
5.7M
72.17
DeiT-Ti
Diffusion Full Wave
224
12
5.7M
72.33 0.
DeiT-Ti
Diffusion + Light Wave
224
12
5.7M
73.09 0.92)
DeiT-Ti + FeatScale
Diffusion
224
12
5.7M
72.35
DeiT-Ti + FeatScale
Diffusion Full Wave
224
5.7M
72.62 (1 0.26)
Quantifying Over-Smoothing. Beyond cosine similarity, we quantify the spectral gap of the attention matrix and node-feature variance. Wave dynamics show a smaller gap and larger variance on CV tasks (Table definitions and additional results on trends of cosine similarity are in Appendix A.2.2
Table 4: Over-smoothing diagnostics on CV tasks. Values are calculated over validation images. Spectral gap is 1 (2( for the attention matrix. Lower gap ({) and higher node (t) indicate mitigated over-smoothing and Inter-class variance is reported only as an auxiliary separability measure.
Dynamics
Spectral gap (1 122))
Node-feature var.
Inter-class var.
Diffusion
0.836 0.00345
2.480 + 0.0778
0.195
+ Full Wave
0.629 = 0.00887
2.609 + 0.0902
0.211
+ Light Wave
0.730 = 0.00842
2.109 + 0.0696
0.308

---

5.2.2 Scaling to Deeper Models
We additionally evaluate CaiT-XXS- on ImageNet- IK under the same training recipe the baseline 40). The wave residual improves Top-1 accuracy by 1.0 point. The 0.9M parameter reduction arises solely from omitting extra class-attention block; all other settings remain unchanged (Table
Table 5: ImageNet- IK (CaiT-XXS-24)) Numbers in (1 indicate Top-1 gain over the diffusion-only baseline. The training recipe and input resolution identical the baseline (40.
Method
Residual
Input Size
#Layer
#Param
Top-1 Acc (%)
CaiT-XXX-24
Diffusion
224
24
12.0M
77.6
Wavy Transformer
Diffuse
Full Wave
224
11.1M
78.6 (1 1.0)
5.3
Experiments: Sparse-Graph Tasks
We further evaluate OGBN-Arxiv and OGBN-Proteins 23 using DIFFormer
under the authors' training protocol and hyperparameters. Diffusion denotes vanilla DIFFormer; + Light Wave denotes addition of our momentum-only wave residual. Table highlight representative depths. Complete results are provided in Appendix A.3.2 + Light Wave substantially mitigates depth collapse on OGBN-Arxiv and delivers a consistent ROC-AU gain on OGBN-Proteins.
45
Table 6: OGB highlights (mean + standard deviation over 3 seeds). Arxiv reports accuracy and Proteins reports ROC-AUC; all values are percentages. denotes absolute improvement.
Dataset
Metric #Layer
Diffusion
Light Wave
A
OGBN-Arxiv
Ace ROC-AUC
7
24.44 + 4.51
66.73 + 0.33
+42.29
OGBN-Proteins
5
69.42 + 2.31
80.14 + 0.67
+10.72
5.4 Computational Cost
For cost-sensitive settings, we adopt the momentum-one only variant (Light Wave), which preserves the second-order residual with negligible overhead. On both BERT and DeiT-Tiny, throughput and memory stay within few percent of the diffusion baseline (Table 7); complete cost tables, including the Full Wave variant, are provided in Appendix A..4
Table 7: Computational efficiency of each residual connections on 4 Nvidia V100.
Model
Variant
Inference
Training
Peak GPU Mem
BERT
Diffusion 101.6
415.6
18.31
BERT
Light Wave
101.3
436.2
18.69
DeiT-Tiny
Diffusion 2631.1
618.6
8.25
DeiT-Tiny
Light Wave
2644.2
617.6
9.14
6 Conclusions
In this paper, we investigated the dynamics embedded in attention layers from the viewpoint of graph neural diffusion and introduced Wavy Transformer. We first established an equivalence between the intrinsic dynamics driven by attention layers and graph neural diffusion this perspective, the ovor-smngt behavior can be understood as the dissipative nature of diffusion. Based on this insight, we proposed Wavy Transformer, which can be seamlessly integrated into standard diffusive transformer architecture. Our extensive experiments on NLP, CV, and sparse graph tasks showed that can capture feature dynamies that qualitatively different from those driven by conventional transformers, and that integrating it into existing models enhances performance without increasing the number of training parameters or requiring additional hyperparameter tuning. We believe this work represents first step toward physics-inspired design of transformer architecture.
complete graph. From
10

---

Acknowledgement
part of numerical simulations were conducted using Earth Simulator JAMSTEC.
References
[1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In IEEE/CVF International Conference Computer Vision, pages 6836-6846,
2021.
[2] Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Lio, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. Advances in Neural Information Processing Systems, pages 18527-18541, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berer, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1900 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229, 2020.
[5] Ben Chamberlain, James Rowbottom, Maria Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. International Conference on Machine Learning, pages 1407-1418, 2021.
[6] Jeongwhan Choi, Scoyoung Hong, Noscong Park, and Sung-Bac Cho. Gread: Graph neural reaction- diffusion networks. In International Conference on Machine Learning, pages 5722-5747, 2023.
[7] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Annual Meeting of the Association for Computational Linguistics, 2019.
[8] Tingting Dan, Jiaqi Ding, Ziquan Wei, Shahar Z. Kovalsky, Minjeong Kim, Won Hwa Kim, and Guorong Wu, Re-think and re-design graph neural networks in spaces of continuous graph diffusion functionals. In seveeth Conference Neural Information Processing Systems, pages 59375-59387, 2023.
[9] Congyue Deng, Brandon Y. Feng, Cecilia Garraffo, Alan Garbarz, Robin Walters, William T. Freeman, Leonidas Guibas, and Kaiming He. Denoising hamiltonian network for physical reasoning, 2025.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database, 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[11] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 1418144162200 2019.
[12] Yihe Dong. Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
[14] Moshe Eliasof, Eldad Haber, and Eran Treister. PDE-GCN:
Novel architectures for graph neural networks motivated by partial differential equations. In Advances in Neural Information Processing Systems, pages 1336-3849,, 2021.
[15] Lawrence C. Evans. Partial differential equations, volume 19, American Mathematical Society, 2022.

---

[16] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2023.
[17] Francesco D. Giovanni, James Rowbottom, Benjamin Chamberlain, Thomas Markovich, and Michael M. Bronstein. Understanding convolution on graphs via energies. Transactions on Machine Learning Research, 2023.
[18] Alessio Gravina, Moshe Eliasof, Claudio Gallicchio, Davide Bacciu, and Carola-Bibiane Schönlich. On Oversquashing in Graph Neural Networks Through The Lens of Dynamical Systems. In The 39th Annual AAAI Conference on Artificial Intelligence, volume 39, pages 16906-16914, 2025.
[19] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, 2019.
[20] Andi Han, Dai Shi, Lequan Lin, and Junbin Gao. From continuous dynamics to graph neural networks: Neural diffusion and beyond. Transactions on Machine Learning Research, 2024.
[21] Matthias Hein, Jean- Yves Audibert, and Ulrike von Luxburg. Graph laplacians and their convergence on random neighborhood graphs. Journal of Machine Learning Research, 2007.
[22] Benjamin Hoover, Yuchen Liang. Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed J. Zaki, and Dmitry Krotov. Energy transformer. In International Conference on Neural Information Processing Systems, pages 27532-27559) 2023.
[23] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005. 00687, 2020.
[24] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In Advances in Neural Information Processing Systems, pages 14745-14758, 2021.
[25] Andrej Karpathy, nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs. Sttpa://githus httt::///gttuu..coo/aaatttttttt nanooPP
[26] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[28] Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing transformers via regular- ized nonlocal functionals. Advances Neural Information Processing Systems, pages 10233-80256. 2023.
[29] Tam M. Nguyen, Tan M. Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. In Conference on Neural Information Processing Systems, pages 80233 80256, 2023.
[30] Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan Minh Nguyen. From coupled oscillators to graph neural networks: Reducing ovor-smog via Kuramoto model-based approach. In International Conference on Artificial Intelligence and Statistics, pages 2710-2718, 2024.
[31] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations, 2020.
[32] Yehuda Pinchover and Jacoh Rubinstein. An introduction to partial differential equations, volume 10. Cambridge university press, 2005.
[33] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension text. In Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.
[34] T. Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra, A survey on oversmoothing in graph neural networks. arXiv preprint 10XV 2023.
12

---

[35] T. Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning, pages 18888-18909, 2022.
[36] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 352-364, 2020.
[37] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang. Zhenguo Li, Lingpeng Kong, Stephen M. S. Lee, and James Kwok. Revisiting wvrr-smothhin in BERT from the perspective of graph. In International Conference on Learning Representations, 2022.
[38] Yunchong Song, Chenghu Zhou, Xinbing Wang, and Zhouhan Lin. Ordered GNN: Ordering message pass- ing to deal with heterophily and over-smoothingg In International Conference on Learning Representations, 2023.
[39] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficien image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357, 2021.
[40] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 42, October 2021.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems, 2017.
[42] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task henchmark and analysis platform for natural language understanding. International Conference on Learning Representations, 2019.
[43] Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti antiioversmootiing in deep vision transformers via the fourier domain analysis: From theory to practice. In International Conference on Learning Representations, 2022.
[44] Qitian Wu, Chenxiao Yang. Wentao Zhao, Yixuan Hc, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference Learning Representations, 2023.
[45] Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-hased graph neural networks. In Conference on Neural Information Processing Systems, pages 35084-35106, 2023.
[46] Yonghui Wa, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint /
2016.
[47] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng. Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. International Conference on Machine Learning, pages 10524-10533. 2020.
[48] Haruo Yoshida. Construction of higher order symplectic integrators. Physics letters A, 150K5-7):262-268, 1990.
[49] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint orXiv:2103. /186, 2021.
[50] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards -iory-like visual explanations by watching movies and reading books, In IEEE International Conference on Computer Vision, pages 19-27, 2015.
13

---

A Experimental Details and Additional Results
A.1 NLP Tasks
A.1.1 Implementation Details
Following [II] we construct a English corpus from Wikipedia [1] and BooksCorpus [50], splitting into 95 % training and 5 % validation data. We tokenize using WordPiece-like tokenizer (30k vocabulary). The WordPiece embedding 46] and the dictionary containing 30,000 tokens [ are used in our paper. To pre-proces: text, we use the special token [CLS] as the first token of each sequence and [SEP] to separate sentences in a sequence. Each sequence is up to 128 tokens, with masked language modeling (MLM) applied 15 of tokens and next sentence prediction binary classification of segment continuity.
We implement three types of models with different residual connections such diffuse, wave, and mix in a 24-layer BERT with hidden size 256 and 4 attention heads based on Post-LN Wavy Transformer blocks. As for feed-forward layer, we the intermediate size 1024. For both wave and mixed residual connections, we set the time interval to = 0.5 unless stated otherwise. In this experiment, the mixing coefficient was defined as À sigmoid with being a single scalar parameter initialized to O; consequently, the same À is shared across all feature dimensions. For the mixing strategy with Full Wave, we adopt the velocity-based variant: Y
+Y (1 (A - I)x', As a optimizer, we use Adam W (27] with learning rate 5.0 10 warm-up for first 10 % of steps, then linear decay to step 10k. Also, we evaluate every 5k steps the validation set, reporting PPL and MLM accuracy.
The
of various downstream tasks are shown in Table g
Table 8: Hyper-parameters for different downstream tasks.
GLUE
SQuAD
Batch size
32
16
Weight decay
[0.1, 0.01]
[0.1, 0.01]
Warmup proportion
0.1
0.1
Learning rate decay
linear
linear
Training epochs
3
3
Learning rate
0.00005
0.00005
A.1.2 MLM Convergence
As shown in Table 9 adding wave terms yields mild speedup Mix (+Full) leads early (e.g.. +2.6 40k and +1.7 60k vs. Diffusion) Full Wave largely tracks Diffusion thereafter. In short, wave tends to accelerate convergence slightly, but the residual choice does not substantially alter the overall convergence behavior.
Table MLM accuracy vs. steps.
Model
0
20k
40k
60k
80k
100k
Diffusion
0.0
17.38
32.99
40.88
43.51
44.39
Full Wave
0.0
18.40
33.48
40.80
43.67
44.52
Mix (+Full)
0.0
19.40
35.56
42.60
44.77
45.56
A.2 CV Tasks
A.2.1
Implementation Details
To demonstrate that our proposed approach are beneficial to CV tasks, we choose 12-layer DeiT 39 as our backbone. Also, for evaluating its consistency with the other remedy to avoid over-smoothing. combine approach with existing technique such FeatScale (43). When use 12-layer
14

---

1.0
1.0
Cosine Sinilarity
Cosine Similarity
0.6
0.6
0.4
0.-
Wave DelT
Wave
Wave+ FeatScale
0.0
0.0
10
12
Layer index
Layer index
(a) Comparison between cosine similarities with sigma interval shading the conventional and the DeiT integrated with Wavy Transformer based on Full Wave variant.
(b) Comparison between cosine similarities with sigma interval shading of the DeiT integrated with Wavy Transformer based on Full Wave variant with and without FeatScale.
Figure 4: Comparisons of cosine similarities with I-sigma interval shading for several DeiTT based models.
DeïT, follow the same training recipe, hyper-parametent and data augmentation with 39 43
We
implemented DeiT based on Pre-LN Wavy Transformer blocks considering that original DeiT is based on Pre-LN Transformer block. We set r = 0.5, and we select X = A AX++1 wave + (1 A) X
difluse as
the mixing strategy with Full Wave and initialize the gating parameter À as all-zero vector whose length equals the dimension of the hidden states in the mix variant.
A.2.2 Analysis of Over-smoothing for DeiT-based models
Figure 4 plots similarity curves for the standard DeiT, DeiT integrated with Wavy Transformer based on Full Wave variant, and Wavy Transformer FeatScale variant with 1-sigma interval shading. The results clearly show how Wavy Transformer reshapes feature dynamics relative to the conventional DeiT. Integrating FeatScale slightly reduces over-smoothing overall, although final-layer cosine similarity remains marginally higher than that of Wavy Transformer alone.
A.2.3 Computation of spectral gap and variance metrics
We complement Figure by detailing how compute the three diagnostics used in the main text (Table Results for spectral gap, node-feature variance, and inter-class variance are reported in the main paper; the definitions here are for reproducibility.
Spectral Gap
For each image, average multi-hes head attention over heads obtain row-/right- stochastic matrix A E Raxa of the final layer. Let (Ai) be the eigenvalues of A ordered by magnitude. The per-sample spectral gap is
= 1- 1-
computed by taking the second-largest eigenvalue magnitude A (via an eigen decomposition the head-averaged A), then averaged over the validation set. Smaller 5 indicates weaker mixing and thus less over-smoothing.
Node-Feature Variance Let X E Rad be the token (patch) embeddings from a designated layer evaluation time. We compute the variance across tokens after the final LN for each channel and then average across channels:
NodeVar
Var (Xis).
We report the mean of this scalar validation images.
15

---

Inter-Class Variance Let K. be the set of classes. Tokens inherit their image labels; for each &s K, define the class centroid μ4 as the mean token embedding. We then compute the variance across class centroids and average over channels:
d
InterClass Var = Σ (aTxex((xX.))
3=1
This measures feature separability (often correlating with accuracy) and is not used as an over- smoothing diagnostic.
A.2.4 Ablation of FFN for Velocity
Table I0 shows the result of ablation of FFN for velocity. In the Full Wave setting, the FFN for velocity yields a modest but consistent gain (+0.53 Top-1), indicating its effectiveness. However, considering the added computational cost, the momentum-only Light Wave the more practical choice for cost-sensitive use (see Appendix A.4.
Table 10: Ablation: removing Velocity-FFr -FNN (Dei 150 epochs).
Configuration
Top-1 (%)
A
Full Wave (diff-wave+(TN++NN)
68.92
Without FFN for velocity
68.39
-0.53
A.3 Sparse-Graph Tasks
A.3.1
Implimentation Details
We use the momentum-one Light Wave residual for all sparse-graph experiments; the mixing coefficient À is set scalar parameter initialized to 0 at the start of training and optimized jointly with the network. All other settings follow the DIFFormer protocol unless otherwise noted. Also, adopt the DIFFormer [44] definition of diffusion mixing т. This is corresponding the time interval 72 in Wavy Transformer. Thus, throughout we control both with a single knob and report T; Appendix for more details.
A.3.2 OGB: full results and r-robustness
We report complete results across depths and the coefficient т. larger t, diffusive stacks exhibit collapse at depth, whereas the wave residual mitigates it (Table IT)
Table 11: OGB results (3 seeds). Arxiv: accuracy (%); Proteins: ROC-AUC (%).
Dataset
Depth
Diffusion
Light Wave
Arxiv
3
68.77 + 0.28
66.68 0.66
-2.09
Arxiv
5
61.72 + 1.11
67.06 = 0.03
+5.34
Arxiv
7
24.44 + 4.51
66.73 = 0.33
+42.29
Proteins Proteins
3
79.26 + 0.40
79.58 = 0.27
+0.32
5
69.42 + 2.31
80.14 0.67
+10.72
A.3.3 Citation Graphs Benchmark
we report full results under mixing regimes, 7=0.2 (moderate) and 7-0.5 (strong neighbor mixing), using the same protocol and hyperparameters: numbers are mean and standard deviation across 5 seeds. Across datasets, Light Wave matches Diffusion at shallow depth and markedly stabilizes deeper stacks: at 7=0.2 it reverses collapse on Cora/PubMed at 20 layers, layers; at --0.5 the diffusive baseline collapses broadly,
In Tables 12 and 1
with single significant drop at PubMed while Light Wave mitigates collapse on Cora 12 layers and PubMed 12/20 layers (both variants collapse on Cora layers and Citeseer 10 layers). Significance symbols follow the captions (i: no significant change; W: significant drop; f: both models collapse).
16

---

Table 12: Citation graphs at r 0.2 seeds). f: paired t-test p > 0.05; V: only significant drop.
Dataset
Depth
Diffusion
Liggh Wave
A
Cora
80.02 0.29
78.54 = 1.14
-1.48f
Cora
16
83.28 = 0.76
84.22 = 0.94
+0.94
Cora
20
39.92 = 5.14
85.18 = 0.48
+45.26
Citeseer
72.04 = 0.25
71.74 0.27
-0.301
Citeseer
62.86 = 17.73
69.32 + 13.26
+6.46
Citeseer
10
34.76 1.44
34,92 = 2.49
+0.16
PubMed
75.74 0.73
74.76 = 1.15
-0.98t
PubMed
16
80.46 0.84
78.86 = 0.62
-1.60
PubMed
20
58.68
2.73
79.22 + 0.93
+20.54
Table 13: Citation graphs at 7 0.5. f: both models collapse.
Dataset
Depth
Diffusive
+ Light Wave
A
Cora
12
29.40 + 0.00
60.38 + 24.7
+31.0
Cora
20
29.40 + 0.00
29.00 + 0.00
10.0
Citeseer
10
22.94
0.31
22.84 + 2.17
-0.10F
PubMed
49.16
9.70
69.74 15.7
+20.6
PubMed
20
29.70 + 0.00
40.10 + 0.84
+10.4
A.3.4 Cosine similarity across depth
To quantify over-smoothing for aparse-graph benchmarks, we track the pairwise cosine similarity between node embeddings after each layer in 20-layer stack on Cora with 0.2. We report means over 5 seeds. Higher values (approaching 1.0) indicate stronger representation collapse within layer. As shown in Table 14, the diffusion-only residual rapidly saturates to ~ x1.00, whereas the wave residual maintains substantially lower similarity across depth, preserving representation diversity.
Table 14: Layer-wise pairwise cosine similarity across node features on Cora (20 layers; 5 seeds). Higher means more over-smoothing
Model
Layer
Layer
Layer 8
Layer 12
Layer 16
Layer 20
Diffusion 0.99
1.00
1.00
1.00
1.00
1.00
Wave 0.31
0.058
0.087
0.13
0.23
0.38
A.4 Full Results of Computational Cost
In Tables [6] we report full runtime and memory measurements for three residual variants under the same training scripts and hyperparameters as the accuracy experiments. Metrics include inference throughput (k tokens/s for BERT; img/s for DeiT-Tiny), training iteration time (ms/iter), and peak GPU memory (GB), all on 4x V100. Only the residual dynamics are changed, isolating their compute impact.
A.4.1 Srouughput-Accurace Trade-off Comparison for Full Wave Branch
Table 17 compares the inference throughput (images/s) of various transformer variants, measured on the ImageNet validation set using V100 GPUs and batch size of 256. Introducing wave dynamics (Diffuse+ Wave or Wave alone) reduces throughput by roughly 50% from about 2 600 to 200- 300 images per second which may pose a limitation in real-time applications. However, this slowdown must be balanced against the gains in representational power and accuracy that wave-enhanced models provide. A simple mitigation to insert wavy blocks in only a subset of layers to recover part of the lost throughput. Concretely, replacing only the last six blocks with wavy blocks (Wave (6))
17

---

Table 15: BERT: runtime and memory on 4 V100.
Variant
Inference (k tokens/s)
Training (ms/iter)
Peak GPU (GB)
Diffusion
101.6
415.6
18.31
+ Full Wave
63.3
1031
28.71
Light Wave
101.3
436.2
18.69
Table 16: DeiT-Tiny: runtime and memory on V100.
Variant
Inference (img/s)
Training (ms/iter)
Peak GPU (GB)
Diffusion
2631.1
618.6
8.25
+ Full Wave
1312.2
1023.4
20.31
Light Wave
2644.2
617.6
9.14
increases throughput by about 33 % relative to the full Diffuse+Waave with FeatScale variant (1 649 img/s vs. 241 img/s) while sacrificing 0. 18 pt in Top-1 accuracy (72.44 % vs. 72.62 %). In this experiment, the velocity tensor initialized to zero at the first wavy block.
Table 17: Inference throughput and accuracy on ImageNet. Wave (6) indicates that wavy blocks are inserted only in the final six layers.
Model
Residual Connections
Throughput
Top-1 1 (%)
DeiT-Ti [39
Diffusion
2632.1
72.17
DeiT-Ti + FeatScale 43
Diffusion
2483.2
72.35
DeiT-Ti
Diffusion+ Full Wave
1266.0
72.33
DeiT-Ti + FeatScale
Diffusion+ Full Wave
1241.4
72.62
DeiT-Ti
Full Wave
1312.2
DeiT-Ti + FeatScale
Diffuse+ Full Wave (6)
1649.3
72.44
A.5
Wavy Transformer under causal attention
For supplemental information, we the Light Wave residual under strictly causal attention using nanoGPT [25). We use the
dataset and initialize from an OpenAl GPT-2 checkpoint, keeping the standard GPT causal mask and training loop unchanged. During fine-tuning, we use batch size of with gradient accumulation over 32 steps; training runs for 50 iterations with learning rate of 10-5 and learning-rate decay enabled. Under this setup, the best validation loss, averaged over three seeds, was 3.0178 for the Light Wave mix (scalar À initialized to 0) versus 3.0196 for the diffusion baseline-as an absolute gap 0.0018 (=s 0.06%); while the Light Wave mix slightly better, we do not observe meaningful difference in validation loss. A more comprehensive assessment under causal masking requires substantially longer training; we therefore view this as worthwhile future work, related to potential over- werrssmothhng in causal-attention Transformers. From a diffusive-dyee -iffumiveedynamics perspective, the attention matrix acts as a causal masking it becomes strictly unidirectional along the sequence, causal attention can be viewed as diffusion on a directed complete graph with unidirectional edges.
matrix; under
B Implementation Details of Light Wave
As shown in Section 3.2 if we directly discretize Eq. 99) in time, rather than bypassing it through
Eq. (10). we obtain
=++1=72Ax (1- (x' X
(16)
This equation can be seen as adding the momentum term (X' - X x into the usual diffusive attention without considering the coefficient ? Also, we consider blend the outputs of the diffusive attention layer and the proposed wavy attention layer:
x
A X wave + (1 A) x
diffuse
18

---

where the mixing vector À E [0, 10 is defined as À = sigmoid( 0), with E R a trainable parameter vector.
Decoupling Across Difflusion and Wave. Let Ta and Tw be the (possibly different) time intervals used in the diffusive and wavy paths, respectively. Substituting
X x+1 difluse
(1 - Ta) X
+ Ta AX',
(17)
X (1 (1-17) - x T AX + (x' (X--
(18)
into the blend gives the elementwise form
X = (1- + PAX X(X'-X'')).
;= AT (1 A)ra- (19)
Equation (19 shows that it is not necessary to strictly match between the two paths: only the effective diffusion mix P and the momentum weight À govern the update. Equivalently, by reparame- terizing with single knob = p E [0, 1]. the update reduces the Light Wave form used in our experiments.
C Energy Evolution in Diffusion and Wave Systems
We provide the fundamental theorems [15 energy dissipation in the diffusion equation and energy conservation in the wave equation.
36 that characterize behavior of Eqs. ( and (4):
Theorem Consider solution the diffusion equation. Then, the following holds:
dt ʃ (a(s.t)))
d
=-
/ [v(ss,t) S0.
which implies that the norm of the solution monotonically non-increasing in time.
Theorem Define the energy functional for = x(8,() which is solution to the wave equation
F(() = / /
(2n(s:1) + [vx(s,t) ds.
0t
Then, under suitable smoothness and boundary conditions, we have:
d
E(t) =0
dt
so E(t) is constant in time.
D Appendices of Difference Between Pre-LN and Post-LN
In this section, we discuss the analytical and implemental difference between Pre-LN and Post-LN. This section provides concise review of the differences between Pre-LN and Post-LN, presents physical interpretation of Pre-LN attention diffusion-reaction process, and details of the computational flow Pre-LN and Post-LN Wavy Transformers.
D.1 Pre-LN vs. Post-LN
Figure contrasts the computational flows of the Post-LN and Pre-LN Wavy Transformers. The critical difference is whether LN applied inside outside the residual connection. For reference. the core computational flows of standard Post-LN and Pre-LN transformer blocks; the table was compiled with reference 47. In Post-LN Transformer (the original Vaswani et al. design (4I). each attention or FFN is applied first, and then LN is performed on the sum of the sub-layer output and the residual shortcut. This keeps the summed output "clean", but it is known to slow or destabilize training [7] In Pre-LN Transformer, each block first normalizes its input with LN and then applies the sub-layer (self-attention or (orwafdr before adding the residual connection, in contrast to the original "Post-LN" design that normalizes after the addition. Placing L.N at the block's entrance ensures every sub-layer receives standardized inputs. In the case of Pre-LN, it is reported that the gradient are well-behaved at initialization 47
19

---

(x+11 Y
(xl+1 r'*!)
Layer Normalization
Add
A
Feed Forward
Add
Feed Forward
Layer Normalization
Layer Normalization
Add
Multi-Head Attention
Add
Multi-Head Attention
Layer Normalization
(X', Y
(X Y
(a)
(b)
Figure (a) Post-LN Wavy Transformer layer; (b) Pre-LN Wavy Transformer layer.
Table 18: Post-LN Transformer vs. Pre-LN Transformer
Post-LN Transformer
Pre-LN Transformer
X = Atta(X))
X LayerNorm(X") (X!)
X x + X
X Attn(X!.)
X LLyeeNorrmX(2))
X X + X
X FF
X Norm
X L + X
X FFN((
(+1 (x
X X X
X Final LayerNorm (X (x (X(+)
D.2 Pre-LN vs. Post-LN Implementations in Wavy Transformer
For considering Wavy Transformer with Full Wave setting, the difference between Pre-LN and Post-LN [47 is not trivial. Therefore, this section examines implementation differences between Pre-LN and Post-LN in the context of Wavy Transformer. Building on the Pre-LN and Post-LN bascline implementations shown Table [8 we construct Post-LN and Pre-LN versions of Wavy Transformer, whose details are summarized in Table 19 Conceptually, Wavy Transformer replaces the conventional residual update based on diffusive dynamics with one governed by wavy dynamics.
When the wavy residual update is combined with the conventional (diffusive) attention, the choice between Pre-LN and Post-LN becomes especially significant: mismatch in normalization placement can disrupt the balance between the wavy and diffusive paths and degrade accuracy. Consequently, LN configuration of wavy branch must be aligned with that of the diffusive branch to achieve the intended performance improvements. In our NLP experiments, we adopted Post-LN Wavy Transformer. Conversely, because DeiT (39) is built on a Pre-LN backbone, we used Pre-LN Wavy Transformer for all DeiT-based CV experiments.
20

---

Table 19: Post-LN Wavy-Transformer vs. Pre-LN Wavy-Transformer
Post-LN Tran-Transf
Pre-LN Wavy-Transformer
X = (X.)
X Y LayerNorm (x', Y')
Y =(x-1 x') + Y
X Attn(X!.))
X -YY.+
Y1. =r(X(2 x!) Y
X Y1.2 LayerNorm X Y!1)
X =YYi2 + +X
X Y = ( 4.2 Yi2)
X Y1. -LayerNorm XL.3 Y1.2)
X X + Y Y Y
X Y =FFN(X! Y!.)
x y LayerNorm X Y1.5)
X X + X x Y yd. + Y1.2
X Final, Yinnl + LayerNorm (xl+1 y
D.3 Attention with Pre-LN graph neural diffusion with reaction term
In Section it is shown that the attention with Post-LN can be straightforwardly considered as graph neural diffusion. In addition to that, show that attention with Pre-LN also can be regarded as graph neural diffusion. First all, attention with Pre-LN can be formalized as follow:
X Attn(LN(XX) + +
(20)
We compactly write LN transformation in matrix-forn as
LN(X) = X = E XS + M,
(21)
where = diag 0i E RAXR diag 711
Raxd Also, M == - ~ + 1, G
Raxdd where μl ( pa)) o 0n) T. . y== (?......,7a) TT)T. and B (B,, Ba)
Using this representation, can obtain
XW, (XWR)) R)
Att softmax
XWv
(22)
Vd AΣ XSW, MWv
(23)
(ĀΣ XS + M) Wy
(24)
where À softmax
(XW vi
and M AM. Therefore, attention with Pre-LN described
in Eq. 20 can be written
x A + M) Wy + X
(25)
On the other hand, consider the graph neural equation with an external source term and a reaction term follows
ax = XS
(26)
Øt For simplicity, we temporarily ignore the feature transformation by Wv. Straightforwardly consider- ing discretization of this equation with 7 1.0 recover the update rule of the attention with Pre-LN written as (2)). The continuous counterpart can be considered as the following the diffusion equation with external source term and reaction term:
@x(s, t) Öt
div( Vz(ss))(ss
b(u,t) ((,,t))
(27)
diflusion
source
reaction
In some cases, Pre-LN Transformers have been shown to outperform their Post-LN counterparts 47]. Furthermore, in GNNs, incorporating diffusion iiffusion-rracction reaction dynamics has led to performance gains (6) Together with the interpretation of Pre-LN attention as a diffusion-reaation process, these findings suggest that the superiority of Pre-LN arises from the implicit dynamics it introduces, which help mitigate over-smoothing and enhance model performance. In this sense, physical-dynamics perspective offers promising basis for clarifying transformer behavior.
21

---

E Intuitive Explanation of Over-smoothing
The graph diffusion equation
5 can be rewritten as
0X
Öt = AX -X-XXXX
(28)
Here, we define the attention-weighted weighted average of the hidden states as X = AX, where is right- stochastic matrix. This relation clearly explain why transformers suffer from over-smoothing: if some a hidden state X exceeds its attention- attention-weighted average X. it is driven downward, whereas if it falls below X, it is pulled upward. Consequently, as the hidden states propagate through many layers, an effect analogous to integrating the diffusion process over a long time, every component X converges toward the common value characterized by X. The farther token is from that local mean, the stronger the restoring force, so sigh-nariance patterns disappear first. What remains is the slowly changing, globally coherent structure, which is exactly the smoothing effect in the neural graph diffusion. We can summarize as follows. Discretization of Eq. (28) gives:
X X
X = r (X! x/). O<т<
(29)
By simple case analysis:
< 0, X X
X - -
0, X X
(30)
0,
X X
Hence each step strictly reduces the deviation from the mean:
x [+] X = (1-r) -1- X < | - X
(31)
F Energy Evaluation in Diffusion and Wave Dynamics on Graphs
In this section, to indicate intuitive discussion of diffusion and wave dynamics graphs from an energetic viewpoint, we assume here time-independent attention matrix A (e.g. Aij = A x using the initial features) and impose the symmetry condition A.j Ajs. These simplifying assumptions do not hold exactly in real transformers; they are introduced here solely for analytical simplicity and an intuitive understanding of their hidden-state dynamics. Note that this section presents the discrete-graph analogue of Appendix C
and should be compared with it.
Graph Neural Diffusion as Gradient Flow
Consider the following potential energy:
(X))
Σx A) X
(32)
Here, the gradient of U() with respect to X is
(X)
Ise (I A)i
X, +
x Ink)
(33)
0x
i.j
(I-A)) X,,
(34)
where I is identity matrix and we used the symmetry condition Aij = Aji. Therefore, the graph neural diffusion can be written as
0x aU OX
(35)
0t
This shows that the hidden-state dynamics in transformers, which implicitly rely on graph neural
diffusion, can be interpreted as the gradient flow of the potential energy in Eq. (32). A similar discussion appears in 229 22], and more general mathematical discussion is given in [16].
Let the node features X evolve according the graph neural diffusion in Eq. 7 Using the symmetry of A and the row-stochastic condition Σ, Aij = 1, we can rewrite the potential as
1 Σ Ai X - x,1
U
(36)
4
1
22

---

where 1 - xx,1 (Xy - (X) X.). yields
Taking the time derivative along the gradient flow
dU(x) dt
OU ( 0X
0x
(x) 0X
=
-I = X| S 0.
(37)
Ot
where (.-)) denotes the Frobenius inner product, i.c., (Y, Z) tr( Y Z). If we define the attention-
weighted average X = AX (cf. Appendix E), then
dx) - X X dt
1 x.I
(38)
This demonstrates that the graph neural diffusion decreases the potential energy (66 monotonically, and that UXX can be interpreted the sum of squared differences between each feature and its attention-weightee average, implying convergence toward a uniform state. This provides a clear explanation for the over-smoothing observed in transformers from the energetic viewpoint.
Energy Conservation of Wave Dynamics on Graphs In contrast to the graph neural diffusion, wavy dynamics introduced into Wavy Transformer conserve their energy in the same way continu- ous dynamics on smooth manifold. To show that, consider the following energy for (X, Y) governed by Eq. I1
E(X, Y) = ク/ Σ 1| + Σx (I -1- A) X,
(39)
1,j
Taking the time derivative and using X; = Y, and Y = -(I - A) X, we obtain
dE(X)
dt = Σv. + Σ (I A)iz X,
(40)
1
= Σ Y Σ (I A) + A) =0
(41)
1.
Hence, the total energy E(X, Y) remains constant over time, i... throughout the entire forward pass, demonstrating that the wavy dynamics on the graph exactly conserve energy.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction faithfully represent the paper's contributions and scope, with every major claim clearly substantiated by the methods, experiments, and analyses presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper adequately discusses the limitations the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiattinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: The paper lists every assumption and supplies full and correct proofs for each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper gives necessary detail to reproduce the main experiments and verify its claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code were supplied reviewers anonymous zip archive, and full public links will be added in the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guudees/Coeettbim.ss
caaassionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuaaaeeeae
Sttelee//ooSSSSeissSioica www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper writes out every training and test detail including data splits, hyper- parameters, optimizer choice, and how they were chosen. So, the results are clrer.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports appropriate information about statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper even enumerates the exact GPU models, so the needed resources are fully clear.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////.oorppss.cc/p//tttttte
Answer: [Yes]
Justification:The research adheres the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: As the study is purely theoretical and currently has no direct social impacts, the authors did not include an explicit discussion of potential societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets correctly cited and their terms of use is properly respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
latasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the new assets are well documented and the documentation is provided alongside them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study involves no crowdsourcing or human-subject experiments, so participant instructions, screenshots, and compensation details are not applicable and were not provided.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study involves no crowdsourcing or human-subject experiments, so participant instructions, screenshots, and compensation details are not applicable and were not provided.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Large language models are not part of the paper's core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
30


Reviews:
[{"review_id": "V9bOHuy8aZ", "reviewer": "Reviewer_sFbK", "summary": "This paper addresses the over-smoothing problem in deep Transformer models by reinterpreting attention layers as a form of graph diffusion on a complete graph. Based on this view, the authors propose the Wavy Transformer, which replaces the diffusive dynamics with wave-based (second-order) dynamics that preserve energy and reduce smoothing. The architecture also introduces corresponding modifications to feed-forward and normalization layers to maintain consistency with the proposed dynamics. Experiments on both NLP and vision tasks demonstrate consistent improvements with minimal parameter overhead.", "questions": "1. How does the method scale to larger models?\n2. Could you comment on the runtime/memory cost introduced by the velocity tracking, compared to a standard Transformer block?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n1. Offers a solid theoretical interpretation of attention as graph diffusion, grounding the over-smoothing issue in physical dynamics.\n2. Introduces a novel wave-based update rule that mitigates over-smoothing by preserving energy and promoting oscillatory behavior.\n3. Demonstrates consistent gains across NLP and CV tasks.\n\n## Weaknesses\n1. Evaluation is limited to relatively small models (BERT hidden size 256, DeiT-Ti), with no experiments on large-scale settings.\n2. Computational cost (runtime, memory) is not discussed, even though parameter count remains unchanged, the added velocity tracking may incur overhead.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive assessment and for the clear questions on scale and computational cost. \n\nWe (i) **add deeper models in vision & graph**, (ii) **detail runtime/memory**, and (iii) answer your points.\n\n---\n\n# 1. Main concerns\n\n1. **Scalability** – behavior on deeper / larger models?  \n2. **Runtime & memory** – overhead of the extra velocity state?\n\n---\n\n# 2. Additional large-depth experiments\n\n## 2.1 ImageNet-1K  (CaIT-XXS-24)\n**Table 1 Comparison with CaIT-XXS-24**\n\n| Model                      | Top-1 (%) | Δ ↑      | Params(M) |\n| -------------------------- | --------- | -------- | --------- |\n| CaIT-XXS-24                | 77.6      | -        | 12.0      |\n| Wavy Transformer 24 Layers | 78.6      | **+1.0** | 11.1      |\n\nSame training recipe as CaIT; LayerScale 1e-5, no extra class-attention block.\n\n_Implementation note._ By omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) yet achieves a +1 pt Top-1 gain.\n\n## 2.2 Citation-graph benchmarks\n\nWe adopt **DIFFormer** (Wu et al. 2023) and _only_ add the wave update without adding Velocity-FFN/LN (τ = 0.2). All hyperparameters remain unchanged. We report mean ± SD over 5 seeds.\n\n**Table 2 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)**\n| Dataset  | Depth | DIFFormer&nbsp; | **DIFFormer+Wave&nbsp;** |        Δ ↑ |\n| -------- | :---: | :-------------: | :----------------------: | ---------: |\n| Cora     |   4   |  80.02 ± 0.29   |     **78.54 ± 1.14**     |     –1.48† |\n|          |  16   |  83.28 ± 0.76   |     **84.22 ± 0.94**     |      +0.94 |\n|          |  20   |  39.92 ± 5.14   |     **85.18 ± 0.48**     | **+45.26** |\n| Citeseer |   4   |  72.04 ± 0.25   |     **71.74 ± 0.27**     |    –0.30 † |\n|          |   8   |  62.86 ± 17.73  |    **69.32 ± 13.26**     |      +6.46 |\n|          |  10   |  34.76 ± 1.44   |     **34.92 ± 2.49**     |      +0.16 |\n| Pubmed   |   4   |  75.74 ± 0.73   |     **74.76 ± 1.15**     |     –0.98† |\n|          |  16   |  80.46 ± 0.84   |     **78.86 ± 0.62**     |     –1.60▼ |\n|          |  20   |  58.68 ± 2.73   |     **79.22 ± 0.93**     | **+20.54** |\n\n† Paired *t*-test (5 matched seeds): *p* > 0.05 → **no statistically significant degradation**.\n\n▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.\n\n\nFull results (τ = 0.2 / 0.5) will be included in the final version.\n\n**Table 3 Cosine similarity (Cora, 20 layers)**\n| Layer    | 1           | 4            | 8            | 12          | 16          | 20           |\n| -------- | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ |\n| Diff     | 0.99±9.6e-6 | 1.00±5.1e-7  | 1.00±1.8e-8  | 1.00±7.0e-9 | 1.00±1.0e-9 | 1.00±1.4e-10 |\n| **Wave** | 0.31±9.1e-4 | 0.058±5.3e-3 | 0.087±6.5e-3 | 0.13±3.8e-3 | 0.23±5.7e-4 | 0.38±1.7e-3  |\n\nAs depth increases, **Diffusion saturates to ≈ 1.00**, showing severe over-smoothing, whereas the **Wave-enhanced model stays below 0.38**, preserving representational diversity and enabling the large accuracy gains highlighted above.\n\n---\n\n# 3. Computational cost (4 × V100) — **Light vs. Full Wave**\n\n**Table 4 Comparison of runtime and memory costs of BERTs**\n|                              | Inference (k tokens/s) | Training (ms / iter) | Peak GPU memory (GB) |\n| ---------------------------- | ---------------------- | -------------------- | -------------------- |\n| Diffusive                    | 101.6                  | 415.6                | 18.31                |\n| Diff+Full Wave (Hamiltonian) | 63.3                   | 1031                 | 28.71                |\n| Diff **+Lighter Wave**       | **101.3**              | **436.2**            | **18.69**            |\n\n**Table 5 Comparison of runtime and memory costs of DeiT-Tiny**\n|                              | Inference (img/s) | Training (ms / iter) | Peak GPU memory (GB) |\n| ---------------------------- | ----------------- | -------------------- | -------------------- |\n| Diffusive                    | 2631.1            | 618.6               | 8.25                 |\n| Diff+Full Wave (Hamiltonian) | 1312.2            | 1023.4               | 20.31                |\n| Diff **+Lighter Wave**       | **2644.2**        | **617.6**           | **9.14**             |\n\n**Notes**\n\n- **Full Wave (original proposal).**  \n  Implements a discrete **Hamiltonian** form of the wave equation by introducing an explicit _velocity_ state plus Velocity-FFN and Velocity-LN. While it raises accuracy on some tasks, the extra state and auxiliary blocks slow both inference and training, as shown above.\n\n- **Lighter Wave (second-order, momentum-only).**  \n  Responding to the reviewer’s cost concern, we removed the auxiliary Velocity-FFN/LN and kept only the momentum term  \n  $$\n  x_{l+1}=x_l+\\operatorname{attn}(x_l)+\\beta\\bigl(x_l-x_{l-1}\\bigr),\n  $$\n  where $\\beta$ is the mixing parameter. This preserves the physical insight, yet matches baseline throughput (NLP -0.30 %, CV +0.5 %) and memory (NLP +2.1 %, CV +10.7 %, still < 1 GB).\n\n- **Effect on deep graphs.**  \n  Applying **Lighter Wave** to 20-layer citation-graph models still _vastly_ outperforms diffusion (+45 pt on Cora, +20 pt on PubMed; see Table 2).\n\n> Guided by the reviewer’s feedback, we now present **Lighter Wave** as the _practical option_: it preserves our physical motivation while keeping compute overhead within a few percent. We thank the reviewer for inspiring this simplification.\n\n---\n\n# 4. Responses to the reviewer’s questions\n\n|#| Question (paraphrased)                                                | Answer                                                                                                                                                                                                                                                                                                               |\n|--| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n|**Q1**| How does the method scale to larger models? | On a 24-layer CaIT-XXS and citation graphs, wave update boosts accuracy by **+1 pt** on ImageNet and up to **+45 pt** on graphs respectively without tuning. Because the update is local to each block, scaling hidden size or depth follows the same complexity curve as a standard Transformer. |\n|**Q2**| Runtime/memory overhead?                    | See Tables 4 and 5.                                                                                                                                                                                                                                                                                                                |\n\n---\n\n# 5 Planned manuscript updates\n\n- **Experiments** – add comparison with CaIT-XXS and the results of graph benchmarks including the trends of cosine similarity for each case of Cora/Citeseer/Pubmed.\n- **Cost analysis** – insert full table for computational cost with the lighter wave variant.\n- **Method** - add formalization of the lighter wave update.\n\n* All numerical details and training scripts will be released upon acceptance.\n\n---\n\n## Remark — Ongoing fine-tuning experiments (GPT-2 / GPT-2-Large)\n\nWe are currently fine-tuning **GPT-2 (124 M)** and may also fine-tune **GPT-2-Large (774 M)** on the OpenWebText corpus, comparing the diffusive baseline with our wave update. Experiments are ongoing; full results will be reported in the camera-ready.\n\n---\n\nWe hope these additions fully address your questions and demonstrate that the proposed wave dynamics are both scalable and efficient. \n\n_Thank you again for your encouraging review._"}, {"review_id": "r7S84GCjae", "reviewer": "Reviewer_o9CG", "summary": "This paper establishes a link between the dynamics of stacked attention layers and diffusion processes on a complete graph, revealing over-smoothing as a consequence of dissipative diffusion dynamics. Motivated by this insight, the authors propose the Wavy Transformer, which introduces a novel attention mechanism based on second-order wave-like dynamics. Unlike diffusion, wave dynamics preserve energy and exhibit oscillatory behavior, helping to potentially mitigate over-smoothing. The authors find that Wavy Transformer achieves consistent performance improvements on NLP and CV tasks, without significant increases in parameters or hyperparameter tuning.", "questions": "- Since the computational cost seems to be mainly discussed for inference of vision models, how does the inference computational performance of Wavy Transformer compare for NLP models?\n- How much computational overhead does the method introduce during training? Additionally, how does incorporating wave dynamics affect training convergence? Are there notable differences in convergence behavior compared to standard transformers?\n- Part of the motivation for this method was to mitigate over-smoothing in transformers. However, based on Figures 2 and 5, wave dynamics don’t appear to reduce over-smoothing (at least based on the token-wise cosine similarity metric), although some differences emerge in its layer-wise behavior. Could the authors clarify this, and if this is indeed the case, could they elaborate on other explanations or benefits of Wavy Transformer beyond over-smoothing?\n- In Section 5.1.4, the authors note oscillatory behavior in cosine similarity for wavy residual connections, but this behavior seems less clear in the DeiT experiments (Figures 5 and 6). Are there reasons for this, and any insights into which behavior might be expected or beneficial for the model? Also, does such behavior persist or vary across prompts and different datasets?\n- Have the authors considered applying Wavy Transformer to decoder-only architectures for language tasks as well?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Proposes a novel approach that modifies transformer dynamics using physical principles, offering a principled approach aimed at reducing over-smoothing.\n- Provides clear explanations and construction of the proposed architecture components.\n\nWeaknesses:\n\n- A potential drawback is the added computational overhead introduced by the method. In the case of vision tasks, the authors report that the inference throughput drops by about 50% when using wave dynamics, which could limit practical usage of the method.\n- See the Questions section for more discussion", "was_revised": true, "final_justification": "My main concerns have mostly been addressed. The authors now provide a more thorough cost analysis of their proposed method and introduce a lighter variant of the wave mechanism to mitigate the inefficiencies present in the original version. Furthermore, they leverage spectral gap and node-feature variance analyses to better illustrate the effects of their proposed wave transformer.", "rebuttal": "We thank the reviewer for the detailed and constructive feedback.  \nWe (i) **present a full cost analysis**, (ii) **link reduced over-smoothing to higher accuracy**, and (iii) answer your specific questions.\n\n---\n\n# 1. Main concerns\n1. **Computational cost** – — vision-inference drops ≈ 50 %; missing NLP/train numbers  \n2. **Over-smoothing** – cosine similarity still high in Figs. 2 & 5  \n3. **Clarifications** – convergence speed, DeiT oscillation, decoder-only use\n\n---\n\n# 2. Computational cost (4 × V100) — **Light vs. Full Wave**\n\n**Table 1 Comparison of runtime and memory costs of BERTs**\n|   | Inference (k tokens/s) | Training (ms / iter) | Peak GPU memory (GB) |\n| ---- | ----- | ------ | ------ |\n| Diffusive   | 101.6  | 415.6     | 18.31     |\n| Diff+Full Wave (Hamiltonian) | 63.3    | 1031                 | 28.71                |\n| Diff **+Lighter Wave**       | **101.3**              | **436.2**            | **18.69**            |\n\n**Table 2 Comparison of runtime and memory costs of DeiT-Tiny**\n|         | Inference (img/s) | Training (ms / iter) | Peak GPU memory (GB) |\n| ----- | --- | ----- | ------- |\n| Diffusive       | 2631.1   | 618.6     | 8.25         |\n| Diff+Full Wave (Hamiltonian) | 1312.2            | 1023.4               | 20.31                |\n| Diff **+Lighter Wave**       | **2644.2**        | **617.6**           | **9.14**             |\n\n_Notes:_\n\n- **Full Wave (original proposal).**  \n  Implements a discrete **Hamiltonian** form of the wave equation by introducing an explicit _velocity_ state plus Velocity-FFN and Velocity-LN. While it raises accuracy on some tasks, the extra state and auxiliary blocks slow both inference and training, as shown above.\n\n- **Lighter Wave (second-order, momentum-only).**  \n  Responding to the reviewer’s cost concern, we removed the auxiliary Velocity-FFN/LN and kept only the momentum term  \n  $$x_{l+1}=x_l+\\operatorname{attn}(x_l)+\\beta\\bigl(x_l-x_{l-1}\\bigr),$$\n  where $\\beta$ is the mixing parameter. This preserves the physical insight, yet matches baseline throughput (NLP -0.30 %, CV +0.5 %) and memory (NLP +2.1 %, CV +10.7 %, still < 1 GB).\n\n- In the citation-graph benchmarks (Table 4), we employ this lighter version of the Wavy Transformer, which outperforms its diffusive counterpart in deeper settings.\n\n> Guided by the Reviewer’s cost concern, we now present **Lighter Wave** as the _practical option_: it preserves the physical insight while keeping compute within a few percent of the baseline. We thank the reviewer for inspiring this simplification.\n\n---\n\n# 3. Convergence (BERT MLM on Wikipedia + BookCorpus)\n**Table 3 Comparison of convergence trends**\n| Step (×10³) | Diffusive (%) | Wave (%) | Mix (%) |\n| ----- | ----- | ---- | ------ |\n| 0  | 0.0           | 0.0      | 0.0    |\n| 20  | 17.38   | 18.40    | 19.40  |\n| 40  | 32.99   | 33.48    | 35.56  |\n| 60   | 40.88         | 40.80    | 42.60  |\n| 80   | 43.51         | 43.67    | 44.77  |\n| 100   | 44.39         | 44.52    | 45.56  |\n\n*Mix = diffusion + wave outputs; reaches 44 % **20 k steps earlier**.\n\n---\n\n# 4. Citation-graph benchmarks and cosine similarity trend\n\nWe adopt **DIFFormer** (Wu et al. 2023) and _only_ add our wave update (τ = 0.2). All hyper-parameters remain unchanged. We report mean ± SD over 5 seeds.\n\n**Table 4 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)**\n| Dataset  | Depth | DIFFormer | **DIFFormer+Wave** |        Δ ↑ |\n| -- | :---: | :-----: | :----: | ---------: |\n| Cora     |   4   |  80.02 ± 0.29   |     **78.54 ± 1.14**     |     –1.48† |\n|    |  16   |  83.28 ± 0.76   |     **84.22 ± 0.94**     |      +0.94 |\n|  |  20   |  39.92 ± 5.14   |     **85.18 ± 0.48**     | **+45.26** |\n| Citeseer |   4   |  72.04 ± 0.25   |     **71.74 ± 0.27**     |    –0.30 † |\n|   |   8   |  62.86 ± 17.73  |    **69.32 ± 13.26**     |      +6.46 |\n|    |  10   |  34.76 ± 1.44   |     **34.92 ± 2.49**     |      +0.16 |\n| Pubmed   |   4   |  75.74 ± 0.73   |     **74.76 ± 1.15**     |     –0.98† |\n|          |  16   |  80.46 ± 0.84   |     **78.86 ± 0.62**     |     –1.60▼ |\n|          |  20   |  58.68 ± 2.73   |     **79.22 ± 0.93**     | **+20.54** |\n\n† Welch *t*-test *p* > 0.05 → **no statistically significant degradation**.\n\n▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.\n\nFull results (τ = 0.2 / 0.5) will be included in the final version.\n\n**Layer-wise cosine similarity on Cora, 20 layers**  \n\n**Table 5 Cosine similarity (Cora, 20 layers)**\n| Layer     | 1           | 4            | 8            | 12          | 16          | 20           |\n| --------- | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ |\n| Diffusion | 0.99±9.6e-6 | 1.00±5.1e-7  | 1.00±1.8e-8  | 1.00±7.0e-9 | 1.00±1.0e-9 | 1.00±1.4e-10 |\n| **Wave**  | 0.31±9.1e-4 | 0.058±5.3e-3 | 0.087±6.5e-3 | 0.13±3.8e-3 | 0.23±5.7e-4 | 0.38±1.7e-3  |\n\n> **Diffusion saturates to ≈ 1.00**, showing severe over-smoothing, whereas the **Wave-enhanced model stays below 0.38**, preserving representational diversity and enabling the large accuracy gains highlighted above.\n\n# 5. Spectral gap and node-feature variance (DeiT-Tiny, ImageNet-1K val dataset: 50k images)\n**Table 6 Spectral gap & Node-feature variance (DeiT-Tiny, ImageNet val dataset)**\n| Dynamics (Model) | Spectral Gap (mean ± var) | Node-feature Variance (mean ± var) | Notes                                  |\n| -- | -- | -- | -- |\n| **Diffusion**    | 0.836 ± 0.00345    | 2.480 ± 0.0778      | e.g., features after final LayerNorm   |\n| **Wave** | 0.629 ± 0.00887   | 2.609 ± 0.0902        |  |\n| **Random** | 0.984 ± 0.000069       | 0.728 ± 0.0499     | randomly initialized (untrained) model |\n\nHere the spectral gap is defined as $1-\\lambda_{2}$, where $\\lambda_{2}$ the second-largest eigenvalue (in magnitude) of the right-stochastic attention matrix $A$. A smaller gap, together with higher node-feature variance, indicates reduced over-smoothing beyond what cosine similarity alone captures.\n\n---\n\n# 6. Image classification: **CaIT-XXS-24 on ImageNet-1K**\n**Table 7 Comparison with CaIT-XXS-24**\n| Model | Top-1 (%) | Δ ↑ | Params (M) |\n| -- | -- | -- | -- |\n| CaIT-XXS-24 | 77.6 | - | 12.0 |\n| Wavy Transformer 24 Layers | 78.6 | **+1.0** | 11.1      |\n\nSame training recipe as CaIT; LayerScale 1e-5, no extra class-attention block.\n\nImplementation note. By omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) yet achieves a +1 pt Top-1 accuracy gain.\n\n---\n\n# 7. Responses to the reviewer’s questions\n| #     | Question (paraphrased)  | Answer                                                                                                                                                                                                                                                                                            |\n| -- | -- | -- |\n| **Q1** | _Inference cost for NLP?_  | Table 1.                                                                                                                                                                                                                                                                                                                                                                                                            |\n| **Q2** | _Training overhead & convergence?_      | Tables 1, 2, and 3.                                                                                                                                                                                                                                                                                                                                                                                                  |\n| **Q3** | _Cosine similarity high in vision/NLP—why useful?_ | In CV/NLP, Wavy Transformer keeps cosine similarity high yet still improves accuracy because **spectral gap falls from 0.836 to 0.629 and node-feature variance rises** (Table 6), indicating richer and more discriminative features even when pairwise angles stay similar.                                                                                                                                                                                                               |\n| **Q4** | _Oscillation weaker in DeiT?_           | One reason might be Layer-Norm placement. As noted in Appendix B.3 of the manuscript, in Pre-LN designs such as DeiT, the normalization effectively introduces a reaction-like term, which may alter the dynamics. A complete quantitative comparison is non-trivial and remains future work.       |\n| **Q5** | _Decoder-only applicability?_           | Yes. We ran a **quick test** on a tiny GPT-like model (context 256 tokens, 6 layers × 6 heads, 384 dims). Adding wavy update improved **validation loss drops from 1.470 to 1.460** on Shakespeare dataset. Although shallow, this implies that wave dynamics also benefit causal attention. We are currently fine-tuning **GPT-2 (124 M)** and may also fine-tune **GPT-2-Large (774 M)** on the OpenWebText corpus, comparing the diffusive baseline with our wave update. Experiments are ongoing; full results will be reported in the camera-ready. |\n---\n\n# 8. Planned manuscript updates\n\n- **Cost and convergence analysis** – insert full table for computational cost with the lighter wave variant and convergence curves.\n- **Experiments** – incorporate graph benchmark results, layer-wise cosine similarity trends with the results of spectral gaps and node-feature variance, experimental results for the lighter wave variant and short discussion of decoder-only applicability.\n- **Method** - add formalization of the lighter wave variant.\n- All numerical details and training scripts will be released upon acceptance.\n---\n\nWe hope these additions address your concerns and kindly ask you to reconsider your rating.\n\n_Thank you again for your thorough evaluation._"}, {"review_id": "ErYZ0xn6rD", "reviewer": "Reviewer_pC7f", "summary": "The paper discusses the equivalence between hidden state dynamics and graph neural diffusion on a complete graph, and interprets over-smoothing as a consequence of the dissipative nature of diffusion dynamics.", "questions": "See weaknesses.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The research topic and its application to text is interesting.\nHowever, the biggest problem is that the paper presents methods and theoretical results/interpretations that have been extensively studied in Graph Neural Networks research (and does not cite them). In particular:\n   - The idea that over-smoothing is a consequence of the dissipative nature of diffusion dynamics has been previously established and discussed   [1,3]\n   - The introduction of wavy dynamics to mitigate over-smoothing is not new and has been implemented in GNNs  [2, 3]. Wave equation based transformer blocks have also been studied in other contexts [4]\n\n\nMoreover, the paper extensively discusses GNN and interprets attention layers as \"graph neural diffusion on a complete graph\". However, the experimental validation does not present any direct experimental results on traditional graph datasets.\n\nOverall, the paper presents an interesting idea, which however needs to be better positioned and contrasted with existing works. This would also imply adding baselines in the experimental sections. \n\n[1] Gravina et al., 2025. On Oversquashing in Graph Neural Networks Through the Lens of Dynamical Systems\n\n[2] Rusch, et al. 2024 Graph-Coupled Oscillator Network\n\n[3] Eliasof et al., 2021. PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations\n\n[4] Deng et al., 2025. Denoising Hamiltonian Network for Physical Reasoning", "was_revised": true, "final_justification": "While I maintain some reservations about the paper's current state, mainly related to the planned integration of the missing related work which has significant overlap with the current paper, the authors added experiments on larger and newer graph datasets. Although the inclusion of these experiments is a welcome addition, the performance reported is not yet competitive, as it remains significantly lower than standard baselines. Nevertheless, I have increased my score to acknowledge the authors' responsiveness.", "rebuttal": "We thank the reviewer for the constructive comments. \n\nWe (i) **clarify how our work differs from prior work**, (ii) **add graph benchmarks**, and (iii) provide a supplemental ablation isolating FFN for velocity.\n\n---\n\n# 1. Main concerns\n\n1. **Positioning vs. prior work**  \n2. **Lack of traditional graph experiments**  \n\n---\n\n# 2. Relation to prior work\n\n**Table 1 Relation to prior work**\n|                                               | **GNN family**                                         | **Transformer family**                                |\n| --------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------- |\n| **Diffusion (1‑st order)**                    | _Vanilla_ GNN (diffusion on a sparse graph)            | _Vanilla_ Transformer (diffusion on a complete graph) |\n| **Wave / Oscillatory (2‑nd order) or Hybrid** | Graph‑CON (Rusch et al. 2024) · PDE‑GCN (Eliasof et al. 2021) | **★ Wavy Transformer** (ours)                         |\n\n- **Already cited in the manuscript.** \n   - *Graph-CON* and *PDE-GCN* are already cited as refs [30] & [13], respectively. They investigate incorporating non-diffusive dynamics into **sparse-graph GNNs**, whereas our work targets **complete-graph attention**.  \n- **Scientific focus.**\n  - Gravina et al. (2025) study **over‑squashing** on sparse graphs; our focus is **over‑smoothing** on complete‑graph. However, since it is related to our work and should be cited, we will cite it in the camera-ready version.\n  - Deng et al. (2025) primarily enforce Hamiltonian constraints in Transformer-based networks for physical reasoning, and **does not analyze the attention-driven dynamics of hidden states**, that is the main focus of our work.\n- **Novelty.** \n   - To our knowledge, this is the **first work to introduce a wave update into a Transformer with complete-graph attention**. \n   - Interpretations of Transformer dynamics from a continuum-physics perspective (e.g., diffusion or wave dynamics) remain scarce. \n   - We further explore Velocity-FFN and Velocity-LN as optional components, completing a fully “wavy” block that extends the standard Transformer architecture.\n\n---\n\n# 3. Citation-graph benchmarks \n\nWe take **DIFFormer** (Wu et al. 2023), fix τ = 0.2, and add our wave update. All other hyperparameters and training scripts stay identical, isolating the effect of the wave term.\n\nWe evaluate on the three citation networks most commonly used in prior work, reporting mean ± SD over 5 seeds.\n\n**Table 2 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)**\n| Dataset  | Depth | DIFFormer | **DIFFormer+Wave** |        Δ ↑ |\n| -------- | :---: | :-------------: | :----------------------: | ---------: |\n| Cora     |   4   |  80.02 ± 0.29   |     **78.54 ± 1.14**     |    –1.48 † |\n|          |  16   |  83.28 ± 0.76   |     **84.22 ± 0.94**     |      +0.94 |\n|          |  20   |  39.92 ± 5.14   |     **85.18 ± 0.48**     | **+45.26** |\n| Citeseer |   4   |  72.04 ± 0.25   |     **71.74 ± 0.27**     |    –0.30 † |\n|          |   8   |  62.86 ± 17.73  |    **69.32 ± 13.26**     |      +6.46 |\n|          |  10   |  34.76 ± 1.44   |     **34.92 ± 2.49**     |      +0.16 |\n| Pubmed   |   4   |  75.74 ± 0.73   |     **74.76 ± 1.15**     |    –0.98 † |\n|          |  16   |  80.46 ± 0.84   |     **78.86 ± 0.62**     |     –1.60▼ |\n|          |  20   |  58.68 ± 2.73   |     **79.22 ± 0.93**     | **+20.54** |\n\n† Paired *t*-test (5 matched seeds): *p* > 0.05 → **no statistically significant degradation**. \n\n▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.\n\n## τ-robustness — cases where the diffusion baseline collapses\n**Table 3 Graph benchmarks (DIFFormer + Wave, τ = 0.5, 5 seeds)**\n| Dataset      | Depth | DIFFormer | **DIFFormer+Wave** |   Δ ↑ |\n| ------------ | :---: | :--------------------------: | :---------------: | ----: |\n| **Cora**     |  12   |        29.40 ± 0.00 †        | **60.38 ± 24.7**  | +31.0 |\n|              |  20   |        29.40 ± 0.00 †        | **29.00 ± 0.00**  | ±0.0‡ |\n| **Citeseer** |  10   |        22.94 ± 0.31 †        | **22.84 ± 2.17**  | -0.1‡ |\n| **Pubmed**   |  12   |        49.16 ± 9.70 †        | **69.74 ± 15.7**  | +20.6 |\n|              |  20   |        29.70 ± 0.00 †        | **40.10 ± 0.84**  | +10.4 |\n\n\n† **Collapse**: the diffusive baseline fails.\n\n‡ 20-layer Cora and 10-layer Citeseer collapse for both models.\n\n**Key points**\n\n- When τ is large (**0.5**), the diffusive baseline **collapses** (accuracy ≈ 29 pt) on deep stacks.\n- The **wave update mitigates this collapse**, recovering **+20–31 pt** (Pubmed/Cora 12-layer).\n- Higher variance (Cora 12-layer/Pubmed 12-layer) arises from seed-to-seed variance during collapse recovery.\n\n## Layer-wise cosine similarity on Cora, 20 layers\n\n**Table 4 Cosine similarity (Cora, 20 layers, τ = 0.2)**\n| Layer    | 1           | 4            | 8            | 12          | 16          | 20           |\n| -------- | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ |\n| Diff     | 0.99±9.6e-6 | 1.00±5.1e-7  | 1.00±1.8e-8  | 1.00±7.0e-9 | 1.00±1.0e-9 | 1.00±1.4e-10 |\n| **Wave** | 0.31±9.1e-4 | 0.058±5.3e-3 | 0.087±6.5e-3 | 0.13±3.8e-3 | 0.23±5.7e-4 | 0.38±1.7e-3  |\n\n> As depth increases, **Diffusion saturates to ≈ 1.00**, showing severe over-smoothing, whereas the **Wave-enhanced model stays below 0.38**, preserving representational diversity and enabling the large accuracy gains highlighted above.\n\n---\n\n# 4. Ablation on ImageNet-1K (DeiT-Tiny, 150 epochs)\n\n**Table 5 Velocity-FFN ablation**\n| Configuration                    | Top‑1 (%) | Δ vs. baseline |\n| -------------------------------- | --------- | -------------- |\n| Baseline (Wavy Transformer)      | 64.52     | —              |\n| Wavy Transformer without Vel-FFN | 64.35     | **-0.17**      |\n\n---\n\n# 5. Planned manuscript updates\n\n- **Related‑work**: Gravina et al. (2025) will be cited in Introduction. Also, added 2×2 matrix to clarify the difference from the previous studies including papers you mentioned.\n- **Experiment**: We will add the results of graph benchmarks including the trends of cosine similarity for each case of Cora/Citeseer/Pubmed. Also, we will added results of ablation study as an Appendix.\n- All numerical details and training scripts will be released upon acceptance.\n\n---\n\nWe hope these additions address your concerns and would appreciate it if you could kindly reconsider your rating.\n\n_Thank you again for your time and consideration._"}, {"review_id": "qPlrvwCHqD", "reviewer": "Reviewer_YBPV", "summary": "The paper proposes a modification to the transformer architecture based on the wave equation. The authors interpret the existing transformer layers as a discretization of the diffusive heat equation. Under this interpretation, they argue that the over-smoothing issue of the transformer -- where representations of different tokens become too similar at deep layers -- is because the tokens follow the dynamics of heat diffusion on a complete graph. They then modify the transformer layers to mimic the discretization of the wave equation, which does not follow diffusive dynamics. They conduct experiments to show that this modification (sometimes in combination with the original transformer layers) gives better performance on NLP and CV tasks.", "questions": "1. As mentioned above, given that the wavy transformer results in higher cosine similarity between representations, how do we explain the effects of the modification? If it does not reduce over-smoothing, then what problem is it solving?\n2. Is over-smoothing a phenomenon observed in models that use causal attention as well? If so, how do we understand it from the diffusive dynamics perspective?\n3. In essence, the paper posits that representations evolve layer-wise through an oscillatory process. Why would that be the case?\n4. You’re proposing the wave equation as an example for a non-diffusive process. Why the wave equation? There are other PDEs that would also not diffuse.\n5. From the last few paragraphs in Subsection 3.2, it’s not crystal clear what is your proposed modification. It seems like there are several options considered.\n6. It would be useful to include an ablation study showing the impact of Equation 15.\n7. “the wavy residual connection alone just achieves nearly the same-level performance as the diffusive residual connection” — Do the authors have a hypothesis for why this occurs?", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written with a clear logical flow that is easy to follow. The relevant background material is clearly presented. \n2. The motivation for the proposed modification is clearly explained. \n3. The proposed modification seeks to address a relevant issue observed by many other works.\n\nWeaknesses: \n1. The experimental results supporting the proposed modification are not particularly strong. The range of tasks selected is small, and where there are improvements, they seem marginal. Since the modification is quite fundamental, I expect to see a wider range of tasks and larger performance improvements.\n\n2. While the cosine similarity for the wavy transformer exhibits oscillations across layers (which is different from the monotone increase of the original transformer), the representations of the wavy transformer almost always have higher cosine similarity. This suggests that the proposed modification does not reduce over-smoothing. The authors state that the wavy transformer “extracts fundamentally different features”, but this is not a sufficient explanation of the effects of the proposed modification and why we should expect it to be better. This also seems to contradict the original motivation to alleviate over-smoothing.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts in addressing my concerns and incorporating numerous new experimental results. However, the extent of the revisions is considerable to the point that it warrants a careful rereading and reevaluation of the manuscript from scratch, after all changes have been incorporated. For this reason, I am not able to recommend acceptance at this stage and have decided to retain my original score.", "rebuttal": "We thank the reviewer for the insightful feedback. We (i) **extend the experiments**, (ii) **revisit over-smoothing**, and (iii) answer your questions.\n\n---\n\n# 1. Main concerns\n\n1. **Limited evaluation / marginal gains**\n2. **Cosine similarity appears higher** — unclear whether over-smoothing is mitigated.\n3. **Design questions** — wave-equation choice, causal attention, Equation 15 options, ablation.\n\n---\n\n# 2. Extended evaluation\n\n## 2.1 Citation-graph benchmarks\n\nWe adopt **DIFFormer** (Wu et al. 2023) and add our wave update (τ = 0.2). All hyperparameters remain unchanged. We report mean ± SD over 5 seeds.\n\n**Table 1 Graph benchmarks (DIFFormer + Wave, τ = 0.2, 5 seeds)**\n| Dataset  | Depth | DIFFormer&nbsp; | **DIFFormer+Wave&nbsp;** |        Δ ↑ |\n| -------- | :---: | :-------------: | :----------------------: | ---------: |\n| Cora     |   4   |  80.02 ± 0.29   |     **78.54 ± 1.14**     |    –1.48 † |\n|          |  16   |  83.28 ± 0.76   |     **84.22 ± 0.94**     |      +0.94 |\n|          |  20   |  39.92 ± 5.14   |     **85.18 ± 0.48**     | **+45.26** |\n| Citeseer |   4   |  72.04 ± 0.25   |     **71.74 ± 0.27**     |    –0.30 † |\n|          |   8   |  62.86 ± 17.73  |    **69.32 ± 13.26**     |      +6.46 |\n|          |  10   |  34.76 ± 1.44   |     **34.92 ± 2.49**     |      +0.16 |\n| Pubmed   |   4   |  75.74 ± 0.73   |     **74.76 ± 1.15**     |    –0.98 † |\n|          |  16   |  80.46 ± 0.84   |     **78.86 ± 0.62**     |     –1.60▼ |\n|          |  20   |  58.68 ± 2.73   |     **79.22 ± 0.93**     | **+20.54** |\n\n† Paired *t*-test (5 matched seeds): *p* > 0.05 → **no statistically significant degradation**.\n\n▼ Only significant drop (–1.6 pt); deeper 20-layer recovers +20 pt.\n\n## 2.2 ImageNet-1K  (CaIT-XXS-24)\n\n**Table 2 Comparison with CaIT-XXS-24**\n| Model                      | Top-1 (%) | Δ ↑      | Params (M) |\n| -------------------------- | --------- | -------- | --------- |\n| CaIT-XXS-24      | 77.6      | -        | 12.0      |\n| Wavy Transformer 24 Layers | 78.6      | **+1.0** | 11.1      |\n\nSame recipe as CaIT; LayerScale 1e-5, no extra class-attention block.\n\n_Implementation note._\nBy omitting the extra class-attention block, our model uses 0.9 M fewer parameters (≈ 7 %) and still gains +1 pt Top-1 accuracy.\n\n---\n\n# 3. Over-smoothing revisited\n\n**Table 3 Cosine similarity (Cora, 20 layers)**\n\n| Layer    | 1           | 4            | 8            | 12          | 16          | 20           |\n| -------- | ----------- | ------------ | ------------ | ----------- | ----------- | ------------ |\n| DIFFormer     | 0.99±9.6e-6 | 1.00±5.1e-7  | 1.00±1.8e-8  | 1.00±7.0e-9 | 1.00±1.0e-9 | 1.00±1.4e-10 |\n| **Wave** | 0.31±9.1e-4 | 0.058±5.3e-3 | 0.087±6.5e-3 | 0.13±3.8e-3 | 0.23±5.7e-4 | 0.38±1.7e-3  |\n\nAs depth increases, **diffusion saturates to ≈ 1.00**, showing severe over-smoothing, whereas the **wave-enhanced model stays below 0.38**, preserving representational diversity and enabling the large accuracy gains highlighted above.\n\n**Table 4 Spectral gap & Node-feature variance (DeiT-Tiny, ImageNet-1K val dataset: 50k images)**\n| Dynamics (Model) | Spectral Gap (mean ± var) | Node-feature Variance (mean ± var) | Notes                                  |\n| ---------------- | ------------------------------ | ------------------------------- | -------------------------------------- |\n| **Diffusion**    | 0.836 ± 0.00345                | 2.480 ± 0.0778                  | e.g., features after final LayerNorm   |\n| **Wave**         | 0.629 ± 0.00887                | 2.609 ± 0.0902                  |                                        |\n| **Random**       | 0.984 ± 0.000069               | 0.728 ± 0.0499                  | randomly initialized (untrained) model |\n\nHere the spectral gap is defined as $1-\\lambda_{2}$, where $\\lambda_{2}$ the second-largest eigenvalue (in magnitude) of the right-stochastic attention matrix $A$. A smaller gap, together with higher node-feature variance, indicates reduced over-smoothing beyond what cosine similarity alone captures.\n\n---\n\n# 4. Ablation on ImageNet-1K (DeiT-Tiny, 150 epochs)\n**Table 5 Velocity-FFN ablation**\n\n| Configuration                           | Top‑1 (%) | Δ vs. baseline |\n| --------------------------------------- | --------- | -------------- |\n| Full Wavy (diff+wave+FFN+LN)            | 68.92     | —              |\n| Without FFN for velocity (diff+wave+LN) | 68.39     | **-0.53**      |\n\n---\n# 5. Responses to the reviewer’s questions\n\n| #      | Question   | Answer |\n| ------ | --- | -- |\n| **Q1** | Cosine similarity is higher—how does the wave help?         | In CV/NLP, Wavy Transformer e keeps cosine similarity high yet still improves accuracy because **spectral gap falls from 0.836 to 0.629 and node-feature variance rises** (Table 4), indicating richer and more discriminative features persist and over-smoothing is effectively mitigated even though the angles between tokens remain almost unchanged.                                                             |\n| **Q2** | Does over-smoothing occur with causal attention? | Yes, over-smoothing weakens but persists. Information flows only one way, so hidden states retain finite gaps instead of collapsing to a single value, which can be understood as a **unidirectional diffusion**. A simple toy test of unidirectional diffusion confirms that some smoothing remains with finite gap; a full investigation is future work. Also, **quick test of tiny GPT-style decoder** (256-token context, 6 layers, 384 channels, 6 heads) shows adding wavy update improved **validation loss drops from 1.470 to 1.460** on Shakespeare dataset, implying that wave updates already help causal decoders at this scale.|\n| **Q3** | Why assume oscillatory layer dynamics?           | We adopt wave dynamics—a conservative, oscillatory mechanism—as a simple way to preserve feature diversity; this choice is not unique, and other conservative update schemes could serve the same purpose.                                                                                                                                                                                                                                                                                                                                                                                   |\n| **Q4** | Why choose the wave equation?                    | The (isotropic) wave equation is the simplest second-order non-diffusive hyperbolic PDE; when discretised, it introduces only one additional buffer (a velocity or previous-step state) and its symplectic updates come with a closed-form energy conservation bound.                                                                                                                                                                                                                                                                                                                                                                                                |\n| **Q5** | Several modification options?                    | Coupling the wave and diffusive updates is non-trivial—a point we will state explicitly in the camera-ready version to avoid confusion. We test two variants: (i) **output fusion**, which merges the diffusive and wave attention outputs, and (ii) **velocity injection**, which feeds the diffusive output into the velocity state. In every case the **wave update** is retained, so the central contribution—the wave-based update mechanism—remains intact.                                                                  |\n| **Q6** | Impact of Equation 15 (ablation)?                | See Table 5.                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n| **Q7** | Why does wave residual alone match diffusive?    | As shown in Section 3.2 (lines 173–180) of the manuscript, the wave update can be conceptually understood as a diffusive update augmented by a single momentum term. Because that **diffusive core is still embedded even in the pure wave update**, we conjecture that wave may reach performance comparable to purely diffusive schemes on some tasks. We will note this briefly in the camera-ready version.                                                                                                                                                                                                                                                                                                                            |\n---\n\n# 6. Planned manuscript updates\n\n- **Typos & clarity** — correct typo and tighten any ambiguous expression in last few paragraphs in section 3.2.\n- **Related-work** – explicitly add a note on causal attention.\n- **Experiments** – incorporate citation-graph benchmark results, layer-wise cosine-similarity trends with results of the spectral gaps and node-feature variances, and the ablation table.\n- All training scripts will be released upon acceptance.\n\n---\n\n## Remark — Ongoing fine-tuning experiments (GPT-2 / GPT-2-Large)\n\nWe are currently fine-tuning **GPT-2 (124 M)** and may also fine-tune **GPT-2-Large (774 M)** on the OpenWebText corpus, comparing the diffusive baseline with our wave update. Experiments are ongoing; full results will be reported in the camera-ready.\n\n---\n\nWe hope these additions fully address your questions and demonstrate the broader impact of Wavy Transformer. We would appreciate it if you could reconsider the overall score in light of this evidence.\n\n_Thank you again for your time and consideration._"}]

Author rebuttal:

Meta-review: It has been known that GNNs implement a form of dissipative (heat) diffusion dynamics that can result in spatial over-smoothing.  The use of (oscillatory) wave equations has also been explored to mitigate over-smoothing, but the authors extend the paradigm to all-to-all attention based transformer networks.  The modification results in better performance in NLP and vision tasks.

The paper is motivated and written with excellent clarity.
The proposed method is likely not the last word on reducing information loss in graph neighborhood compression, but may be a good tool in the graph representation toolbox.
Reviewers YBPV and pC7f have helped ground the work better wrt prior work.
All reviewers helped motivate the authors to put in a terrific amount of effort during rebuttal, leading to more experiments and a much better understanding of the results.

Even after allowing for the discount applied by some reviewers to the originality and impact, the paper would srike up interesting conversation if presented as a poster.
Decision: Accept (poster)