Transformer Block Coupling and its Correlation with Generalization in LLMs
Authors: Murdock Aubry
Haoming Meng
Anton Sugolov
Vardan Papyan
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
transformers
hidden representations
Primary area: interpretability and explainable AI
TL;DR: We discover coupling in transformer block Jacobians and show a relationship with generalization.

Abstract: Large Language Models (LLMs) have made significant strides in natural language processing, and a precise understanding of the internal mechanisms driving their success is essential. In this work, we analyze the trajectories of token embeddings as they pass through transformer blocks, linearizing the system along these trajectories through their Jacobian matrices. By examining the relationships between these block Jacobians, we uncover the phenomenon of **transformer block coupling** in a multitude of LLMs, characterized by the coupling of their top singular vectors across tokens and depth. Our findings reveal that coupling *positively correlates* with model performance, and that this relationship is stronger than with other hyperparameters such as parameter count, model depth, and embedding dimension. We further investigate how these properties emerge during training, observing a progressive development of coupling, increased linearity, and layer-wise exponential growth in token trajectories. Additionally, experiments with Vision Transformers (ViTs) corroborate the emergence of coupling and its relationship with generalization, reinforcing our findings in LLMs. Collectively, these insights offer a novel perspective on token interactions in transformers, opening new directions for studying their mechanisms as well as improving training and generalization.

Published as a conference paper at ICLR 2025
TRANSFORMER BLOCK COUPLING AND ITS CORRE- LATION WITH GENERALIZATION IN LLMs
Murdock Aubry" Haoming Meng'
Anton Sugolov*
Vardan Papyan
University of Toronto
ABSTRACT
Large Language Models (LLMs) have made significant strides in natural language processing, and a precise understanding of the internal mechanisms driving their success essential. this work, we analyze the trajectories of token embeddings as they pass through transformer blocks, linearizing the system along these trajec- tories through their Jacobian matrices. By examining the relationships between these block Jacobians, we uncover the phenomenon of transformer block cou- pling in a multitude of L.Ms, characterized by the coupling of their top singular vectors across tokens and depth. Our findings reveal that coupling poxitively cor- relates with model performance. and that this relationship is stronger than with other hyperparameters such as parameter count, model depth, and embedding di- mension. We further investigate how these properties emerge during training, ob- serving a progressive development of coupling, increased linearity, and layer-wise exponential growth in token trajectories. Additionally, experiments with Vision Transformers (ViT') corroborate the emergence of coupling and its relationship with generalization, reinforcing our findings in L.MMs. Collectively, these insights offer a novel perspective on token interactions in transformers, opening new di- rections for studying their mechanisms as well as improving training and general- ization.
INTRODUCTION
Transformers Vaswani et al. 2017 can be represented discrete, nonlinear, coupled dynamical systems, operating in high dimensions (Greff al., 2016 Papyan et al., 2017: Haber & Ruthotto
2017 Ee 2017 Ebski et al.) 2018 Chen et al. 2018 Bai et al. [019 Rothauge 20,1 Gai & Li & Papyan| 2023). Viewing the skip connections as enabling a discrete time step, we represent the hidden representations dynamically evolving through the layers of the network. The term nonlinear refers to the nonlinear transformations introduced by activation functions, and cou- pled refers the interdependent token trajectories that interact through the MLP and self-attention
Zhang 2021
layers.
In work, investigate whether there identifiable structural characteristics across 30+ pre- trained L.Ms,, analyze their emergence with training, and examine their relationship with general- ization performance. During inference, as token embeddings pass through the network, we analyze their layer-wise trajectories and lincarize the effect of transformer blocks on the token embeddings across model depth study the relationships between blocks. To this end, we compute the cobians of distinct connections between layers and tokens, perform singular value decompositions (SVDs), and compare the resulting singular vectors. This approach measures the degree of coupling between singular vectors to capture the operational coordination of blocks as they act tokens. This perspective raises several key questions:
Q1. What regularity properties do these trajectories individually exhibit, and how are the blocks related each other? More concretely, what is the relationship between the block Jacobians across different tokens and transformer layers?
Q2. How do the properties of hidden representations and their interactions emerge during training?
Q3. Are any of these properties linked to the generalization capabilities of LLMs?
"Equal contribution. Source code available at hanp://gthhh.ccm//aplooo/nnnnggg

---

Published as a conference paper at ICLR 2025
Sbx Ed
Sox 12b
0.15
Nomo Coupling
LLN Sanchrran
0.10
0.69
0.50
0.91
0.92
0.38
0.54
0.95
30
10
(a) Correlation with Generalization
(b) Emergence with Training
1
11
13
13
41
[4
14
0
22
[1
(c) Pythia 12B Checkpoint 1
(d) Pythia 12B Checkpoint 4000
(+) Pythia 12B Checkpoint 143000
Figure Transformer Block Coupling Measurements. (a) The plot illustrates the correlation between average coupling (taking K inj dmodel) and benchmark scores across LLMs, showing that
higher coupling corresponds to improved performance, with regression fit yielding an R² value of 0.8 with a significant p-value 9.99 x 10-10 10 (b) The mean normalized coupling (with K 10) is plotted as function of training checkpoints for Pythia 12B and 6.9B (Biderman et al., 2023), measured at steps 128, 256. 512, 1k; 2k, 128k, 143k. (c-e) Adjacency plots illustrate the mean coupling scores between pairs of layers. Each node represents a layer, and edge weight and opacity indicate the strength of depth-wise normalized coupling. Visualizations are provided for checkpoints 1, 4k, and 143k of Pythia 12B.
1.1 CONTRIBUTIONS
We investigate the motivating questions across an extensive collection of openly- LLMs, most having over 1B parameters, trained by over independent organizations, with varying training methods and data (Appendix A. Through our experiments, we identify consistent properties that characterize the transformer block coupling phenomenon.
1. Coupling.
The singular vectors of the Jacobians of the transformer blocks are coupled across depth (Figures 3 and tokens (Figures 4 [5|
16
18
19 20 in several open source LLMs
(Table (I)) Furthermore, coupling across Jacobians emerges with training (Figures [(b]
6(b) 16), and the coupling strength becomes more pronounced between adjacent
layers with training, indicating a layer-wise locality in the interactions (Figures ( (c-e).
2. Generalization. The strength of coupling is correlated with benchmark performance on the
HuggingFace Open LLM Leaderboard (Beeching et al. 2023 (Figures [[a) 8) Addi-
tionally, coupling is more strongly correlated with generalization than parameter budget, model depth, and token embedding dimension (Figure Experiments with ViTs support the relationship observed in LLMs (Figure 6(a)).
3. Regularity. Linearity in hidden trajectories emerges with training (Figures 10 (a)
23
24), aligning with behaviour previously observed in ResNets Li & Papyan (2023) Expo- nential growth occurs in contiguous token representations as function of depth (Figures
5(b)
22
23 26), starkly contrasting linear growth previously observed.
2

---

Published as a conference paper at ICLR 2025
We provide perspective on token embedding interactions by examining layers of transformers through their Jacobian matrices. Our results display the effects training on transformer blocks, and opens potential avenues for promoting generalization in similar architectures.
BACKGROUND ON LARGE LANGUAGE MODELS
LLMs may be described as a deep composition of functions that iteratively transform token embed- dings. In the input layer, 1 = 0, the text prompts are tokenized and combined with the positional encodings to create an initial high-dimensional embedding, denoted by x! E Rdoss for the 1th token. When these embeddings are stacked, they form matrix:
(a 22
( E Buxddo
(1)
The embeddings then pass through L transformer blocks:
x
Nock
Nick
Black XL_
(2)
= black (X denotes the embeddings after the 1th block, consisting of causal multi-headed attention (MHA), a feed-forward network (FFN), and normalization layers (LN) with residual con- nections:
W+1 (x') MHA(LN(X*))
(3)
g+ (x') = LN(X' 1++1
(4)
n (*'((x)) =
+ FFN(G't
(5)
block (x') = XX + (x'),
where the MHA, LN, FFN are implicitly indexed by layer. Among many models (Appendix [T) an additional rotary positional embedding (RoPE, [Su et al. (2023)) is applied in the MHA layer. In the final representation, typically additional layer normalization is applied:
Fhlock ( (XL-1) LN(X h² + (XL-1))
(7)
The output XL of the final block PL is passed into bias-free linear layer M R.dce Xdased, with dvocab denoting the size of the token vocabulary and dmodel is the dimension of the token embeddings. This layer M computes final-layer logits for each token embedding, 4 Mat' The prediction for the next token is then determined by selecting the maximal logit value: arg maXpeu Ly
METHODS
3.1 COUPLING SINGULAR VECTORS JACOBIANS
Jacobians. Coupling is investigated through analyzing the linearizations of transformer blocks which is given by their Jacobian matrices
J/
Oz (X b2
(8)
defined for each layer E (1,... L). and pair of tokens 41 l2 E (1.....n). Note that this is the Jacobian matrix for each transformer block without the contribution of the skip connection from the input of the block, similar to the quantity measured by Li & Papyan (2023) which strictly analyzes the case where t fz.
Due to the causal structure of the representations, J/ 0 whenever 4 > l2 Hence, we restrict attention to the where I 12.
Singular value decomposition (SVD). We compute the SVD of the Jacobians:
J = U.
3

---

Published as a conference paper at ICLR 2025
Constructing
Constructing A
Depth-wise Coupling
of (x) J(X)
J USV
f ((X-)))
141
OX
A A = UJ
nl (1)))))
dr
Token-wise Coupling
X X
1
X
I
XI+1
Self Coupling
J
(1
A441 4++1
Tokens
J a
(1+)))));
12
2 0
Context Coupling
30
1/2 (((--)))
1441 1234
4 O
J
(1((()))
dxs
Depth
Figure 2: Transformer Block Coupling. A visualization of the various types of transformer block coupling with brief instructions on computing both the Jacobians J and coupling matrices A (Section 3.1). The coupling measurement quantifies the alignment and agreement between the interactions of embeddings connections within the network. The colored subscripts in the sample matrices A indicate the specific connections being compared.
where U1, Vi E IR dlansd ddaass are the matrices of left and right singular vectors respectively, and Si Bedmott X denssel contains the singular values
Coupling Measurement. To measure coupling between two Jacobians
J/,,, U
and
JK = UY
we define the coupling matrix
Ale Muegu UV. Vu. = U Vr...
for 1' E (LI....,,, and f1, f1, t2, E {l..... n), where Ur K and Vr.k are the matrices of top K singular vectors. If the singular vectors of distinct Jacobians strongly aligned, then the coupling matrix A should be strongly diagonal and approximately equal to the matrix, St,, of the top K singular values of J To quantify the top K mis-coupling, we define
||- t2 S1,KIFF
mi (J Fkr
|| 1lp
where si,K is the vector of the top K singular values of J II-Ile denotes the Frobenius norm and Il-llp the vector p-norm. Typically, we take p = 1, which corresponds to normalizing by the sum of the top K singular values of Jr. We then define the top K coupling to be ск = 1 - m--
We also generalize this analysis the alignment between left and right singular vectors of different Jacobians by considering the matrix
D B1b4g = Vi Uy
Depth-wise Coupling. To analyze coupling across transformer blocks, fix I, and measure align- ment between J/ and J through the matrix Al" across layers 1, (l...., L)
Token-wise Coupling We also quantify the coupling across tokens in several ways:
Note that the superscripts f1, fa, indicating the tokens, are omitted for clarity in the expression for the SVD.
In the matrix A, we write the single subscript for clarity.
4

---

Published as a conference paper at ICLR 2025
Self-coupling. By fixing two layers 1,1' fl...., L). analyze the where the input and output tokens are the same. Explicity, compare The and Ji across t, {I...., n2), which represents the coupling across tokens for token's 's effect on its own trajectory.
Context Coupling. We consider the context tokens' impact trajectory by measuring coupling between J/ and J Iige across t2, 122 &1 (fixing the imput token to be the same) and also between J/ye and J lit, across 4,, 1 S (fixing the output token be the same).
3.2 LINEARITY OF TRAJECTORIES
Linearity in intermediate embeddings is quantified with the line-shape score (LSS), defined by Gai
& Zhang 2021)
LSS
(10)
where 27 2.. i.e., the imput embeddings passed the LLM, and o is defined recursively as
= 1
for I....,L.
(11)
Note that LSS 2 1, with LSS if and only if the intermediate representations x!,,- . xf form co-linear trajectory.
.......
3.3 JAYER-WISH EXPONENTIAL GROWTH
We measure the presence exponential spacing (expodistance) the hidden trajectories. Assuming exponential growth of the embedding norms they flow through the hidden layers, estimate ||إا||| = | e" [=>--11 for some fixed R over all layers 1.... L. We quantify the validity of this representation by measuring the coefficient of variation of a/, given
a R In
(12)
for each layer 7 and token i. Under exponential growth, it expected that of is independent of depth. We therefore denote the expodistance (ED) of the trajectory of the ith token of a given sequence by
Vana
ED;
(13)
(Avg,ot)2
This measurement motivated by the discussion in Section 6.1 the parametrization in Ap- pendix A.5 and empirical evidence from Figure 27h. It also serves as method to assess the validity
of the Innarization presented in Equation 14
3.4 VISION TRANSFORMER TRAINING
For further investigation of coupling in transformers, consider Vision Transformers (ViTs) fol- lowing DEiT TTouvron et al. 2021). We train ViTs for classification on CIFARIO (Krizhevsky) 2009) with varied stochastic depth rate for embedding size 192, depth 24, and 3 attention heads. Please see Appendix A.7 for further details.
EVALUATION
4.1 SUITE OF LARGE LANGUAGE MODELS
Our study evaluates comprehensive collection of LLMs (see Appendix A.1 that were indepen- dently trained by various individuals and organizations. These models, provided through Hugging- Face (Wolf et al. 2020), vary in terms of parameter budgets, number of layers, hidden dimensions,
5

---

Published as a conference paper at ICLR 2025
Layers
Layers
Layen
0.0
(a) Llama-3 8B Untrained
(b) Llama-3 8B Trained
Figure 3: Transformer Block Coupling across Depth. The figure shows Jacobian coupling across transformer blocks 9 to 16, using the prompt "What is the capital of France? The capital is" to trace the final token's 's trajectory. In trained models (bottom row), the diagonal pattern with minimal off-diagonal values indicates alignment of Jacobians, where top singular vectors of J diagonalize J Untrained models do not exhibit such coupling. Further details are in the Appendix A.8 (Figure
Best viewed in color.
and training tokens. Moreover, analyze the dynamics of each measurement throughout training by deploying the Pythia Scaling Suite (Biderman et al. 2023). summary of the models under consideration is presented in Table Appendix|A.. A.1 and further details in Appendix(A.d
4.2 PROMPT DATA
We evaluate these LLMs using prompts of varying length, ambiguity, and context, sourced from the test set of ARC (Clark et al. 2018), GSM8K (Cobbe et al. 2021). HellaSwag (Zellers et al., 2019 MMLU (Hendrycks et al. 2021). Truthful QA (Lin al. 2022). and Winogrande (Sakaguchi et al. 2019). These set the performance benchmarks on the HuggingFace Open LLM Leaderboard (Beeching et al. (023) and provide representative evaluation of performance on many language tasks.
RESULTS
5.1 COUPLING OF JACOBIANS ACROSS DEPTH
In trained LLMs, we observe coupling the top singular vectors of the Jacobians across depth (Figure bottom matrix subplots. This is consistently observed across various LLMs considered. On the other hand, in untrained models (Figure top row), there is no coupling of Jacobians across different depths. There is coupling along the diagonal, however, because each Jacobian trivially diagonalized by its singular vectors. This, in addition Figure [(b] suggests that coupling across depth emerges through training.
b row), evident in the low non-diagonal values with visible diagonal present in the
5.2 COUPLING OF JACOBIANS ACROSS TOKENS
We analyze the coupling of singular vectors of Jacobians across tokens. For input and output tokens that the same (J and J Figure 4b. observe strong coupling, indicating that token's interactions along its trajectory are coupled with others. For context tokens, coupling is examined by fixing the input token (Juita and JrL' Figure 1.5) the output token (Jf1f2 and f Figure

---

Published as a conference paper at ICLR 2025
16
While context coupling exists, its strength varies across token pairs. Untrained models show
such coupling with randomly fixed layers 1, in general.
Tokens
Tokens
Max
Tokens
(a) Llama-3 8B Untrained
(b) Llama-3 8B Trained
Figure 4: Transformer Block Coupling across Tokens (Self Coupling). The figure shows Jacobian coupling for the same input and output token across tokens, visualized using the absolute values of A (with fixed layers 1,)"). In trained models (bottom row), the strong diagonal and small off-diagonal values indicate coupling, while no such coupling is present at initialization (top row). Additional plots are in Appendix A.8 (Figure
5.3 EMERGENCE OF COUPLING WITH TRAINING
Coupling emerges through training for the evaluated LLMs, including coupling across depth (Figure 3) and across tokens (Figures 13 16.. Further, evaluate layer-wise coupling at intermediate training checkpoints of Pythia 6.9B and 12B (Biderman et al.) 2023) Figure T(b), and observe that
coupling is generally low at initialization and increases persistently throughout training. Moreover, there is a clear sense of locality in the strength of coupling which is visually displayed in Figure
Ii-e)
The gradual growth of coupling observed in Figure (bbaalalllls the logarithmic increase in accu- racy during training for Pythia 12B and Pythia 6.9B (Biderman al. (023). This highlights the relationship between coupling and performance, since both properties emerge at similar training iterations and rates.
5.4 COUPLING IN VISION TRANSFORMERS
To further investigate coupling, train several ViTs CIFAR-10. Our experiments confirm that coupling emerges with training (Figure 6(b) and exhibits a strong positive correlation with test ac- curacy (Figurel6(a) These findings suggest that coupling is general phenomenon in transformers and reinforces our observations in LLMs. Notably, we measure coupling in ViTs on train data while its correlation with accuracy persists data, highlighting coupling as a structural property of the model related to generalization.
5.5 CORRELATION WITH GENERALIZATION
For each LLM, we measure the average coupling across depth for prompts in the 6 evaluation datasets (Section 4.2 where for each prompt, Cx cx (J,,, (J", J.'') ) averaged over layers 1, 1 [1..... L We then plot the coupling values against the benchmark scores across several LLMs (Figure (
Our results reveal a positive correlation between coupling and performance benchmark scores, ex- ceeding the significance of correlations with other model hyperparameters (Figure This observa-

---

Published as a conference paper at ICLR 2025
tion suggests compelling relationship between stronger coupling of singular vectors of Jacobians Je and improved generalization.
We hypothesize that simple trajectories may lead to better generalization. This idea aligns with with several generalization bounds in machine learning (Arora et al. 2018). which suggest that models with lower complexity tend towards better generalization. Furthermore, previous work (Novak et al., 2018) demonstrated that the Frobenius norm of input-output Jacobians is related to generalization, suggesting that coupling, structural property derived from block Jacobians, may also be related to generalization.
101
Contiiitnt Narastinn
daron Mastal meseentte falve
Good beteeheeee Code Mourit
Marii
Misical
Come
(a) Linearity
(b) Expodistance
Figure 5: Regularity of Trajectories. The line-shape score (LSS) of embedding trajectories, as discussed in Section [56 computed on 1,200 prompts of the HuggingFace Open LLM Leaderboard (Section 4.2) for variety of trained (black) and randomly initialized (blue) L.MSs (Appendix [A.1) Median values over all prompts are plotted and are accompanied with uncertainty intervals depicting the inter-quartile range of the results for each model. Models are sorted by number of parameters.
0.70
0.8
Accurary
0.7
Coupling
0.66
0.64
SD 0.025
0.62
0.4
SD
0.075
0.05
0.3
0.93
0.075
0.60
0.62
0.64
0.66
0.68
0.70
100
200
300
400
500
Coupling
(a) Corrolation with Test Accuracy
(b) Emergence with Training
Figure 6: Transformer Block Coupling in ViTs. (a) CIFARIO test accuracy against coupling
for stochastic depths (0.025, 0.05, 0.075) (Section 3.4h. Coupling and accuracy evaluated
training checkpoints (see (b)), showing positive correlation (R² = 0.93). (b) Coupling at training checkpoints {50, 100,.. ... 550) for transformers trained with stochastic depths (0.025, 0.05, 0.075).
8

---

Published as a conference paper at ICLR 2025
5.6 REGULARITY IN EMBEDDING TRAJECTORIES
By analyzing token trajectories as representations propagate through the layers, we identify several structural properties. Notably, we observe considerable degree of linearity in the hidden trajec- tories across all examined LLMs. The LSS of the trajectories has an average value 4.25 in trained models, compared 6.54 at initialization, and is supported by the low variation across benchmark prompts (Figure 5). Linearity increases with training at varying depths of Pythia 12B (Figure [25] This linear and expansive behavior of the representations is further illustrated in Llama-3 70B, MPT 30B, and NeoX 20B through low dimensional projections of embedding trajectories (Figure (TO..
Among the LLMs studied, most hidden trajectories exhibit exponential growth that emerges with training (Figure 5(b)). The low coefficient of variation across prompts indicates the robustness of this property across diverse tasks. contrast, measurements at initialization show equally (rather than cxponentially) distanced trajectories, as reflected by the low coefficients of variation in the norms of their laser-wise differences (Figure 22) Under certain assumptions, exponential spacing motivated through coupling, as discussed in Section 6.1 and Appendix A.5
DISCUSSION
6.1 EMERGENCE REGULARITY WITH COUPLING
Under certain assumptions, the increased linearity and exponential spacing observed in many models can analyzed as a consequence of coupling. Considering input embeddings x? wf...., xn and the linearization of the last token embedding z given by J (= (x?....,=%)
S/1 = =1 J n.n ,....,,,,))) =0) (
To simplify notation, we write zy x', J¹ n,n (2)....,,.................. .....,,,) J The representations follow the linearized equation:
r = x¹ + Jjr (I Ji)a'.
Expanding across layers, the entire system can be approximated by the product
x² -1+ + - J1)2
Assuming perfect coupling of singular vectors across depth (and between left and right singular vectors), we have Jj US, As discussed in Appendix A.5 assuming the singular values across depth are approximately constant (S S), this equation predicts that the norm of would exhibit exponential growth if zg is some eigenvector J. Expanding and S.
danke
Σ
where uy and 8j represent the singular vectors and singular values of the Jacobian, respectively. In general, trajectories not expected be perfectly linear unless x² aligns with eigenvector of J. However, in our experiments, we observe a notable tendency towards linearity, suggesting that the representations align progressively during training with the eigenvectors of the coupled Jacobians.
RELATED WORK
Residual Networks. ResNets (He et al. 2016) have been viewed as an ensemble of shallow net- works Veit et al. 2016), with studies delving into the scaling behaviour of their trained weights (Cohen et al. 202 1). The linearization of residual blocks by their Residual Jacobians was first ex- plored by Rothauge et al. (2019), who examined Residual Jacobians and their spectra in the context of stability analysis, and later by Li Papyan (2023) who discovered Residual Alignment. Cou- pling of Jacobian singular vectors in LLM transformers extends previous results for Resnets (Li & Papyan 2023). We show coupling in j/ across various tokens, which specific to tokenization in transformers, in addition to demonstrating coupling of Juula across I, which was also identified analogously in ResNets (Li & Papyan 2023). Further comparison is included in Appendix A.3
9

---

Published as a conference paper at ICLR 2025
Neural Ordinary Differential Equations. Neural ODEs (Chen et al. 2018) view ResNets discretized dynamical process, with past work (Sander et al. 2022) showing the convergence of Residual Networks to system of linear ODE, with some extensions transformers (Zhong al. 2022 et al.
2021) The emergence of coupling in transformers suggests that a discretization of simple fterative process emerges in L.Mss.
In-context learning. LLMs can perform tasks through examples provided in a single prompt,
demonstrating in-context learning (von Oswald al. 2023
Bai et al. 2024 Ahn al. [023
Akyürek et al., 2023) Xie et al. 2021; Hahn & Goyal 2023
Xing et al. 2024). ). Studies suggest
trained self-attention layers implement gradient-descent like updates across depth to minimize the
MSE of a linear model:
Frinl min ||Ar - b112.
These updates take the form:
Z++1 = +1+ A)xx e.
Coupling across depth suggests similarity in the matrices eAT A.
Hidden Representation Dynamics. Prior research interprets deep neural networks through dynam- ical systems, revealing that training trajectories align with geodesic curves (Gai & Zhang 2021 and partition activation space into basins of attraction (Nam et al. 2023). For further related works, see
(Geshoovkki et al. 2023b a Tarzanagh al. 2023 Valeriani et al. 2023 Hosseini Fedorenko)
2023).
Structure in Hidden Representations. Neural Collapse (Papyan et al. 2020) highlights emergent
regularity in last-layer representations, with subsequent studies exploring hidden-layer structures and their theoretical underpinnings (Wang et al. (2024a) Parker et al. 2023 Zangrando et al. 2024
Garrod & Keaatig| 2024 Wang et al.] 2024b] Hoyt & Owen|22222 2021 Arous et al. 2023 Zarka al,
2021 Ben-Shaul & Dekel] 2022; Papyan, 2020; Sūkeník et al.) 2023). In LLMs, recent works
identity uniform token structures (Wu & Papyan 2024) and low low-dimesiional hidden trajectories
Sarrati et al. 2024). Our work examines local token interactions through Jacobian dynamics across
all layers.
CONCLUSION
contribute to the understanding of the mechanies underlying transformer architectures through an analysis of the trajectories of token embeddings and their interactions. Our research builds on the understanding of transformer architectures by revealing the coupling, across depth and token, of singular vectors in the Jacobians of transformer blocks for multiple LLMs trained by various organizations, as well as ViTs. We establish a correlation between the strength of this coupling and benchmark performance on the HuggingFace Open LLM Leaderboard, highlighting the significance of transformer block coupling for generalization. These findings open avenues for future research, encouraging deeper exploration into the connections between regularity of hidden representations, training methods, and generalization.
Our primary goal
ACKNOWLEDGMENTS
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). This research was supported in part by the Province of Ontario, the Government of Canada through CIFAR, and industry sponsors of the Vector Institute (www www.wveeoorrssiitt.cmmmm This research was also supported in part by ca/) and the Digital Research Alliance
ai heiparrnnersigge htt.t//rssssspsecrrrrttetattatte
Compute Ontario (ht.tps: http:///wwwwwww///ttt..ooo
of Canada (https: /tttps://allaaneeaan.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning, 2023.
Al@Meta. Llama 3 model card. 2024.
URL https: sttas://gthuu..cco/mmtttttt www.:uwww.oo//tt.aaaa.aeaaa /aama//aeeaaaaaaa///aaa
1lama3/blbb/ M/MODEL, CARD md
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou, What learning
algorithm is in-context learning? investigations with linear models, 2023.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-ar performance. 2023.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International conference on machine learning, pp- 254-263. PMLR, 2018.
Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. High-dimensional sgd aligns with emerging outlier eigenspaces, 2023.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL org/ abs 2401. 15024
hhttps://aae
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL https: /aaxivvv...... wwrexiv.org//.. abs/ abs 2309.1660000
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural
Information Processing Systems, 32, 2019.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open I1m leaderboard. https: 11m leaderboard 2023.
//wgggingface.ccommmm cttp:/cccccppa/eaaeeeeeee htwp://wwww....gggggggaaeeeeeeee
dd Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Alexander Cloninger, Timothy Doster, Tegan Emerson, Manohar Kaul, Ira Ktena, Henry Kvinge, Nina Miolane, Bastian Rieck, Sarah Tymochko, and Guy Wolf (eds.), Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, volume 196 of Proceedings of Machine Learning Research, pp. 37-47 PMLR, 25 Feb 22 Jul 2022. URL https: https::/proccediin mlr. wwwppppwwss.....t ben www.lhha..aaaa....m halx://eee///////eeaaaaaaaaaaaa.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Mesh- Tensorflow, March 2021. URL /ooi. org/ Mesh-Tensorflowr
10. 5281/ http//eeoo....o 5297715 If you use this software, please cite it using these metadata.
11

---

Published as a conference paper at ICLR 2025
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shiv- anshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks, 2021.
Amir Ben Dror, Niv Zehngut, Avraham Raviv, Evgeny Artyomov, Ran Vitek, and Roy Jevnisek. Layer folding: Neural network depth reduction using activation linearization, 2021. URI.
htttps://axxiv http://aaivvvvaa/aeeeeeee
09309
Stanistaw Jastrz Ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio.
Residual connections encourage iterative inference. In International Conference Learning Representations, 2018.
Weinan Ee. A proposal on machine learning via dynamical systems. Communications in Mathemat- ics and Statistics, 5:1-11, 02 2017. doi: 10. 10.10077-403344-1--0003--
Yonina C. Eldar and Moshe Mishali. Robust recovery of signals from a structured union of sub- spaces. IEEE Transactions on Information Theory, 55(11):5302-533 2009. doi: 10.1109/TIT. 2009 1039411
Sara Elkerdawy, Mostafa Elhoushi, Abhineet Singh, Hong Zhang, and Nilanjan Ray. To filter prune, or to layer prune, that is the question, 2020. URL 05667
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models, 2023. URL httts:///ariiv: http://...r.aaa////////// hitppps//arxvv.orrr///////.....0 10924
Yonggan Fu, Haichuan Yang. Jiayi Yuan, Meng Li, Cheng Wan, Raghuraman Krishnamoorthi, Vikas Chandra, and Yingyan Lin. Depthshrinker: A new compression paradigm towards boosting real- hardware efficiency of compact neural networks, 2022. URL https::/aaxti: org/ abs/
22066.00 00843
Kuo Gai and Shihua Zhang. A mathematical principle of deep learning: Learn the geodesic curve in the wasserstein space, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002: 2020.
Connall Garrod and Jonathan P. Keating, Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model, 2024.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clus- ters in self-attention dynamics. 2023a.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical per- spective on transformers, 2023b.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:/612.0777,, 2016.
12

---

Published as a conference paper at ICLR 2025
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magnusson, Yizhong Wang. Shane Arora, David Atkin- son, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacoh Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Worts- man, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo. Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024.
URL http::tttt/eett /ttp://aaxvvv.v... wttp://gg//aboaeooo /2L 00838
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2024. URL atttss://arxio org/abs/
2403. 17887
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang. Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL htLhtts::/// /ttp://///xxx..///// Abb/2306.11644
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. CoRR, abs/1705.03341, 2017. URL (ttp:: http://www.gg////////ooa// 03341
Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10. 10.1109/CVPR.2016.900
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Eghbal A. Hosseini and Evelina Fedorenko. Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language, 2023. URL /Rttsss:///xxxiiu org. abs /et...231.... 04930
Christopher R. Hoyt and Art B. Owen. Probing neural networks with t-sne, class-specific projections and a guided tour, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -hu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https: //
09685
neural Metod Jazbec, Patrick Forré, Stephan Mandt, Dan Zhang, and Eric Nalismick. Early-exit neural net- works with nested prediction sets, 2024. URL /tttass//axxxvvoo////b/////////
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023.
Jinuk Kim, Marwa EI Halabi, Mingi Ji, and Hyun Oh Song. Layermerge: Neural network depth compression through layer pruning and merging, 2024. URL httpt::///exii..ot/ a
2406. 12837
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL
wwtp:/iwwwwr//////////////asse// 2205
11 916
Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Toronto, 2009.
13

---

Published as a conference paper at ICLR 2025
Vedang Lad, Wes Gumee, and Max Tegmark. The remarkable robustness of Ilms: Stages of infer- ence?, 2024. URLhttps: /tt.://:///xxv..oo///////////00 19384
Bei Li, Quan Du Tao Zhou, Shuhan Zhou, Xin Zeng. Tong Xiao, and Jingbo Zhu. Ode transformer:
An ordinary differential equation-inspired model for neural machine translation, 2021.
Jianing Li and Vardan Papyan. Residual alignment: Uncovering the mechanisms of residual net-
works. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank ap- proximation via alternating minimization, 2016. URL
02262
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. URL https::/aaxxx........ org/
abs/ 2309. 05463
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021. URL
04647
Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campe- san, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, Mohammed AI- Yafeai, Hamza Alobeidli, Leen Al Qadi, Mohamed El Amine Seddik, Kirill Fedyanin, Reda Alami, and Hakim Hacid. Falcon2-111 11b technical report, 2024. URL https:://// http://axxvvvoo abs /aass/240 2407 14885
Andrew Nam, Eric Elmoznino, Nikolay Malkin, Chen Sun, Yoshua Bengio, and Guillaume Lajoie. Discrete, compositional, and symbolic representations through attractor dynamics, 2023.
Roman Novak, Yasaman Bahri, Daniel A. Abolañia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study, 2018. URL https://
Vardan Papyan. Traces of class/cross- class structure pervade deep learning spectra, 2020.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via con- volutional sparse coding. The Journal of Machine Learning Research, 18(1):8877-2388, 2017.
Vardan Papyan, X. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40); 24652 24663, 2020. doi: 10. 10.10733/naa. 20135p99..005559111 abs/ 2bs/10.. 10..0....1a1ass... 20155091177
URL https: /wwp: www. pnas org/doi/
Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks, 2023.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, 2017.
abs/ 1tt./771..0 04735
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Kai Rothauge, Zhewei Yao, Zixi Hu, and Michael W. Mahoney. Residual networks as nonlinear systems: Stability analysis using linearization, 2019.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt. Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llaaa: Open foundation models for code, 2024.
14

---

Published as a conference paper at ICLR 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale, 2019.
Michael E. Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations?, 2022.
Raphaël Sarfati, Toni J. B. Liu, Nicolas Boullé, and Christopher Earls. Lines of thought in large language models, 2024. URL https::/aaxxi...ooooo
2410. 01545
Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should add early exits to neural networks? Cognitive Computation, 12(5):95-966,, June 2020. ISSN 1866-9964. doi: 12.10077/12559-00000977777 5 612559-020-0973---
URL http: /ttt:://ox..ci.or//111000 wtg/:..........o.. 10077
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. 2023.
org/abs/
2104. 09844
Peter Súkeník, Marco Mondelli, and Christoph Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model, 2023.
Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-marge token selection in attention mechanism, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, An- tonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christine Muraru, Grigory Rozhdestvenskly, Hen- ryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean- Cean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuta, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis. Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. 2024.
MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023a. URL W.LWww.......................... www..oosaiee. www.osaacml.... www.://saammm..comom..oooom
30b Accessed: 2023-06-22.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable
11ms, 2023b. URL www. www.mosaai mosai cml com www.coooooooooo.o..o..oooooooooo http//lllo//ppp/// -7b Accessed: 2023-05-05.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2021.
URL httt:p://aaxxvv.oo wtt.orggg.bb//..o.a/// 2atp://////.iiv.oor///////22.... 12877
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anna Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
15

---

Published as a conference paper at ICLR 2025
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, llyyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Al- berto Cazzaniga. The geometry of hidden representations of large transformer models, 2023.
Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki M. Asano, and Tijmen Blankevoort. The IIm surgeon, 2024. URL htthttt:: / http//aww...../..aaaa///// htrgg/bb//230 2ttps/231..1222
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. 30, 2017. URL htttps://proceednngss neurips htg.ee/dddeaaeeeeieisss..oeeeeee File/ 31le/3f5ee255 3ww.eee24337edeeeeeeeeee
cc/paper files/paperr22117/ www.aaappppp...
Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks, 2016.
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent, 2023.
Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Understanding deep representation learning via layerwise feature compression and discrimination, 2024a.
Sicong Wang, Kuo Gai, and Shihua Zhang. Progressive feedforward collapse of resnet training, 2024b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing, 2020.
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models, 2024. URL
Sang Michael Xie, Aditi Raghunathan, Percy Liang. and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint LuXiv:2111.0200 2021.
Yue Xing, Xiaofeng Lin, Namjoon Suh, Qifan Song, and Guang Cheng. Benefits of trans- former: In-context learning in linear regression tasks with unstructured data. arXiv preprint Ntl::2402.00743.3 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://axxix: org/ abs /tbb://447.1066
Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Guglielmi, and Francesco Tud- isco. Neural rank collapse: Weight decay and small within-class variability yield low-rank bias, 2024.
John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks, 2021.
16

---

Published as a conference paper at ICLR 2025
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019.
Yaofeng Desmond Zhong, Tongtao Zhang, Amit Chakraborty, and Biswadip Dey. A neural ode interpretation of transformer layers, 2022.
A
APPENDIX AND SUPPLEMENTARY MATERIAL
A.1 SUITE OF LARGE LANGUAGE MODELS AND PROMPT DATA
We evaluate a comprehensive collection LLMs on datasets from the HuggingFace Open LLM
leaderboard (Beeching et al. 2023).
Base models. Falcon (40B, 7B) (Almazrouei et al. 2023). Falcon2 (11B) Malartic et al.
2024
Llama-3 (70B, 8B) (AI@ Metal 2024). Llama-2 (70B, 13B. 7B) Touvron
2023
MPT (30B, 7B) (Team] 2023a b). Mistral v0.1 (7B) (Jiang et 2023). Gemma (7B, 2B)
(Team al., 2024). Gemma 1.1 (7B, 2B), Phi (1B, 1,5B, 2B) (Gunasekar al.
2023 Li
et al. (023), OLMo (1B, 7B) (Groeneveld et al. (2024), Qwen (Bai et al.
2023 Yang et al.,
2024 ) NeoX (20B) (Black et al. 2022), Neo (2.7B) (Black et al. 2021 Gao et al.
2020).
Pythia (I.4B, 2.8B, 6.9B, 12B) Biderman et al. 2023). and GPT-2 (I.BB, 774M, 355M, 117M) (Radford et al. 2019).
Fine-tuned models. CodeLlama (34B, 13B, 7B) Rozière al.
2024). CodeLlama Instruct (34B,
13B, 7B) (Rozière et al. 2024). Mistral-vO. Instruct (7 (Jang et al. 2023), Mistral-v0.2
Instruct (Jiang al.
2023). CodeGemma (Team et al. 2024).
Datasets. ARC (Clark al.
2018 HellaSwag (Zellers et al.
2019) MMLU (ennryykks et al.
2021}, Truthful QA
Lin ct al. 2022). WinoGrande (Sakaguchi et al. 2019).
17

---

Published as a conference paper at ICLR 2025
Table LLMs featured in the experiments throughout paper. Included in the table is the param- eter budget of each model, the embedding dimension, the number of training tokens, and the Open LLM leaderboard (Beeching et al. 2023 benchmark score.
MODEL
PARAM.
LAYERS (L)
DIM. (duoma.)
TOKENS
SCORE
PNS.DING
LLAMA-3
70 B
80
4096
15 T
ROPE
B
4096
15T T
62.35
ROPE
LLAMA-2
70 B
8192
2T 66.05
RoPE
13 B
40
5120
2T
55.69
RoPE
7 B
32
4096
TT
50.97
RoPE
CODELLAMA
34 B
48
8192
2T
55.33
ROPE
5120
45.82
ROPE
7B
32
4096
2T
39.81
RoPE
CODELLAMA (IT)
3
48
8192
2T
43.0
RoPE
B3
40
5120
2T
37.52
RoPE
7B
4096
2T
40.05
RoPE
ORCA
40
5120
2T
58.64
RoPE
7 B
32
4096
2T
54.55
ROPE
FALCON
40 B
60
8192
T
58.07
ROPE
B
4544
T
44.17
RoPE
FALCON (IT)
40 B
8192
IT
43.26
ROPE
7 B
32
4544
1.5T
ROPE
FALCON2
11B B
60
4096
64.28
ROPE
MISTRAL-VVO.I
B
32
4096
60.97
ALIBI
MISTRAL.-V.. (IT)
7 B
32
4096
54.96
AL.BI
MISTRAI--V0.2 (IT)
7 B
32
4096
65.71
ALıBi
MISTRAL-VO.3
7B
32
4096
60.28
ALIBI
VICUNA
4096
$2.06
RoPF
MPT
30 B
48
7168
1
66.98
AL1B1
7B
32
4096
1 T
56.84
ALIB1
OLMO
7B
32
4096
2.5 T
43.36
RoPE
16
2048
T
36.78
RoPE
OLMO-1.7
7B
32
4096
2.5 T
52.82
RoPE
PHI-2
2 B
32
2560
1.4 T
61.33
ROPE
1.5
2048
B
47.69
ROPE
2048
54 B
47.69
RoPE
GEMMA
7 B
28
3072
6 T
64,29
RoPE
2B
18
2048
6
42.75
RoPE
GEMMA-I.1
7B
3072
T
60.09
ROPE
2B
18
2048
6T
30.0
RoPE
CODEGEMMA
7B
28
3072
6T
56.73
RoPE
2 B
18
2048
6 T
32.19
RoPE
QWENI
14 B
5120
66.7
ROPE
7B
32
4096
61.76
RoPE
4B
40
2560
57.05
RoPE
188
24
2048
46.55
RoPE
0.5 B
24
1024
38.62
ROPE
QWEN2
B
3584
68.4
RoPF
NEO
20 B
44
6144
41.69
RoPE
2.7 B
32
2560
0.42T
36.20
SINE
PYTHIA
12 B
36
5120
0.3
38.82
ROPE
6.9
4096
0.3
39,30
RoPE
2.8
32
2560
0.3
37.09
RoPE
1.4B B
24
2048
0.3
34.75
ROPE
IB
16
2048
0.3
32.78
RoPE
GPT-2
I
1600
34.12
SINE
774
36
1280
32.07
SINE
355 M
24
1024
29.87
SINE
117
12
768
28.53
SINE
18

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL METRICS
A.2.1 VISUALIZATION OF TRAJECTORIES WITH PCA
Each token, with initial embedding zR. forms a trajectory z!. ......,,2 2 as passes through the I. transformer blocks. The dynamics in high-dimensional space visualized through 2- dimensional principal component (PC) projection, PC₁, fitted to the last layer embeddings X = (af 2z. .... 2/). The projected embeddings, (x)() PCz(x,),... (af( are plotted for each of the i = = I,..., trajectories.
A.3 COMPARISON TO RESNETS
Coupling, Our results complement and build upon those of (Li & Papyan 2023 who have ob- served the coupling of singular vectors of Residual Jacobians in classification ResNets. We observe coupling across depth in a wide range of L.Ms,, where Jacobians are evaluated at the sequence of embeddings, with respect to the current token, whereas in RA Jacobians are evaluated at a single representation. Additionally, with transformers pling not only across depth but also across tokens. We observe coupling in LLMs across tokens in a variety of ways. Further, we consider and analyze the relationship between coupling and generalization.
may analyze cou-
Linearity and Equidistance. Linearity in hidden trajectories, observed in ResNets Li & Pa- pyan (2023): Gai & Zhang (2021). also emerges with training in LLMs. The mean LSS value among the evaluated LLMS 4.24 (Figure 5(a). greater than LSS measurements observed for ResNets (Gai & Zhang 2021), page 18) which range between 2.0-3.0 (due
to varying trajectory length and hidden dimension). In both architectures, training induces improved linearity and regularity (Figure L0p in trajectories. In contrast to ResNets, trajec- tories are not equidistant, instead showing exponential growth between layers ((b). We quantify this spacing through low coefficient of variation, displaying the presence of exponential growth in token trajectories. In both classifier ResNets and LLM transformers, there is evident level of regularity in hidden representations (Figure IOP.
Rank of Jacobians. Li & Papyan 2023) show that Residual Jacobians have rank at most C, the number of classes. This analogous result automatically holds for LLMs since the vocabu- lary size is significantly greater than the embedding dimension of the transformer blocks.
Singular Value Scaling. Li & Papyan 2023) observe that top singular values of Residual Jaco- bians scale inversely with depth. In trained LLMs, however, top singular values do not show a consistent depth scaling across models (Figure 30 notably differing from classification ResNets. In addition, the distribution of singular values at each layer varies significantly be- tween models. Singular value scaling is more present in untrained transformers (Figurd2) likely caused by additional layer normalizations in residual blocks (Section 4.1
A.4 SIGNIFICANCE COUPLING
The transformer block coupling phenomenon offers insight into several prominent practices LLM research, as summarized in Table
The coupling phenomenon provides insight into the internal operations of transformer. We hy- pothesize that during training, the LLM learns to represent embeddings in specific low-dimensional subspaces (Eldar& & Mishali 2009). Given an input, the first layer converts the input into embed- dings within one of these learned subspaces. Each subsequent transformer layer modifies these embeddings. potentially moving them different subspaces. Strong coupling between consecutive layers suggests that the LLM tends towards representations. in the same or similar subspaces across many layers. Weak coupling suggests that the subspaces may change between layers, though usu- ally gradually, and that adjacent layers still operate in relatively similar subspaces. Previous works have shown (Lad et al. 2024: Gromov et al., 2024) that the early and late layers of language models behave differently. which may be understood through coupling; tokens remain in similar spans, then transition to a different subspace, continuing within new span that is consistent in the remainder of the transformer.
19

---

Published as a conference paper at ICLR 2025
The emergence of coupling with training steps (Figure D may provide insight into the dynamics. Under full coupling and a difference equation approximation, the representations evolve as
uy + 8z)'ujj
where us and X denote the eigenvectors and eigenvalues of the Jacobian, respectively. In this case, the gradients of the loss L with respect prediction y, x² are represented by
OL Ox (x 1)
desodd
u 8,) 8 x²),
Due to the coupling, the dynamics exhibit either exponential growth decay in different subspaces, depending on whether 8j is greater or less than 1, which known to cause challenges for opti- mization as in past work dynamical isometry (Pennington et al. 2017). We infer that increasing coupling during training makes optimization progressively more difficult. Conversely, as training progresses, it becomes harder to achieve stronger coupling, and is consistent with the logarithmic trend in Figure I
Table Significance of the Coupling Phenomenon. table which highlights the implications of transformer block coupling to variety of effors in machine learning research.
Current Research Practice
Key Idea
Our Contribution
Compressing models by merging blocks (Fu et al. 2022
Combine adjacent trans- former blocks to reduce model size without signifi- cant performance loss.
Demonstrates that merging is effective because blocks be- come strongly coupled during training.
Kim et al. 2024)
Compressing models by prun-
Remove certain transformer blocks while preserving func- tionality.
Explains that pruning works because the coupling ensures redundancy across blocks.
ing blocks (Elkerdawy al.
2020
Kim et al.] [2024] Dror
al. 202 Fang al. 2023
Compressing models by projecting weight matrices (Ashkboos et al. 2024)
Reduce dimensionality by projecting weights into smaller subspaces.
Shows that coupling induces low-dimensional subspace in which blocks' weights are aligned.
Studying the effect of trans- former block permutations
Investigate whether permut- ing the order of blocks affects model performance.
Explains why permutations have minimal impact: strong coupling creates structural ro- bustness.
(Hu et al.]
2021 Mahabadi
al.
2021
der Ouderaa
et al.
2024 Li et al. 2016)
Early exiting in LLMs (Scar-
Allow models to exit compu- tation early based on task con- fidence.
Reveals that early exiting works because representa- tions progress linearly along shared trajectory due to coupling.
dapane et al. 2020
Jazbec
al.
2024)
A.5 DYNAMICAL MOTIVATION
The equality top left and right singular vectors suggests that the linearizations form a simple linearized system that acts representations. Consider difference equation
.+1 A x'
(14)
Its solution the final L is given by
L + Ar) x
(15)
1=1
20

---

Published as a conference paper at ICLR 2025
Expanding the brackets shows that x' can be thought of collection of many paths of various lengths, due to the binomial identity. This agrees with Veit et al. (2016) which views ResNets as
x' === + A + A1-2) ... (1 A1) x0
(16)
However, Veit et al. (016) do not make any assumptions about the alignment of the various A matrices. The coupling phenomenon suggests the model as implementing the simpler system
(1+
(17)
where all the A matrices are aligned. One benefit of this interpretation that, we can write x' in simple closed form, as above. We quantify the similarity of hidden trajectories to the evolution of the above difference equation, in order to detect the emergence of a simple linearization to rep- resentations. The emergence of increased linearity and exponential spacing in many LLMs can be analyzed as a result of coupling under some conditions on the spectral decomposition of the Jaco- bians. Consider input embeddings .f....,. rn and the linearization of the last token embedding s given by nn (af......................... ):
1 (I J (o). 20
(18)
We simplify notation and write oy, = x², _n (ff..... ........)))))))) J Under the assumption of spectral coupling, J = USSV v USU and the linearized effect of the last token is
x' = [ UUSVV Yx = (Tir+s)) UT
(19)
Suppose that x²' Ug is the k-th left singular vector of J'. It follows that x IL st )²² where sy denotes the k-th singular value at layer 1. The exponential spacing measurement is 8 2t =.... sk. Explicitly, under such assumption,
(1 +
motivated by the consistent choice Sμ:
x² = =1 + =x" and by Equation
12 for each
(1 +1+00 8k ||oll
a = in
= Im(1 + sR)
ED O,
(1+ sk) |
that is, the coefficient of variation 0 across l. In addition, if x' (1 + st. )z)=²'1 it is clear that the trajectory would form a perfect line, yielding LSS by the discussion Section 3.2 In general, trajectories are not expected to be perfectly linear unless x aligns with an eigenvector of J
A.6 LLM EVALUATION AND IMPLEMENTATION DETAILS
The source code used to produce the results reported in this experiment has been included as supple- mental material. Models with varying parameter sizes are loaded on GPUs with appropriate memory requirements: NVIDIA A40 (Naaram 2 40B), NVIDIA Quadro RTX 6000 for Gemma variants and when (40B > Ryuaram > 13B), and NVIDIA Tesla T4 when (13 2 Mpparm) except Gemma variants. 1,200 prompts from the OpenLLM leaderboard were evaluated in variable batch sizes were queued on SLURM cluster, with appropriate adjustments depending on the memory required to load the LLM.
13B 2 Apurrm: 100 prompts per batch, except Gemma variants, which used 25 prompts per batch. The larger memory requirement for Gemma variants is likely due to the much larger vocabulary size in the model.
40B > Tparamm 13B: 10 prompts per batch, except NeoX 20B which used 100 prompts
per batch.
Tparamm 2 40B: 50 prompts per batch.
Due to the high memory requirement for computing block Jacobians, for experiments involving the Jacobians, NVIDIA Quadro RTX 6000 was used additionally for 13B > Пpuammm 2 7B and corresponding models were quantized. Additionally, coupling of singular vectors of Jacobians was computed on a smaller subset of the dataset prompts.
The computational complexity for the metrics utilized in this paper follows:
21

---

Published as a conference paper at ICLR 2025
Weight decay
0.210
0.005
0.01
0.205
0.04
0.06
Coupling
0.200
0.2
0.195
0.190
0.00
0.10
0.20
0.30
Stochastic depth
Figure 7: Coupling against Stochastic Depth Rate. Plots are generated for each weight decay in {0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.2.
Coupling.
Coupling requires computing the Jacobians for each transformer block, and for- ward pass and backward pass required (note that we compute the Jacobians on a block level). Once the Jacobians are obtained, it requires computing truncated singular value decomposition of each Jacobian. The time complexity of computing the truncated SVD of rank for has time complexity O ), so the asymptotic time complexity of computing the coupling score between two connections is (c)
d matrix (ddk),, where k << d. Computing A from the SVDs then
LSS. For each trajectory, the time complexity of computing the LSS is O(Ld) where is the num- ber of layers and is the hidden dimension. Therefore, for a prompt containing T tokens, the total time complexity for each prompt is (d) (in addition to single forward pass of the model).
Expodistance.
Similarly, computing the expodistance of a single trajectory has time complexity ((Td). Therefore, for prompt containing T tokens, the total time complexity for each prompt is O(TLd) (in addition to a single forward pass of the model).
A.7 VIT TRAINING DETAILS
For further investigation of coupling in transformers, we train 3 Vision Transformers (ViTs) follow- ing the default configurations of DEiT training (Touvron et al. (02T) CIFAR10 (Krizhevsky) 2009). For fixed ViT architecture with embedding dimension 192. depth of 24 layers, and attention heads, vary the weight decay 0.05 and stochastic depth rate {0.025, 0.05, 0.075). Opti- mization uses ADAM optimizer with 5e - learning rate and cosine scheduler for 600 epochs with 10 epochs of linear warmup. Training proceeds with data mixup using 0.8 Optimization pro- ceeds on 4 NVIDIA Tesla T4 GPUs with 128 batch size and data parallelization, with total training time being approximately 6 hours.
A.8 ADDITIONAL EXPERIMENTAL RESULTS
22

---

Published as a conference paper at ICLR 2025
(a) 1% of Hidden Dimension
(b) 3% of Hidden Dimension
(c) 5% of Hidden Dimension
Figure 8: Coupling plotted against benchmark score for varying percentage of top singular vectors. This indicates that the relationship robust across various proportions of top singular vectors.
Ig
(a) Score vs. Number of Layers
(b) Score vs. Embedding Dimen- sion
(c) Score vs. Number of Parame-
ters
Figure 9: LLM number of layers, embedding dimension, and number of parameters, against score.
25
33
-28
-38
14/5
260
Llama-3 70B
(b) MPT 30B
NeoX 20B
Figure 10: Trajectories of Hidden Representations. Visualization of the layer-wise trajectories of hidden representations in Llama 70B, MPT 30B, and NeoX 20B the prompt: What the capital of France? The capital is. Trajectories of tokens are plotted in latent space, visualized with 2-dimension principal component projection. clear directed and outward growth is visible in each token trajectory.
23

---

Published as a conference paper at ICLR 2025
7pta thaiquee
Pitaa 120
Figure 11: Evolution of Hidden Trajectories Throughout Training. Principle component visual- izations of the hidden trajectories in Pythia 12B at training checkpoints 1, 4000 and 143000 on the prompt: what is the capital of France? The capital is.
Us
The
Palion
CPP2 1.58
309
20
400
28.38 15-30 15.38
12.44
15.3 D011
Figure 12: Hidden Trajectories in LLMs. Principal components of the trajectories of the hid- den representations through various LLMs (columns, decreasing in model size, see Table in the prompt: What is the capital of France? The capital is. Top row: all layers. Middle Row: layers in shallower transformer blocks (layers specified above plot). Bottom Row: layers in deeper transformer blocks (layers specified above plot). Trajectories of each input token (last token 'is' is plotted in black) are plotted in latent space, visualized with 2-dimension principal component projection. Representations proceed in distinct outward directions, especially the sec- ond half of transformer blocks (lower row) during which the norm of representations increases, with possible abrupt change in the last layer (outer points upper row). A clear direction of movement is visible in each token trajectory.
24

---

Published as a conference paper at ICLR 2025
(a) Gemma 7B Untrained
(b) Gemma 7B Trained
(c) Phi-2 Untrained
(d) Phi-2 Trained
Figure 13: Transformer Block Coupling across Depth.
(a) Gemma 7B Untrained
(b) Gemma 7B Trained
(c) Phi-2 Untrained
(d) Phi-2 Trained
Figure 14: Transformer Block Coupling across Tokens (Same input and output tokens).
25

---

Published as a conference paper at ICLR 2025
(a) Llama-3 8B Untrained
(b) Gemma 7B Untrained
(c) Phi-2 Untrained
(d) Llama-3 8B Trained
(c) Gemma 7B Trained
(f) Phi-2 Trained
Figure 15: Transformer Block Coupling across Token (Fixed input). The figure illustrates cou- pling of Jacobians, with fixed input token, across tokens. More specifically, in the matrix plot located at entry (f2, t22), the absolute values of the entries of matrices Afitttut2 are visualized (with randomly fixed layers 1, ('), In the trained plots (bottom row), the off-diagonal entries being close to with visible diagonal indicates coupling of these Jacobians. This coupling, however, seems to be more evident for certain token pairs and less for others. At initialization (top row), there is no such coupling across tokens, Additional visualizations are included Appendix A.8 (Figure [9)
26

---

Published as a conference paper at ICLR 2025
(a) Llama-3 8B Untrained
(b) Gemma 7B Untrained
(c) Phi-2 Untrained
(d) Llama-3 8B Trained
(e) Gemma 7B Trained
(f) Phi-2 Trained
Transformer Block Coupling across Token (Fixed output). The figure illustrates coupling of Jacobians, with fixed output token, across tokens. More specifically, in the matrix plot located at entry (41, 41. the absolute values of the entries of matrices A 11t22/12 are visualized (with randomly fixed layers 1, I'). In the trained plots (bottom row), the off-diagonal entries being close to 0 with visible diagonal indicates coupling of these Jacobians. This coupling again seems to be more evident only for certain token pairs. At initialization (top row), there is no such coupling across tokens. Additional visualizations included in Appendix A.8 (Figure 20
Figure 16:
27

---

Published as a conference paper at ICLR 2025
(a) Llama-2 13B
(b) Llama-2 7B
(c) Orca-2 7B
(d) Vicuna 7B
(c) Falcon 7B
Mistral 7B
Gemma 7B
(h) Gemma 2B
Figure 17: Additional plots of Coupling across depth. The figure illustrates the coupling of Jaco- bians across transformer blocks 16.
(a) Llama-2 13B
(b) Orca-2 13B
(c) Ilama-2 7B
(d) Orca-2 7B
(c) Vicuna 7B
(f) Falcon 7B
Figure 18: Additional plots of Coupling across Tokens (same input and output tokens).
28

---

Published as a conference paper at ICLR 2025
(a) Llama-2 13B
(b) Orca-2 13B
(c) Llama-2 7B
(d) Orea-2
(e) Vicuna 7B
(I) Falcon
Figure 19: Additional plots of Coupling across Tokens (fixed input).
(a) Llama-2 13B
(b) Orca-2 13B
(c) Llama-2 7B
(d) Orca-2 7B
(e) Vicuna 7B
(I) Falcon 7B
Figure 20: Additional plots of Coupling across Tokens (fixed output).
29

---

Published as a conference paper at ICLR 2025
1.20
533
302
30
K3-
430
434
(a) Llama-3 8B
(b) Gemma 7B
(c) Phi-2
304
K3
Laper
(d) Llama-2 7B
(c) Gemma 2B
(f) Phi- 1.5
Figure 21: Zero-shot Chain of Thought on Depth-wise Coupling across Layers. We compare the normalized coupling prompts from GSM8k with that of the same prompts appended with "Lett' think step by step.", which we refer to as the Zero-Shot CoT (Kojima et al. (023) prompts. For more thorough analysis, we measure how much each layer is coupled with all other layers, as shown in the figures above. Firstly, the coupling across layers exhibits distinct behaviors across different models, but with noticeable similarities within the same model families In the LLaMA models, coupling starts off lower, increases in the middle layers, then decreases before showing a slight increase again the final layers. In the Gemma models, coupling begins relatively high and steadily decreases toward the end of the network. In contrast, the Phi models exhibit significantly lower coupling in the first layer, followed by immediate increase, and then slight decrease in coupling toward the final layers. The CoT prompt produces similar coupling patterns to the standard prompt, with slight variations in coupling strength. Specificallyy, in the LLaMA models, the CoT prompt consistently results in higher coupling across layers. For the Gemma models, the CoT prompt leads to similar overall coupling levels, though some layers exhibit slightly lower coupling and others slightly higher. On the other hand, Phi-2 shows consistently lower coupling with the CoT prompt, while Phi-1.5 is marginally higher. This variability in behavior, along with the similarities within model families, is likely due to differences in training methods and data across organizations, while models within the same family are trained with potentially similar methodologics.
Coeditiient of Vartation
00397
w0300
w8385
0.334
03333
pimmos 1120
colegermnne
Cadad Sh- prthuu-b abs
meo-2 70
Cada Th-lsstrerddh
Codel Lamp--2
Iakon -ittee
gemamo
2006
Codel Laoas Thold
Haaaed-h bd
ciouee 7hay1
Mismal BBaatt--ttttttaoaa B--fsttecc/-...
Llamsa-5 CodulJama-hn Lama--ma-nn Ad
bd
Creca 220
grommo 1.1 1.76
Mistral-P0 78.011
Aeeaa Jauts: gemoue-1
Mitmnl 7B Bol.aanne
phi12
mat-
rppl 100b
Figure 22: Coefficient of variation of layer-wise equidistance. Variation of layer-wise equidis- tance (Section 3.3 computed over 1,200 prompts from the HuggingFacc Open LLM Leaderboard datasets (Section 4.2) on a suite of untrained LLMs (Appendix(A.1) Plotted are the median values over all prompts, and are accompanied with uncertainty intervals depicting the inter-quartile range of the results for each model. The models are sorted by increasing benchmark performance.
30

---

Published as a conference paper at ICLR 2025
Figure 23: Various Measurements of Representations. All measurements were made on 100 prompts taken from the WikiText 2 datasets. (Row 1) Norm of hidden representations as a function of layer depth. (Row 2) Line Shape Score (LSS) of the hidden trajectories depth. (Row 3) Mean equidistance of contiguous hidden trajectories as function of depth. (Row 4) Entropy logit vectors as function of depth. Noted in most plots a line where the behaviour of the measurement drastically changes.
function of layer
Pyth 1.45
Chacqpai: 143300
a 2.81
pym
2,5
Moon
2.0
15 Window
30
Number
Figure 24: Linearity Emerges with Training. Two plots displaying the evolution of the linearity of the token trajectories through training. (Left) The LSS as function of training checkpoint for the variants of the Pythia Scaling Suite entire prompt. (Right) The mean LSS as function of layer depth measured at various checkpoints throughout the Pythia 12B model. Here, the LSS computed on window layers of width 11, centred at the value given by the x-axis.
al. (2023). Here, the LSS is measured over each
Linearit of Trajectories
-1.5
-2.0
Extremum
+3.
10a
10
10a 10f
10" 10 10"
102
10'
10a
Training Checkpoint
Training Checkpoint
Training Checkpoint
(a) Layer = 5
(b) Layer 1 = 15
(c) Layer = 30
Figure 25: Emergence of Linearity with Training. Average linearity of a trajectory at block depths I {5, 15, 15,30) 30) evaluated for Pythia 12B (Biderman al. 2023) checkpoints (1,2,4. 2,2,4...., 256, 512, 1k, 2k, 4k: 128k, 143k) The linearity is given by the negative LSS, and computed on window of 11 layers centered each depth
31

---

Published as a conference paper at ICLR 2025
Coefiteiont of Variation
10"
10a
Mean
1.30
10-1
10
10"
10-1
10-1.
10
10
109 10
10
100
10a
10
109
Training Chackpoint.
Training Checkspoint
Training Checkpoint
(a) Layer 5
(b) Layer
(c) Layer 25
Figure 26: Expodistance at Fixed Layers. Plotted mean expodistances a function of training checkpoint at various depths of the network. The values at a given depth are the mean expodistance layer window of width 11 centred at said depth 100 MMLU prompts plotted at each layer.
1400
Checkpoint
10a
1200
Checkpoint 143000
Coefficient. of Varnation
Moon Representation
1000
800
10-1
600
10-2
400
200
10-3
Checkpoint
Checkpoint 143000
5
20
35
30
Layer
Layer Window Centre
Figure 27: Norm and Expodistance During Training. (Left) Plotted is the norm of the repre- sentations function of depth various training checkpoints. Observed is the transition form log-like growth in early stages to exponential-like growth, particularly through layers 5 through 20, training evolves. (Right) Plotted is the expodistance the give depth, each computed at variety of training checkpoints.
layer window of width 11 centred at
Cosine simi anit unperturbed
1.1
Hist layer
leyer
0.8
Cosine simierky
0.4
0.2
0.0
8.033
02000
Roise level
Figure 28: Perturbation Experiments with Llama-3 SB. The last token embedding is perturbed
with various noise levels, and compared with the true embedding at the first and last layers. The trend shows that at small noise levels, cosine similarity with the true embedding remains somewhat high, and is significantly lower at the first layer.
32

---

Published as a conference paper at ICLR 2025
2500
1.0
2000
Singular Value
0.8
Valuue Value
1500
0.6
1000
0.4
0.2
500
0.0
0
10
20
30
10
20
30
Depth
Depth
(a) Ilama-3
1.4
1.2
20
1.0
Singular Value
Singular
0.8
15
0.6
10
0.4
5
0.2
0.0
5
10
15
20
25
5
10
15
20
25
Depth
Depth
(b) Gemma 7B
3.5
3.0
Singular Value
2.0
(iingula
1.5
1.0
Depth
Depth
(c) GPT-2 XL.
Figure 29: Scaling of Singular Values of Residual Jacobians (Untrained).
33

---

Published as a conference paper at ICLR 2025
14
0.30
Singular Value
12
1/ Singular
0.25
0.20
0.15
0.10
0.05
10
20
10
20
30
Depth
Depth
(a) Llama-3 8B
0.275
Singular Value
0.250
Singular Value
5
0.225
0.200
0.175
0.150
1
20
Depth
Depth
(b) Gemma 7B
0.5
Singuler Value
Singular Value
30
0.3
20
0.1
0.0
10
20
40
10
20
40
Depth
Depth
(c) GPT-2 XL
Figure 30: Scaling of Singular Values of Residual Jacobians (Trained).
Triind
I
LSS
phi-1 of
phi-l
Endegemmngg 2h
gemman22
1-26
Mistral-T fostruct vi
nen- Th
Mistral- 78 "B-lovtrie B-lostricc-- vi
pythus- ah
falcon-- "b
falcom- Th "h-istrus ieisstrcct
mptt71 7b
contegome pntegommma-7
gemmit-T th
Heatn 7b-s1
Mistral vil
2.th
gomuaa I-th
Mata- Lecta-llant Liama's an
CodeLlaama Lama (it he
Grear 2:1 ah
Llama- 13tb hf
Codet Codetinma 34b 34b-1f
mip--306
falcon falcom-le dil--metrrge
fatcon 4th
Mota Llame---- 2-70-
Codel Lama- Limmn:
CadeLJama- 7h
Codel Jame Lib
Figure 31: LSS sorted by LLM parameters.
34

---

Published as a conference paper at ICLR 2025
BOODX
60
10
Tratntng Thkaen
Figure 32: Plotting the number of training tokens against its LLM Huggingface Benchmark score.
pou
100
Aigemmnt
Posttional Encoci ng
hellewngg
muttigg ja Wragannde
Positional encoding and Alignment
(b) Prompt lengths
Figure 33: Other plots.
35


Reviews:
[{"review_id": "cupyM21bFS", "reviewer": "Reviewer_bxkM", "summary": "This paper presents an insightful exploration into the internal dynamics of Large Language Models (LLMs). The authors propose a novel framework to evaluate the coupling of transformer blocks through their Jacobian matrices, offering a structured perspective on their inter-token and cross-layer relationships. The study concludes that this coupling is positively correlated with model generalization performance and appears to be a more significant factor than other hyperparameters such as parameter budget and model depth.", "questions": "See weaknesses section above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of using Jacobians to study transformer block coupling provides a fresh perspective on understanding the internal mechanics of LLMs.\n\n- The authors conduct an extensive empirical analysis across a variety of LLMs, lending credence to their hypothesis regarding the significance of transformer block coupling.\n\n- The paper is well-organized, with clear definitions and logical flow, making it easy to follow the complex concepts and their implications.", "weaknesses": "- The most concern lies in the strength of the generalization claims made by the authors. The paper presents empirical evidence suggesting a strong correlation between transformer block coupling and model performance on Open LLM LeaderBoard. However, it seems that some new models (e.g., LLaMA 3, Phi-2) which involve more pertaining steps (tokens) may have a more significant increase in performance than coupling. Revisiting the Figure 2, the coupling seems to emerge at certain training steps and then remain plain. So if the correlation only holds on a limited scope of pretraining steps?\n\n- The model performance is evaluated on Open LLM LeaderBoard. However, for some reasoning tasks, there may be some emergent capabilities [1] that only larger models have. In such settings, is the correlation between coupling and model performance still holding?\n\n[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xUF8j06KPn", "reviewer": "Reviewer_n94A", "summary": "the paper analyzes and correlates jacobian singular value decompositions cross tokens and cross layer-depths. as a result, they define a 'coupling metric', and show that high coupling apparently correlates to high benchmark values.\nas such some intel on the inner processings of llm's is generated but the interpretation is widely left to the user.", "questions": "1) what is the value of fig 6/9?\n2) what are your computational costs?\n3) anything surprisingly found in the jacobians?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper offers a new perspective on by coupling training dynamics with model performance. \n- the method demonstrates somewhat robust as shown with the regression fit\n- potentially high impact on model architecture/ training and diagnosis (if better understood)", "weaknesses": "the interpretation of this metric is vastly unclear. it is kinda clear that model training dynamics converge over training steps, in particular with common decaying lr-schedulers applied. so i wonder rather if one can interpret the score as a 'model training convergence rate' rather then performance-metric. falcon-7b (the entire model cluster > -.7 in fig1) demonstrates that high-coupling does not necessarily correlate with openllm-benchmarks afterall, tracing the activations/ gradients already shows such a convergence as well. afterall, on a converged model, activations only nuanced shift between layers. \n\nsimilarly i'm unsure how to interpret fig 6 etc.\ncomputational costs of the methods are also not discussed (?).\n\noverall i appreciate the solid work and do think there is merit in this methodology per se. however more focus should be made to practical implications of this rather abstract metric, and rigorously discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0lEVvIXBm", "reviewer": "Reviewer_UDjb", "summary": "This paper studies the inference dynamics in LLMs, by looking at several properties of the \"trajectories\" that tokens representations form as they evolve through the network. It is found that the linearized approximations of different blocks in a trained transformer tend to align with each other (in the sense of their top singular vectors being aligned), and that the magnitude of this alignment correlates with overall model performance. Several other properties of the trajectories are also discussed.", "questions": "Figure 3: The LSS is defined as $\\frac{L}{\\lVert \\tilde{x}^l_{i}-\\tilde{x}^0_{i} \\rVert}$ -- clearly a ratio of two positive numbers. How can it be that LSS values reported in Figure 3 are negative?. The authors themselves mention that \"Note that $\\text{LSS}\\geq1$\" (in Line 216) which is inconsistent with Fig. 3\n\nFigures 4 and 5: Readability will be greatly improved if some indications of the relevant \"axes\" is added to the figure itself (Layer/Time in Fig 4/5; embedding dimension on one of the panels; ). Since the different models basically exhibit \"identical\" results (at least qualitatively), I will advise the authors to keep just one column (one model) for each Fig in the main text, and use the freed space to better visual explanation (other models can be moved to Supplementary).\nEven more importantly: a scale/colorbar should be included. Currently it's not even clear if different sub-panels have the same scale or no. The choice to present absolute values is also a bit odd and not well-motivated.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper offers an intriguing observation, expanding on previous works that have reported similar phenomena in resnets. It makes a strong use of the large number of openly available LLMs to study hidden representations. The observation that the \"coupling\" performance correlates with model performance is of potential interest to the community and could encourage further research.", "weaknesses": "The paper seems to be somewhat missing a more coherent organization and message. In particular:\n- What is the motivation to consider the additional metrics of \"Linearity\" (Section 3.2) and \"Layer-wise exponential growth\" (Sec 3.3)? Can these be related, theoretically or empirically, to the \"coupling\" phenomena that is the main focus of the paper? (There's a small attempt to answer that in Section 6.2, I don't think it is very satisfying -- see below).\n- The coupling phenomena is presented as an \"interesting observation\", but the paper doesn't offer much more than that, and so the reader is left somewhat unsatisfied. I think it would significantly improve the manuscript if the authors would study this phenomena in a more \"functional\" way -- is this a uniform behavior for all types of tasks/prompts, or are there some variations that we can understand? Is coupling \"important\" to either learning or inference, or is it an epiphenomena?\n- Another possible way of making the findings more meaningful is a more mechanistic understanding of the phenomena (for example, does it happen in small models, trained on synthetic data? and so on; although this might be further away from the paper's overall approach).\n\nOther than that, there are some \"local\" issues:\n\nLine 35: The explanation for what a \"discrete non-linear dynamical system\" is, is pretty confusing. Almost every term seems to be at least inaccurate:\n- \"The term discrete reflects the network finite depth\" (line 38): the term discrete is typically referring to the fact that *time* is discrete in these models, and that the dynamics are described in terms of finite differences rather than rate of change (as would be in a differential equation). It would still be \"discrete\" even if the network depth would be infinite (i.e., in discrete-time RNNs). The converse is also not true, a continuous-time dynamical system can be evaluated/evolved for finite time.\n- \"coupled refers to the interdependent token trajectories enabled by self-attention\": this is inaccurate, the dynamics will be coupled even in a simple MLP (in fact even in a linear network -- consider the dynamics $\\mathbf{x}^{(t+1)}=\\mathbf{A}\\mathbf{x}^{(t)}$ : for a non-diagonal $\\mathbf{A}$, the dynamics is coupled because there are interactions between $x_i$ and $x_j$ through $\\mathbf{A}$.)\n- \"dynamical is due to the residual connections spanning various layers\": this is also an odd statement. the term \"dynamics\" is typically understood simply to refer to the fact that the state is changing with time. Consider the dynamical system from the previous bullet (as an example for dynamical system without the residual connections).\nIf the authors find it crucial to use non-standard interpretations of standard terms, they should at least note that, and hopefully provide a reasoning for their choice.\n\n\n\nFigure 2 and section 5.3: The fact that the rise in \"coupling\" is so fast seems like a strong indication that the correlation between coupling and performance (as shown in Fig 1) is due to some other variables/confounders, and that coupling in itself has no \"causal\" importance. The reason is that, presumably, performance decreases gradually and doesn't plateau after 10-20K training steps, like coupling. It could still be the case that coupling early-on facilitates further training (even if not a necessary component of inference in a trained model). In that sense, the authors statement  \"this finding suggests that the development of training methods to amplify coupling across transformer blocks may lead to favourable model performance\" (Line 431) seems to be a strong claim, as the authors haven't demonstrated any evidence for a causal relationship between coupling and performance.\n\n\nSection 6.2:\nI find the entire explanation rather confusing.\nThe linearization of $F^l$ in and by itself doesn't mean one can approximate the entire dynamical system as a linear dynamical system. The system is linearized *at* the particular input $x$, so we can't read out much about the dynamics of the same input -- one has to introduce a perturbation in $x$ and *then* use the linearized dynamics to study the evolution of this perturbation (and while it's not clear how to perturb input tokens, it is easy to perturb their embeddings in the first layer).\nI would also suggest to the authors to re-write their A.4 appendix in terms of a discrete dynamics (the same arguments can be made, but will be more relatable to the actual content/method of the paper)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79T2gQbfTq", "reviewer": "Reviewer_8ECx", "summary": "The paper proposes Coupling, a metric of a Transformer model.\nIt computes the similarity of two Jacobians, followed by a normalization; each is the sum of non-residual terms by Attention and FFN.\nThe main claim of this paper is that the Couping metric of a model, among a set of public pre-trained LLMs, is correlated with the average performance of a set of downstream tasks from the HuggingFace Open LLM Leaderboard (R^2 value of 0.7214).\nAdditionally, the paper explores the properties of the Transformer models in terms of the line-shape score (LSS), defined by Gai & Zhang (2021), and the exponential spacing (expodistance) of the hidden trajectories.", "questions": "In summary, what can we concretely learn from the result? If we don't consider the quality, it's not very difficult to develop metrics that show a correlation but are ultimately not useful. Playing devil's advocate for a moment, for example, one of the most boring metrics could be the performance of a single downstream task like GSM8K. It may be correlated with the performance on the LLM Leaderboard. But, of course, this finding is not helpful. Compared with this or other candidates with such correlations, how can we claim that the Coupling metric is more interesting, helpful, or convincing?\n\nThe main claim may be based on Figure 1. But, the result looks very consolidated. Can we believe the presence of the correlation over various models or a limited number of models? For example, what happens if you only examine models that score above 50? Is there still a strong correlation, or does the relationship break down at higher performance levels? For example, if one observes phi-1 and phi-2, the Coupling rule may be broken because they may have similar settings, similar Couplings, and very different performances.\n\nI'm not sure why linearity and exponential growth are discussed and considered as a goodness metric. Playing devil's advocate for a moment again, if we make Transformer blocks, which just scale their inputs by large positive constants, it seems to satisfy the linearity and exponential growth but would fail to achieve the high performance as they the Transformer is just a scaler. Could you provide more discussion and reasons for choosing or designing the metrics for analysis?\n\nFigure 3 (c). Does this show that Linearity disappears at the end of training?\n\nFigure 6. MPT -> Mistral?\n\nCan we see Figure 6-style results of untrained models?\n\nFigure 8. Can we see the results of untrained models?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "To the best of my knowledge, this work is the first to explore the Coupling measures and indicate a correlation between them and the LLM performance on downstream tasks.\nIt also analyzes linearity in intermediate embeddings over the depth and exponential growth of the embedding norms.", "weaknesses": "The correlation between the Coupling metric and performance may not be a helpful or exciting finding. First of all, the paper does not provide any reasonable explanation or hypothesis about (1) why they investigated the Coupling, (2) why the Coupling could reflect performance, and (3) whether the correlation is from a causality (of a direction) or not.\nAdditionally, the paper does not perform intervention experiments, e.g., training a model with a regularization of increasing the Coupling and demonstrating the improvement. In total, it is difficult to conclude that this finding is helpful for the community so far. There are many experimental results, but they are not always well-organized and explained. I would like to know the real contributions of the results.\n\nThe correlation between the Coupling metric and performance might be very consistent. It may disappear in trends of recent high-performance models.\n\nThe contribution of the analysis of linearity and exponential growth is unclear. In my understanding, the analysis is conducted independently from the Coupling. Results are not specific to the models of interest and the Coupling. Many things might be obvious from Li & Papyan (2023). It would be great to add connections between them and Coupling (and the performance, if possible).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)