Lines of Thought in Large Language Models
Authors: Raphaël Sarfati
Toni J.B. Liu
Nicolas Boulle
Christopher Earls
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
latent space
token trajectories
interpretability
transformer
Primary area: interpretability and explainable AI
TL;DR: We study token trajectories in an LLM latent space and find that they cluster along a low-dimensional subspace.

Abstract: Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

Published as a conference paper at ICLR 2025
LINES OF THOUGHT IN LARGE LANGUAGE MODELS
Raphaël Sarfati
Toni J.B. Liu Department of Physics
School of Civil and Environmental Engineering Cornell University, USA
Cornell University, USA toni Moni.lii@corner edd
raphael swrratrriiii @ edu
Nicolas Boullé
Christopher Earls
Department of Mathematics Imperial College London, UK
Center for Applied Mathematics School of Civil and Environmental Engineering Cornell University, USA
n. b.boulll le@ imperial wallaca acc.ur uk
earl sê cornell........
ABSTRACT
Large Language Models achieve next-token prediction by transporting a vector- ized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajecto- ries realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim characterize the statistical proper- ties of ensembles of these 'lines of thought." We observe that independent tra- jectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters ex- tracted from data. We find remarkable that the vast complexity of such large models be reduced to much simpler form, and reflect implications. Code for trajectory generation, visualization, and analysis is available on Github at httpps://giit thubbbb///tuuubu.. cww. wwomrrapsat.. Weleenn--ee -ttooghht
INTRODUCTION
How does large language model (LLM) think? In other words, how does it abstract the prompt "Once upon time, facetious" to suggest adding, e.g., "chatbor", and, by repeating the opera- generate respectable fairy tale à la Perrault? What know is by design. A piece of text is mapped into a set of high-dimensional vectors, which are then transported across their embedding (latent) space through successive transformer layers (Vaswani et al. 2017), each allegedly distilling different syntactic, semantic, informational, contextual aspects of the input (Va- leriani et al. 2023 Song & Zhong, 2024). The final position is then projected onto an embedded vocabulary to create a probability distribution about what the next word should be. Why these vec- tors land where they do eludes human comprehension due to the concomitant astronomical numbers of arithmetic operations which, taken individually, do nothing, but collectively confer the emergent ability of language.
tion, continue
Our inability understand the inner workings of LLMs is problematic and, perhaps, worrisome. While LLMS are useful to write college essays or assist with filing tax returns, they are also often capricious, disobedient, and hallucinatory (Sharma et al. 2023 Zhang al. 2023). That's because, unlike traditional 'if-then' algorithms, instructions have been only loosely, abstractly, encoded the structure of the LLM through machine learning, that is, without human intervention.
In return. language models, trained primarily on textual data to generate language, have demon- strated curious abilities in many other domains (in-context learning), such as extrapolating time series (Gruver et al.) 2024 Liu et al.] 2024), writing music (Zhou et al., (024)) or playing chess (Ru- oss et al. 2024). Such emergent, but unpredicted, capabilities lead questions about what other abilities LLMs may possess. For these reasons, current research is attempting to break down internal processes to make LLMs more interpretable Recent studies have notably revealed some aspects of
And, eventually, more reliable and predictable.

---

Published as a conference paper at ICLR 2025
(a)
(b)
400
200
13g
-500
600 1000
C239 200
e35
Figure 1: (a) Lines of thought (blue red) for an ensemble of 1000 pseudo-sentences of 50 tokens each, projected along the first 3 singular vectors after the last layer (f 24). They appear to form tight bundle, with limited variability around common average path. (b) Representation of the low-dimensional, abboonshaped manifold in S (projected along 3 Cartesian coordinates). Positions are plotted for = 12 (green) to / (yellow).
the self-attention mechanism Vig 2019 patterns of neuron activation
Bricken al. 2023 Tem-
pleton et al. 2024). signatures of world models
? Gurnee & Tegmark
2023 Marks & Tegmark
2023), geometrical relationships between concepts (Jiang et al. 20224. or proposed mathematical models of transformers (Geshkovski et al. 2024).
This work introduces an alternative approach inspired by physics, treating LLM complex dynamical system. We investigate which large-scale, ensemble properties can be inferred experi- mentally without concern for the 'microscopic' details Specifically, we are interested in the tra- jectories, or 'lines of thought' (Lo'T), that embedded tokens realize in the latent space when passing through successive transformer layers (Aubry et al., 2024). By splitting a large input text into N- token sequences, we study LoT ensemble properties to shed light on the internal, average processes that characterize transformer transport.
We find that, even though transformer layers perform 106 10 individual computations, the re- sulting trajectories can be described with far fewer parameters. In particular, we first identify low-dimensional manifold that explains most of LoT transport (see Fig. []. Then, demonstrate that trajectorics can be well approximated by an average linear transformation, whose parameters extracted from ensemble properties, along with random component with well characterized statistics. Eventually, this allows us to describe trajectories as a kind of diffusive process, with a linear drift and a modified stochastic component.
Main contributions.
1. We provide framework to discover low-dimensional structures an LLM's latent space.
2. We find that token trajectories cluster on non-Euclidean, low-dimensional manifold.
3. We introduce a stochastic model to describe trajectory ensembles with few parameters and extend them continuous paths.
"orldd models refers to evidence of (abstract) internal representations which allow L.LMs an apparent understanding of patterns, relationships, and other complex concepts. Such semantic syntactic relationships, architecture specificities, etc.
2

---

Published as a conference paper at ICLR 2025
2 METHODS
This section describes our algorithm for generating and analyzing ensemble of tokens trajectories in the latent space of LLMs. Our code is provided in the corresponding Github repository (Sarfati J.
Language models. We rely primarily on the 355M-paramette parameter ('medium) version of the GPT- model (Radford et al.] 2019). presents the core architecture of ancestral (circa 2019) LLMs: [ It consists of Nr = 24 transformer layers operating in latent 1024. The vocabulary
transformer-based decoder-only!
space S of dimension normalization (Ba et al.) 2016 is applied to the last latent space position before projection onto V to form the logits. (This final normalization is not included in our trajectories.) We later extend our analysis to the Llama 2 7B (Touvron et al. 2023), Mistral 7B v0.1 (Jiang al. 2023). and small Llama 3.2 models (1B and 3B) (MetaAl 2024).
contains Ny 50257 tokens. layer
Input ensembles. We study statistical properties of trajectory ensembles obtained by passing set of input prompts through GPT-2. We generate inputs by tokenizing (Wolf et al., 2020) a large text and then chopping it into "pseudo-sentences" i.c., chunks of fixed number of tokens Nu (see Algorithm I). Unless otherwise noted, N& = 50. These non-overlapping chunks are consistent in terms of token cardinality, and possess the structure of language, but have various meanings and endings (see Appendix [A.I). The main corpus in this study comes from Henry David Thoreau's s Walden, obtained from the Gutenberg Project (Project Gutenberg 2024 N, ~ 3000-14000 pseudo-sentences.
We typically use a set of
Trajectory collection. We form trajectories by collecting the successive vector outputs, within the latent space, after each transformer layer (hidden states). For conciseness, we identify layer number with a notional 'time', t. Even though all embedded tokens prompt voyage across the latent space, only the embedding corresponding onto V) for next-token inference. Hence, here, we only consider the trajectory of this last (or 'pilot') token. The trajectory M' of sentence k's pilot is the sequence of 24 successive time positions (ær (1), (2), (24)). concatenated as a column matrix (Algorithm) Db.
the last token form the logits (by projection
Latent space bases. The latent space is spanned by the Cartesian basis {ci).=....D (the orthogonal set of one-hot unit vectors with a 1 in ithh position, 0 elsewhere). Additionally, we will often refer the bases (t) {u, Ji=I... formed by the left-singular vectors of the singular value decomposition (SVD) of the D x N₂ matrix after layer t: M = UEVT, with M = x (t) Vectors are organized according to their corresponding singular values, a,, in descending order. Note that because trajectory clusters evolve over time there 24 distinct bases.
RESULTS
We present and characterize results pertaining to ensembles of trajectories as they travel within the latent space S.
3.1 LINES OF THOUGHT CLUSTER ALONG SIMILAR PATHWAYS
We endeavor visualize and characterize the trajectories of pilot tokens in the latent space S. The high dimensionality makes it non-trivial. Which projections should we consider?
'Compared to current state-of-the-art models, GPT-2 medium rather unsophisticated. Nevertheless, works. It produces cogent text that addresses the input prompt. Hence, we consider the model already contains the essence of modern LLMs and leverage its agility and transparency for scientific insight.
(LLayerNorm +) Self-attention then (LayerNorm +) Feed-forward, with skip connections around both. 'The idea of using literary piece probe statisties of language was investigated by Markow back in
1913 (Makoov 2006).
3

---

Published as a conference paper at ICLR 2025
Algorithm Trajectory generation in transformer-based model
Input: Large text: "It was the best of times, the worst of times, the age...""
2: Tokenize text into token sequence: [1027, 374, 263, 1267, 287, 1662, 12, .2.
3:
Split token sequence into n-token pseudo-sentences,
374,
$2 [1267, 287, 1662].
4: for each pseudo-sentence 5; do
5: Semantic embedding:
Es = e((1))7),)(374)) v(263)] for $1
6:
E Es + Ep (add positional embeddings P}
7:
for = + 23 do
8:
E++1 TransformerlLayer, (E') (update embeddings through transformer layer)
9:
att + 1) - Ex (extract last token representation)
10:
M. = an 1) {save trajectory array)
end for
12:
end for Output: Final embeddings æ(t + 1) for all pseudo-sentences
13:
There is no reason priori for the Cartesian axes, e,, to align with any meaningful directions, so we seek relevant alternative bases informed by the data. Naturally, we consider the bases ((()) formed by the singular vectors of pilots ensemble after each layer
Using these bases aligned with the data's intrinsic directions, we observe in Fig. Dh that trajectories tend to cluster together, instead of producing an isotropic and homogeneous filling of S. Indeed, LoT' for different, independent pseudo-sentences follow common path (forming bundles), aug- mented by individual variability. Specifically, there exist directions with significant displacement relative to the spread (mean over standard deviation); and positions at different times form distinct clusters, as shown in Fig. A
3.2 LINES OF THOUGHT FOLLOW A LOW-DIMENSIONAL MANIFOLD
We remark Fig. 2h that the intrinsic bases (1)) rotate only slightly across successive timepoints Besides, Fig. 2b shows that the corresponding singular values decay quickly over several orders of magnitude. Both suggest that LoTs may described by low-dimensional curved subspace.
But how many dimensions are relevant? Singular values relate ensemble variance along their corresponding directions. Since the embedding space is high-dimensional, however, the curse of dimensionality looms, hence the significance of Euclidean distances crumbles. To cireumvent this limitation, we consider a more practical metric: how close to the original output distribution on the vocabulary does reduction in dimensionality get us?
To investigate this question, we express token positions ax)) in the singular vector basis ((()):
K
(1)
(0) (0)
u
where the 1 organized by descending order of their corresponding singular values. By default K = D. and the true output distribution p is obtained. Now, examine what happens when, instead of passing the full basis set, we truncate it, after each layer, to keep only the first < D principal components. We compare the resulting output distribution, Pk to the true distribution p using KL divergence DKL k (lp)). ). In Fig. 22. we see that DRL. decreases very slowly with decreasing K, up to about K₂ = 256. At that point, DKL is only about 10% of its uncorrelated baseline value, implying that most of the true distribution is recovered when keeping only about K, = 256, or 25%, of the principal components. In other words, for the purpose of next-token prediction, Lo''s quasi-556-dimensiona.l
4

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
10
90
10*
S.Val. magnitude
10a
10a
109
h
10a
10a
10a
10a
256
1024
S.Val. rank
K
Figure 2: (a) Angle between the first 4 singular vectors at (11, f2), arccos( (41) u, (fa)), for i (1,2,3, 43 (top-left, top-right, bottom-left, bottom-right, respectively). (b) Singular values for I,..., 24 (blue red). Clusters stretch more and more after each layer. The leading singular values, ai(t), have been omitted for clarity. (c) Average (over all trajectories) KL divergence between reduced dimensionality trajectories ouiput and true output distributions, the dimensionality K is increased. The red dashes line shows the average KL divergence for output distributions from unrelated inputs
(baseline for dissimilar distributions).
these principal directions remained constant at each layer, this would imply that 75% of the latent space could be discarded with no consequence. This seems unrealistic. In fact, the principal direc- tions rotate slightly over time, as displayed in Fig. Zp. Eventually, between / = and = 24, the full Cartesian basis & is necessary to express the first singular directions. Thus, we conclude that lines of thoughts evolve on low-dimensional curved manifold of about 256 dimensions, that is contained within the full latent space (Fig. [p))
3.3 LINEAR APPROXIMATION OF TRAJECTORIES
Examination of the singular vectors and values at each time step indicates that LoT bundles rotate and stretch smoothly after passing through each layer (Fig. 2). This suggests that token trajectories could be approximated by the linear transformations described by the ensemble, and extrapolated accordingly, from initial time i later time + Evidently, it is improbable that transformer layer could be replaced by mere linear transformation. We rather hypothesize that, in addition to this deterministic average path, a token's location after layer + 7 will depart from its linear approximation from i by an unknown component wit [ We propose the following model:
x(++ R(t T), x=t) + w(t,
where xet) is the pilot token's position in the Cartesian basis, and R, A are rotation (orthonor- mal) and stretch (diagonal) matrices, respectively. Eq. ( formalizes the idea that, to approximate aell + 7), given ar)), we first project in the ensemble intrinsic basis at / (R x:), then stretch the coordinates by the amount given by A, and finally rotate according to how much the singular direc-
tions have rotated between and t т. R(t r) (see also Fig. A2 in Appendix B). Consequently. we can express these matrices as a function the set of singular vectors (U) and values (E):
R(t) (t), A 7) diag(o, +T/aa,(t)) E(t T)
Fig. 3 shows the close agreement, the ensemble level, between the true and extrapolated positions.
This is confirmed by the observation that the two sets are not separable with a trained linear clas- sifier, including at large t (see Appendix). Eq. (T) is merely linear approximation it is similar to assuming that LoT clusters deform like an elastic solid, where each point maintains the same vicinity, illustrated in Fig. A2 The actual coordinates ought to include additional random component w(1, r), which a priori depends on both and
We emphasize that prompt trajectories are completely deterministic; the stochastic component introduced in the model accounts for the fact that we perform a linear extrapolation based only on token's position at certain time, which unsurprisingly deviates from the true position obtained from processing the full prompt with transformer layers.
5

---

Published as a conference paper at ICLR 2025
1
==
-16
15
=11
1=12
Figure 3: Extrapolated token positions (A) (blue) from t = {12, 14, 16, 14, 18)
+t+
1,...,20 21), compared their true positions x (gray), projected in the (u (0)) planes.
Is possible to express in probabilistic terms? We consider the empirical residuals 8xx(t, r) xtt + r) a(t, r)
between true positions æ and linear approximations ē(t, -) = R(t + (t,, R(1) x()). investigate the distributions and correlations of ōz(tt, across layer combinations (t, t + r).
From the data, Fig. 4 shows that, for all (t, +.)) E fl..... .......23] x (t + 1,..., 24), the ensemble of (t r) has the tollowing characteristies: 1) it is Gaussian, 2) with zero mean, 3) and variance scaling as exp(t + r). In addition, Fig. A3 shows that the distribution is isotropic, with no evidence of spatial cross-correlations Hence, propose:
wt, ~ N(O, ae (ct+)))
(2)
i.e., each coordinate w, of w a Gaussian random variable with mean zero and variance de (t+r). Linear fitting of the logarithm of the variance yields = 0.64 and X = 0.18. Even though this formulation ignores some variability across times and dimensions, it is useful minimal modelling form to describe the ensemble dynamics with as few parameters as possible.
3.4 LANGEVIN DYNAMICS FOR CONTINUOUS TIME TRAJECTORIES
Just like the true positions ær (t), matrices R and A are known (empirically) only for integers values Can extend Eq. interpolate R and A between their known values (Absil et al. (2008.. Specifically, R(t) remains orthogonal and rotates from its endpoints; singular values be interpolated by spline function.
of R
continuous time parameter
[1, 2422 Indeed, it is possible to
That is, after each layer.

---

Published as a conference paper at ICLR 2025
In return, this allows interpolate trajectories between transformer layers Thus, extend Eq. (T) to a continuous time variable t, and write in infinitesimal terms the Langevin equation for the dynamiis:
dx((t) (()) (())((t) z(t) dt val dw(t).
(3)
where S = diag (d./o;) and dl(()) is differential of Wiener process (Pavliotis, 2014p. We defer the mathematical derivation to Appendix A.2 This equation artificially extends LoT's to continuous paths across S. It provides stochastic approximation to any token's 's trajectory, all times t.
0.02
0.54
9.06
0.08
0.1
1.5
2.5
0.
15
(a)
(C)
(1///))
(lop (o))
(())
13
17.
2
10
14
18
22
10
18
22
10
18
22
Figure 4: Statistics of (1,(1, T): mean variance o², excess kurtosis Brackets (...) denote
average over directions e; (see Fig. [4.[[4]fr details). (a) For all (4, (1,++),,14 = 0 (that is, u/o a & 1). (b) log(o²) increases linearly in time, only depends 7 (c) The excess kurtosis (kurtosis minus 3) remains close to O, indicating Gaussianity (except in early layers).
3.5 PLAKER-PLANCK FORMULATION
Eq. stochastic differential equation (SDE) describing individual trajectories with random
component. Since the noise distribution is well characterized (see Eq. ) we can write an equiva- lent formulation for the deterministic evolution of the probability density P(x,,) of tokens
time Pavlioois 2014). The Fokker-Planck equation associated to Eq. (3) reads:
a t) -v Ot
[
1 + RŠR x: x(( t) + aλe (2P,ztt).
(4)
This equation captures trajectory ensemble dynamics in much simpler form, and with far fewer parameters, than the computation actually performed by the transformer stack on the fully embedded prompt. The price paid for this simplification is probabilistic, rather than deterministie, path for LoTs. We now test our model and assess the extent and limitations our results.
TESTING AND VALIDATION
4.1 SIMULATIONS OF THE STOCHASTIC MODEL
We test our continuous-time model described above. Due to the high dimensionality of the space,
numerical integration of the Fokker-Plane Planck equation, Eq. (4) computationally prohibitive. stead, we simulate ensemble of trajectories based on the Langevin formulation. Eq. 3 The
technical details are provided in Appendix
The results presented in Fig. show that the simulated ensembles closely reproduce the ground truth of true trajectory distributions. We must note that Eqs. (3) and { are not path-independent;
These interpolated positions do not hold any interpretive value, but may be insightful for mathematical purposes.
Also known Kolmogorov forward equation.

---

Published as a conference paper at ICLR 2025
++r-13 (+1=13
1+1=11
11: 111
11 w
11
Figure 5: Simulated distributions for t 12, t (u1, U2) plane (top row) and the (u3, u1) plane (bottom row). Distributions have been approximated from ensemble trajectories, 10 trajectories for each initial point. Background lines indicate true distributions, thin lines on top indicate simulations.
{12, 13, 14, 15, 16), projected the
therefore, their solution depend the value of R(t). S(t) at all time t. Since there is 'true' value for the matrices in betweeeen layers, the output of numerical integration naturally depends on the interpolation scheme. Hence, discrepancies are to be expected.
4.2
NULL TESTING
We now examine trajectory patterns for non-language inputs and untrained models.
4.2.1 GIBBERISH
We generate non-language ('gibberish") sseuud-sentences by assembling N-token sequences of ran- dom tokens in the vocabulary, and pass them as input to GPT-2. The resulting trajectories also cluster around a path similar to that of language. However, the two ensembles, language and gibberish, are linearly separable at all layers (see Fig. A5 in Appendix B.6), indicating that they travel on two distinct, yet adjacent, manifolds.
4.2.2 UNTRAINED & ABLATED MODELS
We compare previous observations with the null baseline of untrained model.
First, we collect trajectories of the Walden ensemble passing through reinitialized version of GPT- (the weights have been reset to a random seed). We observe that while LoT's get transported away from their starting point, the trajectories follow straight, quasi-parallel paths, maintaining their vicinity (see Fig. A5 Furthermore, the model of Eqs. ( and (2) does not hold: Fig. A6 shows that the variance of dæ does not follow the exp(t 7) scaling, and the distributions far from Gaussian.
Next, consider ablated model, where only layers 13 to 24 have been reinitialized. When reaching the untrained layers, the trajectories stop and merely diffuse about their t = 12 location (Fig. A5).
In conclusion, upon training, the weights evolve to constitute a specific type of transport the latent space.
8

---

Published as a conference paper at ICLR 2025
4.3 RESULTS WITH OTHER MODELS
We repeat the same approach with a set of larger and more recent LLMs. We collect the trajectories of the Walden ensemble in their respective latent spaces.
Llama 2 7B. We first investigate the Llama 2 7B model (Touvron et al. 2023 Remarkably, the pattern of GPT-2 repeats. Token positions stretch using the singular vectors and values of the ensemble. The residuals are distributed as those of GPT-2, with wt, ~ ( (t*(t))), see Fig. A7 The values for the parameters a and X, however, differ from those of GPT-2 (here,
+ T can be extrapolated from & by rotation and
5.4,
0.27).
Mistral 7B. Trajectories across the Mistral 7B (v0.1) model (Jiang et al. 2023 also follow the same pattern (Fig. A8). We note, however, that Eq. 4 only holds up until layer 31. It seems as though the last layer is misaligned with the rest of the trajectories, as linear extrapolation produces an error that is much larger than expected.
Llama 3.2. The last layer anomaly is also apparent for Llama 3.2 IE 13 both in the mean and variance of (t, 16) (see Fig. A99. However, the rest of the trajectories follows Eq. (I. The same pattern is observed for Llama 3.2 3B
14 in Fig. A10
is noteworthy that these three recent models feature the same anomaly the last layer. The reason is not immediately evident, and perhaps worth investigating further. In addition, we remark that all models also show deviations from predicted statistics across the very first layers (top-left corners). We conjecture that these anomalies might be an effect of re-alignment or fine-tuning, as the first and last layers are the most exposed to perturbations which might not propagate deep into the stack.
CONCLUSION
Summary. This work began with the prospect of visualizing token trajectories in their embedding space S. The space is not only high-dimensional, but also isotropic: all coordinates are a priori equivalent Hence, sought directions and subspaces of particular significance in shaping token trajectories
To some kind of "eigenvectors' of the transformer stack.
Instead of spreading chaotically, lines of thought travel along www-dimeniionl manifold. We used this pathway to extrapolate token trajectories from a known position at / to a later time, based on the geometry of the ensemble. Individual trajectories deviate from this average path by a random amount with well- well-defined statistics. Consequently, we could interpolate token dynamics to continuous time in the form of a stochastic differential equation, Eq. ( The same ensemble behavior holds for various transformer-basee pre-trained LLMs, but collapses for untrained (reinitialized) ones.
This approach aims to extract important features of language model internal computation. Unlike much of prior research on interpretability. it is agnostic to the syntactic and semantic aspects of inputs and outputs. We also proposed geometrical interpretations of ensemble properties which avoid relying on euclidean metrics, as they become meaningless in high-dimensional spaces.
Limitations. This method is limited to open- spercsource models, as requires extracting hidden states; fine-tuned, heavily re-aligned models might exhibit different patterns. In addition, it would be com- pelling to connect the latent space with the space of output distributions, for example by investigating the relative arrangement of final positions with respect embedded vocabulary. However, this is complicated by the last layer normalization which typically precedes projection onto the vocabulary. This normalization has computational benefits, but its mathematical handling is cumbersome: it is highly non-linear as it involves the mean and standard deviation of the input vector.
Decoderrooly, 32 layers, 4096 dimensions: released July 2023 by Meta
12 "Decoder-only. 32 layers, 4096 dimensions: released September 2023 by Mistral AL.
Decodee-ooly, 16 layers, 2048 dimensions; released September 2024 by Meta AI.
" Decoderronly, 28 layers, 3072 dimensions; released September 2024 by Meta AI.
15 "nnlike other types of datasets where different dimensions might have well-defined meaning, for example: temporature, pressure, wind speed, etc.
Andd hence defining next-token distribution outputs

---

Published as a conference paper at ICLR 2025
Implications. Just like molecules in gas birds in flock, the complex system formed by billions of artificial neurons in interaction exhibits some simple, macroscopic properties. It can be described by ensemble statistics with well defined random component. Previously, (Aubry et al. (2024) had also uncovered specific dynamical features, notably token alignment, in transformer stacks of wide variety of trained models.
Patterns are explanatory. Our concern here has been primarily to discover some the mechanisms implicitly encoded in the weights of trained language models. Yet, there are also concrete and potentially practical implications to our findings.
For interpretability, finding low-dimensional structures is consequential. It is one of the most effi cient ways to break down the inherent complexity of large models into more elementary constituents. Our dynamical system approach reveals a surprising dimensionality reduction of token embeddings. It suggests, notably, that the true "meaning" of embeddings is contained within individual variability (possibly orthogonal to the average pathway collectively followed by all LoTs). This also merely first-order approximation, which could be extended to more complete and precise equations, where the "noise term" " becomes smaller and smaller. Eventually, we anticipate the possibility for hybrid architectures where the deterministic part of trajectories is delegated to a small system of equations, while the variable part, where meaning is encoded, is handled by neural network; potentially with many fewer weights.
Our theoretical model Eqs. 3 and not only reveals low-dimensionality. but also extends token trajectories to continuous paths. In the past, the Neural Ordinary Differential Equation paradigm by Chen et al. (2019) showed that converting a discrete neural network into a continuous dynamical system had many advantages. Notably, it offers opportunities for compression and stability, while pointing towards efficient hybrid architectures. Our paper demonstrates that transformers can also been seen through the lens of dynamical systems, with a similar continuous extension seen in neural ODEs.
Finally, the new methodology that introduced is portable and widely applicable. Incidentally, can also serve as a diagnostic method to highlight intrinsic differences between transformer layers. Fig. A8 to Fig. ATO for example, show significant deviations in the last layer (and to a lesser extent in the early ones). This suggests that these layers achieve different kind of processing than intermediate layers, possibly following fine-tuning and/or re-alignment. It's not immediately obvious to us how these "anomalies" could be detected through different approach.
ACKNOWLEDGMENTS
This work was supported by the SciAI Center, and funded by the Office of Naval Research (ONR), under Grant Numbers 200014-23-1-2729 and 400014-23-1-2716.
REFERENCES
P.-A. Absil, Mahony, and Sepulchre. Optimization AatiiiizaaionnAoaiiiiiiii Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008. ISBN 978-0-691-13229-33
Murdock Aubry, Haoming Meng. Anton Sugolov, and Vardan Papyan. Transformer Alignment in Large Language Models. arXiv preprint arXiv:2407.078100 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: 160::0607.655 2016.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity:: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
-eeeerrcciiuu circuits..... pub/
2023 htt////oosemmnnicccccm features wtw.uurrreeiinnexeeeee wnnei.iixx.////i html
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations, 2019. URL htt.ttsss//axxxv.rr////////
07366
10

---

Published as a conference paper at ICLR 2025
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clus- ters in self-attention dynamics. In Advances in Neural Information Processing Systems, vol- ume 36, 2024.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. In Advances in Neural Information Processing Systems, volume 36, 2024.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.3867, 2024.
Toni JB Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher Earls. LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law. arXiv preprint arXiv:2402.00795, 2024.
A. A. Markov. An example of statistical investigation of the text eugene onegin concerning the connection of samples in chains. Science in Context, 19(4):591-600 2006. doi: 10.1017) 50269889706001074.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint 200p222202..4 2023.
MetaAl. Llama 3.2 model card, https: wtt.://ttmm/mammmaaaaaaaaa 1mwa:/1mmmmmmmmmmmmmmmm
/llob/mai//modee http///oeelllmmmaaaaaaaaa 2 /OODEL _ar____CRRDD__ mdl 2024. Accessed: 2024-09-25.
Grigorios A Pavliotis. Stochastic processes and applications, volume 60. Springer, 2014.
Harshwardhan Praveen, Nicolas Boullé, and Christopher Earls. Principled interpolation of green's functions learned from data. Comput. Methods Appl. Mech. Eng., 409: 15971, 2023.
Project Gutenberg. Project Gutenberg. https: wttp:///wwwwgteeebrrrrrrrrggg ortg//out//////////o//// 2024. Ac- cessed: 2024-09-07.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, and Tim Genewein. Grandmaster-level chess without search. 2024. URL
bittps://azee wwtp//t.////.ooooo.ot..../.....t 2402. 2etp///2222444444
Raphaël Sarfati, Toni J.B. Liu, Nicolas Boullé, and Christopher Earls. Code for Lines of Thoughts in LLMs.
comttuub.com/rreee http//ub..oooorpppra//eeeeeeee of of-thugght
Mrinank Sharma, Meg Tong. Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Tim- othy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy language models. 2023. URL https:::: /ttpp::://axiiii wttpv//rggbbbbb///3.e 13548
Jiajun Song and Yiqiao Zhong. Uncovering hidden geometry in transformers via disentangling position and context, 2024. URL [RRL/ttts://axxiie org/ http://bs//31.0.. 04861
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey. Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL https::////........ bttpss:/ttraassoomeeee ceaerriiiritttts pub/ scal www.--mooommmatiiiiitooo
html

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09228. 2023.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Al- berto Cazzaniga. The geometry of hidden representations of large transformer models, 2023. URL https::/tttttttaae httpa/v.rrggbbb///3//
00294
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, (86))2557-- 2008. URL http: /
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural In- formation Processing Systems, volume 30, 2017.
Jesse Vig. Visualizing attention in transformer-based language representation models, 2019. URL
htttps://arxv..rg//// wttp://p///aa..ooooo 02679
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface transformers: State-of- the-art natural language processing. 2020. URL https: /
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's s song in the ai ocean: A survey on hallucination in large language models, 2023. URL https:// www.vv.rrvvvv//a////// org/ abs 2200/339.0000 01219
Ziya Zhou, Yuhang Wu, Zhiyue Wu, Xinyue Zhang, Ruibin Yuan, Yinghao Ma, Lu Wang, Em manouil Benetos, Wei Xue, and Yike Guo. Can Ilms "reason" in music? evaluation of Ilms' capability of music understanding and generation. 2024. URL htttttss//xxxii.. www.:oorgaaa...... 2407. 21531
12

---

not been made by my townsmen concerning my mode to pardon me if I undertake to answer some of writer, first or last, simple and sincere would fain say something, not so much concerning Brahmins sitting exposed four fires and looking more incredible and astonishing than the scenes which I daily and farming tcols; for these are more easily acquired How many poor immortal soul have I met into the soil for compost. By a seeming fate as Raleigh rhymes it in his sonorous way iil, are too clumsy and tremble too much be preserved only by Published as a conference paper at ICLR 2025
Appendix
A ADDITIONAL METHODS AND DERIVATIONS
A.1 PSELDO-SENTENCES
Random sample of 10-token pseudo-sentences (non-consecutive) extracted from Walden. Similar chunks, but 50 tokens, were passed through GPT2 to form trajectories.
the bloom fruits,
A.2 LANGEVIN EQUATION DERIVATION
Starting from
+t++)) R(t + ((; + w(t,
with 7). = E(1 (() and assuming now that 7 are variables in R. as 7 goes to 0 we can approximate:
R(I+T) R R(I) + +i((
and
-(++ E() + +11(1,,
leading to:
A(t,T) = (3(t)+75(0) E = -1)
Hence:
R( R((+T)((,7) (R(I)+ + - (I++1(0)=1()) R(!)T
I (i(()) + ((()((()(())
given that RR = I and with S(I) = diag (In (t)) and thus S(t) = diag(o./o.).
The variance of the noise term is given by:
var a exp(A(t + 7)) exxp(At) AT))
The increment of variance over time r is:
8[var] atexp(At)r.
This means the noise term can be expressed as:
w r) caxcxp(At)s n
where 7 a vector of standard Gaussian random variables.
Putting everything together:
x<1 T) x01) = (((()) + R())(()(()))) x<(1) + Vadexp(A()T (1))
13

---

Published as a conference paper at ICLR 2025
And finally: dx (t) [(())R(t) ((()(()))(()) + Vadexp(At) dwo(t, +
with (t) Wiener process.
A.3 NUMERICAL. INTEGRATION
Numerical integration of Eq. (3) requires to interpolate the singular vectors and values, and their derivatives, at non-integer times.
Interpolation of (scalar) singular values is straightforward. We polynomial interpola- tion scheme for each value, and compute the corresponding polynomial derivative. This yields (l)/a((() for every coordinate at any time E [1,24], and hence S(t).
Interpolating sets of orthogonal vectors presents significant challenges. A rigorous approach in- volves performing the interpolation within the compact Stiefel manifold, followed by reprojection onto the horizontal space Praveen et al. (2023). However, this method computationally expen- sive and can introduce discontinuities, which are problematic for numerical integration. To address these issues, we used an approximation based on the matrix logarithm, which simplifies the process while maintaining an acceptable level of accuracy. To interpolate between U1 and U2 at t1, 12, we compute the relative rotation matrix R U, U2 and interpolate using
U(t) U1 expx Ium R).
(5)
where (L- (1/((12 f1) and with Inm, expM denoting the matrix logarithm and exponential, respectively This also yields the derivative ( IU Inm R| (t2 - +1). Indeed:
U- d dt exp (a(t) R) Uj In Rexp (t In 60 R.
B SUPPLEMENTARY FIGURES AND SCHEMATICS
B.1 TRAJECTORY CLUSTERING
In Fig. AT we show evidence of trajectory clustering in the latent space. In particular, all pilot tokens get transported away from the origin (or their starting point) by comparable amount, resulting in narrow distributions along the first singular direction. Another signature of clustering is the fact that token positions at different times form distinct clusters, showed by low-dimensional t-SNE representation (van der Maaten & Hinton 2008).
B.2 TRAJECTORY EXTRAPOLATION
In Fig. A2 we provide a schematic to explain the reasoning behind Eq. (.. If the cluster rotated and stretched like solid, the position of point æ' at t' could be inferred exactly from it position æ at f, using the formula outlined. However, unsurprisingly, the token ensemble does not maintain its topology and the points move around the clusters, requiring the stochastic term w injected in Eq. (T).
B.3 SEPARABILITY OF TRUE AND EXTRAPOLATED POSITIONS
To characterize the similarity between the ensemble of true positions at t T, xtt + r), from the evaluate how much the
positions extrapolated from f, =(t, 7) R(t + A) (t. 7) =(()) two sets can be separated with linear classifier. We train a Support Vector Machine Model with a linear kernel for each set of extrapolations (z)(,,)) (70/30 train/test). We then apply the classifier to predict whether points the test set are true or extrapolated. In Table I we report results for the panels corresponding Fig. The accuracy the classifier lies in the 50%-60% range, barely above random guessing (50%).
(A) Σ A and In the inverse function: Iny (exPae (A))
14

---

Published as a conference paper at ICLR 2025
0.32
80
0.08
ESNE Dimension
par
0.02
50
150
250
400
-60
-40
-20
x(3) (1)(()
tSNE Dimension 1
Figure Al: (Left) Distributions along the first singular vector at different times. (Right) Low-
dimensional (t-SNE) visualization of the clustering of tokens, notably across different times. Same color legend.
Lun 12
2: + 221
62
e1
Figure A2: Extrapolation between & and t'. The extrapolated location x' corresponds to the rotated and stretched position of æ. Given that ū Re, 7 R' and R _R. we have ē' R v R' v and thus i7' R'R iz.
15

---

Published as a conference paper at ICLR 2025
Table Accuracy of linear classifier separate a:( + r) and (tt, 7), ) percent.
f ==
12
14
t =
16
18
f 8+ 13
48
t 7 15
51
49
t+ = 17
56
53
47
1+ 19
54
55
46
8+ 21
56
60
61
55
B.4 NOISE STATISTICS
Fig. A3 provides additional details pertaining the distribution of residuals 6 Since they many dimensions and time points, it gives only representative snapshots. It intends to substantiate the results that:
the dæ are Gaussian (Fig. A3AA A);
the variance is exponential in (1 + r), with dependency on / (Fig. A3
all components dx; of óæ have the same distribution (Fig. A3C), i.e., isotropy:
there are no spatial corss-corrl,stion, i.e. (ōxςōιz) 5.j (Dirac function) (Fig. A3 D)
19 (A)
(8)
10
287
30
15
28
13 5x.
10
15
8x.
Figure A3: Statistics of 8x. (A) Empirical PDF of 5x (10, + r). with + 14,11, The
curves appear Gaussian. (B) Variance of da; for 1...8. -. for 4,8, 12, 16 and
(C) Empirical PDF of f1, 14) for i = 1... 1024. The curves are similar for almost all coordinates. (D) Cross correlations of ôx, and òzz
B.5 DETAILS ON NOISE AGGREGATED STATISTICS (FIG.
Fig. A4 explains how the noise plots such as Fig. are created. We use ensemble averages (...) of the absolute values for |pal, |rc] since are interested in the average distances from 0.
B.6 NULL TESTING
Fig. A5 shows the trajectories of language vs gibberish, as well as the linear separability of the two ensemble. also shows trajectories for untrained GPT-2 shell, and model with only the last 12 layers reinitialized.
B.7 RESULTS WITH OTHER MODELS
16

---

Published as a conference paper at ICLR 2025
(t. (t (t,
ensemble (A)
pdf
t+7
^
6r 6x ...5z (x)
01
61
0g
62
dimension (
8x 14)
03
6xx!"
: :
M1024
10024
K1024
(logo?)
{[<))
Figure A4: Schematic to explain the noise figures such as Fig. Each square represents a summary statistics. Specifically, the square at (t, + 7) represents the distribution of (6x (t., t + r) k. with k indexing individual tokens. The dæ along each coordinate form distribution, from which one can extract the corresponding (4i ai, K; (mean, variance, kurtosis). These ID moments are then averaged along all coordinates i ((p))s, etc.), forming the value displayed in the square.
17

---

Published as a conference paper at ICLR 2025
198
0.998
800
-400
0.996
-200
10
400
0.982
800
-600
11000
12
16
20
24
U1
20
10
86
-10
-12
Figure A5: (Top-left) Trajectories of non-language (red) language (black), plotted in the same axes (10-token pseudo-sentences). (Top-right) Accuracy of linear separability between language and non-language for each layer. Obtained by training Perceptron (train/test: 0.7/0.3; 14000 trajectories). (Bottom-left) Trajectories in the untrained GPT-2 model. They are transported in straight lines. (Bittom-right) Trajectories in the mixed model. After being transported by trained layers 1-12, the trajectories stop. Layers 13-24 with random weights do not transport tokens any further.
0.02
(a)
(u/al)
(log(o"))
[K]]
17
21
10
22
2
Figure A6: GPT-2 UNTRAINED.
The averaged excess kurtoses ([sl) fall in the 1-1.5 range, indicating strong -onggaussianity. The variance does not scale solely with f + т.
18

---

Published as a conference paper at ICLR 2025
63
0.
03
0.4
0.5
2
0.2
R44
0.6
(a)
(1//al))
(log' (o²))
(k))
13
17
21
23
2
10
14
18
22
26
30
10
14
18
22
26
10
14
18
22
26
Figure A7: LLAMA 2 7B: noise statistics, dx(t, + = x(t + all Cartesian dimensions, for 1000 trajectories (50-token chunks). (a) Mean over standard deviation. (b) Logarithm of variance. (c) Excess kurtosis (0 means Gaussian).
at, r), averaged (...) over
(a)
(c)
(lop(a"))
(())
5-
9-
13
29
18
Figure A8: MISTRAL 7B v0. The last layer (32) appears have an anomalously large variance.
(a)
(b)
(C)
(u//a())
(log(a")
([x))
Figure A9: LLAMA 3.2 IB. This small model all present an out-of-distribution last layer.
19

---

Published as a conference paper at ICLR 2025
0.02
0.04
9.06
0.08
6.1
-35
1.5
2.5
01
02
0.4
8.6
(C)
(log(o"))
((xl)
5-
17
21
23
10
26
10
26
10
Figure A10: LLAMA 3.2 3B. The last layer anomaly also present.
20


Reviews:
[{"review_id": "GoHVe0EFEw", "reviewer": "Reviewer_AKd3", "summary": "The paper aims to study the statistical properties of what they call *lines of thought*; trajectories traced by the embedded tokens through the latent space while traversing successive transformer layers. The key observation is that independent trajectories cluster along a low-dimensional manifold, and that their paths can be approximated using a simple dynamics model.", "questions": "In addition to the concerns raised in the weaknesses,\n\n- I find the point being made in the paragraph between lines 195-201 very interesting, but I am then immediately confused by the following paragraph. Could you please clarify this?\n\n- Could you please clarify the paragraph between lines 248-251?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- I find the question posed, and the consequent findings very interesting\n\n- I really appreciate the development of a linear approximation to the distribution of trajectories.", "weaknesses": "- One of my biggest gripes with the paper is, while I wanted to be excited about the findings, a recurring question that I had was \"why should I care\"? It is my opinion that the authors should invest in a motivation for why the reader should care about the presented findings. It's not clear to me what the takeaways are, or more concretely, how we can utilize the observation to develop better LLMs for instance.\n\n- A second gripe, which perhaps goes hand-in-hand with my first one, is that I often felt the paper could have used a bit more hand holding, or even been organized better. To make this concrete, in Section 3.1, the findings are presented first and then the methodology employed to find them was presented, which to me was a bit confusing. Section 3.2 starts abruptly with the sentence \"The fast decay of the singular values...\". Section 3.4 is titled \"Langevin Dynamics...\" with no mention of Langevin Dynamics. Furthermore, I often found myself having to reference many different figures in different parts of the paper while reading a single paragraph. All of this made it harder to follow the paper closely.\n\n- (Stylistic nitpicking) I think some of the writing maybe embellishes on details that maybe are not very important e.g. the methodology section starts with what I believe to be details that can be abstracted away. I also think the footnotes are somewhat excessively used.\n\n- I'm not really sure what is meant by \"pilot\" in this context. My understanding is that a pilot is  \"done as an experiment or test before introducing something more widely.\"\n\n- The authors claim that the pseudo-sentences are \"independent\" (line 171) which is not immediately clear to me given that the pseudosentences are chunks produced from a single piece of writing.\n\n- I find the notation used in the algorithm confusing. Why do we sometimes use parentheses and other times square brackets? Also, wouldn't indexing with t+1=25 at  the last iteration be undefined?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ikkJ4vCJ6n", "reviewer": "Reviewer_u8bS", "summary": "This work studies the dynamics of the embedding space as the input tokens go through the layers of a transformer architecture. The authors break the input prompt into sequences of tokens and study the embeddings of the last token as it propagates through the layers. Using PCA-like projections, it's shown that these embeddings approximately lie in a low-dimensional manifold. Then, by rephrasing these trajectories using linear approximations, the authors attempt to model it via Langevin dynamics which gives rise to the standard Fokker-Planck probability flow. One of the main conclusions is that transformers can be \"distilled\" into few parameters. To validate their assumptions, the authors experimentally study these trajectory on non-language inputs and untrained models and show that their ideas weakly hold. The target audience are people interested in the physics and interpretability of large language models.", "questions": "Some questions were raised above. In addition\n\n- This work reminds me of neural ODEs. Are there any connections between them and the viewpoint this paper explores?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- There has been a flurry of works trying to understand the inner workings of LLMs. Therefore, this direction is relevant and interesting.\n\n- This work studies this problem from a unique flow-based perspective by studying the dynamics of the embeddings as they evolve in the layers. The perspective is novel to the best of my knowledge and may potentially lead to a new perspective or algorithm to improve interpretability of LLMs.", "weaknesses": "- While potentially interesting, the paper feels too vague and very high-level without any concrete theoretical or experimental contributions.\n\n- No new theoretical contributions are made, other than standard langevin dynamics formulations of their ideas. The projection to lower dimensions and linear approximations are also somewhat too lossy, as the authors note, so it's not clear how well the observations here actually hold in real life.\n\n- Experiments seem limited to a few models and as the authors note, the last layer seems to form an outlier for the Mistral 7B and the Llama models. While the authors suggest some reasons for this, it's not inherently clear why these happen and bring the central hypotheses into question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYTfHvsn2t", "reviewer": "Reviewer_k1Ts", "summary": "This paper studies the trajectory of token embedding across layers. Inspired by dynamic system, they model the trajectories as diffusive process with a linear drift and a modified stochastic component.", "questions": "1. It is unclear to me what does this sentence (”the uncertainty (or stochasticity) introduced here accounts only for the loss of information of considering the token without its prompt.”) in footnote 8 mean?\n2. For figure 3, why u_2 and u_3 without u_1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea of this paper is kind of interesting.", "weaknesses": "1. the Gaussian assumption seems to not hold in early layers.\n2. It is unclear if the same type of paths would hold for larger and more complex model as we already see problems with newer model like LLaMA-3. I think it makes sense to model the intermediate layers with diffusion process but early and last layers might not work not well.\n3. The theory here does not lead to any practical predictions. For example, can you use this model to predict next token?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "75DJDB59vn", "reviewer": "Reviewer_UUNh", "summary": "The authors describe a framework that allows tracing the path taken by the activations in latent space during a forward pass on some input text. They note that despite the complexity of computing a forward pass, the trajectory taken is rather simple, and the activations live on a low-dimensional manifold that moves through latent space as a function of the number of layers processed. They provide a stochastic model that describes the trajectory, with vastly fewer parameters than the network itself. The approximation closely agrees with the actual activations, and can be used to extrapolate the behavior of the model.", "questions": "Questions: (annotated with line number/section)\n125: The vector for every token is projected to form the logits, not just the last vector. For inference, we only care about the logits in the last token'th position, but that doesn't mean the logits must only come from there.\n130: Algorithm 1 is overkill. Just saying \"we cache the activations between each transformer block in the sequence position of the last token over a forward pass for each psuedo-sentence $s_i$\" would be sufficient. The algorithm is not doing anything interesting that the transformer would not already do during a forward pass (other than the caching of activations).\n145: It is advised for ICLR to notate the last token position for layer $t+1$ as $\\mathbf{E}^{t+1}_{:, end}$ rather than $\\mathbf{E}(t+1)[:, end]$ (check the ICLR formatting guide).\n152: The latent space is obviously spanned by a Cartesian basis. Any such space is.\n155: You then revert to the ICLR standard notation for indexing a matrix that you did not use on line 145. Pick one and be consistent.\n175-176: Replace \"Because the Cartesian axes, $\\mathbb{e}_i$, are unlikely to align with trajectories meaningful directions,\" with \"There is no reason a priori we would expect the Cartesian axes to align to meaningful directions for the trajectories of the activations\"\n177: Concatenating the $\\mathbb{x}_k(t)$? Over what dimension? $k$ or $t$ or something else?\n154: Does it makes sense to plot the trajectories such that at each layer we use a different basis? Seems like you'd want to use the same basis throughout the entire forward pass if possible, as then each step along the trajectory is measured in terms of something different. I guess maybe the size of the principal component, with respect to the most important eigenvector is still interesting. This was partially addressed by Fig. 2a. showing that all the dimensions are important at some point during the forward pass, and that the low-dimensional manifold \"wanders around\" the entire latent space.\n\n\nSection 3.2: At first glance this seems a reasonable choice of metric: The KL divergence between the normal output distribution, and the output distribution after having performed dimensionality reduction via SVD. Is this a metric that is defined elsewhere in the literature? Cite if so. I wonder if SVD is the best thing to do here: Is there a better choice of low-dimensional manifold that can only be obtained via a non-linear transformation? Perhaps if an autoencoder was trained that compresses/uncompresses the latent space on forward passes, could you squeeze the activations into an even lower dimensional space while preserving the KL divergence?\nHowever, what isn't clear to me is that how much displacement of KL divergence is a lot, and why $K_0 = 256$ was the value for which \"most\" of the true distribution is recovered. What is \"most\"? What makes a KL-divergence of ~0.7 \"not much\"? It would be good to have some sort of sense of scale what the downstream effects are (e.g. measure the performance on some benchmark before and after throwing away 75% of the activations, and see how much \"dumber\" the model gets.)\n\nFigure 2(c): What precisely is the baseline defined as? The KL divergence between the output distributions conditioned on two unrelated inputs, or with random noise injected into the unembedding matrix, or something else?\n\nFigure 3: It's not clear how to compare the true and extrapolated positions. If the cluster comparison is desired, maybe colour one red, and the other blue, but both have transparency? So the purple region is the overlap? It's hard to see with the grey just covering the blue. I also would have liked a more principled metric to measure how well the model suggested in Equation (1) captures the dynamics of the activations, and what component remains unexplainable.\n\n255-256: The rotation matrix $\\mathbf{R}(t)$ was substituted out for $\\mathbf{U}(t)$, but $\\mathbb{\\Lambda}(t,\\tau)$ was not substituted out for $\\mathbb{\\Sigma}(t+\\tau)\\mathbb{\\Sigma}^{-1}(t)$. I think either substitute both (my preference) or neither.\n\n619: Forgot to bold $\\Lambda$", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Strong evidence provided to back up the claims made. The authors do a great job of explaining what it is they are measuring and why.\n* The result is fascinating, computationally cheap to obtain, model agnostic, and is empirically backed by measuring the KL-divergence of the distribution after down-projecting the dimensionality.\n* The theoretical justification on the proposed model is well motivated and explained in the appendix.\n* See questions", "weaknesses": "* Some claims on the similarity of extrapolated token positions are a bit weaker, and are primarily based on visually comparing the extrapolated and actual activations projected down to a low-dimensional space. I would have liked to have seen a more concrete metric of comparison, together with some baseline values to compare against to indicate that they really are close.\n* I would have liked to have seen some applications of this discovery, or at least conjectures for what it could be used for. For example, could one project down the weight matrices into the space provided, and obtain a network that has similar performance and is much smaller, at the expense of off-distribution task degradation? Essentially, a way to distil a pre-trained network down if we only care about performance on one task?\n* I would elaborate on the potential use-case for a continuous interpolation between layers. Is there any sort of intuitive meaning we can assign to this?\n* See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)