Lines of Thought in Large Language Models

## Abstract

Abstract Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers.
The resulting high-dimensional trajectories realize different contextualization, or ‘thinking’, steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these ‘ lines of thought .’ We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data.
We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

## 1 Introduction

How does a large language model (LLM) think?
In other words, how does it abstract the prompt “Once upon a time, a facetious” to suggest adding, e.g., “transformer”, and, by repeating the operation, continue on to generate a respectable fairy tale à la Perrault?
What we know is by design.
A piece of text is mapped into a set of high-dimensional vectors, which are then transported across their embedding (latent) space through successive transformer layers (Vaswani et al., ), each allegedly distilling different syntactic, semantic, informational, contextual aspects of the input (Valeriani et al., ; Song & Zhong, ).
The final position is then projected onto an embedded vocabulary to create a probability distribution about what the next word should be.
Why these vectors land where they do eludes human comprehension due to the concomitant astronomical numbers of arithmetic operations which, taken individually, do nothing, but collectively confer the emergent ability of language.

Our inability to understand the inner workings of LLMs is problematic and, perhaps, worrisome.
While LLMs are useful to write college essays or assist with filing tax returns, they are also often capricious, disobedient, and hallucinatory (Sharma et al., ; Zhang et al., ).
That’s because, unlike traditional ‘if-then’ algorithms, instructions have been only loosely, abstractly, encoded in the structure of the LLM through machine learning, that is, without human intervention.

In return, language models, trained primarily on textual data to generate language, have demonstrated curious abilities in many other domains (in-context learning), such as extrapolating time series (Gruver et al., ; Liu et al., ), writing music (Zhou et al., ), or playing chess (Ruoss et al., ).
Such emergent, but unpredicted, capabilities lead to questions about what other abilities LLMs may possess.
For these reasons, current research is attempting to break down internal processes to make LLMs more interpretable.(^1^11And, eventually, more reliable and predictable.)
Recent studies have notably revealed some aspects of the self-attention mechanism (Vig, ), patterns of neuron activation (Bricken et al., ; Templeton et al., ), signatures of ‘world models’(^2^22 World models refers to evidence of (abstract) internal representations which allow LLMs an apparent understanding of patterns, relationships, and other complex concepts.)
(Gurnee & Tegmark, ; Marks & Tegmark, ),
geometrical relationships between concepts (Jiang et al., ), or proposed mathematical models of transformers (Geshkovski et al., ).

This work introduces an alternative approach inspired by physics, treating an LLM as a complex dynamical system.
We investigate which large-scale, ensemble properties can be inferred experimentally without concern for the ‘microscopic’ details.(^3^33Such as: semantic or syntactic relationships, architecture specificities, etc.) Specifically, we are interested in the trajectories, or ‘lines of thought’ (LoT), that embedded tokens realize in the latent space when passing through successive transformer layers (Aubry et al., ).
By splitting a large input text into $N$-token sequences, we study LoT ensemble properties to shed light on the internal, average processes that characterize transformer transport.

We find that, even though transformer layers perform $10^{6}-10^{9}$ individual computations, the resulting trajectories can be described with far fewer parameters.
In particular, we first identify a low-dimensional manifold that explains most of LoT transport (see Fig. 1).
Then, we demonstrate that trajectories can be well approximated by an average linear transformation, whose parameters are extracted from ensemble properties, along with a random component with well characterized statistics.
Eventually, this allows us to describe trajectories as a kind of diffusive process, with a linear drift and a modified stochastic component.

##### Main contributions.

- 1.
We provide a framework to discover low-dimensional structures in an LLM’s latent space.
- 2.
We find that token trajectories cluster on a non-Euclidean, low-dimensional manifold.
- 3.
We introduce a stochastic model to describe trajectory ensembles with few parameters.

Figure: Figure 1: (a) Lines of thought (blue to red) for an ensemble of 1000 pseudo-sentences of 50 tokens each, projected along the first 3 singular vectors after the last layer ($t=24$). They appear to form a tight bundle, with limited variability around a common average path. (b) Representation of the low-dimensional, ribbon-shaped manifold in $\mathcal{S}$ (projected along 3 Cartesian coordinates). Positions are plotted for $t=12$ (green) to $t=24$ (yellow).

## 2 Methods

This section describes our algorithm for generating and analyzing an ensemble of tokens trajectories in the latent space of LLMs.

##### Language models.

We rely primarily on the 355M-parameter (‘medium’) version of the GPT-2 model (Radford et al., ).
It presents the core architecture of ancestral (circa 2019) LLMs: transformer-based, decoder-only.(^4^44 Compared to current state-of-the-art models, GPT-2 medium is rather unsophisticated. Nevertheless, it works. It produces cogent text that addresses the input prompt. Hence, we consider the model already contains the essence of modern LLMs and leverage its agility and transparency for scientific insight.)
It consists of $N_{L}$ = 24 transformer layers(^5^55(LayerNorm +) Self-attention then (LayerNorm +) Feed-forward, with skip connections around both.)
operating in a latent space $\mathcal{S}$ of dimension $D=1024$.
The vocabulary $\mathcal{V}$ contains $N_{\mathcal{V}}=50257$ tokens.
A layer normalization (Ba et al., ) is applied to the last latent space position before projection onto $\mathcal{V}$ to form the logits. (This final normalization is not included in our trajectories.)
We later extend our analysis to the Llama 2 7B (Touvron et al., ), Mistral 7B v0.1 (Jiang et al., ), and small Llama 3.2 models (1B and 3B) (MetaAI, ).

##### Input ensembles.

We study statistical properties of trajectory ensembles obtained by passing a set of input prompts through GPT-2.
We generate inputs by tokenizing (Wolf et al., ) a large text and then chopping it into ‘pseudo-sentences’, i.e., chunks of a fixed number of tokens $N_{k}$ (see Algorithm 1).
Unless otherwise noted, $N_{k}=50$.
These *non-overlapping* chunks are consistent in terms of token cardinality, and possess the structure of language, but have various meanings and endings (see Section A.1).
The main corpus in this study comes from Henry David Thoreau’s Walden, obtained from the Gutenberg Project (Project Gutenberg, ).(^6^66 The idea of using a literary piece to probe statistics of language was investigated by Markov back in 1913 (Markov, ).)
We typically use a set of $N_{s}\simeq 3000\text{--}14000$ pseudo-sentences.

##### Trajectory collection.

We form trajectories by collecting the successive vector outputs, within the latent space, after each transformer layer (hidden_states).
For conciseness, we identify layer number with a notional ‘time’, $t$.
Even though all embedded tokens of a prompt voyage across the latent space, only the vector corresponding to the last token is eventually projected onto $\mathcal{V}$ to form the logits.
Hence, here, we only consider the trajectory of this last (or ‘pilot’) token.
The trajectory ${\bm{M}}_{k}$ of sentence $k$’s pilot is the sequence of $24$ successive time positions $\{{\bm{x}}_{k}(1),{\bm{x}}_{k}(2),\ldots,{\bm{x}}_{k}(24)\}$, concatenated as a column matrix (Algorithm 1).

Figure: Algorithm 1 Trajectory generation in transformer-based model

##### Latent space bases.

The latent space is spanned by a Cartesian basis, i.e., the orthogonal set of one-hot (unit) vectors $\mathcal{E}=\{{\bm{e}}_{i}\}_{i=1\dots D}$.(^7^77With a 1 in $i^{\mathrm{th}}$ position, 0 elsewhere.)
Additionally, we will often refer to the bases $\mathcal{U}(t)=\{{\bm{u}}_{i}^{(t)}\}_{i=1\dots D}$ formed by the left-singular vectors of the singular value decomposition (SVD) of the $D\times N_{\mathrm{s}}$ matrix after layer $t$: ${\bm{M}}={\bm{U}}\mathbf{\Sigma}{\bm{V}}^{\top}$, with
${\bm{M}}_{:,k}(t)={\bm{x}}_{k}(t)$.
Vectors ${\bm{u}}_{i}$ are organized according to their corresponding singular values, $\sigma_{i}$, in descending order.
Note that because trajectory clusters evolve over time there are 24 distinct bases.

## 3 Results

We present and characterize results pertaining to ensembles of trajectories as they travel within the latent space $\mathcal{S}$.

### 3.1 Lines of thought cluster along similar pathways

We first observe in Fig. 1a that pilot trajectories tend to cluster together, instead of producing an isotropic and homogeneous filling of $\mathcal{S}$.
Indeed, LoTs for different, *independent* pseudo-sentences follow a common path (forming *bundles*), despite individual variability.
Specifically, there exist directions with significant displacement relative to the spread (mean over standard deviation).
In addition, positions at different times form distinct clusters, as shown in Fig. 6.

Properly visualizing these trajectories is difficult, due to the high dimensionality of $\mathcal{S}$.
Because the Cartesian axes, ${\bm{e}}_{i}$, are unlikely to align with trajectories meaningful directions, we seek relevant alternative bases, informed by the data.
After each layer $t$, we perform the singular value decomposition of the matrix formed by concatenating the ${\bm{x}}_{k}(t)$ to obtain a basis $\mathcal{U}(t)$ aligned with the data’s intrinsic directions.
In the following, we leverage these time-dependent singular vectors and values to investigate ensemble dynamics and low-dimensional structures.

### 3.2 Lines of thought follow a low-dimensional manifold

The fast decay of the singular value magnitudes seen in Fig. 2b suggests that LoTs may be described by a lower-dimensional subspace.
But how many dimensions are relevant?
Singular values relate to ensemble variance along their corresponding directions. Since the embedding space is high-dimensional, however, the curse of dimensionality looms, hence the significance of Euclidean distances crumbles.
To circumvent this limitation, we consider a more practical metric: how close to the original output distribution on the vocabulary does a reduction in dimensionality get us?

To investigate this question, we express token positions ${\bm{x}}(t)$ in the singular vector basis $\mathcal{U}(t)$:

$$ ${\bm{x}}(t)=\sum_{i=1}^{K}a_{i}^{(t)}{\bm{u}}_{i}^{(t)},$ $$

where the ${\bm{u}}_{i}^{(t)}$’s are organized by descending order of their corresponding singular values.
By default $K=D$, and the true output distribution ${\bm{p}}^{\mathcal{V}}$ is obtained.
Now, we examine what happens when, instead of passing the full basis set, we truncate it, after each layer, to keep only the first $K<D$ principal components.
We compare the resulting output distribution, $\mathbf{p}^{\mathcal{V}}_{K}$ to the true distribution $\mathbf{p}^{\mathcal{V}}$ using KL divergence $D_{\mathrm{KL}}(\mathbf{p}^{\mathcal{V}}_{K}\|\mathbf{p}^{\mathcal{V}})$.
In Fig. 2c, we see that most of the true distribution is recovered when keeping only about $K_{0}=256$, or $25\%$, of the principal components.
In other words, for the purpose of next-token prediction, LoTs are quasi-256-dimensional.

If these principal directions remained constant at each layer, this would imply that $75\%$ of the latent space could be discarded with no consequence.
This seems unrealistic.
In fact, the principal directions rotate slightly over time, as displayed in Fig. 2a.
Eventually, between $t=1$ and $t=24$, the full Cartesian basis $\mathcal{E}$ is necessary to express the first singular directions.
Thus, we conclude that lines of thoughts evolve on a low-dimensional curved manifold of about 256 dimensions, that is contained within the full latent space (Fig. 1b).

Figure: Figure 2: (a) Angle between the first 4 singular vectors at $(t_{1},t_{2})$, $\arccos({\bm{u}}_{i}^{(t_{1})}\cdot{\bm{u}}_{i}^{(t_{2})})$, for $i=\{1,2,3,4\}$ (top-left, top-right, bottom-left, bottom-right, respectively). (b) Singular values for $t=1,\dots,24$ (blue to red). Clusters stretch more and more after each layer. The leading singular values, $\sigma_{1}(t)$, have been omitted for clarity. (c) Average (over all trajectories) KL divergence between reduced dimensionality trajectories output and true output distributions, as the dimensionality $K$ is increased. The red dashes line shows the average KL divergence between unrelated distributions (baseline for dissimilar distributions).

### 3.3 Linear approximation of trajectories

Examination of the singular vectors and values at each time step indicates that LoT bundles rotate and stretch smoothly after passing through each layer (Fig. 2).
This suggests that token trajectories could be *approximated* by the linear transformations described by the ensemble, and extrapolated accordingly, from an initial time $t$ to a later time $t+\tau$.
Evidently, it is improbable that a transformer layer could be replaced by a mere linear transformation.
We rather hypothesize that, in addition to this deterministic average path, a token’s location after layer $t+\tau$ will depart from its linear approximation from $t$ by an unknown component ${\bm{w}}(t,\tau)$.(^8^88 We emphasize that trajectories are completely deterministic; the uncertainty (or stochasticity) introduced here accounts only for the loss of information of considering the token without its prompt.)
We propose the following model:

$$ ${\bm{x}}(t+\tau)={\bm{R}}(t+\tau)\mathbf{\Lambda}(t,\tau){\bm{R}}(t)^{\top}{ \bm{x}}(t)+{\bm{w}}(t,\tau),$ (1) $$

where ${\bm{x}}(t)$ is the pilot token’s position in the Cartesian basis, and ${\bm{R}},\mathbf{\Lambda}$ are rotation (orthonormal) and stretch (diagonal) matrices, respectively.
Eq. 1 formalizes the idea that, to approximate ${\bm{x}}(t+\tau)$, given ${\bm{x}}(t)$, we first project ${\bm{x}}$ in the ensemble intrinsic basis at $t$ (${\bm{R}}^{\top}{\bm{x}}$), then stretch the coordinates by the amount given by $\mathbf{\Lambda}$, and finally rotate according to how much the singular directions have rotated between $t$ and $t+\tau$, ${\bm{R}}(t+\tau)$ (see also Fig. 7 in Appendix B). Consequently, we can express these matrices as a function of the set of singular vectors (${\bm{U}}$) and values ($\mathbf{\Sigma}$):

$$ ${\bm{R}}(t)={\bm{U}}(t),\quad\mathbf{\Lambda}(t,\tau)=\text{diag}(\sigma_{i}(t +\tau)/\sigma_{i}(t))=\mathbf{\Sigma}(t+\tau)\mathbf{\Sigma}^{-1}(t).$ $$

Fig. 3 shows the close agreement, at the ensemble level, between the true and extrapolated positions. This is merely a linear approximation as it is similar to assuming that LoT clusters deform like an elastic solid, where each point maintains the same vicinity.
The actual coordinates ought to include an additional random component ${\mathbf{w}}(t,\tau)$, which a priori depends on both $t$ and $\tau$.

Figure: Figure 3: Extrapolated token positions $\tilde{{\bm{x}}}^{(k)}$ (blue) from $t=\{12,14,16,18\}$ to $t+\tau=\{t+1,\dots,21\}$, compared to their true positions ${\bm{x}}^{(k)}$ (gray), projected in the $({\bm{u}}_{2}^{(t)},{\bm{u}}_{3}^{(t)})$ planes.
Refer to caption: extracted/5960418/fig/fig-extrapolation-v07.png

Is it possible to express ${\bm{w}}$ in probabilistic terms?
We consider the empirical residuals

$$ $\delta{\bm{x}}(t,\tau)={\bm{x}}(t+\tau)-\tilde{{\bm{x}}}(t,\tau)$ $$

between true positions ${\bm{x}}$ and linear approximations $\tilde{{\bm{x}}}(t,\tau)={\bm{U}}(t+\tau)\mathbf{\Lambda}(t,\tau){\bm{U}}(t)^{
\top}{\bm{x}}(t)$.
We investigate the distributions and correlations of $\delta{\bm{x}}(t,\tau)$ across layer combinations $(t,t+\tau)$.

From the data, Fig. 4 shows that, for all $(t,t+\tau)\in\{1,\dots,23\}\times\{t+1,\dots,24\}$, the ensemble of $\delta{\bm{x}}(t,\tau)$ has the following characteristics: 1) it is Gaussian, 2) with zero mean, 3) and variance scaling as $\exp(t+\tau)$.
In addition, Fig. 8 shows that the distribution is isotropic, with no evidence of spatial cross-correlations . Hence, we propose:

$$ ${\textnormal{w}}_{i}(t,\tau)\sim\mathcal{N}(0,\alpha e^{\lambda(t+\tau)}),$ (2) $$

i.e., each coordinate ${\textnormal{w}}_{i}$ of ${\mathbf{w}}$ is a Gaussian random variable with mean zero and variance $\alpha e^{\lambda(t+\tau)}$.
Linear fitting of the logarithm of the variance yields $\alpha\simeq 0.64$ and $\lambda\simeq 0.18$.
Even though this formulation ignores some variability across times and dimensions, it is a useful minimal modelling form to describe the ensemble dynamics with as few parameters as possible.

### 3.4 Langevin dynamics for continuous time trajectories

Just like the true positions ${\bm{x}}(t)$, matrices ${\bm{R}}$ and $\mathbf{\Lambda}$ are known (empirically) only for integers values of $t$.(^9^99That is, after each layer.)
Can we extend Eq. 1 to a continuous time parameter $t\in[1,24]$?
Indeed, it is possible to *interpolate* ${\bm{R}}$ and $\mathbf{\Lambda}$ between their known values (Absil et al., ).
Specifically, ${\bm{R}}(t)$ remains orthogonal and rotates from its endpoints; singular values can be interpolated by a spline function.

In return, this allows us to interpolate trajectories between transformer layers.(^10^1010 These interpolated positions do not hold any interpretive value, but may be insightful for mathematical purposes.)
Thus, we extend Eq. 1 to a continuous time variable $t$, and write in infinitesimal terms:

$$ $d{\bm{x}}(t)=\left[\dot{{\bm{R}}}(t){\bm{R}}(t)^{\top}+{\bm{R}}(t)\dot{{\bm{S} }}(t){\bm{R}}(t)^{\top}\right]{\bm{x}}(t)\,dt+\sqrt{\alpha\lambda\exp(\lambda t )}\,d{\mathbf{w}}(t),$ (3) $$

where $\dot{{\bm{S}}}=\text{diag}\left(\dot{\sigma_{i}}/\sigma_{i}\right)$ and $d{\mathbf{w}}(t)$ is a differential of a Wiener process (Pavliotis, ).
We defer the mathematical derivation to Section A.2.
This equation artificially extends LoTs to continuous paths across $\mathcal{S}$.
It provides a stochastic approximation to any token’s trajectory, at all times $t$.

Figure: Figure 4: Statistics of $\delta{\bm{x}}(t,\tau)$: mean $\mu$, variance $\sigma^{2}$, excess kurtosis $\kappa$. Brackets $\langle\dots\rangle$ denote average over directions ${\bm{e}}_{i}$ (see Fig. 9 for details). (a) For all $(t,t+\tau)$, $\mu\simeq 0$ (that is, $\mu/\sigma\ll 1$). (b) $\log(\sigma^{2})$ increases linearly in time, only depends on $t+\tau$. (c) The *excess* kurtosis (kurtosis minus 3) remains close to 0, indicating Gaussianity (except in early layers).

### 3.5 Fokker-Planck formulation

Eq. 3 is a stochastic differential equation (SDE) describing individual trajectories with a random component.
Since the noise distribution is well characterized (see Eq. 2), we can write an equivalent formulation for the *deterministic* evolution of the probability density $P({\bm{x}},t)$ of tokens ${\bm{x}}$ over time (Pavliotis, ).
The Fokker-Planck equation(^11^1111 Also known as Kolmogorov forward equation.)
associated to Eq. 3 reads:

$$ $\frac{\partial P({\bm{x}},t)}{\partial t}=-\nabla_{{\bm{x}}}\cdot\left[\left( \dot{{\bm{R}}}{\bm{R}}^{\top}+{\bm{R}}\dot{{\bm{S}}}{\bm{R}}^{\top}\right){\bm {x}}P({\bm{x}},t)\right]+\frac{1}{2}\alpha\lambda e^{\lambda t}\,\nabla_{{\bm{ x}}}^{2}P({\bm{x}},t).$ (4) $$

This equation captures trajectory ensemble dynamics in a much simpler form, and with far fewer parameters, than the computation actually performed by the transformer stack on the fully embedded prompt.
The price paid for this simplification is a probabilistic, rather than deterministic, path for LoTs.
We now test our model and assess the extent and limitations of our results.

## 4 Testing and validation

### 4.1 Simulations of the stochastic model

We test our continuous-time model described above.
Due to the high dimensionality of the space, numerical integration of the Fokker-Planck equation, Eq. 4, is computationally prohibitive.
Instead, we simulate an ensemble of trajectories based on the Langevin formulation, Eq. 3.
The technical details are provided in Section A.3.

Figure: Figure 5: Simulated distributions for $t=12$, $t+\tau=\{12,13,14,15,16\}$, projected on the $\left({\bm{u}}_{1},{\bm{u}}_{2}\right)$ plane (top row) and the $\left({\bm{u}}_{3},{\bm{u}}_{4}\right)$ plane (bottom row). Distributions have been approximated from ensemble trajectories, 10 trajectories for each initial point. Background lines indicate true distributions, thin lines on top indicate simulations.
Refer to caption: x1.png

The results presented in Fig. 5 show that the simulated ensembles closely reproduce the ground truth of true trajectory distributions.
We must note that Eqs. 3 and 4 are not path-independent; therefore, their solution depend on the value of ${\bm{R}}(t)$, ${\bm{S}}(t)$ at all time $t$. Since there is no ‘true’ value for the matrices in-between layers, the output of numerical integration naturally depends on the interpolation scheme. Hence, discrepancies are to be expected.

### 4.2 Null testing

We now examine trajectory patterns for non-language inputs and untrained models.

#### 4.2.1 Gibberish

We generate non-language (‘gibberish’) pseudo-sentences by assembling $N$-token sequences of random tokens in the vocabulary, and pass them as input to GPT-2.
The resulting trajectories also cluster around a path similar to that of language.
However, the two ensembles, language and gibberish, are linearly separable at all layers (see Fig. 10 in Section B.5), indicating that they travel on two distinct, yet adjacent, manifolds.

#### 4.2.2 Untrained & ablated models

We compare previous observations with the null baseline of an untrained model.

First, we collect trajectories of the Walden ensemble passing through a reinitialized version of GPT-2 (the weights have been reset to a random seed).
We observe that while LoTs get transported away from their starting point, the trajectories follow straight, quasi-parallel paths, maintaining their vicinity (see Fig. 10).
Furthermore, the model of Eqs. 1 and 2 does not hold; Fig. 11 shows that the variance of $\delta{\bm{x}}$ does not follow the $\exp(t+\tau)$ scaling, and the distributions are far from Gaussian.

Next, we consider an ablated model, where only layers $13$ to $24$ have been reinitialized.
When reaching the untrained layers, the trajectories stop and merely diffuse about their $t=12$ location (Fig. 10).

In conclusion, upon training, the weights evolve to constitute a specific type of transport in the latent space.

### 4.3 Results with other models

We repeat the same approach with a set of larger and more recent LLMs.
We collect the trajectories of the Walden ensemble in their respective latent spaces.

##### Llama 2 7B.

We first investigate the Llama 2 7B model (Touvron et al., ).(^12^1212 Decoder-only, 32 layers, 4096 dimensions; released July 2023 by Meta AI.)
Remarkably, the pattern of GPT-2 repeats.
Token positions at $t+\tau$ can be extrapolated from $t$ by rotation and stretch using the singular vectors and values of the ensemble.
The residuals are distributed as those of GPT-2, with ${\textnormal{w}}_{i}(t,\tau)\sim\mathcal{N}(0,\alpha e^{\lambda(t+\tau)})$, see Fig. 12.
The values for the parameters $\alpha$ and $\lambda$, however, differ from those of GPT-2 (here, $\alpha\simeq-5.4,\lambda\simeq 0.27$).

##### Mistral 7B.

Trajectories across the Mistral 7B (v0.1) model (Jiang et al., )(^13^1313 Decoder-only, 32 layers, 4096 dimensions; released September 2023 by Mistral AI.)
also follow the same pattern (Fig. 13).
We note, however, that Eq. 4 only holds up until layer 31.
It seems as though the last layer is misaligned with the rest of the trajectories, as linear extrapolation produces an error that is much larger than expected.

##### Llama 3.2.

The last layer anomaly is also apparent for Llama 3.2 1B(^14^1414 Decoder-only, 16 layers, 2048 dimensions; released September 2024 by Meta AI.), both in the mean and variance of $\delta{\bm{x}}(t,16)$ (see Fig. 14).
However, the rest of the trajectories follows Eq. 1.
The same pattern is observed for Llama 3.2 3B(^15^1515 Decoder-only, 28 layers, 3072 dimensions; released September 2024 by Meta AI.) in Fig. 15.

It is noteworthy that these three recent models feature the same anomaly at the last layer.
The reason is not immediately evident, and perhaps worth investigating further.
In addition, we remark that all models also show deviations from predicted statistics across the very first layers (top-left corners).
We conjecture that these anomalies might be an effect of re-alignment or fine-tuning, as the first and last layers are the most exposed to perturbations which might not propagate deep into the stack.

## 5 Conclusion

##### Summary.

This work began with the prospect of visualizing token trajectories in their embedding space $\mathcal{S}$.
The space is not only high-dimensional, but also isotropic: all coordinates are a priori equivalent.(^16^1616Unlike other types of datasets where different dimensions might have well-defined meaning, for example: temperature, pressure, wind speed, etc.)
Hence, we sought directions and subspaces of particular significance in shaping token trajectories(^17^1717And hence defining next-token distribution outputs), some kind of ‘eigenvectors’ of the transformer stack.

Instead of spreading chaotically, lines of thought travel along a low-dimensional manifold.
We used this pathway to extrapolate token trajectories from a known position at $t$ to a later time, based on the geometry of the ensemble. Individual trajectories deviate from this average path by a random amount *with well-defined statistics*.
Consequently, we could interpolate token dynamics to a continuous time in the form of a stochastic differential equation, Eq. 3.
The same ensemble behavior holds for various transformer-based pre-trained LLMs, but collapses for untrained (reinitialized) ones.

This approach aims to extract important features of language model internal computation.
Unlike much of prior research on interpretability, it is agnostic to the syntactic and semantic aspects of inputs and outputs.
We also proposed geometrical interpretations of ensemble properties which avoid relying on euclidean metrics, as they become meaningless in high-dimensional spaces.

##### Limitations.

This method is limited to open-source models, as it requires extracting hidden states; fine-tuned, heavily re-aligned models might exhibit different patterns.
In addition, it would be compelling to connect the latent space with the space of output distributions, for example by investigating the relative arrangement of final positions with respect to embedded vocabulary.
However, this is complicated by the last layer normalization which typically precedes projection onto the vocabulary.
This normalization has computational benefits, but its mathematical handling is cumbersome: it is highly non-linear as it involves the mean and standard deviation of the input vector.

##### Implications.

Just like molecules in a gas or birds in a flock, the complex system formed by billions of artificial neurons in interaction exhibits some simple, macroscopic properties.
It can be described by ensemble statistics with a well defined random component.
Previously, Aubry et al. () had also uncovered specific dynamical features, notably *token alignment*, in transformer stacks of a wide variety of trained models.

That’s not to say that reduced complexity representations are necessarily useful in practice.
Individual trajectory variability persists, and is essential to accurately predict next tokens and continue textual inputs.
Thus, it is not immediately apparent to us whether the low-dimensionality structures identified could lead to avenues for compressing or ablating transformers, although it might.

Yet, patterns are explanatory.
Our concern here has been primarily to discover some of the mechanisms implicitly encoded in the weights of trained language models.
Further investigations could extend this methodology to more thoroughly identify and characterize the dynamics of tokens.

#### Acknowledgments

This work was supported by the SciAI Center, and funded by the Office of Naval Research (ONR), under Grant Numbers N00014-23-1-2729 and N00014-23-1-2716.

## Appendix A Additional methods and derivations

### A.1 Pseudo-sentences

Random sample of 10-token pseudo-sentences (non-consecutive) extracted from Walden.
Similar chunks, but of 50 tokens, were passed through GPT2 to form trajectories.

### A.2 Langevin equation derivation

Starting from

$$ ${\bm{x}}(t+\tau)={\bm{R}}(t+\tau)\mathbf{\Lambda}(t,\tau){\bm{R}}(t){\bm{x}}(t )+{\bm{w}}(t,\tau),$ $$

with $\Lambda(t,\tau)=\mathbf{\Sigma}(t+\tau)\mathbf{\Sigma}^{-1}(t)$, and assuming now that $t,\tau$ are variables in $\mathbb{R}$, as $\tau$ goes to 0 we can approximate:

$$ ${\bm{R}}(t+\tau)\approx{\bm{R}}(t)+\tau\dot{{\bm{R}}}(t)$ $$

and

$$ $\mathbf{\Sigma}(t+\tau)\approx\mathbf{\Sigma}(t)+\tau\dot{\mathbf{\Sigma}(t)},$ $$

leading to:

$$ $\mathbf{\Lambda}(t,\tau)\approx\left(\mathbf{\Sigma}(t)+\tau\dot{\mathbf{ \Sigma}}(t)\right)\mathbf{\Sigma}^{-1}(t)={\bm{I}}+\tau\mathbf{\Sigma}^{-1}(t) \dot{\mathbf{\Sigma}}(t).$ $$

Hence:

$$ $\displaystyle{\bm{R}}(t+\tau)\mathbf{\Lambda}(t,\tau){\bm{R}}(t)^{\top}$ $\displaystyle\approx\left({\bm{R}}(t)+\tau\dot{{\bm{R}}}(t)\right)\left({\bm{I }}+\tau\dot{\mathbf{\Sigma}}(t)\mathbf{\Sigma}^{-1}(t)\right){\bm{R}}(t)^{\top}$ $\displaystyle\approx{\bm{I}}+\tau\left(\dot{{\bm{R}}}(t){\bm{R}}(t)^{\top}+{ \bm{R}}(t)\dot{{\bm{S}}}(t){\bm{R}}(t)^{\top}\right),$ $$

given that ${\bm{R}}{\bm{R}}^{\top}={\bm{I}}$ and with ${\bm{S}}(t)=\text{diag}\left(\ln{\sigma_{i}(t)}\right)$ and thus $\dot{{\bm{S}}}(t)=\text{diag}(\dot{\sigma}_{i}/\sigma_{i})$.

The variance of the noise term is given by:

$$ $\text{var}=\alpha\exp(\lambda(t+\tau))\approx\alpha\exp(\lambda t)(1+\lambda \tau).$ $$

The increment of variance over time $\tau$ is:

$$ $\delta[\text{var}]=\alpha\lambda\exp(\lambda t)\tau.$ $$

This means the noise term can be expressed as:

$$ ${\bm{w}}(t,\tau)=\sqrt{\alpha\lambda\exp(\lambda t)\tau}\cdot\vec{\eta},$ $$

where $\vec{\eta}$ is a vector of standard Gaussian random variables.

Putting everything together:

$$ ${\bm{x}}(t+\tau)-{\bm{x}}(t)=\tau\left(\dot{{\bm{R}}}(t){\bm{R}}(t)^{\top}+{ \bm{R}}(t)\dot{{\bm{S}}}(t){\bm{R}}(t)^{\top}\right){\bm{x}}(t)+\sqrt{\alpha \lambda\exp(\lambda t)\tau}\,\mathbf{\eta}(t).$ $$

And finally:

$$ $d{\bm{x}}(t)=\left(\dot{{\bm{R}}}(t){\bm{R}}(t)^{\top}+{\bm{R}}(t)\dot{{\bm{S} }}(t){\bm{R}}(t)^{\top}\right){\bm{x}}(t)dt+\sqrt{\alpha\lambda\exp(\lambda t) }\,d{\bm{w}}(t),$ $$

with $d{\bm{w}}(t)$ a Wiener process.

### A.3 Numerical integration

Numerical integration of Eq. 3 requires to interpolate the singular vectors and values, and their derivatives, at non-integer times.

Interpolation of (scalar) singular values is straightforward.
We use a polynomial interpolation scheme for each value, and compute the corresponding polynomial derivative.
This yields $\dot{\sigma_{i}}(t)/\sigma_{i}(t)$ for every coordinate $i$ at any time $t\in[1,24]$, and hence $\dot{{\bm{S}}}(t)$.

Interpolating sets of orthogonal vectors presents significant challenges. A rigorous approach involves performing the interpolation within the compact Stiefel manifold, followed by a reprojection onto the horizontal space Praveen et al. (). However, this method is computationally expensive and can introduce discontinuities, which are problematic for numerical integration. To address these issues, we used an approximation based on the matrix logarithm, which simplifies the process while maintaining an acceptable level of accuracy.
To interpolate between ${\bm{U}}_{1}$ and ${\bm{U}}_{2}$ at $t_{1},t_{2}$, we compute the relative rotation matrix ${\bm{R}}={\bm{U}}_{1}^{\top}{\bm{U}}_{2}$ and interpolate using

$$ ${\bm{U}}(t)={\bm{U}}_{1}\exp_{\mathrm{M}}(\alpha\ln_{\mathrm{M}}{{\bm{R}}}).$ (5) $$

where $\alpha=(t-t_{1})/(t_{2}-t_{1})$ and with $\ln_{\mathrm{M}},\exp_{\mathrm{M}}$ denoting the matrix logarithm and exponential, respectively.(^18^1818 $\exp_{\mathrm{M}}({\bm{A}})=\sum{\bm{A}}^{k}/k!$ and $\ln_{M}$ is the inverse function: $\ln_{\mathrm{M}}\left[\exp_{\mathrm{M}}({\bm{A}})\right]={\bm{I}}$.)
This also yields the derivative $\dot{{\bm{U}}}(t)=\left[{\bm{U}}\ln_{M}{{\bm{R}}}\right]/(t_{2}-t_{1})$.
Indeed:

$$ $\dot{{\bm{U}}}={\bm{U}}_{1}\cdot\frac{d}{dt}\exp\left(\alpha(t)\ln{{\bm{R}}} \right)={\bm{U}}_{1}\dot{\alpha}\ln{{\bm{R}}}\exp{\alpha(t)\ln{{\bm{R}}}}=\dot {\alpha}{\bm{U}}\ln{{\bm{R}}}.$ $$

## Appendix B Supplementary figures and schematics

### B.1 Trajectory clustering

In Fig. 6, we show evidence of trajectory clustering in the latent space.
In particular, all pilot tokens get transported away from the origin (or their starting point) by a comparable amount, resulting in narrow distributions along the first singular direction.
Another signature of clustering is the fact that token positions at different times form distinct clusters, as showed by low-dimensional t-SNE representation (van der Maaten & Hinton, ).

Figure: Figure 6: (Left) Distributions along the first singular vector at different times. (Right) Low-dimensional (t-SNE) visualization of the clustering of tokens, notably across different times. Same color legend.
Refer to caption: x2.png

### B.2 Trajectory extrapolation

In Fig. 7, we provide a schematic to explain the reasoning behind Eq. 1.
*If* the cluster rotated and stretched like a solid, the position of a point ${\bm{x}}^{\prime}$ at $t^{\prime}$ could be inferred *exactly* from it position ${\bm{x}}$ at $t$, using the formula outlined.
However, unsurprisingly, the token ensemble does not maintain its topology and the points move around the clusters, requiring the stochastic term ${\mathbf{w}}$ injected in Eq. 1.

Figure: Figure 7: Extrapolation between $t$ and $t^{\prime}$. The extrapolated location ${\bm{x}}^{\prime}$ corresponds to the rotated and stretched position of ${\bm{x}}$. Given that $\vec{u}={\bm{R}}\vec{e}$, $\vec{u}^{\prime}={\bm{R}}^{\prime}\vec{e}$ and ${\bm{R}}^{-1}={\bm{R}}^{\top}$, we have $\vec{e}={\bm{R}}^{\top}\vec{u}={\bm{R}}^{\prime\top}\vec{u}^{\prime}$ and thus $\vec{u}^{\prime}={\bm{R}}^{\prime}{\bm{R}}^{\top}\vec{u}$.
Refer to caption: x3.png

### B.3 Noise statistics

Fig. 8 provides additional details pertaining to the distribution of residuals $\delta_{x}$.
Since they are many dimensions and time points, it gives only representative snapshots.
It intends to substantiate the results that:

- •
the $\delta{\bm{x}}$ are Gaussian (Fig. 8A);
- •
the variance is exponential in $(t+\tau)$, with no dependency on $t$ (Fig. 8B);
- •
all components $\delta x_{i}$ of $\delta{\bm{x}}$ have the same distribution (Fig. 8C), i.e., isotropy;
- •
there are no spatial cross-correlations, i.e. $\langle\delta x_{i}\delta x_{j}\rangle=\delta_{ij}$ (Dirac function) (Fig. 8D).

Figure: Figure 8: Statistics of $\delta\mathbf{x}$. (A) Empirical PDF of $\delta x_{42}(10,t+\tau)$, with $t+\tau=12,14,16$. The curves appear Gaussian. (B) Variance of $\delta x_{i}$ for $i=1\dots 8$, for $t=4,8,12,16$ and $t+\tau>t$. (C) Empirical PDF of $\delta x_{i}(12,14)$ for $i=1\dots 1024$. The curves are similar for almost all coordinates. (D) Cross correlations of $\delta x_{i}$ and $\delta x_{j}$.
Refer to caption: x4.png

### B.4 Details on noise aggregated statistics ( Fig. 4 )

Fig. 9 explains how the noise plots such as Fig. 4 are created.
We use ensemble averages $\langle\dots\rangle$ of the *absolute values* for $|\mu_{i}|,|\kappa_{i}|$ since we are interested in the average *distances* from 0.

Figure: Figure 9: Schematic to explain the noise figures such as Fig. 4. Each square represents a summary statistics. Specifically, the square at $(t,t+\tau)$ represents the distribution of $\{\delta{\bm{x}}^{(k)}(t,t+\tau)\}_{k}$, with $k$ indexing individual tokens. The $\delta{\bm{x}}$ along each coordinate $i$ form a distribution, from which one can extract the corresponding $\mu_{i},\sigma_{i},\kappa_{i}$ (mean, variance, kurtosis). These 1D moments are then averaged along all coordinates $i$ ($\langle\mu_{i}\rangle_{i}$, etc.), forming the value displayed in the square.
Refer to caption: x5.png

### B.5 Null testing

Fig. 10 shows the trajectories of language vs gibberish, as well as the linear separability of the two ensemble.
It also shows trajectories for an untrained GPT-2 shell, and a model with only the last 12 layers reinitialized.

Figure: Figure 10: (Top-left) Trajectories of non-language (red) vs language (black), plotted in the same axes (10-token pseudo-sentences). (Top-right) Accuracy of linear separability between language and non-language for each layer. Obtained by training a Perceptron (train/test: 0.7/0.3; 14000 trajectories). (Bottom-left) Trajectories in the untrained GPT-2 model. They are transported in straight lines. (Bottom-right) Trajectories in the mixed model. After being transported by trained layers 1-12, the trajectories stop. Layers 13-24 with random weights do not transport tokens any further.
Refer to caption: extracted/5960418/fig/figS_null_testing_v01.png

### B.6 Results with other models

Figure: Figure 11: GPT-2 untrained. The averaged excess kurtoses $\langle|\kappa|\rangle$ fall in the 1–1.5 range, indicating strong non-gaussianity. The variance does not scale solely with $t+\tau$.

Figure: Figure 12: Llama 2 7B: noise statistics, $\delta{\bm{x}}(t,t+\tau)={\bm{x}}(t+\tau)-\tilde{{\bm{x}}}(t,\tau)$, averaged $\langle\cdots\rangle$ over all Cartesian dimensions, for 1000 trajectories (50-token chunks). (a) Mean over standard deviation. (b) Logarithm of variance. (c) Excess kurtosis (0 means Gaussian).

Figure: Figure 13: Mistral 7B v0.1. The last layer (32) appears to have an anomalously large variance.

Figure: Figure 14: Llama 3.2 1B. This small model all present an out-of-distribution last layer.

Figure: Figure 15: Llama 3.2 3B. The last layer anomaly is also present.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: In addition to the concerns raised in the weaknesses,

- I find the point being made in the paragraph between lines 195-201 very interesting, but I am then immediately confused by the following paragraph. Could you please clarify this?

- Could you please clarify the paragraph between lines 248-251?
soundness: 3
strengths: - I find the question posed, and the consequent findings very interesting

- I really appreciate the development of a linear approximation to the distribution of trajectories.
summary: The paper aims to study the statistical properties of what they call *lines of thought*; trajectories traced by the embedded tokens through the latent space while traversing successive transformer layers. The key observation is that independent trajectories cluster along a low-dimensional manifold, and that their paths can be approximated using a simple dynamics model.
weaknesses: - One of my biggest gripes with the paper is, while I wanted to be excited about the findings, a recurring question that I had was "why should I care"? It is my opinion that the authors should invest in a motivation for why the reader should care about the presented findings. It's not clear to me what the takeaways are, or more concretely, how we can utilize the observation to develop better LLMs for instance.

- A second gripe, which perhaps goes hand-in-hand with my first one, is that I often felt the paper could have used a bit more hand holding, or even been organized better. To make this concrete, in Section 3.1, the findings are presented first and then the methodology employed to find them was presented, which to me was a bit confusing. Section 3.2 starts abruptly with the sentence "The fast decay of the singular values...". Section 3.4 is titled "Langevin Dynamics..." with no mention of Langevin Dynamics. Furthermore, I often found myself having to reference many different figures in different parts of the paper while reading a single paragraph. All of this made it harder to follow the paper closely.

- (Stylistic nitpicking) I think some of the writing maybe embellishes on details that maybe are not very important e.g. the methodology section starts with what I believe to be details that can be abstracted away. I also think the footnotes are somewhat excessively used.

- I'm not really sure what is meant by "pilot" in this context. My understanding is that a pilot is  "done as an experiment or test before introducing something more widely."

- The authors claim that the pseudo-sentences are "independent" (line 171) which is not immediately clear to me given that the pseudosentences are chunks produced from a single piece of writing.

- I find the notation used in the algorithm confusing. Why do we sometimes use parentheses and other times square brackets? Also, wouldn't indexing with t+1=25 at  the last iteration be undefined?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Some questions were raised above. In addition

- This work reminds me of neural ODEs. Are there any connections between them and the viewpoint this paper explores?
soundness: 2
strengths: - There has been a flurry of works trying to understand the inner workings of LLMs. Therefore, this direction is relevant and interesting.

- This work studies this problem from a unique flow-based perspective by studying the dynamics of the embeddings as they evolve in the layers. The perspective is novel to the best of my knowledge and may potentially lead to a new perspective or algorithm to improve interpretability of LLMs.
summary: This work studies the dynamics of the embedding space as the input tokens go through the layers of a transformer architecture. The authors break the input prompt into sequences of tokens and study the embeddings of the last token as it propagates through the layers. Using PCA-like projections, it's shown that these embeddings approximately lie in a low-dimensional manifold. Then, by rephrasing these trajectories using linear approximations, the authors attempt to model it via Langevin dynamics which gives rise to the standard Fokker-Planck probability flow. One of the main conclusions is that transformers can be "distilled" into few parameters. To validate their assumptions, the authors experimentally study these trajectory on non-language inputs and untrained models and show that their ideas weakly hold. The target audience are people interested in the physics and interpretability of large language models.
weaknesses: - While potentially interesting, the paper feels too vague and very high-level without any concrete theoretical or experimental contributions.

- No new theoretical contributions are made, other than standard langevin dynamics formulations of their ideas. The projection to lower dimensions and linear approximations are also somewhat too lossy, as the authors note, so it's not clear how well the observations here actually hold in real life.

- Experiments seem limited to a few models and as the authors note, the last layer seems to form an outlier for the Mistral 7B and the Llama models. While the authors suggest some reasons for this, it's not inherently clear why these happen and bring the central hypotheses into question.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It is unclear to me what does this sentence (”the uncertainty (or stochasticity) introduced here accounts only for the loss of information of considering the token without its prompt.”) in footnote 8 mean?
2. For figure 3, why u_2 and u_3 without u_1?
soundness: 3
strengths: 1.The idea of this paper is kind of interesting.
summary: This paper studies the trajectory of token embedding across layers. Inspired by dynamic system, they model the trajectories as diffusive process with a linear drift and a modified stochastic component.
weaknesses: 1. the Gaussian assumption seems to not hold in early layers.
2. It is unclear if the same type of paths would hold for larger and more complex model as we already see problems with newer model like LLaMA-3. I think it makes sense to model the intermediate layers with diffusion process but early and last layers might not work not well.
3. The theory here does not lead to any practical predictions. For example, can you use this model to predict next token?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Questions: (annotated with line number/section)
125: The vector for every token is projected to form the logits, not just the last vector. For inference, we only care about the logits in the last token'th position, but that doesn't mean the logits must only come from there.
130: Algorithm 1 is overkill. Just saying "we cache the activations between each transformer block in the sequence position of the last token over a forward pass for each psuedo-sentence $s_i$" would be sufficient. The algorithm is not doing anything interesting that the transformer would not already do during a forward pass (other than the caching of activations).
145: It is advised for ICLR to notate the last token position for layer $t+1$ as $\mathbf{E}^{t+1}_{:, end}$ rather than $\mathbf{E}(t+1)[:, end]$ (check the ICLR formatting guide).
152: The latent space is obviously spanned by a Cartesian basis. Any such space is.
155: You then revert to the ICLR standard notation for indexing a matrix that you did not use on line 145. Pick one and be consistent.
175-176: Replace "Because the Cartesian axes, $\mathbb{e}_i$, are unlikely to align with trajectories meaningful directions," with "There is no reason a priori we would expect the Cartesian axes to align to meaningful directions for the trajectories of the activations"
177: Concatenating the $\mathbb{x}_k(t)$? Over what dimension? $k$ or $t$ or something else?
154: Does it makes sense to plot the trajectories such that at each layer we use a different basis? Seems like you'd want to use the same basis throughout the entire forward pass if possible, as then each step along the trajectory is measured in terms of something different. I guess maybe the size of the principal component, with respect to the most important eigenvector is still interesting. This was partially addressed by Fig. 2a. showing that all the dimensions are important at some point during the forward pass, and that the low-dimensional manifold "wanders around" the entire latent space.


Section 3.2: At first glance this seems a reasonable choice of metric: The KL divergence between the normal output distribution, and the output distribution after having performed dimensionality reduction via SVD. Is this a metric that is defined elsewhere in the literature? Cite if so. I wonder if SVD is the best thing to do here: Is there a better choice of low-dimensional manifold that can only be obtained via a non-linear transformation? Perhaps if an autoencoder was trained that compresses/uncompresses the latent space on forward passes, could you squeeze the activations into an even lower dimensional space while preserving the KL divergence?
However, what isn't clear to me is that how much displacement of KL divergence is a lot, and why $K_0 = 256$ was the value for which "most" of the true distribution is recovered. What is "most"? What makes a KL-divergence of ~0.7 "not much"? It would be good to have some sort of sense of scale what the downstream effects are (e.g. measure the performance on some benchmark before and after throwing away 75% of the activations, and see how much "dumber" the model gets.)

Figure 2(c): What precisely is the baseline defined as? The KL divergence between the output distributions conditioned on two unrelated inputs, or with random noise injected into the unembedding matrix, or something else?

Figure 3: It's not clear how to compare the true and extrapolated positions. If the cluster comparison is desired, maybe colour one red, and the other blue, but both have transparency? So the purple region is the overlap? It's hard to see with the grey just covering the blue. I also would have liked a more principled metric to measure how well the model suggested in Equation (1) captures the dynamics of the activations, and what component remains unexplainable.

255-256: The rotation matrix $\mathbf{R}(t)$ was substituted out for $\mathbf{U}(t)$, but $\mathbb{\Lambda}(t,\tau)$ was not substituted out for $\mathbb{\Sigma}(t+\tau)\mathbb{\Sigma}^{-1}(t)$. I think either substitute both (my preference) or neither.

619: Forgot to bold $\Lambda$
soundness: 3
strengths: * Strong evidence provided to back up the claims made. The authors do a great job of explaining what it is they are measuring and why.
* The result is fascinating, computationally cheap to obtain, model agnostic, and is empirically backed by measuring the KL-divergence of the distribution after down-projecting the dimensionality.
* The theoretical justification on the proposed model is well motivated and explained in the appendix.
* See questions
summary: The authors describe a framework that allows tracing the path taken by the activations in latent space during a forward pass on some input text. They note that despite the complexity of computing a forward pass, the trajectory taken is rather simple, and the activations live on a low-dimensional manifold that moves through latent space as a function of the number of layers processed. They provide a stochastic model that describes the trajectory, with vastly fewer parameters than the network itself. The approximation closely agrees with the actual activations, and can be used to extrapolate the behavior of the model.
weaknesses: * Some claims on the similarity of extrapolated token positions are a bit weaker, and are primarily based on visually comparing the extrapolated and actual activations projected down to a low-dimensional space. I would have liked to have seen a more concrete metric of comparison, together with some baseline values to compare against to indicate that they really are close.
* I would have liked to have seen some applications of this discovery, or at least conjectures for what it could be used for. For example, could one project down the weight matrices into the space provided, and obtain a network that has similar performance and is much smaller, at the expense of off-distribution task degradation? Essentially, a way to distil a pre-trained network down if we only care about performance on one task?
* I would elaborate on the potential use-case for a continuous interpolation between layers. Is there any sort of intuitive meaning we can assign to this?
* See questions

Meta-review: This paper describes an interesting phenomena in language models about the structure of the dynamics of residual stream updating across transformer layers. Reviewers found the idea interesting, the presentation clear, and were overall positive. However, all reviewers were unclear on the importance of this observation. It does not seem useful as either a subroutine in an engineering application, nor as a test of some scientific hypothesis. That is, no reviewer was clear on what it would mean to build on the results here. I strongly encourage the authors to elaborate on this aspect of the results in order to achieve significant impact.

Decision: Accept (Poster)