Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language.
Authors: Eghbal A. Hosseini
Evelina Fedorenko
Venue: NeurIPS 2023 poster (2023)
Keywords: (Cognitive/Neuroscience) Language
Structured Prediction
(Application) Natural Language and Text Processing

Abstract: Predicting upcoming events is critical to our ability to effectively interact with our
environment and conspecifics. In natural language processing, transformer models,
which are trained on next-word prediction, appear to construct a general-purpose
representation of language that can support diverse downstream tasks. However, we
still lack an understanding of how a predictive objective shapes such representations.
Inspired by recent work in vision neuroscience Hénaff et al. (2019), here we test a
hypothesis about predictive representations of autoregressive transformer models.
In particular, we test whether the neural trajectory of a sequence of words in a
sentence becomes progressively more straight as it passes through the layers of the
network. The key insight behind this hypothesis is that straighter trajectories should
facilitate prediction via linear extrapolation. We quantify straightness using a 1-
dimensional curvature metric, and present four findings in support of the trajectory
straightening hypothesis: i) In trained models, the curvature progressively decreases
from the first to the middle layers of the network. ii) Models that perform better on
the next-word prediction objective, including larger models and models trained on
larger datasets, exhibit greater decreases in curvature, suggesting that this improved
ability to straighten sentence neural trajectories may be the underlying driver of
better language modeling performance. iii) Given the same linguistic context, the
sequences that are generated by the model have lower curvature than the ground
truth (the actual continuations observed in a language corpus), suggesting that
the model favors straighter trajectories for making predictions. iv) A consistent
relationship holds between the average curvature and the average surprisal of
sentences in the middle layers of models, such that sentences with straighter neural
trajectories also have lower surprisal. Importantly, untrained models don’t exhibit
these behaviors. In tandem, these results support the trajectory straightening
hypothesis and provide a possible mechanism for how the geometry of the internal
representations of autoregressive models supports next word prediction.

Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language
Eghbal A. Hosseini
Brain and Cognitive Sciences. McGovern Institute for Brain Research
Massachusetts Institute of Technology
Cambridge, MA, 02139 ehoseini phoseini@mit..om edu
Evelina Fedorenko
Brain and Cognitive Sciences, McGovern Institute for Brain Research
Massachusetts Institute of Technology
Cambridge, MA, 02139 evelina9@mit.c edu
Abstract
Predicting upcoming events critical our ability effectively interact with our environment and conspecifics. In natural language processing, transformer models, which are trained on next-wen word prediction, appear to construct ceneral-purpose representation of language that can support diverse downstream tasks. However, we still lack an understanding of how predictive objective shapes such representations. Inspired by recent work in vision neuroscience Hénaff et al. (2019), here test a hypothesis about predictive representations of autoregressive transformer models. In particular, we test whether the neural trajectory of sequence of words in a sentence becomes progressively more straight as it passes through the layers of the network. The key insight behind this hypothesis is that straighter trajectories should facilitate prediction via linear extrapolation. We quantify straightness using a I- dimensional curvature metric, and present four findings in support of the trajectory straightening hypothesis: i) In trained models, the curvature progressively decreases from the first the middle layers of the network. ii) Models that perform better the next-word prediction objective, including larger models and models trained on larger datasets, exhibit greater decreases curvature, suggesting that this improved ability to straighten sentence neural trajectories may be the underlying driver better language modeling performance. iii) Given the same linguistic context, the sequences that are generated by the model have lower curvature than the ground truth (the actual continuations observed in language corpus), suggesting that the model favors straighter trajectories for making predictions. iv) A consistent relationship holds between the average curvature and the average surprisal sentences in the middle layers of models, such that sentences with straighter neural trajectories also have lower surprisal. Importantly, untrained models don't exhibit these behaviors. In tandem, these results support the trajectory straightening hypothesis and provide a possible mechanism for how geometry the internal representations of autoregressive models supports next word prediction.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Biological systems, like brains, and artificial systems, like deep neural networks, construct internal representations in the service of their internal or external objectives. Certain objectives appear to yield representations that are useful across diverse behaviors. For example, representations that are predictive of incoming input have been argued to be useful in both biological systems across perception, action, and cognition (Rao and Ballard 1999 Palmer et al., 2015) Shadmehr et al., 2010 Hohwy et al. 2008|Jessupp et al. 2010: Shain et al. 2020 Frank et al. 2015) and in artificial systems across domains (van den Oord et al., 2018 Radford et al. 2018). Two general approaches have been commonly used in modeling predictive processing in the brain. The first approach leverages information theory (Shannon 1949) to quantify the relationship between the past and current neural states and future inputs (e.g., Bialek et al. 2007; Tishby et al. 2000) Wiskott and Sejnowski] 2002 Palmer et al.] 2015). A key limitation of this kind of approach is that they do not specify how the information about past inputs is actually used to make predictions (see Hénaff 2018 for discussion). spenifs-secciiiclllly interactions between lower- level and higher-level areas- that give rise to a predictive, top-down signal, and the bottom-up error signal (c.g.. Rao and Ballard 1999). This approach faces the challenge of specifying what information is represented in high-level cortical areas, which, for many domains, remains not well understood.
The second approach instead focuses circuit-level
Recently, in the context of visual processing, Henaff (2018; see also Hénaff et al.) 2019) have developed an approach to temporal prediction at an intermediate level of abstraction. In contrast to the information-theory- theoory-roouudeed approaches, which focus on predicting upcoming inputs Palmer et al. 2013, this approach focuses on the internal representation states and on predicting future internal states. The critical insight comes from vision: because a sequence of visual inputs to the retina evolves in nonlinear manner, and are difficult to extrapolate, visual system performs series of transformation to make them casier to predict. The representation of input sequence is transformed to result in straighter the trajectory in the internal state, and allow for linear extrapolation of future states of the sequence. Hénaff et al. (019) found support for this straightening hypothesis in behavioral psychophysies experiments and in neural recordings in the early visual areas of macaques (Hénaff ett al. 2021). They also tested the predictions of this hypothesis in AlexNet (Krizhevsky et al. 2012) an early convolutional neural network for vision, but did not observe representation straightening. They hypothesized that representation straightening may only emerge in systems where the objective has to do with prediction (cf. AlexNet where the objective function is object categorization).
Language is a domain where information unfolds over time and where prediction is a natural objective function. Indeed, many successful language models use next-word prediction as their core training objective (e.g.. Radford et al., 2018). As a result, representational straightening seems priori plausible a mechanism for linguistic prediction. Here, evaluate the straightening hypothesis across four computational experiments. In Experiment 1, we show that across a corpus of approximately 8.5K genaanteeeeated sentences, the average curvature of sentences decreases gradually between the input layer and the deep layers. In Experiment 2, we show that larger models and models that are trained on larger datasets show a greater degree of representation straightening, suggesting that straighter internal representations is what allows for better predictive performance. In Experiment 3, we perform a critical comparison between natural, human-generater sentences and
sentences (created by providing the models with the first few words of the natural sentences) and show that the model-generated sentences have straighter trajectories. Finally, in Experiment 4, we relate sentence curvature to surprisal, a measure of how expected words are in context (Shannon 1949), which has been shown predict human behavior and neural responses to language (Levy 2008 Smith and Levy, 2013; Willems et al.) 2016, Henderson et al. 2016) Lopopolo et al. 2017; Shain et al. 2020 Heilbron et al., 2022). Jointly, these results provide evidence for representation straightening as a mechanistic hypothesis about the computations that allow transformer language models to construct a predictive representation the service of their behavioral objective.

---

2 Methods
2.1 Models
We focused on autoregressive language models (Brown et al. 2020; Radford et al. 2018), specifically the GPT model family (Figure 1A). These models explieitly trained the next-word prediction objective. Each layer consists of main blocks: (i) 1st layer normalization, (ii) self-attention, (iii 2nd layer normalization, and (iv) the feed-forward layer. For most analyses, we used GPT2-XL (48 layers, 1600 embedding dimensions, 25 heads, 1558M parameters). To examine the effects on representational geometry of model training. we used GPT2 (12 layers, 768 embedding dimensions, 12 heads. 1 17M parameters; Radford et al., 2018); and to examine the effects of model size, we additionally used GPT2-Large (36 layers, 1280 embedding dimensions, 20 heads, 774M parameters; Radford et al. 20188) GPT2-Medium (24 layers, 1024 embedding dimensions, 16 heads, 345M pa- rameters; Radford et al. 2018), and distilIGPT2 (6 layers, 768 embedding dimensions, heads, 82M parameters: Sanh et al. 2019). For all analyses, we used pre-trained models from the HuggingFace
library (Wolf et al. 2019)
2.2 Curvature estimation
We developed a curvature metric using the neural trajectory of words (tokens) in sentence that was used in all analyses. To make sure definition of curvature is similar (Hénaff al. 2019), which would facilitate relating our findings to the prior work that we are building on, we adopted their metric and simply equated words (tokens) visual frames. We used tokens for all analyses except for the analysis where we related curvature to n-gram word surprisal, in which case we used words instead of tokens. Given sequence of words, W1, W22 .... Wn, we first extracted the activation weights from each layer of the network, starting from the first contextualized layer Lo- Considering layer Lp activation as states xF, .r.2..... ....,, we then computed vectors vf, ... as the difference between adjacent states vf 2 z (Figure IB). We calculated eurvature as the angle between these vectors, namely:
vp 1 11 11
c
arccos
We computed average curvature across the sentence.
Cº
We then computed a change in curvature for each sentence between each layer P and the first layer, obtaining a value for each layer:
AC. C.
Ch
&n
Finally, we computed the average change in curvature across all sentences for each layer P as:
PCP
AC
2.3 Sentence corpus
We used the Universal Dependencies corpus (de Marneffe et al. 2021) to sample diverse set of sentences. The corpus includes texts on diverse topics that come from books, newspapers, and web-based sources. We filtered sentences to only include 100K most common nouns in English Brants and Franz, 2006) and additionally removed abbreviations and capitalized words. In addition, ensure that have sufficient sensitivity estimate curvature value per sentence, the sentences were constrained to between 6 and 19 words. We then used the resulting 8408 sentences to extract model representations, and refer to this corpus as UDsubset8408.
2.4 Model training
In order to investigate the effects on representational geometry of model training, we trained GPT2 (12 layers) model with context window of 1024 using the GPT-NEOX library which is distributed
3

---

training framework that utilizes the DeepSpeed library (Aminabadi et al., 2022; Black et al. 2022).. For the training datasets, following Hosseini Hosseeii et al. 2022] we combined Book Copus and English Wikipedia (Zhu et al. 2015 Liu et al. 20199 in 1:3 ratio, and created 4 different datasets consisting of million, 10 million, 100 million, and billion tokens. The details of the training are similar to Hosseini et al. (022)) In particular, models with random weight initialization were trained on the next-word prediction objective, with context size of 1024 tokens, and batch size of 128, on 4 NVIDIA RTX A6000 GPUs, with maximum training duration of week. We trained each model until it reached its best validation loss for next-word prediction, with the same validation dataset used across models. (In addition to the critical goal of evaluating the effects of model training on representational geometry, this analysis helps evaluate the robustness of the main results to implementation details.)
2.5 Model sequence generation
In order to test whether models favor straight trajectories, we compared the curvature for a set of natural, human-generated sentences and sentences generated by the model from the same ini- tial prompt. To perform this experiment, we selected a subset of the UDsubset840$ corpus (see Sentence Corpus) such that each sentence contained at least tokens, which amounted to 5815 sentences. These sentences (cut off at 10 tokens) constituted the ground truth condition. To create the condition, we provided the first 3 tokens to the model, and allowed the model to generate the remaining 7 tokens in greedy fashion (i.e., by having the model select the token with maximum probability at each step).
2.6 Relating sentence surprisal to curvature
To investigate the relationship between curvature and a behaviorally relevant measure of human language processing, we turned to surprisal. Surprisal, a measure of how unexpected word is in particular context, has been shown relate comprehension difficulty in both behavioral investigations and brain imaging studies (c.g., (Levy 2008; Smith and Levy 2013) Willems et al. 2016 Henderson al. 2016 Lopopolo et al. 2017: Shain al. [2000] Heilbron et al. 2022)). For each of the sentences in the UDsubset840$ corpus, we computed an average sentence surprisal using the 3-gram measure (Brants and Franz 2006 Piantadosi et al. 2011).
Surprisal(w;; [w,|w,,,,, (x-1)
log2 P P( W, 2, Wn-1)
We then computed a Pearson correlation between average sentence surprisal and average sentence curvature for each layer of the model.
3 Results
3.1 Experiment 1. The curvature of sentence representations decreases across the model layers.
We first tested whether the trajectory of sentences becomes progressively more straight in the internal states of the model across layers. We computed the average sentence curvature for the 8,408 sentences in the UDsubset8408 set across all layers of the GPT2-XL model (see Methods). In Figure 2A, we plot for each layer (column), the difference in curvature relative to the curvature in the first layer for each sentence (row). We indeed observed substantial drop in curvature from the early to the middle layers (as evidenced by darker colors in middle). Beyond the middle layers, where we see a reduction in curvature for all sentences, the curvature trajectories are somewhat variable across sentences: for some sentences, the curvature remains relatively constant from the middle to the late layers, for others, it increases towards the values in the carly layers. The increase in curvature (on average) in the later layers plausibly due to the fact that the model needs to eventually map its representations back to words in the output, and the word space is inherently nonlinear.
To ensure that curvature reduction not due to the model architecture alone, we performed the same analysis across the layers of an untrained GPT2-XL model (Figure 2B). We did not observe any systematic change in the curvature values across the layers. To better illustrate the difference in the curvature patterns between the trained vs. the untrained model, we selected 300 sentences with maximum curvature drop (the biggest change between the first layer and any of the subsequent layers) for the trained model and 300 sentences with maximum curvature drop for the untrained model. The

---

Autorogressive
The quick brown Tox jumps over the lazy
quick
brown
fox
Jumps
over
lazy
dog
Early layers
Deep layers
Sentence trajectory:
less straight
more straight
Untrained Model
-
Trained Model
C2
o C Cee-ggiege 0
AC
of.
straightenii gihteiiiigg
Curvature change: ACP
AC
model layers
Figure 1: A. Top: The structure of autoregressive language models, like GPT2, used in the current study. The representation each word (token) is extracted from each individual layer of the model. Bottom: The putative sentence trajectory in the state space of the model shown for earlier vs. deeper layer. In early layers, the trajectories are less straight, which makes the next state more difficult to predict (depicted by wide angle in the shaded red region). In deeper layers, the trajectories are more straight, which makes the next state easier to predict (depicted by a narrower angle in the shaded red region). B. Sentence curvature computation. The graph shows putative trajectory for a sample 8-word-long sentence the multi-dimensional representational space of the model. For each sentence in each layer, computed sentence curvature the average of the angles between the vectors that connect adjacent words (C1, C2, and so on). We then computed a change in sentence curvature between each layer and the first layer. Finally, we computed the average change in curvature between each layer and the first layer across a large set of natural sentences (see Methods). C. The predictions of the representation straightening hypothesis for trained (blue dots) versus untrained (gray dots) models across layers. The y-axis represents the amount of curvature change between each layer and the first layer (lower values correspond to greater change curvature). The hypothesis predicts that for trained, but not untrained, models, the curvature of sentences should decrease from the early to the deeper layers so as to enable efficient next-word prediction (the red-shaded portion of the graph). The curvature may then go back up for the layers that are closest to the output layer because the word space highly nonlinear.
curvature values show a clear separation between the trained and the untrained model after layer 10 for these sentences (Figure 2C), as well for the full of 8,408 sentences (Figure 2D). Overall then, we found a robust reduction in the curvature of sentence representations, from early to middle layers of GPT2-XL, and only for the trained version of the model.
3.2 Experiment 2. The curvature of sentence representations decreases to greater extent in larger models and with more training.
In the literature on foundation models, model performance on the next-word prediction task scales proportionally to the model size and the training dataset size (Kaplan al. 2020). We tested whether curvature reduction may provide mechanistic-level explanation for these relationships in terms of internal model states.
To test the effect on curvature of model size, we computed the average sentence curvature for the same set of 8,408 sentences used in Experiment across all layers of class GPT2 models that vary in the number of parameters (82, 117, 345, 774, and 1,558 million parameters). Replicating the basic finding for GPT2-XL. from Experiment we observed drop in curvature from the early to the deeper layers in the four smaller models (GPT2-large, GPT2-medium, GPT2, and distilIGPT2; Figure 3A). Critically, larger models exhibited larger decreases in eurvature. The average change in curvature shows logarithmic relationship with model size (Figure 3B).
5

---

GPT2-XL (trained)
10
Curvature change (deg)
Curvature change (deg)
Trained
Sentence number
Untrained
2K
-5
-5-
6K
-10
-10
-15
-15
-20
-20
10
20
30
40
0
10
20
30
40
GPT2-XL (untrained)
Curvature change (deg)
Trained Untraiined
Sentence number
2K
0
6K
-15
10
20
30
40
0
10
20
30
40
Model layer
Model layer
Figure 2: A-B. Curvature changes (relative to the first layer) across the layers of the network (GPT-2-XL) (columns) for sentences in UDsubset840S set (rows). For the trained model (A), consistent drop in the curvature is observed from the early to middle layers of the network. No such drop seen for an untrained GPT-2-XL model (B). Curvature changes across the layers of the trained and untrained network (red and gray dots, respectively) for a set of 300 sentences selected separately for the trained and untrained models as having a maximum curvature drop. Individual lines correspond to sentences. A clear separation between the trained and untrained model is observed after layer 10. D. Average curvature changes across the layers of the trained and untrained network (red and gray dots, respectively) for all sentences in UDsubset8408 set. Error bars show standard deviation over sentences in each layer.
To test the effect curvature of training corpus size, independent of model size, trained four GPT2 models (12 layers) using datasets with a controlled number of words, similar to Hosseini scaled logarithmically in size (1 million, 10 million, 100 million, and billion words). After training, we computed the average sentence curvature for our set of 8,408 sentences across all layers of each model. As be seen in Figures 3C-D, the model trained million words exhibits minimal change in curvature (close to an untrained model). The models trained 10 million and 100 million words exhibit progressively larger decreases in eurvature. Moreover, the layer with the largest curvature reduction shifts from the earlier to the deeper layers as function of training corpus size. Interestingly, the model trained billion words does not show further reduction in curvature, reaching a similar level of curvature reduction as the model trained on 100 million words.
al. 2022 The datasets
Thus, both model size and training corpus size affect the curvature of sentence representations, such that larger models and models trained more data achieve straighter representations in the deep model layers. We hypothesize that, mechanistically. the greater degree of representation straightening what leads to better next-word prediction performance in larger models and models trained more data.
3.3 Experiment 3. The model favors straight trajectories during language generation.
If models rely on representation straightening to make predictions about upcoming words, then the trajectories of sentences that are generated by the model should be straighter than natural, human- produced sentences, given that next-word prediction is not the only objective that guides human language production (Figure 4A). This constitutes perhaps the most direct test of the representation straightening hypothesis. To test whether models favor straight sentence trajectories, we designed

---

DistIGPT2
GPT2-L
Curveture change (deg)
GPT2
GPT2-XL
-2
GPT2-M
Curvature change
-6
-10
-10
-12
-12
-14
10
20
30
40
10a
10'
Model layer
Model
(millions of parameters)
0
Curvatture change (dog)
Curvature (deg)
GPT-2 trained on
+
million
million
10" (millions words)
10'
10"
layer
Training
A. Curvature changes (relative to the curvature in the first layer) across the layers of the network for five GPT2-class models different sizes (each model a line; darker lines=smaller models; each dot is layer) for the 8,408 sentences in the UDsubset8408. For all models, consistent drop in curvature is observed from the early to deeper layers of the network. However, larger models, with better next-word prediction performance, exhibit greater curvature reduction. B. The relationship between model size and curvature change for the layer with the largest average curvature reduction. C. Curvature changes across the layers of the network for four versions of GPT2 (gpt-neox implementation; Black et al. 2022) trained on different-size corpora (each model is a line; darker lines=models trained on smaller corpora; each dot layer). For all models except for the one trained on the smallest corpus (IM tokens), consistent drop in curvature is observed from the early to deeper layers of the network. Models trained on more data exhibit greater curvature reduction, but this effect appears to plateau with datasets larger than 100M tokens. (Note that here we examine curvature changes across layers (relative the first layer): absolute curvature continues decrease for larger training datasets (not shown). D. The relationship between training corpus size and curvature change for the layer with the largest average curvature reduction.
Figure 3:
controlled experiment. We first selected subset of sentences from our set of 8,408 sentences that consist of at least 10 tokens (n=5,815 sentences). These corpus-extracted sentences constitute our ground-truth condition. We then created alternate versions of these sentences by providing the model (GPT2-XL) with the first tokens each sentence and allowing the model to generate a sequence of 7 tokens continuation (model-generatee condition; for example sentence pairs, Figure 4B). then compared the curvature for the ground-truth sentences (cutting them off at 10 tokens) vs. the model-generated sentences.
The pattern of curvature change is similar between the two conditions in the early layers. Critically, starting around layer 7, the model-generater sentences exhibit a larger drop in curvature, and this between-condition difference increases over the subsequent layers, peaking around layer 20 (Figure 4B). The sharper decrease in curvature for the model-generated sentences is predicted by the representational straightening hypothesis. These results also generalize to smaller models and to prompts of different lengths (not shown here).
7

---

A
Representation in deep layer
o00e
Example:
Context
young pup and over very good player, he is
Ground-truth
The
Curvature change (dag)
Model-generation
Ground-truth (less straight)
broad
jumpa
-8
-8
Model-pe generation (more straight)
-10
Model layer
Figure 4: A. The predictions of the representation straightening hypothesis for the ground-truth (natural, buman-produces sentences (red line) vs. model-generated sentences (blue line). The hypothesis predicts that the trajectories of the model-generated sentences should be straighter given that linear extrapolation in the internal state space hypothesized to serve as the critical prediction mechanism. In other words, if the model is internally producing low-curvature trajectory, then self-generated sequences should have lower curvature than sequences generated by humans (given that next-word prediction is not the only objective that guides human language production). B. Curvature changes across the layers of the network for 5,815 pairs of 10-token sentences: the ground-trun sentences (red line; each dot is layer) come from the UDsubset8408 set and the penel-ggeerated sentences (blue) are generated from the prompt consisting of the first three tokens of the ground sentences (the model generates the subsequent 7 tokens using a greedy approach; Methods). Model- generated sentences show a greater drop curvature reduction relative to the ground truth sentences. An example of ground-truth vs. model generated sentence is shown on top of the panel.
3.4 Experiment 4. The curvature of sentence representations correlated with sentence surprisal.
In Experiments 1-3, have focused on language models and established that models reduce the curvature of sentence trajectories in their internal state spaces. Using a set of 8408 sentences, we found consistent curvature reduction in the deep model layers across this corpus. However, also observed some variability among the individual sentences their curvature patterns across the layers (Figure 1C). In Experiment 4, we attempted to connect this variability in the geometry of sentence representations in the model to some linguistic features of the sentences that we know affect human language processing. In particular, we focused on surprisal-a measure of how expected word is given context. Surprisal has been shown to affect behavioral (Levy) 2008 Smith and Levy 2013) and neural (Willems et al., 2016 Shain et al. 2020) responses language. So we asked whether average sentence surprisal (we used 3-gram surprisal, averaging across the words to derive a single measure for each sentence; see Methods) relates the curvature of the sentence's 's trajectory in the model space. If higher curvature in the model space corresponds to greater difficulty in predicting the next word, we should observe a positive relationship between curvature and surprisal: sentences that are overall less predictable (more surprising) should have higher curvature. Moreover, this correlation should only be observed in deeper layers (Figure 5A).
The results are shown in Figures 5B-C. As expected, for the untrained model, we found no relationship between surprisal and curvature correlation across layers, except for layer 0). However, for the trained model, the correlation starts to increase from the early layers toward the deeper layers, peaking around layer 20 (Figure 5C). This result suggests that the degree which the model straightens sentence trajectory internally is associated with how surprising the sentence is behaviorally. These results also generalize to smaller models (not shown) and to other surprisal metrics (other n-gram metrics and PCFG-parser-baase surprisal; see Supplementary Information).
4 Discussion
In this work, building on Hénaff and colleagues' proposal for primate vision (Hénaff et al. 2019
Hénafff 2018 Hénaff et al. 2021). we established neural trajectory straightening as a representa- tional hypothesis about how neural network language models perform next-word prediction. Models consistently reduced sentence curvature from early to middle layers, and this effect only observed
8

---

A
Early loyers
130
GPT2-XL (Layer 1)
Curvature
0.20
e 00000
Trained Untrained
condlation
120
Cornelation between
curveture and surpsjal
0.15
Avg.
0.0
significant
curvanuro
110
0.05
surprisal
GPT2-XL (Layer
Deep
120-
0.10
Curvature
0 0000
High correiation
110
0.00
100
-0.05
20 Model layer
40
Average surprisal
Figure 5: A. A hypothesized relationship between curvature and surprisal in the early layers (top) and the middle layers (bottom) of the network. In early layers the representations are close to input, the curvature is not correlated with sentence surprisal. In deep layers however the curvature is predictive model states, so there is consistent correlation between the sentence curvature and sentence surprisal. B. The relationship between average sentence surprisal (3-gram surprisal; see Methods) and average sentence curvature for sentences in the UDsubset84088 set in layer 1 (top) vs. layer 20 (bottom). The lines show a linear regression fit to the data (for illustrative purposes). C. Pearson correlation between average sentence surprisal (3-gram) and average sentence curvature across the layers of the network for an untrained model (gray-contour dots) and trained model (black-contcc for the layers where the correlation reached significance are color-filled (grey for untrained; red for trained). For the untrained model, no relationship is observed in any of the layers, except for layer O. For the trained model, there is consistent increase correlation between curvature and surprisal from the early to middle layers.
dots). The data points
for trained models (Figure 2). Model size and training dataset size affected the model's ability to reduce curvature (Figure 3), and model-generated sentences exhibited lower curvature compared to natural human--enerated sentences (Figure 4). Finally, average sentence curvature correlated with average sentence surprisal in the middle layers of the model Figure 5).
Our results sit squarely within the efficient coding framework and establish temporal prediction as form of efficiency that is born out of training autoregressive transformer models on next-word prediction. These findings may also explain why representations from the middle layers of transformer language models align best with human neural responses during language processing (e.g., |Schrimpf
ct al. [221 Goldstein et al., 2022, Caucheteux and King 2022 Caucheteux et al. 2023 Toneva and and Huth 2018).
Wehbe
Using representational geometry to understand the relationship between the internal workings of artificial and biological systems and their behavior has gained momentum in recent years( Wang al. 2018 Remington et al.) 20188) Mante al., 2013 Chung et al. 2018 for a review, see Chung and Abbott 2021). For example, in a deep neural network of vision, [Cohen et al., 2020 showed how the geometry of object manifolds changes across model layers and how it relates to model performance in object categorization. In the domain of lang, Hewitt and Manning 2019 found that the representations the middle layers of BERT best capture the hierarchical structure of sentences In another line of work, Mamou et al. 2020 used manifold analysis to uncover how different features of words (e.g., part of speech) and sentences become separable in the deep layers of language models like BERT and GPT. More recently, Valeriani et al., 2023 investigated how geometric properties, such as intrinsic dimensionality, change across the layers of the transformer models, and found that dimensionality increases across layers before sharply decreasing in deeper layers of the bidirectional transformer models. These prior studies provide insights into the geometric propertics of vision and language models, but, to our knowledge, no prior study has evaluated representationall-eww hypothesis that connects language model behavior (next-word prediction) to neural trajectories of individual sentences.
In the representation straightening hypothesis, the problem of predicting the next token/word reformulated as predicting the next state in the model's internal representation. Upon predicting the next state, the model can connect this new state to the features of the input/output (i.e., to words). Explicitly defining model's objective to build predictive representation over its internal

---

representation, and not output, and would be an interesting future direction (Olshausen and Field 1996). Another direction would be evaluate representation straightening in human behavioral or neural responses to language. Finally, if representation straightening is general mechanism for temporal prediction, should be evident in other systems, biological and artificial, that have a core prediction objective; it would be important to evaluate the generality of this mechanism.
is important to note that in the representation straightening hypothesis, the problem of predicting the next woren/word is reformulated as predicting the next state in the model's internal representation. Upon predicting the next state, the model can then connect this new state to the features of the input/ouiput (i.e., to words). Explicitly defining such transformation stages, would allow the model to build a predictive representation without any apparent behavior and would be an interesting future direction. Another exciting direction would be to evaluate representation straightening in human neural language data direction. Finally, if representation straightening is a general mechanism for temporal prediction, it should be evident in other systems, biological and artificial, that have core prediction objective; it would be important to evaluate the generality of this mechanism.
5 Broader Impact and Limitations
Our work puts forward and provides support for a general hypothesis at the representational level about mechanism that allows large language models achieve good performance on next word prediction and potentially downstream tasks. This work adds to the growing body of research on the interpretability of Al models. A better, more mechanistic understanding of these models, and potentially other models with prediction objectives, can both i) suggest ways to improve model efficiency and robustness, and ii) provide insights into the relationship between neural networks and the human language system.
We acknowledge that our work could be improved in several respects. The results they stand are compatible with at least two possibilities: (i) that predicting future inputs intrinsically gives rise to implicit next-state prediction, thus directly favoring linear state dynamics, (ii) that predicting future inputs in domains like language benefits from slowly-changing contextual information, thus indirectly favoring slower (and more approximately linear) state dynamics. These possibilities can be distinguished in the future by training models on artificially created datasets that vary in the length of context that affects the predictability of an incoming element. If the effects we report here obtain across these different training datasets, that would support the first possibility; if instead the effects only hold for models trained on data where relatively long predictive contexts, that would support the second possibility.
Furthermore, we have not evaluated the effects on sentence curvature of other training objectives or fine-tuning for downstream tasks. Doing so can help understand the selectivity of the observed effects (i.e., do sentence representations get straightened in the middle layers only under the pressure of the next-word prediction objective?) and their robustness to adding other objectives to a pre-trained model. We have also not causally tested the straightening hypothesis, which would require ablating the model in such a way that only curvature is affected, and testing how next-word prediction behavior changes.
is also important to note that we are not claiming that representation straightening is the only mechanism that models rely to gain linguistic competence. However, to the extent that prediction is a core part of language learning and processing (in artificial as well as biological systems), we are showing that targeted inspection of geometric properties of sentence representation gives rise to hypothesis about how prediction may be implemented in language models.
6 Acknowledgements
The authors would like to thank Eero Simoncelli, Olivier Hénaff, Yoon Bao, and Cory Shain for helpful insights and discussions, Anya Ivanova, Chengxu Zhuang for feedback on the manuscript, as well as members of EvL_ab at MIT. EF was supported by NIH grants R01-DC01695, and U01- NS121471, and by funds from the McGovern Institute for Brain Research, the Brain and Cognitive Science Department, the Simons Center for the Social Brain, and MIT Quest for Intelligence.
10

---

References
Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. (2022). DeepSpeed inference: Enabling efficient inference of transformer models at unprecedented scale.
Bialek, W., van Steveninck, R. R. d. R., and Tishby, N. (2007). Efficient representation as design principle for neural coding and computation.
Black, S., Biderman, S., Hallahan, E., Anthony, Q.. Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L.. Tow, J., Wang, B., and Weinbach, S. (2022). GPT-NcoX-2005 An Open- autoregressive language model.
Brants, T. and Franz, A. (2006), Web 1t 5 5-gram version LDC2006T13. Web Download.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are Few-Shot learners.
Caucheteux, C.,, Gramfort, A., and King, J.-R. (2023), Evidence of predictive coding hierarchy in the human brain listening to speech. Nat Hum Behav, 7(3):430-44
Caucheteux, C. and King, J.-R. (2022). Brains and algorithms partially converge in natural language processing. Commun Biol, 5(1):134.
Chung, S. and Abbott, L. (2021). Neural population geometry: An approach for understanding biological and artificial neural networks. Curr. Opin. Neurobiol. 70:137-144.
Chung, S., Lee, D. D., and Sompolinsky, H. (2018). Classification and geometry of general perceptual manifolds. Phys. Rev. X, 8(3):031003.
Cohen, U.. Chung, S., Lee, D., and Sompolinsky, H. (2020). Separability and geometry of object manifolds deep neural networks. Nat. Commun., 11(1):746.
de Mamneffe, M.-C., Manning, C. D., Nivre, J., and Zeman, D. (2021). Universal dependencies. Comput. Linguist. Assoc. Comput. Linguist., pages 1-54.
Frank, S. L., Otten, L. J., Galli, G., and Vigliocco, G. (2015). The ERP response to the amount of information conveyed by words in sentences. Brain Lang., 140:1-11.
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, A., Feder, A., Emanuel, D., Cohen, A., Jansen, A., Gazula, H., Choe, G., Rao, A., Kim, C., Casto, C., Fanda, L.,
Doyle, W., Friedman, D., Dugan, P., Melloni, L., Reichart, R., Devore, S., Flinker, A., Hasenfratz, L., Levy, O., Hassidim, A., Brenner, M., Matias, Y.. Norman, K. A., Devinsky, O., and Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nat. Neurosci. 25(3):369-380.
Heilbron, M., Armeni, K., Schoffelen, J.-M., Hagoort, P., and de Lange, F. P. (2022). A hierarchy of linguistic predictions during natural language comprehension. Proc. Natl. Acad. Sci. U. S. A., 119632):22011968119
Hénaff, O. (2018). Testing Mechanism for Temporal Prediction in Perceptual, Neural, and Machine Representations. PhD thesis, New York University, Ann Arbor, United States.
Hénaff, O. J., Bai, Y., Charlton, J. A., Nauhaus, I., Simoncelli, E. P., and Goris, R. L. T. (2021). Primary visual cortex straightens natural video trajectories. Nat. Commun, 12(1):5982.
Hénaff, O. J., Goris, R. L. T., and Simoncelli, E. P. (2019). Perceptual straightening of natural videos. Nat. Neurosci.
Henderson, M., Choi, W., Lowder, M. W., and Ferreira, (2016). Language structure the brain:
A fixation-related IMRI study of syntactic surprisal in reading. Neuroimage, 132:293-300.

---

Hewitt, J. and Manning, C. D. (2019). A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-4138.
Hohwy, J., Roepstorff, A., and Friston, K. (2008). Predictive coding explains binocular rivalry: An epistemological review. Cognition, 108(3)6687-71.
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., and Fedorenko, E. (2022). Artificial neural network language models align neurally and behaviorally with humans even after developmentally realistic amount of training.
Jain, S. and Huth, A. (2018). Incorporating context into language encoding models for IMRL. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31, pages 6628-6633 Curran Associates, Inc.
Jessup, R. K., Busemeyer, J. R., and Brown, J. W. (2010). Error effects in anterior cingulate cortex reverse when error likelihood is high. Journal of Neuroscience, 3069):3467-347..
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R.,. Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolu- tional neural networks. volume 60 of 25, pages 84 94.90.
Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3): 117-17777
Liu. Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach.
Lopopolo, A., Frank, S. L., van den Bosch, A., and Willems, R. M. (2017). Using stochastic language models (SLM) to map lexical, syntactic, and phonological information processing in the brain. PLoS One, 12(5):e0177794
Mamou, J., Le, H., Del Rio, M., Stephenson, C., Tang, H., Kim, Y., and Chung, S. (2020). Emergence of separable manifolds in deep language representations.
Mante, V., Sussillo, D., Shenoy, K. V., and Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78 84.
Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 38 (6583):607-669.
Palmer, S. E., Marre, O., Berry, 2nd, M. J., and Bialek, W. (2015). Predictive information in a sensory population. Proc. Natl. Acad, Sci. U. S. A., 112(22):6008-6613.
Piantadosi, S. T., Tily, H., and Gibson, E. (2011). Word lengths are optimized for efficient communi- cation. Proc. Natl. Acad. Sci. U. S. A., 108:9):552-3322..
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understand- ing by generative pre-training.
Rao, and Ballard, D. H. (1999). Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-fiels effects. Nat. Neurosci., 2(1):79-87.
Remington, E. D., Narain, D., Hosseini, E. A., and Jazayeri, M. (2018). Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron, 1019..
Sanh, V., Debut, L.,, Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proc. Natl. Acad. Sci. U. S. A., 118(45).
12

---

Shadmehr, R., Smith, M. A., and Krakauer, J. W. (2010). Error correction, sensory prediction, and adaptation in motor control. Annu. Rev. Neurosci, 33:89-108.
Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., and Fedorenko, E. (2020). fMRI reveals language-specifie predictive coding during naturalistic sentence comprehension. Neuropsychologia, 138:107307.
Shannon, C. E. (1949). Communication theory of secrecy systems. Bell Syst. Tech. J., 2564):656-
Smith, J. and Levy. R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 320(3)302--319,
Tishby, N., Pereira, F. C., and Bialek, W. (2000). The information bottleneck method.
Toneva, M. and Wehbe, L. (2019). Interpreting and improving natural-language processing (in machines) with natural language- (in the brain). In Wallach, H., Larochelle, H., Beygelz- imer, A., dAlché-Bue, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. pages 14954 14964. Curran Associates, Inc.
Valeriani, L., Doimo, D., Cuturello, F., Laio, A., Ansuini, A., and Cazzaniga, A. (2023). The geometry of hidden representations of large transformer models.
van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding.
Wang, J., Narain. D., Hosseini, A., and Jazayeri, M. (2018). Flexible timing by temporal scaling of cortical responses. Nat. Neurosci,, 21(1):102-110.
Willems, R. M., Frank, S. L., Nijhof, A. D., Hagoort, P., and van den Bosch, A. (2016). Prediction during natural language comprehension. Cereb. Cortex, 26(6):25062216.
Wiskott, L. and Sejnowski, T. (2002). Slow feature analysis: unsupervised learning of invariances. Neural Comput., 14(4):715-770.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. (2019). HuggingFace's transformers: State-of-the of natural language processing.
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.
13


Reviews:
[{"review_id": "B6jyY6kRwt", "reviewer": "Reviewer_ycZy", "summary": "This paper provides evidence that autoregressive language models - specifically the GPT family straighten the internal trajectory of word sequences, making them more linear, in order to better predict next words. They show that trained models decrease sequence curvature across layers, larger models straighten more, model-generated sentences are straighter, and curvature correlates with unpredictability.", "questions": "Does straightening occur in LSTM or GRU sequential models? Or is it unique to transformer self-attention?\nIf we remove layers or transformer components like self-attention or the feed forward layer is straightening impeded?\nDoes straightening also happen in other transformer-based models but with a different pertaining objective such as BERT ?\n", "limitations": "The authors have a sufficient limitations section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces computational evidence for the trajectory straightening hypothesis using a simple and intuitive curvature metric and backs it up with various experiments. It is also well-written and raises exciting questions about the working/interpretability of these models.", "weaknesses": "The paper did not perform any ablation studies to see what is causing the trajectory straightening. Removing different components of the transformer like the feed-forward layer, and seeing the effects on straightening may lead to some more insights. It is not clear if straightening depends on the transformer architecture specifically or also occurs in other model architectures like LSTMs or vanilla RNNs Do similar dynamics occur in MLPs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\nThe paper did not perform any ablation studies to see what is causing the trajectory straightening. Removing different components of the transformer like the feed-forward layer, and seeing the effects on straightening may lead to some more insights. It is not clear if straightening depends on the transformer architecture specifically or also occurs in other model architectures like LSTMs or vanilla RNNs Do similar dynamics occur in MLPs?\n\nWe appreciate the reviewer for pointing this out. To address this, we performed ablation on GPT2-XL autoregressive model (Supplementary figure 1). We performed ablation at different layers of the network [5,15,25,35,45], and for each layer on individual modules [attention head, attention projection, MLP]. To do the ablation in each module we set the weights with identity matrix and biases to 1. Specifically, for attention head we replace the weight for Key matrix to identity, for attention projection we replace the weights with identity, and for MLP we replace the two weight matrices (h*4h, 4h*h, where h=hidden size=1600) such that the effective weight is identity. This approach allowed us to do minimal disruption in on model and observe how the representation of ablated layer is transformed.  For computational considerations, in each experiment we tested 500 randomly sentences sampled uniformly. \nWe observed 2 main effects. First, ablation of early layers has more consequence on curvature in the succeeding layers. Second, we observed that the ablation of attention mechanism has leads to largest deficit in reducing curvature in the succeeding layers. This suggest that attention mechanism is causing the straightening. \nWe also tested recent transformer like RNN model ((Peng et al. 2023)) and observed similar straightening to transformers (Supplementary figure 3). \n\nQuestions:\nDoes straightening occur in LSTM or GRU sequential models? Or is it unique to transformer self-attention? If we remove layers or transformer components like self-attention or the feed forward layer is straightening impeded? Does straightening also happen in other transformer-based models but with a different pertaining objective such as BERT?\n\nWe thank reviewer for bringing this issue up. This is a very important question that we intend to address in our continuation of this work. Our current results points towards objective function as the main driver of straightening. As reviewer suggested, we tested the straightening hypothesis in a bidirectional model ( Bert-large-uncased, Supplementary figure 2).  We observed that a different pattern of curvature across layers. Early layers do not show a gradual drop in curvature, something that we observed in autoregressive transformer models. instead, there was a drop in curvature in deeper layers. Importantly there is no reliable relationship between surprisal and curvature for across the layers. What could contribute to curvature reduction in deeper layers of BERT. It is possible that a masked language modeling objective still share some similarities to an autoregressive objective. For example, cases masked words that are towards late part of a sequence of tokens, the model can use information from past words to predict missing masked token. \n\nLimitations:\nThe authors have a sufficient limitations section.\n \n"}, {"review_id": "Mmk4pNseBD", "reviewer": "Reviewer_S19m", "summary": "This work examines the an hypothesis regarding neural trajectory straightening as a mechanism, by which neural language models achieve next word prediction. Specifically, this hypothesis connects between the objective of next word prediction and extrapolation to the embedding of the next word in neural representation space. They define layer curvature based on prior work and find that (1) autoregressive LMs consistently reduced their curvature from early to middle layers, and this effect was only observed for trained models; ,odel size and training dataset size affected the model’s ability to reduce curvature; model-generated sentences exhibited lower curvature compared to natural human-generated sentences; average curvature correlated with average sentence surprisal in the middle layers of the model.\n", "questions": "- Why only train and not compare to other models?\n- The y-axis in your reported plots is \"curvature change\" What was the distribution of the original absolute curvature angle in the first layer? If you report only a relative number, how can I know that curvature drop makes it close to zero  doesn’t end up in a negative degree, rendering the conclusion on \"straightening\" incorrect? To be clear I'm pretty sure / hopeful that the authors will have a good answer and that \"straightening\" indeed takes place, but this represents some weakness in the presentation. \n- I found this specific sentence very odd: \"The results generalize to smaller models and other surprisal metrics (other n-gram metrics and PCFG-parser-based surprisal - not shown here)\" why do the authors mention other related results but do not show them?\n- Why did you use n-gram surprisal and not LM perplexity in the forth experiment? The first is model agnostic and the latter is model related, so maybe you should have ran both, but I find perplexity very natural to use here.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper explores an interesting hypothesis regarding the connection between an internal geometric property of Transformer based LMs and their performance. If this was not examined in prior work, I find the questions posed in all 4 experiments novel and interesting, and the results non-trivial. In particular, I liked the thoroughness in testing both different model sizes and different training set sizes for the same model size, that did a good job in removing an important confounder in my opinion (though not enough discussion and experimentation regarding the point in 1B tokens training set that broke the trend)\n", "weaknesses": "I find that this paper can be strengthened from several different angles: \n\n- Discussion of prior work: I am not familiar with literature on geometric interpretability of language models. This paper is on this exact topic but does not convey sufficient background on related work.\n- Question scope: the focus on one specific geometric measure limits this paper’s strength, to me it seems that several related measures can be examined. Alternatively, though it’s intuitive, I find the focus on this specific geometric measure as not sufficiently motivated. \n- Depth of investigation: For each experiment, only the basic setup was ran and often there was not sufficient discussion on the outcome or follow up experimentation (eg, what happens in the second half of the network? Why did the 1B token experiment not show the same trend as 1M, 10M, 100M?)\n- Several experimental design choices were not sufficiently motivated (Why only one dataset of 8,408 sentences? Why constrain sentences to be between 6 and 19 words long, and to not contain abbreviations or uncommon words?)\n- writing and presentation. There were several clumsy sentence phrasings (eg, first intro sentence) and some typos (eg, mid sentence capitalization line 121). More importantly, some core quantities were not adequately presented (eg, no formula given for the employed 3-gram surprisal metric), and the figures were generally pretty hard to decipher (eg, What does figure 4A mean? I didn’t understand what quantity is referred to by the title: “The predictions of the representation straightening hypothesis”. What do the axes of this plot correspond to?). As mentioned above, I found the results sections not written well enough, often reiterating the premise and the intuition and not conveying and discussing the actual experimental outcome clearly enough. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Discussion of prior work: \nThanks for the reviewer for bringing this issue to our attention, we will include a more through discussion of prior work in our revised manuscript. There are a number of prior work that utilized geometric approaches to understand the internal representation of language models. (Hewitt and Manning 2019) used linear transformation to identify projection of over model representation that is maximizing similarity to syntactic distances in a parse tree, and found that middle layers of BERT provide best representation.In another line of work (Mamou et al. 2020) used manifold analysis to uncover how different features of words and sentences, such as part of speech ) become separable in deep layers of language models like BERT, and GPT. More recently, (Valeriani et al. 2023) have investigated how the geometric properties such as intrinsic dimension changes across the layers of transformer models, finding that intrinsic dimension first increases before sharply decreeing in deeper layer of the bidirectional transformer models. While these works provide insight to geometrical properties of these network, to our knowledge no prior work have tested a representational level hypothesis that connect behavior of the network (next word prediction) with neural trajectories of individual sentences. \n\nScope: \nThe reviewer brings up a good point, and we will include more information regarding our motivation to focus on straightening. We specially focused on straightening because it provides a testable prediction about how model behavior is shaped by its internal representation. Figure 4 is one such test in which we used our geometric measure to reveal a possible mechanism leads model to deviate from natural language, mainly that their generated sequences exhibit more straightening compared to human produced language.  \nDepth of investigation: \n\nWe agree with the reviewer that there are aspects of the results that we did not discuss thoroughly. We hope to investigate these phenomena in future work, and tried to address some of the follow up experiments in the supplementary section.\n\nexperimental design : \nWe thank the reviewer for bringing these to our attention, and will fix the errors and clarify other topics in the updated version of the paper. \n1.\tChoice of dataset: we used universal dependencies because we wanted to be able to quantify the sentences and phrases along multiple lexical, semantic and syntactic feature. UD corpus provides precomputed values for syntactic features. We further restricted our sentences to include 100K most common words in English (using google ngram dataset). Our choice of sentence length was to ensure we have enough sensitivity in measuring average curvature. Sentences that are very long would result in averaging out the differences that we are interested in. \n2.\tWe will modify figure 4A to emphasize that it is for illustrative purposes and will clarify the detail. the axes in this figure correspond to state spaces built by units in layer P of the network. The trajectories represent evolution of unit activity when the model is generating a sentence vs when it is exposed to a veridical sentence. If the unit activity is favoring a straight trajectory, then the model-generated sentences would be have lower curvature (red compared to blue). Figure 4.B shows the results of an experiment in which we provided a model with 3 context tokens and recorded its generated sequence for 7 tokens, along with the model representation for the generated sequences, and contrasted it with a condition we provided the full 10 token sequence to the model. We then compare the curvature for generated to the ground-truth, and observed that model generated sequences achieve lower curvatures, in line with our prediction in figure 4.A. \n\nQuestions:\nWhy only train and not compare to other models?\nwe added new models in supplementary figure 3 (RWKA, GPT-NEOX, OPT,GPT2-XL). We observed similar decrease in the curvature in trained version of model, as well as similar correlation between curvature and surprisal.  \nThe y-axis in your reported plots is \"curvature change\" What was the distribution of the original absolute curvature angle in the first layer? If you report only a relative number, how can I know that curvature drop makes it close to zero doesn’t end up in a negative degree, rendering the conclusion on \"straightening\" incorrect? To be clear I'm pretty sure / hopeful that the authors will have a good answer and that \"straightening\" indeed takes place, but this represents some weakness in the presentation.\n\nWe focused on curvature change so that we can compare models in their performance for curvature reduction. However, we agree that the absolute curvature values are important to consider as well. To this end we added supplementary figure 6 to clarify this. First using simulation, we showed that for a trajectory over random points in space, the average curvature is distributed around 120 degrees. For many of the models we tested, early layers indeed have a curvature in the range of 120, and there is a drop in the curvature in deep layers of the network.\nI found this specific sentence very odd: ...\n\nWe are sorry for not including this in the main part of manuscript. We included the additional surprisal metric as well as the relationship between surprisal and curvature for a number of new models in the supplementary information supplementary figure 4. \nWhy did you use n-gram surprisal and not LM perplexity in the forth experiment? The first is model agnostic and the latter is model related, so maybe you should have ran both, but I find perplexity very natural to use here.\n\nWe picked a model agnostic measure so we could test it across many models as the reviewer mentioned. But we agree and included a new figure in the supplementary that is relating curvature to LM perplexity. We included the results for the main model (GPT-XL) and few other models in supplementary figure 3.\n"}, {"review_id": "C2zPg9Ra86", "reviewer": "Reviewer_Sn24", "summary": "This work investigates whether large language models learn to \"straighten\" the word-by-word representation of sentence as it passes through the model layers. The word-by-word curvature of the sequence embeddings is defined as the angle between two consecutive word embeddings (i.e. arccos of the cosine similarity of consecutive word embeddings from a particular layer). The idea is that a \"straighter\" trajectory would enable generalization via extrapolation. This work tests a number of models from the GPT-2 family of various sizes and shows that the word-by-word sequence curvature decreases from the early to middle layers (relative to the first layer of the model), and then increases towards the later layers.", "questions": "Major:\n\nQ1. What angles are considered to lead to a \"straight\" trajectory, and what are the individual values of the curvatures for each layer? Does straight mean closer to 0? Is the curvature considered to increase as arcos increases? As in, the curvature is the highest for arccos = pi when the vectors point in completely opposite directions?\n\nQ2. Currently the average of the curvature between consecutive words/tokens in the sentence is considered, but how does the curvature actually evolve through the sentence? This question also somewhat relates to L127-129: “If the model is achieving next word prediction by reducing the curvature in the trajectory of its internal states over the course of a sentence, then we should observe a reliable decrease in the average curvature across the model layers.” The entailment is not clear to me here. If the model is reducing the curvature over the course of the sentence, then I would expect that there will be a decrease in the curvature that relates to the position of the sentence. It’s not clear why one would expect to see the suggested result across different layers of the network. \n\nQ3. Relatedly, how do the learned positional encodings of GPT-2 interact with the results in this work? Specifically, I am wondering if the reason that the randomly initialized model does not show any substantial changes in curvature is due to the poor representations of position. Perhaps a better baseline would be a model that has pretrained positional encodings but a random initialization of other embeddings and parameters.\n\nQ4. It seems like the surprisal is measured entirely on the test corpus, but it should be tested w.r.t. the training corpus. The models are learning the statistics of language using the training corpus, so one would predict that these statistics are closer to the ones estimated by the models. Perhaps the correlation between surprisal and curvature can be even higher if the surprisal better captures the training statistics.\n\n\n\nMinor:\n- All the tested models produce token-level embeddings. The curvature computation in Section 2.2 discusses word-level embeddings. Is the curvature computation indeed done on the word-level, and if so, how were the word-level embeddings produced? If the curvature computation is produced by aggregating the token-level embedding within a word using a specific function, such as mean pooling, then how does this function affect the computed curvature? (e.g. when compared to max pooling or taking the last token in the word as the word-level token)\n\n- L6: work from 2019 is hardly recent\n\n- L90 + 6 lines (missing line numbers on page 3): typo “as the difference between to adjacent states“\n\n- Top Fig1, right hand side: words out of order “fox jumps over the” -> “fox jumps the over”\n\n- L234: the earliest citations for the middle layers of language models predicting brain recordings the best is Jain and Huth, 2018 NeurIPS (who show this for LSTM-based models) and Toneva and Wehbe 2019 NeurIPS (who show this for larger transformer and LSTM-based models).\n", "limitations": "Please discuss the fact that there can be other causes for the results you observe (see Weakness 1)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well written, clear, and concise manuscript\n- Investigates a topical question that will be of interest to many in the NeurIPS audience", "weaknesses": "W1. Several times throughout the manuscript (including in the abstract and title), it is claimed that the results that larger models that have better next-word-prediction also have less curved trajectories in the early-to-mid layers suggests that models learn straighter trajectories in order to predict better. There is no evidence in this manuscript to support this claim. There is only evidence of correlation between the two, and not of causation. These claims need to be dialed way down and qualified. There can be other causes that lead to both straighter trajectories and better prediction performance. For example, the finding in the later part of the paper that sentence surprisal is correlated with curvature can be exactly this cause: it is possible that the most likely next word is the one that leads to the most \"straight\" trajectory. Therefore, a model which learns to predict the most likely next word (i.e. a language model) and achieves a good performance, will also have a straight trajectory. The interesting question is why the most likely next word would lead to a straight trajectory, but that is not answered by the current work.\n\nW2. The manuscript heavily leans on a hypothesis developed by previous work (Henaff et al. 2018/9) but it is not clear how the curvature measure defined in the current work is related to the one developed by previous work. This needs to be clarified.\n\nW3. A few possible confounders for the results. See Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. \nwe agree with the reviewer that the causation is a harder question to answer. And will emphasize in the updated version of the paper that we are showing the evidence at the correlation in this work. We intend to extend our work in causation direction by training models and biasing their representation toward specific curvature values and observe their next word prediction performance. However, we are connecting the next word prediction as behavior of the model to straightening as a hypothesis about evolution of neural trajectories that yield next word prediction behavior. the reviewer points to this fact here. We certainly not claiming that the straightening is the only features that allow the model to develop linguistic competency. As reviewer pointed out we will make this point clear in the discussion. \nW2. The manuscript heavily leans on a hypothesis developed by previous work (Henaff et al. 2018/9) but it is not clear how the curvature measure defined in the current work is related to the one developed by previous work. This needs to be clarified.\nWe are sorry about that, our work defines the curvature in the trajectory of representation the in the same manner as henaf 2019, we will clarify this in the text.\nW3. A few possible confounders for the results. See Questions below.\nQuestions:\nMajor:\n\nQ1. \nWe consider straightness with respect to initial layer of the network, we have shown in the supplementary material that a random trajectory on average has a curvature close to 120 degrees and this is indeed the case for untrained and early layers models ( supplementary figure 6A). We considered a trajectory straighter when the average angle of a trajectory decreases with respect to angle in the first layer. in none of the networks we observed curvature to be close to pi  \n\nQ2. \nThis is a fair point, however here we are measuring an average measure for curvature over the whole sentence, and not at the level of individual words. It is not straightforward to interpret the word-by-word changes in the curvature, one reason for this is that the error in estimating next state in time is cumulative in the sense that the error in predicting the n+1 token will affect n+2 token prediction and so on. As a result, the later part of the sentence could have more variation in the curvature.\n\nQ3. \nThis is a good point; we validated that positional information is not contributing to curvature in untrained models.to do so we studied a transformer model with Rotary positional encoding (Su et al. 2021; Biderman et al. 2023). Similar to GPT2 model with learnt positional embedding, we observed that the untrained version of the model does not exhibit any reduction in curvature, supplementary figure 3C,D\n    \nQ4. \nThis is a great point, we unfortunately don’t have access to the training corpus for many of these models, but agree that it would be a next step. We will try to address this by using models that are trained on publicly available corpus.\nMinor:\n•\tAll the tested models produce token-level embeddings. The curvature computation in Section 2.2 discusses word-level embeddings. Is the curvature computation indeed done on the word-level, and if so, how were the word-level embeddings produced? If the curvature computation is produced by aggregating the token-level embedding within a word using a specific function, such as mean pooling, then how does this function affect the computed curvature? (e.g. when compared to max pooling or taking the last token in the word as the word-level token)\nFor word level embedding we average the activation of tokens that compose the word, and from that point on the curvature computation is identical between the two settings. \n•\tL6: work from 2019 is hardly recent\n•\tL90 + 6 lines (missing line numbers on page 3): typo “as the difference between to adjacent states“\n•\tTop Fig1, right hand side: words out of order “fox jumps over the” -> “fox jumps the over”\n•\tL234: the earliest citations for the middle layers of language models predicting brain recordings the best is Jain and Huth, 2018 NeurIPS (who show this for LSTM-based models) and Toneva and Wehbe 2019 NeurIPS (who show this for larger transformer and LSTM-\n\nWe thank the reviewer for point these out and we will update the manuscript accordingly \n"}, {"review_id": "02gl4vdtVL", "reviewer": "Reviewer_WYqK", "summary": "This paper hypothesizes that the deep, casually-masked transformer models learn to predict by linearizing representational trajectories. This hypothesis is rooted in observations from the neuroscience literature. The hypothesis is tested through experiments that probe:\n\n   (1) the degree to which representation curvature decreases with network depth,\n   (2) the relationship between curvature and model performance,\n   (3) the curvature of representations of model-generated text,\n   (4) the relationship between text surprisal (entropy?) and curvature.\n\nCurvature is defined in the sense of pairwise cosine-similarity between adjacent representations, averaged across sequences of text.", "questions": "Is there a connection between straightening and Koopman operator theory? For Koopman, non-linear dynamics are explicitly linearized in a latent Hilbert space. Could we view the behavior of trained autoregressive transformers as approximations to a Koopman operator?\n\nDo you believe this result is specific to the transformer architecture? Would we also expect to observe this straightening in state-space models, or causally masked convnets?\n\nDo we know whether text in the Universal Dependencies dataset might have been part of the GPT-2 training data? How might this affect the results if this data was indeed included during training?\n\nWhy are the generated sequences for Experiment #2 so short (7 tokens, with a 3 token prompt)?\n\nWhy measure surprisal in Experiment #4 using trigrams? Would an information measure like entropy be more natural?", "limitations": "As the authors acknowledge, the stronger hypothesis--that straightening is a consequence of the predictive loss--is not supported by the experiments in this paper. This would require experiments involving, e.g., MLM or classification tasks to observe whether straightening also appears in these settings.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The straightening hypothesis is interesting, and the experiments convince me that transformers do exhibit straightening behavior. The experiments appear to be generally well executed (but see my questions below). Experiments #1 and #2 in particular clearly establish the pattern of increased straightening as a function of model depth, model size, and optimization steps.\n\nIt seems possible that this observation of straightening could be important and exciting to the neuroscience community, but I do not have the right background to make that judgement.", "weaknesses": "It is not clear to me why the straightening hypothesis is important. Accepting that LLM's do indeed straighten trajectories, what should I do with this knowledge? The conclusion gestures at the possibilities with respect to interpretability of models and revealing \"when and how they could fail and suggest ways to make models more efficient and robust.\" But the connection between straightening and these broader goals (which are undoubtedly of relevance to a broader NeurIPS community) are are not clear to me.\n\nI am very open to an argument that more clearly makes the case straightening is important: either from the perspective of its importance to the neuroscience community, or for its potential significance to the broader machine learning community (as hinted at in the conclusion).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Strengths:\n\n\nWe agree with the reviewer on this point, and to our knowledge, no prior studies in neuroscience have investigated geometrical properties of language networks related to straightening and connecting it behavior of these models. \n\nWeaknesses:\n\nWe thank you the reviewer for bringing this issue. We think the building a hypothesis at the level of representation, and not behavior, is the first step in understanding model behavior. Diagnosing failure modes of the model at the level of behavioral is useful but not enough to reveal what change in the internal representation could avoid them. Our work develops a hypothesis instead at the level of representations. In figure 4 for example we showed that model generated sentences diverge from original sentences, and this is evident in the difference in the curvature between the two conditions. One could foresee that using for example a new decoding strategy could make the two curvature more similar and thus the model generate sequences to ground truth text. We demonstrated this in supplementary figure 5, as some decoding strategies are closer to ground-truth than greedy sampling. \nMoreover, the straightening can be used to design more efficient models. For example Olshausen and Field (Olshausen and Field 1996) showed that a sparsity prior over internal representation can push the neural network to develop receptive fields that are similar to that of  biological visual system. We aim to use the straightening as an inductive bias over the model representation and train new models suitable for next word prediction tasks. \n\nWe will certainly elaborate on these points in the updated version of the work \n\nQuestions:\nIs there a connection between straightening and Koopman operator theory? For Koopman, non-linear dynamics are explicitly linearized in a latent Hilbert space. Could we view the behavior of trained autoregressive transformers as approximations to a Koopman operator?\nThis is a very interesting proposal and we thank the reviewer to bring this to our attention. This is beyond the scope of the work, and we can hint towards it in the discussion part of the work \n\nDo you believe this result is specific to the transformer architecture? Would we also expect to observe this straightening in state-space models, or causally masked convnets?\nIn supplementary figure 3, we show suggestive evidence that the state-space models exhibit similar behavior when trained on next word prediction objective., and in supplementary figure 2   we showed that bidirectional models behave differently from unidirectional models. Unfortunately, we are not aware of models that are causally masked convnet in text model. We would appreciate if the reviewer could point us to one so we can test their curvatures. \n\nDo we know whether text in the Universal Dependencies dataset might have been part of the GPT-2 training data? How might this affect the results if this data was indeed included during training?\n(search if UD was a part of openwebtext).\nGiven that the training dataset for GPT2 is not publicly available, we instead compare model behavior on openwebtext and UD corpus supplementary figure 6, we do observe similar curvature reduction between both datasets. \n\nWhy are the generated sequences for Experiment #2 so short (7 tokens, with a 3 token prompt)?\nWe chose 3 token as a prompt for 2 reason,\n1.\tWe wanted to limit the amount of veridical information the model is exposed to before generating the sequence of tokens \n2.\tThe definition of straightening requires at least 3 initial points, so we wanted to give the model the minimal amount of information, in order to have the model produce its prior over trajectories \n\nWhy measure surprisal in Experiment #4 using trigrams? Would an information measure like entropy be more natural?\nWe included in supplementary figure3G,H correlation between model generated surprisal and curvature, and observed a similar relationship as the main manuscript. We hope this addresses this question \n\nLimitations:\nAs the authors acknowledge, the stronger hypothesis--that straightening is a consequence of the predictive loss--is not supported by the experiments in this paper. This would require experiments involving, e.g., MLM or classification tasks to observe whether straightening also appears in these settings.\n\nWe agree and in future work we aim to address this. \n"}, {"review_id": "lAiw39uewD", "reviewer": "Reviewer_TdxE", "summary": "In transformers architecture a word sequence is first represented by a sequence of word embeddings. This sequence of vector is then iteratively transformed by the successive transformer layers of the model. This paper proposes to characterize the trajectories taken by word embedding sequences. Features are introduced to better understand what happens during these transformations: the curvature along one dimension is estimated by looking the angle of the \"curve\" (the arcos) and trying to make correlation with \"surprisal\".\n", "questions": "The notion of straithening is a nice idea and a important contribution\nof the paper. Maybe it deserves a discussion.  Why it is estimated\nlike that ? Is there other features that can characterize the\ncurvature ? Maybe they are less tractable, but they are important. It\ncould be nice to relate this notion to the manifold defined by the\ntrajectories.\n\nWhy the change in curvature is only measured relatively with the first layer ?  \nWhy only looking at the cosinus ?  The cosinus cannot distinguish the sign of the angle.\n", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is important to understand what is going on inside transformers and what kind of transformation are learnt by the model. Looking at the trajectories is clearly a good idea. This notion of curvature tries to characterize the surprisal observed in the data based on a intuitive assumption. Some experimental results show that the curvature can indeed capture something of the iterative transformations.\n\n", "weaknesses": "While the starting idea is nice, the submission needs to be improved. Many important points remain unclear. Here is a list in reading order (more or less).\n\n- Concerning the dataset: UD is a multilingual dataset, what is the language ? Why selecting only very short sentences ?\n- For the models used, it is really messy. Are the models retrained from scratch ? finetuned ? All along the paper, we never know.\n- On the same topic, what does \"untrained model\"  mean ? Is it randomly initialized ? Only pre-trained ? Why it is a good basis for comparison ? \n- This a bit similar for evaluation data: you could define the different datasets once and make clear reference afterwards. \n- The decoding strategy maybe importance for some measurement. Maybe the greedy choice is not the best.\n- The definition of surprisal could be given and related to the perplexity/NLL which can be the optimization criterion.\n- At the end the statistical observation are not so impressive, or I did not clearly understand. The claim are not so clear at the end. It is difficult to really conclude that the prediction aims at  linearizing the input through a series of transformations. \n\n- You could use latex reference with section numbers (see eg line 130).\n- The colors used for some figures (3 and 5) are really difficult to distinguish even on a color printed version, or with the pdf.\n- The figure captions are very long ! \n\n\nAs a conclusion, I really liked the starting idea of the paper and I think that it deserves further improvement before submission.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\nWhile the starting idea is nice, the submission needs to be improved. Many important points remain unclear. Here is a list in reading order (more or less).\n\nConcerning the dataset: UD is a multilingual dataset, what is the language ? Why selecting only very short sentences ?\nWe focused on only English sentences in UD corpus. We limited the number of words in the sentence so that at we don’t get averaging effect since we are looking at an average metric. we used universal dependencies because we wanted to be able to quantify the sentences and phrases along multiple lexical, semantic and syntactic feature. UD corpus provides precomputed values for syntactic features. We further restricted our sentences to include 100K most common words in English (using google ngram dataset). Our choice of sentence length was to ensure we have enough sensitivity in measuring average curvature. Sentences that are very long would result in averaging out the differences that we are interested in.\n\nFor the models used, it is really messy. Are the models retrained from scratch ? finetuned ? All along the paper, we never know.\nWe are sorry for this overlooking this. Yes, the models were trained on next word prediction objective from scratch. We will address this in the updated version of the paper\n\nOn the same topic, what does \"untrained model\" mean ? Is it randomly initialized ? Only pre-trained ? Why it is a good basis for comparison ?\nThe untrained model is indeed randomly initialized (same procedure as (Radford et al.,2020.)) we believe it is a good basis for comparison because the representation is not yet shaped by the training objective, and only shaped by the architecture and operations such as layer norm. the lack of evidence for curvature in untrained model further suggests that it is a consequence of predictive objective that the model is trained on. \n\nThis a bit similar for evaluation data: you could define the different datasets once and make clear reference afterwards.? \nWe will update the manuscript accordingly \nThe decoding strategy maybe importance for some measurement. Maybe the greedy choice is not the best.\nFor completeness we have included in supplementary figure 5 additional decoding strategies. However for the purpose of interrogating the internal representation of the model, we think the greedy is the most direct one. \n\nThe definition of surprisal could be given and related to the perplexity/NLL which can be the optimization criterion.\nWe included a supplementary figure 3G,H to address reviewer’s concern. We observed similar relationship between surprisal and curvature for model generated surprisal. \n\nAt the end the statistical observation are not so impressive, or I did not clearly understand. The claim are not so clear at the end. It is difficult to really conclude that the prediction aims at linearizing the input through a series of transformations.\nWe would appreciate if the reviewer could point us to specific observations so that we could address their concern. \n\nYou could use latex reference with section numbers (see eg line 130).\n\nThe colors used for some figures (3 and 5) are really difficult to distinguish even on a color printed version, or with the pdf.\nWe appreciate this comment and can address specific figures that the reviewer think would benefit from restructuring. \n\nThe figure captions are very long !\nWe will update the figure captions in the updated manuscript to fix this. \nAs a conclusion, I really liked the starting idea of the paper and I think that it deserves further improvement before submission.\n\nWe thank the reviewer for the positive outlook, and will integrate all their input to improve the work before submission \n\nQuestions:\nThe notion of straithening is a nice idea and a important contribution of the paper. Maybe it deserves a discussion. Why it is estimated like that ? Is there other features that can characterize the curvature ? Maybe they are less tractable, but they are important. It could be nice to relate this notion to the manifold defined by the trajectories.\nWe think the current estimation is the simplest way of parametrizing the neural sentence trajectory. We agree that connecting trajectories to the manifolds that govern their evolution in the internal representation of the model would be an exciting next direction \n\nWhy the change in curvature is only measured relatively with the first layer ?\nWhy only looking at the cosinus ? The cosinus cannot distinguish the sign of the angle.\nWe measured the curvature change from the first layer to be able to compare models. We have included original curvature values in the supplementary figure. \nWe always observed positive curvatures in model representations, as can be seen in the supplementary figures 1, 2, 3,and 6\n"}]

Author rebuttal:

Meta-review: The paper examines whether transformers conduct “trajectory straightening”. Straighter trajectories, it is argued, should facilitate generalization. The work proposes an easy to understand curvature metric and conducts several experiments using it, showing that indeed, in later layers the layer representations become straighter. The reviewers are leaning positive and the rebuttal alleviated some concerns. I think this is an interesting finding with interesting connections to previous work, and it opens up further study into how different architectures behave, how language models generalize, etc.

I strongly encourage the authors to revise the manuscript to incorporate the reviewers' suggestions. In particular, I recommend that the claims around trajectory straightening be dialed way down and qualified — it would be best to simply state it as a hypothesis without assuming any causation, and then conducting experiments to examine the hypothesis.
Decision: Accept (poster)