Large language models transition from integrating across position-yoked, exponential windows to structure-yoked, power-law windows
Authors: David Skrill
Samuel Victor Norman-Haignere
Venue: NeurIPS 2023 poster (2023)
Keywords: language modeling
temporal integration
transformers
timescales
model interpretation
TL;DR: New methods for studying temporal integration reveal that LLMs have stereotyped integration windows that obey a simple functional form, vary substantially across layers, and become yoked to structural boundaries at late network layers.

Abstract: Modern language models excel at integrating across long temporal scales needed to encode linguistic meaning and show non-trivial similarities to biological neural systems. Prior work suggests that human brain responses to language exhibit hierarchically organized "integration windows" that substantially constrain the overall influence of an input token (e.g., a word) on the neural response. However, little prior work has attempted to use integration windows to characterize computations in large language models (LLMs). We developed a simple word-swap procedure for estimating integration windows from black-box language models that does not depend on access to gradients or knowledge of the model architecture (e.g., attention weights). Using this method, we show that trained LLMs exhibit stereotyped integration windows that are well-fit by a convex combination of an exponential and a power-law function, with a partial transition from exponential to power-law dynamics across network layers. We then introduce a metric for quantifying the extent to which these integration windows vary with structural boundaries (e.g., sentence boundaries), and using this metric, we show that integration windows become increasingly yoked to structure at later network layers. None of these findings were observed in an untrained model, which as expected integrated uniformly across its input. These results suggest that LLMs learn to integrate information in natural language using a stereotyped pattern: integrating across position-yoked, exponential windows at early layers, followed by structure-yoked, power-law windows at later layers. The methods we describe in this paper provide a general-purpose toolkit for understanding temporal integration in language models, facilitating cross-disciplinary research at the intersection of biological and artificial intelligence.

Large language models transition from integrating across position-yoked, exponential windows to structure-yoked, power-law windows
David Skrill
Department of Biostatistics and Computational Biology
University of Rochester Medical Center
Rochester, NY 14642
Bavid_skrill@uret www.ettt.......rr............... edu
Sam V. Norman-Haigner
Depts. of Biostatistics and Computational Biology, Neuroscience
University of Rochester Medical Center Depts, of Brain and Cognitive Sciences, Biomedical Engineering University of Rochester Rochester, NY 14642
samuel norman wwrmmannaaigneereuurii.. rochester. edu
Abstract
Modern language models excel integrating across long temporal scales needed to encode linguistic meaning and show non-trivial similarities to biological neural systems. Prior work suggests that human brain responses to language exhibit hierarchically organized "integration windows" that substantially constrain the overall influence an input token (c.g., word) on the neural response. However, little prior work has attempted to use integration windows to characterize com- putations in large language models (LLMs). We developed a simple word-swap procedure for estimating integration windows from black-box language models that does not depend on access to gradients or knowledge the model architecture (e.g., attention weights). Using this method, show that trained L.LMs exhibit stereotyped integration windows that are well-fit by a convex combination an exponential and power-law function, with partial transition from exponential power-law dynamics across network layers. We then introduce metric for quanti- fying the extent which these integration windows vary with structural boundaries (e.g., sentence boundaries), and using this metric, we show that integration win- dows become increasingly yoked structure later network layers. None of these findings were observed in an untrained model, which as expected integrated uniformly across its input. These results suggest that LLMs learn to integrate information in natural language using a stereotyped pattern: integrating across seiition-ykkede exponential windows at early layers, followed by structure-yoked, power-law windows at later layers. The methods we describe in this paper provide a general-purpose toolkit for understanding temporal integration in language models, facilitating cross-disciplinary research at the intersection of biological and artificial intelligence.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Natural language hierarchically structured at many scales (e.g., phrases, sentences, and narrative structures), and the temporal extent of these structures is highly variable. Intelligent systems must therefore have mechanisms to flexibly integrate across diverse and variable temporal scales to infer meaning from linguistic inputs (Hochreiter and Schmidhuber [1996]. Tallec and Ollivier (2018), Mahto et al.] (0200). Large language models (LLMs) excel at this task as demonstrated by their impressive performance
wide range of linguistic benchmarks (Radford et al. [2019), , Devlin et al.
[2199] [Brown et al. (020), [Raffel et al.] [2020]. [Chowdhery et [2022] [ouuron et al. (2223] Recent studies have shown that features from these models are also highly predictive of human brain
responses high-level language regions of the human cortex (Schrimpf et al. [2021]. Jain et al.
(2023), suggesting that LLMs may be powerful tools for understanding and modeling temporal integration in the brain (Caucheteux et al. (021bb).
Prior work has shown that brain responses to speech and language exhibit structured "integration windows" that substantially constrain the influence of an input token (e.g., word) the neural response (Fig.
Integration
Potential functional forms
window
Model Position
TAA (Theunissen and Miller [995]. Lerner al. Chien Chien and Honey 2020],
the
dog
Sonroisseu
fmallls
Norman-Haignere et al. [2022]. Inte-
gration windows have been studied for
decades in the neuroscience literature and
Structure-yoked integration
Position-yuked integration
are thought to be central to the hierar- chical organization of the brain (Hickok
MA M
and Poeppel 2007], Sharpee et al. [[011]. Haignere al. (0222). ). By comparison, much less is known about whether LLMS exhibit struc- tured integration windows, which rel- evant for understanding the mechanisms used by these models to integrate across the multi-scale structure of language, and whether these mechanisms bear any resem- blance to those used by the brain to encode linguistic meaning.
Lemner et al. 2011J.
Sentence
Figure Schematic of integration windows. A, Illustra- tion a causal, exponentially distributed integration window where nearby words in the past have greater influence on the response. B, Example integration windows with different func- tional forms: exponential (blue). power-law (pink), gamma- distributed (green), and uniform (brown). C, structure- yoked integration window that dynamically contours to lin- guistic boundaries (c.g., sentence breaks). D, A position- yoked window that is unaffected by structural boundaries.
Modern attention-basee architectures can in principle integrate uniformly across their entire input (Vaswani et al. (2077). However, as we demonstrate, training can alter the effective integration window of network and cause that window to differ substantially from the integration window implied by a model's architecture (Luo et al. [2016). Keshishian et al. (2021), and little prior work has attempted to empirically measure the integration window of trained LLMs. address this gap, we developed set of general-purpose methods for estimating integration windows from black-box language models and applied these methods to characterize LLM integration windows. We focused initially on GPT-2 (Radford et al. (019)) because it is popular, publicly available, has shown good brain predictivity (Schrimpf et al. 2021 Jain et al. (2023), D. and, like the brain, is causal. We assessed the generality of our findings by repeating analyses for two other LLMs (LLaMA and roBERTa: Liu et al. 2019 Touvron et al. 2023 and varying important properties of our analysis.
Our paper makes the following contributions:
We introduce a word-swap procedure for systematically measuring integration windows from language models, which does not depend on any information about the model architecture.
We show that LLMs exhibit stereotyped integration windows that substantially constrain the influ- ence of an input token and follow simple function form that is well-fit by a convex combination of an exponential and power-law function.
Using these convex fits, we show that that integration windows expand substantially in later network layers driven in part by shift from exponential to power-law dynamics.

---

We introduce a metric for quantifying the degree to which integration windows vary with structural boundaries in language.
Using this metric, we show that there is substantial increase in structure-yoked integration across the layers of LLMs.
We show that these properties (1) are present across three different LLM models (2) generalize across different structural lengths, types, and boundaries and (3) are absent from an untrained model, demonstrating that they are learned from the structure of natural language.
Code implementing methods and corresponding analyses is available at https: htte::://tttbbt com/ @GIril//eemprra.ntttrraaioo git
2
Relationship with prior work
large body of prior work has focused on characterizing attention weights in LLMs, and how these weights vary with linguistic structure, word position, and network layer (Clark et al. (209). Vig and Belinkov (2019) Abnar and Zuidema (2020). Multiple studies have observed that attention weights align with syntactic and semantic information, with lower layers preferentially encoding local syntactic structure and higher layers encoding more complex semantic relationships (Tenney) et al. (019)). Tenney et al. [2019] demonstrated that LLMs effectively rediscover the classical NLP pipeline, with regions of the trained networks corresponding to part-of-speech (POS) tagging. parsing. named entity recognition (NER), semantic roles, and coreference. There is also evidence that the range of attended-to tokens expands from lower higher network layers (Vig and Belinkov [2019]). potentially increasing the timescale of analysis.
However, none of these prior studies have assessed whether LLMs exhibit integration window that constrains the influence of an input token on a unit's response. Indeed, was not obvious a priori whether LLMs would have an integration window at all, given that transformer models can in principle integrate uniformly over their entire input. Moreover, attention-based metrics are limited to transformer models and thus cannot be used to evaluate and compare machine learning models with distinct architectures (e.g., LSTMs) or compare biological and artificial systems. By comparison, our methods are applicable to any language model, and using these methods, we show that LLMs exhibit integration windows that substantially constrain the influence of an input token on unit responses, vary substantially across layers, obey a simple functional form, and vary with structural boundaries at later network layers.
Prior empirical and theoretical work has characterized temporal dependencies in natural language (Chomsky [965). For example, the mutual information between English words declines with distance according to a power-law (Lin and Tegmark (2016). ), and there have been efforts to incorporate this type of power-law structure into recurrent neural network language models (Tallec and Ollivier 22018) Mahto et al. (2200). ). However, little known about how these temporal dependencies shape integration dynamics in modern LLMs, and whether the different layers of LLMs obey power-law dynamics or some other functional form (e.g., exponential decay).
Keshishian et al. 2021 studied integration windows in automatic speech recognition (ASR) models trained transcribe speech from audio. This study found that integration windows late network layers of DeepSpeech2 (Amodei et al. (2015) expanded and contracted when the audio waveform was physically stretched compressed, suggesting its integration windows reflect structure duration. This study however only investigated time-domain audio models, and as a consequence, many of the methods and findings are not applicable to language models.
3 Measuring overall integration windows with word swaps
An integration window, by definition, constrains the influence of an input token depending on the position of that token (Theunissen and Miller [1995), Norman- et al. (2022) (Fig, [A]. For causal model, such as GPT-2, the integration window extends backwards from the position of the model response since only tokens that precede the model response can influence it. We sought to characterize the functional form of L.LM integration windows (Fig, [B] and test whether they
dynamically vary with linguistic structures (Fig. TT) or, alternatively, are fixed function of relative distance between the token and model response (Fig, ID).
3

---

3.1 Word swap procedure
We measured the response of model units two input sequences with a single swapped word (Fig. 2 For each unit, we then calculated the magnitude of the response difference between the two sequences at each model position, as well as the distance between the position of the swapped word and the position of the model response (A). The integration window was calculated as the distant-dependent change in response magnitudes, averaged across many sequences and words swaps (Fig. [ Word swaps were sampled from list of the 5 most probable words given the context (excluding the actual word) as computed by BigBird roBERTa Zaheer et al.) 2021] (masking out the target word to be swapped).
To formalize this idea, denote the jth word of the ith sequence as wili] and the paired sequence with the keth word swapped as w [j]. The corresponding activation sequences for single model unit denoted by a, [j] and a. [j] (where a = f [w,)][)) of [j]
f(w(")) and f(-) is given by
language model that maps an input sequence to the response of single unit; the unit index is omitted
to simplify notation since all analyses are applied separately each unit). The integration window of
the unit is defined as (I and K are the number of sequences and swaps, respectively):
1
K
[|]|
lassk + A - de ke [k= + All
IK
k=
Because the scale of 0|A] arbitrary, we normalized by its maximum value which empirically always occurred at distance O:
[|]] 0
Onorm [
3.2 Experiments
We applied this procedure to measure integration windows from GPT-2 (using the smallest version of GPT-2 with 124M parameters, 12 layers, and 12 attention heads per layer; embedding dimension set to 768; vocabulary size equal to 50,256). We used sequences of 600 words extracted from the Brown Corpus (Francis and Kuceral [1979] (accessed using the NLTK library: Loper and Bird (2002)). )) In all of our experiments, we removed punctuation, capitalization and any tokens that delineated structural boundaries (e.g., [CLS], [SEP]). We also excluded sequences that contain alpaabhhabt characters.
We found that the influence of word depends strongly the distance between the position of the model and the swapped word (Fig. 2B.. This ceetece-deeeedeet effect varied substantially across layers, with consistently longer integration windows in later network layers, suggesting that LLMs integrate hierarchically across linguistic structure, first analyzing short-term structure and then longer-term temporal structure.
To characterize the shape of these integration windows, we fit a wide variety of functional forms, motivated by the prior literature characterizing heavy-tailed distributions (Newman (2005] J, Mitzen macher [ Altmann et al. [2099] (equations of all forms tested are given in the appendix). Fitting performed by minimizing the mean-squared error between the measured and predicted integration window, using the curve_fit function from the Scipy library (Virtanen et al. (2020)). Model accuracy was measured the mean-squared error test data, separate from that used to fit the model. Results were similar using other performance metrics (Kolmogorov-Sminng -miinov (KS) test-statistic and Bayesian Information Criterion).
We found that a simple, 3-parameter functional form a convex combination of an exponential and power-law provided the best fit across all layers tested (Fig. 2C):
0norm [ ~ c(A + 1) (1 c)e -4
where c is the convex combination parameter (constrained to lie between 0 and 1) that controls the relative contribution of the power-law and the exponential function, and the parameters and b control the rate of decline for these two functions, respectively, with higher values indicating faster decline and thus shorter integration times (a and b were constrained to take values between 0 and 10).
For GPT-2, we found that all of the model parameters changed substantially across layers (Fig. 2 The convex combination parameter (c) increased nearly three-fold from layer I (median value across

---

Word swap procedure
B
Integration window
Model position
Layer
Model swap position
Layer
10
Layer
Onarm A
10
the dog sniffed the fire hydrant
10a
the dog licked the fire hydrant
0
10
10a
model swap pos.
Functional form fit
D
Example unit fits
10
Mean Square Enror Entor
10
n [A]
10
10
10
10
11
10' 102 Layer
A model Layer 6
swap Layer
Layer
Log
Log
Expo-
Power
normal
Cauchy
nential.
law
Data
Exp.
Power
Exp. power
Zipf-
Broken
Exp.
Exp.
Alckscev
stick
Exp.
Power
Convex
Power-law parameter a
Exponential parameter
parameter
0.6
1.5
6.0
Parameter value
GPT-2
0.4
1.0
0.2
0.5
2.0
0.0
0.0
1.5
6.0
0.8
1.0
4.0
LLaMA
0.4
0.5
2.0
0,2
0.0
5
15
35
Layer
Figure 2: Characterizing the functional form of LLM integration windows. A, Schematic illustration of unit response word sequences (blue and light circles) with single word swapped. Unit responses differ after the swap due causality. Integration windows calculated by computing the magnitude of the response difference (vertical bracket) function position of the model response (A, horizontal bracket), averaged across many sequences and swaps (eq T B, Integration windows for individual units from several layers of GPT-2 with the median across units overlaid. The influence of word swap decreases sharply with distance (note logarithmic scale). C. Quantification of the goodness-of-fi for a variety functional forms. The exponential- -owwer were better across layers. D. Integration windows for several example units fit using an exponential, power-law, or convex combination of both (see eq. B. E, Violin plots showing the distribution of fit parameter values across all units from diffcrent layers. The convex combination parameter (c, left) reveals transition from exponential to power-law dynamics across layers, and the power-law (a, middle) and exponential rate (b, right) parameters reveal slowing of integration times each functional form individually. F Violin plots showing the distribution of fit parameter values across all units from different layers of LLaMA?
distance between the position the swapped word and

---

units: 0.108) to 11 (median: 0.274) (p < 0.001; Mann-Whitney U test), indicating a substantial change from exponential to power-law dynamics. Since power-law functions decline much more slowly than exponential functions, this implies a substantial lengthening of integration timescales at later layers. However, we note that even later layers the median convex combination parameter (c) hovered around 0.25, indicating that exponential dynamics remain present. We note that although the values of c are relatively low, the overall contribution of the power-law function to the total mass of the integration window exceeds the contribution of the exponential function due to its heavier tail (by 20-fold in the final layer of GPT-2).
We also observed substantial drops in the power-law (a) and exponential (b) parameters, indicating a slower decline and thus a longer timescale. The power-law parameter showed the biggest decreases at mid-to-late layers where the contribution from the power-law fits was greatest (median value of a decreased two-fold from layers 3 (1.022) to 12 (0.512); p < .0001), while the exponential parameter decreased the most in early-to-mid layers where its contribution was greatest (median value of b decreased six-fold from layers 2 (4.386) to (0.697); p < 0.001). Collectively, These findings demonstrate substantial lengthening of integration timescales across LLM layers, driven by a change from exponential to power-law dynamics, as well as an increase in integration times for each functional form separately.
To assess the generality of our findings. we repeated our analyses using two different models (I): LLaMA a state-of- offtc--o--heeartt causal LLM that is substantially larger than GPT-2 and trained on more data (Touvron et al. [2023] (2) roBERTa a non-causal LLM trained on a masked word prediction task (Liu et al. (2019) D. In LLaMA, we again observed a substantial increase in integration timescales across model layers that was well-predicted by a convex combination of an exponential and power- law (Appendix fig. 1B). Compared with GPTT-2 the change across LLaMA's layers was primarily driven by a change from exponential to power-law dynamics with comparatively little change in the decay rates for each functional form individually (Fig. 22)) For roBERTa, we observed symmetric integration windows (Appendix fig. ID), which when collapsed across past and future integration windows showed similar trends to those for GPT-2 (Appendix fig. 1E).
4 Assessing structure-yoked integration
The temporal extent of linguistic structures, such as sentences, is constantly varying. To account for such variation, LLMs might dynamically adapt their integration window so as to contour to structural boundaries (Fig. [[). We refer to integration windows that vary with structural boundaries as "structure-yoked" and integration windows that are unaffected by structural boundaries and only depend on the distance between model and swap position (A) as "position-yoked" (Fig. ID).
To investigate whether structure-yoker integration is present in LLMs, we repeated our word-swap procedure using sequences composed of five 12-word sentences (Fig. 3A), in order to test whether integration windows varied with the boundary between those sentences. We again removed all punetuation, capitalization, and -enuttre-deeoting tokens (e.g., [CLS], [SEP]) to ensure there were no trivial boundary cues. We found that sentence-fing final words yielded slightly more tokens on average and thus restricted our analysis to sentences comprised only of sngle-token words to avoid this potential confound. When swapping words, we ensured that the mean embedding distance was the same for all positions in the 60-word sequence so as to guarantee that any observed differences did not reflect differences in the embedding layer. Specifically, for each swap, we sampled a desired embedding distance from a uniform distribution and then sampled a word whose distance from the original word was close to this target value when swapped in (the uniform distribution and distance tolerance were hand selected so as to provide a feasible target for the vast majority of words needing swaps; in the rare case when there was not a valid target, we sampled randomly). Sampled words were not constrained to be probable in this analysis, but results were similar using only probable swaps without the distance constraint.
We then applied our word swap procedure to the sequences, but rather than computing a single overall integration window (0[A)), we computed a matrix (0[j, k)) that encodes the change in response magnitudes as a function of both the model position (j) and swap (k) position (rather than the distance (A) between the two) (Fig. 3

---

Éxample sequence constructed from 12-word
Magnitude differences
LLaMA; Magnitude differences
Embeddiig
Layer
Layer
sentences
the state has recently undertaken liabilit insurance for drivers of state somewhere the city big clock soundie ed twelve solemn notes micniigtt when she agrees you can rest assured her position remain unchanged but before diagnose it would offer other evidence practically the only enthusiasm they showed was when they were discussing names
25
Layer 32
Modal Modelpoo posttti
89
Madel pos
25
Layer
Layer
Loyer 12
37
25
37
25
37 49
Swap
25 37 49 Swap position
25 37 49
25 37 49
Sentence Boundary
0.5
Intagration change calculation
Integration change
Layer
Embedding
Layer
Layer
13
Model pine
25
(Mean)
8(0)
37
49
8[k]
Layer 6
Layer
Layer
13 25 49 Swap pos.
colk
0.1
Swap Position
Indiv. units
Mean
Boundary
Boundary change
LLaMA: boundary change
0.15
0.15
0.1
0.1
0.05
0.0
WTE
4 6
8 10 12 Layer
WTE
12
18 Layer
24
30
Figure 3: Emergence of structure-yoked integration across LLM layers. A. An example sequence composed of five 12-word sentences, each with unique color. No information about sentence boundaries was provided the network. The average magnitude of response changes induced by word swaps a function of both the model (rows) and swap (columns) position with sentence boundarics overlaid (gray lines) (for GPT-2). The emergence of structure-yoked integration is visually apparent from the triangular structure present at later network layers (see eq. 4. C, Schematic illustrating the calculation of the integration change metric (see eq. S D. Integration change metric for several layers of GPT-2, showing the emergence of boundary-speciffe change later network layers. Dashed lines indicate sentence boundaries. E, Violin plots showing the distribution of the integration change metric at sentence boundaries across units for GPT-2 layers (see eq, (6) F The average magnitude of response changes for LLaMA as function of model and swap position. G Integration change metric for several layers of LLaMA, showing the emergence of coundar--speciff change at later network layers.
Ølj. k] = laibi a [i]l
(4)
We can think of this matrix reflecting a time-varying integration window where 03[A] @ij.j A] reflects the integration window at model position j.
As expected, when averaged this matrix across units within layer (Fig. 3B), observed strong effect of distance between model and swap position for all contextual layers, which manifests as reduction in integration magnitudes for cells further from the diagonal (the diagonal corresponding to distance O). This distance-dependent decline was slower at later layers, consistent with our previous analyses.
The second feature that is visually apparent from these matrices is "triangular" organization that emerges at later network layers. This triangular organization is caused by a drop in the measured integration window the boundary between sentences, which suggests the emergence of structure- yoked integration. To quantify this effect, we computed measure of the local change in the
7

---

integration window (N was set equal to one less than the structure duration, here N = 12 - 1; results were robust this parameter):
A-1 8|]] = Σ klj.k] Ø - 1,
j=ビーN
This equation measures the average change in the integration window at a given swap position (averaged across model position) in a manner that controls for the effect of absolute distance, since the distance between the swap and model position is the same for the two quantities being subtracted. Visually, this equation corresponds to subtracting two offset, off-diagonal columns from the integration matrix (see Fig. 3C for schematic).
We found that the value of our integration change metric (d(k) was close to O, except at the boundary between sentences later network layers (Fig. 3D). This boundary-specific change demonstrates that there is a drop in the integration window at the juncture between sentences in these layers, but not at other points in the sequence. This pattern suggests that the average value of our integration change metric (6 (6(k) at structural boundaries provides a useful measure of structure-yoked integration:
B-1 11 = Σ
k + 1]
(6)
where S equals the length of each structure (here S = 12 words) and B equals the number of structures (here B sentences). When we plotted this boundary metric across layers (Fig. 3). we observed clear increase in structure-yoked integration from early late network layers (the median value of ō[k] increased from 0.003 (layer 1) to 0.074 (layer 12); < 0.001; Mann-Whitney test). This finding suggests transition from position-yoked to structure- yoked integration across the layers of GPT-2. A similar increase in structure-yoked integration was observed for both LLaMA (Fig. (G)) and roBERTa (Appendix fig. 2B), suggesting that the emergence of structure-yoked integration is common feature of LLMs.
4.1 Generalization across different structural lengths, types, and natural contexts
We next sought to examine the generality and robustness of our effects across different types of linguistic structures and contexts.
We examined whether the degree of structure-yoked integration depends on the structure's temporal extent by varying the length of the component sentences (using our base GPT-2 model). We found that our boundary metric (6) was similar when measured for short (8 words) and long (36 words) sentences (Fig. A), demonstrating generalization across structural extent. We also tested whether similar boundary effect could be observed for a different type of linguistic structure, by repeating our analyses using sequences composed of noun phrases (6 words) rather than sentences (specifically, noun "chunks" " as computed by spaCy, which do not allow nested noun phrases within them; Honnibbll and Montani (0220). We observed a similar increase in our boundary yoking metric across network layers, demonstrating that structure yoking is not specific to sentences (Fig, 4B).
In our original paradigm, we concatenated randomly selected 12-word sentences and the resulting sequences therefore lack coherent linguistic content across sentence boundaries. To test whether structure-yoked integration was present at natural structural boundaries, we repeated our analyses using 84-word sequences, excerpted from paragraphs (Brown Corpus) such that each sequence had a 12-word sentence aligned at the middle of the sequence (Fig. 4E). We observed clear boundary effect at the start and end of this 12-word sentence that increased across network layers (Fig. 4D), demonstrating that our findings generalize to natural structural boundaries. The magnitude of the boundary change was somewhat smaller than that observed for randomly selected 12-word sentences, plausibly due to greater across-sentence integration for natural sequences that have coherent across- sentence linguistic structure.
Finally, we attempted to investigate hierarchical integration of linguistic structures. Specifically, we used ChatGPT-4 generate sequences composed of exactly 3 paragraphs, each with 3, 12-word sentences. We then calculated our boundary strength metric paragraph and sentence boundaries
8

---

A
Sentence length
Noun phrases
Boundary change
Boundary change
8 words
36 words
0.20
.20
0.15
.15
C 0.10
5.10
0.05
0.0
0.0
6 Layer
12
Layer
C
Example aligned natural sequences
these alone usually there factors good planning than impressions site may
cubic inches however the displacement the sylinder can be converted cubic centimeters the compression,
D
Centrally-alignees
E
GPT-4 sequences
sequences
Boundary change
Boundary change
Sentence boundary
Paragraph boundary
.10
.05
5 0.0
0.0
0.0
-05
-05
-05
.10
-10
WTE
WTE
WTE
Layer
Layer
Figure 4: Generalization across different structural lengths, types, and natural contexts. A, A similar 36 words (same format
increase in structure-yokes integration was observed for sentences composed structure-yoked integration as Fig, 3C). Results in all panels are based on GPT-2. B, similar increase in was observed for sequenees composed of noun phrases instead of sentences. C, Examples of two sequences of words that contain a 12-word sentence aligned to the middle of the sequence (indicated by color). D, Boundary change metric calculated from natural soquences with centrally-aligned sentenees, demonstrating the emergence structure yoking for fully naturalistic contexts. E, Boundary change metric computed hierarchically structured sequences composed of 3 paragraphs, each with 3, 12-word sentences (generated by ChatGPT-4). The boundary change metric was computed separately for paragraph boundaries and sentence boundaries (excluding paragraph boundaries). We did not observe greater change at paragraph sentence boundaries.
(excluding paragraph boundaries). We did not observe greater structure yoking at paragraph bound- aries (Fig. E), in contrast with what might expected from hierarchical integration. However, the sequences generated by ChatGPT-4 did not always have clear topic change at paragraph boundaries, which might have limited our ability to detect hierarchical integration.
5 Dependence on training
We tested whether the integration windows observed in trained LLMs were learned from the structure
of natural language by repeating our analyses for an untrained GPT-2 model (Fig. 5 using the default settings from Huggingface; Wolf et al. (2020). We found that integration windows from the model were flat across all network layers, consistent with the idea that attention-baser based architectures are capable of integrating uniformly over their input (Fig, 5A). The integration value for A = 0 was higher than for other distances, likely due to residual connections whose integration window effectively spike at O. When we repeated our structure-yokiie analyses, we found that our boundary metric was close for all network layers (Fig. 5B), providing further validation that our boundary metric provides a meaningful measure of dtrueture-dependdet processing in LLMs.

---

6 Conclusions, limitations, and broader significance
We have introduced simple word-swap procedure for estimating integration windows in black-box neural language models and their dependence on linguistic structure. Using this procedure, we found that LLMs have stereotyped integration windows that substantially constrain the influence of an input word and follow a simple functional form, being well-approximated by a convex combination of an expo- nential and a power-law function. Across the layers of the network, we found there was a substamtial increase in integration times driven partly by a transition from exponential to power-law dynamics. Finally, we found that integration windows at late network layers began to partially contour to the boundaries of linguistic structures, and we developed metric to quantify this counday--speciffe change. All of our findings were robust across different model architectures, the procedure used to swap words, and the length and type of linguistic structures manip- ulated, and were absent from an untrained model, demonstrating that they were learned from the structure of natural language. Thus our findings help to reveal how LLMs learn to flexibly integrate across the multiscale structure of natural language, and provide a flexible toolkit to study temporal integration in black-box language models.
Results for GPT-2 with random weights
Integration Window
Bxer A
Layer
Layer
10
10a
0
10
10'
10a
model swap pos.
B
Sentence Boundary
0.2
0.15
0.1
0.05
Given finite time, there were limits the number of different model architectures and linguistic structures we were able to test. Future work, for example, could examine how temporal integration windows differ between transformer models and recurrent neural networks, as well examine broader range of linguistic structures. While, characterized how structure-yoked integration varies across the layers of L.M,, did not investigate how units within layer vary. For example, units or heads within a layer might specialize for integrating across particular types of linguistic structure, as suggested by prior work (Lakretz et al. [2019), Manning et al. (2020).
WTE
12
Layer
Figure Dependence on training. Key findings are ab- sent from an untrained GPT-2 model demonstrating they learned from the structure of language. A. An untrained model shows flat integration windows with the exception of O. B An untrained model shows no evidence structure- yoked integration.
The goal of our work was to understand existing language models, not advance the state-of-the-arr art. However, we believe there are many potential applications of our work. For example, one could investigate weight initialization schemes (or architectural improvements) that im- pose the functional forms demonstrated here the start of training so that the network only needs to learn variations on this form (e.g.. pruuctur-yooke integration), which might improve speed performance. Our metrics might also provide useful tools for diagnosing model limitations, such as an inability to yoke to larger-scale structures.
Our methods and findings have relevance to understanding neural integration windows in biological neural systems. Currently, little is known about what functional form best describes neural integration windows in the brain and whether/how these windows vary with structural boundaries (Chien and
Honey [020), Norman-Haignere al. [2022). in part due to methodological limitations. LLMs offe-of-art in terms of predicting human brain responses to natural language ( (Schrimpf et al. 202 ),
and there is considerable interest in whether the computations of these systems resemble those in the
brain as well as utilizing these systems to generate new scientific insights (Anderson et al. (202] Caucheteux et al. 2022). Tang al. (2023)) Because our methods are model-agnostic, they are potentially applicable to measuring and modeling integration windows in biological neural systems, and our findings provide testable predictions for how neural integration windows in the brain will be structured if they are similar to those in LLMs.
10

---

Acknowledgements
The research reported in this publication supported by the National Institute of Environmental Health Sciences (NIEHS) and the National Institute of Deafness and Communication Disorders (NIDCD) of the National Institutes Health (NIH) (T32ES007271 support of D.S., ROODC01805 grant to S.N.H.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
References
S. Abnar and W. Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190 4197, Online, July 2020. Association for Computational Linguistics. doi: /0.18653//122020.ccl maap:/.......mmm..333.3.3.3 main.385. URL
https://aclanthologyy wwwp/02....oooooooooo. 2020. acl- mai.mmain... 385
E. G. Altmann, J. B. Pierrehumbert, and A. E. Motter. Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions words. PLOS ONE, 4(11):1-7, 11 2009. doi: 10.1371/ journal.pone. 0007678. URL pulttps:/ddi... org 10. 10.1311///rr///all pone 0007678
D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro. J. Chen, M. Chrzanowski, A. Coates, Diamos, E. Elsen, J. Engel, L. Fan, C. Fougner, T. Han, A. Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, A. Ng. S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, Zhan, and 7. Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin, 2015.
A. Anderson, D. Kiela, J. Binder, L. Fernandino, C. Humphries, L. Conant, R. Raizada, S. Grimm, and E. Lalor. Deep artificial neural networks reveal distributed cortical network encoding propositional sentence-level meaning. Journal of Neuroscience, 41(18):110-4111 4119, May 2021. doi: 10.1523/NEUROSCC.I552 20.2021. URL http:///wwwww.oooooooooooooooooe 18/4100
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language models are few-shot learners, 2020.
C. Caucheteux, A. Gramfort, and J.-R. King. Long-range and hierarchical language predictions in brains and algorithms, 2021a.
C. Caucheteux, A. Gramfort, and J.-R. King. Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects, 2021b.
C. Caucheteux, A. Gramfort, and J.-R. King. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3):400441, 2023. ISSN 2397-3374, doi: 10.1038/44152--022-1111111
https:////// org/ /00 1038/ 012////222122212015055
H.-Y. S. Chien and C. J. Honey. Constructing and forgetting temporal context the human cerebral cortex. Neuron, 106(4))65-686.c11 2020. ISSN 0896-6273. doi: https://doi.or// 10. ww0.://.ccuronn22200022222. p11/80896627220011377
URL www www.sciinneeireet...........ee.. wttp:ttcccoccccii.ccce.o httpa//ccecceeteteccc/c/////eert
N. Chomsky. Aspects the Theory of Syntax. MIT Press, 1965.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko. J. Maynez, A. Rao, Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope. J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, Duke, Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, Dohan, S. Agrawal, M. Omernick, A. M. Dai, S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petroy, and N. Fiedel. Palm: Scaling language modeling with pathways, 2022.
A. Spiridonov, R. Sepassi,

---

K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does BERT look at? an analysis of BERT's attention. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 276-286, Florence, Italy, Aug. 2019. Association for Computational Linguisties. doi: 11W33/11111994933 URL URL.attss///cclaamiooogg org/ W19-4828
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
W. Francis and H. Kucera. The brown university standard corpus of present-day american english. In Classified and Annotated Corpus of Present-Day American English, volume 1, pages 3-36. Brown University Press, 1979.
G. Hickok and D. Poeppel. The cortical organization of speech processing. Nature Reviews Neuroscience, 3(5):393 402, 2007. ISSN 1471-0048. doi: 10.1038/nm2113. URL https //doi. 1038/nrn2113
S. Hochreiter and J. Schmidhuber. Lstm can solve hard long time lag problems. Advances in neural information processing systems, 9, 1996.
M. Honnibal and Montani. spacy: satustrraal-srengthh natural language processing python. Software, 2020. /ttpsttss://spayy
S. Jain, V. A. Vo, L. Wehbe, and A. G. Huth. Computational language modeling and the promise of in silico experimentation. Neurobiology of Language, 2023. doi: 10. aaap//oooaa_00000 URL https://doi
10. attp///a...........11///aaaaaaaa a00011 Advance publication.
M. Keshishian, S. Norman- Haigneee and N. Mesgarani. Understanding adaptive, mul- tiscale temporal integration in deep speech recognition systems. In M. Ranzato, A. Beygelzimer,
Dauphin, P. Liang, and J. W. Vaughan, editors, Advances Neu- ral Information Processing Systems, volume 34, pages 24455-24467. Curran Associates, Inc., 2021. URL https: //poceeed/sscces/sss.. wtwpeeeddinnnnneuiips..e. neurips. catsscccpaagg wccppappr_ftiiee wwt..lsspppppe//22//fi... 1ice220077550003333344141411ed Paper.
Paper pdff
Y. Lakretz, G. Kruszewski, T. Desbordes, D. Hupkes, S. Dehaene, and M. Baroni. The emergence of number and syntax units in Istm language models, 2019.
Y. Lemerr C. J. Honey, L. J. Silbert, and U. Hasson. Topographic mapping of hierarchy of temporal receptive windows using narrated story. Journal of Neuroscience, 31(8):2906 2915, 2011. ISSN 0270-6474. doi: 10. 10.122/EUUOOSC..... 11.21..0.884111101 10.2011. http::tttt:////t content/31/8/2900
jneurosci... hwwpij/ooo...nuroocic......o org/
H. Lin and M. Tegmark. Critical behavior from deep dynamics: A hidden dimension in natural language. 06 2016.
Y. Liu, M. Out, N. Goyal, J. Du M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 12955-295,200 2019.
E. Loper and Bird. Nltk: The natural language toolkit, 2002. URL httph/ttt///////// http:://axxvv.oaa//.//// abs/ cs/0205028
W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL https http:::::////rooeedingggss wttp://ooooeiiioooooooiieeeepeep nwwrips...... wwtpsaeccw.pppp.p..eeoooemm
pdf
Mahto, V. Vo, S. Turek, and A. G. Huth. Multi-timescale representation learning Istm language models. arXiv preprint 22777 2020.
12

---

D. Manning, K. Clark, J. Hewitt, U. Khandelwal, and O. Levy, Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 1 117:30046 30054, 2020.
M. Mitzenmacher. Brief History of Generative Models for Power Law and Lognormal Distributions. Internet Mathematics, 1(2):226 251. 2003.
M. Newman. Power laws, pareto distributions and zipf's law. Contemporary Physics, 46 (5):323-351, 2005. doi: 10. 10.10800001075500000024444 URL https://doi. ort.o.oorg/1.... 1080/ 00107510500052444
S. V. Nornan-Haiinere, L. K. Long, O. Devinsky, W. Doyle, I. Irobunda, E. M. Merricks, N. A. Feldstein, G. M. McKhann, C. A. Schevon, A. Flinker, and N. Mesgarani. Multiscale temporal integration organizes hierarchical computation in human auditory cortex. Nat Hum Behav, (3):455-469) 2022. doi:
URL https: https:///oii //doi. org.org.//.. 1038/
941562-021-012611 Epub 2022 Feb 10.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and
E. Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)e210564611 18, 2021.
T. O. Sharpee, C. A. Atencio, and C. E. Schreiner. Hicrarchical representations in the auditory cortex. Current Opinion in Neurobiology, 2165),661-767, 201 ISSN -938-4388. doi: https://doi.org/10 1016/j.conb.201 .05.02.......... RRRLaatpsss/w....ceeeedeeee 50959438811000957
cww http:/seennnaarricleeeeee Networks, eircuits and computation.
www. wwc.enceddreett.....mc..c..
C. Tallec and Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018. URL httpss://oppenreeviee http://www...etooorrreeesssoooda
Tang, A. LeBel, S. Jain, and et al. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26:858-866, 2023. doi: 10310 14155-023-011344
I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 111111//1//19-11//11 1452. URL https: /estps://aclaste //aaaaaaanntooogooooooo
F. Theunissen and J. P. Miller. Temporal encoding in nervous systems: A rigorous definition. Journal of Computational Neuroscience, 2(2): 162, 1995. ISSN 1573-6873, doi: 18.0////909661885. URL
org/10 1007//F00991888
H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, Rozière, N. Goyal, E. Hambro, F, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, Llama: Open and efficient foundation language models, 2023.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processiny Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:////oooeeeee..eeeeeeeeeeeee neurips cc/ppp/e...ep......eeeeeee http://ww..a.ttteee//eeeee/eeeea
Paper. pdf
Vig and Y. Belinkov. Analyzing the structure of attention in transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNIPP. Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy, Aug. 2019. Association for Computational Linguisties. doi: 10. 18653/v 1/W19-488
URL URLhttts://accaanholloo www.eey-rrg/199-9//-0
13

---

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weekesser, J. Bright, S. van der Walt. M. Brett, Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, i. Polat, Y. Feng. E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen,
A. Quintero, Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:
261-272, 2020. doi: 10.
T. Wolf, V. Sanh, J. Chaumond, C. Delangue, A. Liu, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, Plu, T. Xu, Y. Le Scao, S. Gouws, B. Ginsburg, and N. Shazeer. Transformers: State-of-the- art natural language processing. https://huugingfacee co/ transforrmmsss 2020. Accessed: [Insert Date].
M. Zaheer, G. Guruganesh. A. Dubey, Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang.
L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences, 2021.
14

---

Appendix
Functional forms
Equations for all functional forms tested is below:
[I((=)=))²
Log-Normal: f(z|μ, o) = exp
2
Log-Cauchy: f(x(o, = )²+((u(0)-11
Exponential: f(x/A) exp(-Aa*
Power Law: [())a) = ()-
Zipf-Alekseev: f(alaa, (æ =1)-0-8ln(+t)
Exponential Exponential: [( \1 c2 c exp(-X + +(1 +11 c) exp(->22x)
Exponential Power: ax, c) c. (x+ 1)-0 + (1 - kx
15

---

A LLaMA int. window
B
LLaMA: functional forms
C
LLaMA: example unit fits
Log
10
normal
Layer 5 Layer 20
Log
Mean Square Error
Layer 35
10
Cauchy
10
Exponential
Gnorn [A]
10
nurm
10.
Power-law
10
Zipf-
10
10
Alekseev
Exp. + Exp.
10
10
10 10'
10
20
35
10 10
A model swap pos.
Layer 5 Layer 20 Layer 35
Layer
Exp. + Power
Convex param. с
Power law param. a
Exponential param. b
roBERTa int. window
0.8
Layer Layer Layer 11
0.4
Quurm [A
Param. value
0.6
0.3
3
0.2
0.4
10
2
0.1
0.2
0
0
-102
-10'
10'
102
12
6
12
4 model swap pos.
Layer
Layer
Layer
Appendix Figure 1. Additional plots showing integration windows for LLaMA and BERT. A, Integration windows for individual units from several layers of LLaMA of a word swap decreases sharply with distance (note logarithmic scale). B, Quantification of the goadness-of-fit for variety of functional forms for LLaMA. The iioonential-powerr fits were better at predicting integration windows across all layers, with GPT-2. C, Integration windows for several example units from LLaMA using an exponential, power-law, or convex combination both (see eq. 3 Integration windows for individual units from several layers roBERTa. Negative distances indicate integration across future tokens and positive distances indicate integration across past tokens. F, Violin plots showing the distribution of fit parameter values across all units from different layers of roBERTa. Results were very similar to those for GPT-2.
median across units overlaid. The influence
16

---

roBERTa: Magnitude differences Layer 12
roBERTa: boundary change
0.2
0.1
Model position
25
0.1
0.0
0.0
25 37
49
Layer 12
Swap position
Appendix Figure 2. Emergence of structure-yoked integration in roBERTa. The average magnitude of response changes induced by word swaps as function both the model (rows) and swap (columns) position with sentence boundaries overlaid (gray lines). Results are similar to those for GPT-2, except the effects extend to both past and future tokens. B. Violin plots showing the distribution of the integration change metric at sentence boundaries across units for layers of roBERTa showing substantial increase layers, similar GPT-2.
17


Reviews:
[{"review_id": "MkMYzVHtVQ", "reviewer": "Reviewer_3ifa", "summary": "This paper describes a novel method for measuring and characterizing integration behavior in large language models. This method is used first to look just at temporal integration windows—i.e. how do inputs at different lags affect model activations at a specific time—in GPT-2, revealing a transition from exponential-like to power-law-like behavior across model layers. Second, the authors investigate whether integration is “position-yoked” or “structure-yoked” by performing a similar analysis on strings comprising multiple concatenated sentences. These results show that the degree of structure-yoking, like integration window size, increases consistently across model layers. Finally, these results are replicated on other networks (LLaMA and roBERTa) and with some variations in language input.\n", "questions": "* The analyses consider structure only at the scale of noun phrases (6 words) and sentences (8 to 36 words). What about higher-order structure? Do the same layers also “yoke” to structure at supra-sentential scales, like paragraphs or (maybe possible only with LLaMA) entire narratives? ", "limitations": "The discussion of limitations is clear and complete.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is overall an interesting and well-executed paper, with many strengths:\n* The procedures are well-motivated theoretically, clearly explained, and obviously well-suited to measuring the effects of interest.\n* In particular, the structure-yoking experiment is cleverly constructed and tests a very interesting property of these models. \n* Evaluation is fairly exhaustive, with tests of many networks and different variations on the procedure to ensure robustness.\n* The paper makes contact with much of the relevant literature, linking the work both to machine learning and neuroscience research.", "weaknesses": "* My only actual issue with the paper is that it does not do a terribly good job at answering the question, “so what?” This seems like timely and interesting work, so the authors should be able to say a little about why it’s useful to deeply understand and characterize integration behavior in these networks.\n* Differences between exponential and power law decay are quite difficult to see with linear scales; the authors should try showing the relevant data (especially Figure 2C) using log-log plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive review. \n\nWe chose to address your questions about the impact of this work and the generalization of our paradigm to supra-sentential scales in our general response since similar questions were raised by other reviewers. \n\nWe now plot integration windows on a log-log scale (see Figure 1 of included PDF).\n"}, {"review_id": "rfNaokn1Ky", "reviewer": "Reviewer_KfF7", "summary": "The authors provide an experimental understanding of LLMs that how LLMs have inherent Integration windows to take account of the global meaning of the given sentence, by developing a novel method called “word-swap procedure,” which is model agnostic. The authors investigate the behavior of the Integration window with various control: changing the inner layer, changing the distance from the current position of sentence tokens, or varying sentence structures. The finding is that some trained LLM’s Integration window fit well with a convex combination of an exponential and power law function, resulting in the exponential law window at early layers across position-yoked, and the power law window at later layers followed by structure-yoked.", "questions": "- Can you specify what kind of linguistic structure is used for the experiments of structure-yoked integration in Figure 3?", "limitations": "- Integration window may not appear only for simple text generation, but could be observed in other tasks of NLU, or summarization, or text retrieval (though summarization and text retrieval are belong to generation, these tasks would need more detailed behaviour of Integration window). Could you provide experimental investigations other than simple text generation?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Suggest a metric for quantifying the variance of integration windows\n- propose model agnostic analysis method to understand the integration window inherent LLMs\n- provide an explicit experimental understanding of how LLM’s integration window behaves across layers, and with varying sentence structure.", "weaknesses": "- It seems that the kinds of tested LLMs are quite restricted (only three), which bound the extensiveness of the analyses, let alone that GPT-2 is actually not that large with respect to current LLMs. In addition, for the generalization of the transformer architecture, it seems necessary to add the result of the encoder-decoder transformer.\n- The number of tokens for each sequence seems to be fixed at most 40, but the methods author proposes can be applied to the longer sequence length. It would be helpful to understand the property of the integration window if the author provides enriched experiments with longer sequence lengths.\n- Though the authors exhibit that the integration window of LLM shows different behavior across the layers, that the lower layer follows the exponential law, and that the higher layer follows the power law, the contribution itself seems to be not enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To address your comments, we will test additional, larger models including LongT5. LongT5 is an encoder-decoder architecture that can accept very long sequences and has been trained on multiple tasks, addressing each of the issues that you raised in your review. The T5 model has also shown strong neural predictivity in the brain (Schrimpf et al. (2021) The neural architecture of language: Integrative modeling converges on predictive processing) (LongT5 has not been tested to our knowledge). We plan extend our analyses to much longer sequences (e.g., 1000 tokens).\n\nWe note that our contribution extends beyond simply showing a transition from exponential to power law dynamics. In particular, a key result of our study is that late layers adapt their integration window to structural boundaries in language, while earlier layers do not (or do so weakly). This suggests there is a transition from position-yoked to structure-yoked integration in LLMs, a finding that we replicate across all models tested. Our study also introduces new methods for measuring both the overall integration window and structure yoking. \n\nWe used 12-word sentences for our analyses in Figure 3. Figure 4K shows the results for 8- and 36-word sentences and Figure 4L shows the results for 6-word noun phrases. In our general response, we describe the results of a new analysis using paragraphs composed of three 6-word sentences. All of these analyses qualitatively show the same effect and our analyses with paragraphs further reveal hierarchical structure yoking. We will clearly label the structure tested in the caption of all figures.  \n"}, {"review_id": "7gR0nVsu7X", "reviewer": "Reviewer_hKex", "summary": "This paper studies the relationship between the length of context and its influence on language model outputs across layers and units. They propose a model-agnostic procedure that swaps words at a given distance and measuring the change in the activations. Doing this for a range of distances and many different sequences allows patterns to emerge across layers. Curve-fitting shows that these patterns are a convex combination of exponential and power law functions:  largely exponential for lower layers, suggesting shorter temporal horizons, and gradually shifting to power-law for higher layers, suggesting growing temporal dependence. The parameters of the curves also suggest increasing temporal dependence within the functions. Experiments also show that these temporal horizons are bound to linguistic structure, i.e. sentence boundaries.\n\nThis paper provides empirical evidence for robustness of results by experimenting with different models and varying the details of the proposed procedures and research questions. They also provide evidence for the absence of such patterns in untrained models, which strongly suggests that the patterns discovered emerge from training on natural language.\n", "questions": "Maybe I missed this in the experiments: is it hard to extend this analysis beyond sentences to contextually relevant paragraph-level analyses and demonstrate hierarchical dependence? Figure 3A includes sequences that are contextually unrelated, right?\n\nThe authors may find this older study on recurrent LMs and context interesting: https://arxiv.org/abs/1805.04623 ", "limitations": "As noted above, perhaps a discussion on the potential of this line of work to concretely influence the relevant fields could be useful. Other noted limitations seem pretty thoughtful.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is an interesting study that sheds light on how language models rely on context (temporally) by analyzing changes in activations caused by swapping words at increasing distances. It provides a different and valuable perspective on potential sources of the learning signal within the data and how that may translate to certain behaviors we observe during inference. The paper makes a strong effort to reinforce results by varying the experimental setup and measuring statistical validity of the findings.", "weaknesses": "While the discussion is scientifically engaging, one of the frequently discussed weaknesses of analytic studies such as this one is: how can the findings of such studies provide concrete/actionable improvements and help advance the relevant fields, beyond speculation. Improving our mental models of how LMs work holds plenty of value, but if the authors have ideas or experiments on this it may be worth including them. \n\nThe experimental setup in the main experiments is a bit hard to follow. For example, what is the word replacement procedure used? How much data is used and how is it sourced? It might be worth extracting the details into a separate mini section if possible? Highlighting the findings per experiment maybe via paragraph titles may also improve clarity of the paper.\n\nIs the data used too synthetic or too much of a toy setting? After the initial experiments, is it possible to explore a variant on more naturally occurring data such as that in language modeling datasets? Would this be feasible?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive review. \n\nWe have addressed your question about the impact of our work on relevant fields in our general response because other reviewers raised similar questions. \n\nBelow we have clarified the word swap procedure and the method used to select the swapped word:\n\nWord swap procedure. We first sample an N-word sequence from a corpus. We used the Brown Corpus, but the results were similar using the Book Corpus. For each word in the original sequence, we generate a new sequence with just that word swapped, yielding N sequences each with a single swapped word. We tested several methods for selecting the word to be swapped.   \n\nWord selection procedure. We investigated three procedures for choosing the swapped word. (1) Part-of-speech matched. The simplest procedure randomly selected the swapped word from the set of all words with the same part-of-speech tag. (2) Probably swaps. Probable swaps were randomly sampled from a list of the 100 most probable words given the context (excluding the actual word) as computed by BERT (masking out the target word to be swapped). (3) Embedding-distance matched. The goal of this procedure was to ensure that the average embedding distance between the original and to-be-swapped word was the same for all positions, helping guarantee that any position-dependent effects (e.g., due to structure-yoked integration) could not be explained by the structure of swaps in the embedding layer. Specifically, for each swap, we sampled a desired embedding distance from a uniform distribution and then sampled a word whose distance from the original word was close to this target value when swapped in (the uniform distribution and distance tolerance were hand selected so as to provide a feasible target for the vast majority of words needing swaps; in the rare case when there was not a valid target, we sampled randomly). We found the results were very similar using all three of these procedures. We focused on the results from the simpler, part-of-speech matching procedure for our overall integration window analyses (Figure 2). We used distance matching for our structure yoking analyses since these analyses focused on position-specific effects (Figure 3), and we replicated both our overall integration window and structure yoking analyses using probable swaps (Figure 4I).\n\nWe will follow your advice and clarify these methods in separate mini sections, and we will highlight our results using section and paragraph titles. \n\nWe have addressed your comments about more naturalistic and paragraph-level analyses, in our general response, since a similar question was raised by other reviewers. Specifically, we repeated our analyses using paragraphs composed of three 6-word sentences, and as a consequence, the boundary between each sentence is entirely natural, unlike in our original experiments. The results from this analysis replicate our findings showing a clear change in the integration window at the boundary between sentences and also show hierarchical organization with a greater boundary change for paragraphs compared with sentences (see Figure 2). We are in the process of scaling this up to larger structures using stimuli generated by ChatGPT (e.g., 3 paragraphs each composed of 12-word sentences).\n\nThe only remaining unnatural part of this procedure is the use of sequences composed of multiple fixed duration structures (e.g., 6-word sentences). To address this, we are in the process of performing the following analysis. First, we select completely natural sequences that contain a structure of a given duration (e.g., 12-word sentence) in the middle of the sequence, thus aligning the start and end of this structure across all sequences. We then repeat all of our analyses and check if there is a change in the integration window at the start and end of this single structure. \n\nThank you for noting the Khandelwal et al. paper. We agree this is a relevant study and will reference it in our revised manuscript. Word shuffling might provide an interesting way to examine order-dependent integration in future research. \n \n"}, {"review_id": "2Tf9ydc3LV", "reviewer": "Reviewer_NDTr", "summary": "Transformer models have the potential to acquire essentially arbitrary patterns of attention during training. But what patterns do they acquire in practice? This is the question taken up in the present paper. The data for the paper are 40 word sequences from the classic Brown corpus. The language models examined are GPT-2, LLaMA, and BERT. The paper introduces a word-swap procedure for evaluating integration. It argues that the large language models exhibit a transition from exponential to power-law dynamics across the layers of the network. It describes the power-law windows as structure-yoked (in the context of the study, this means yoked to sentence boundaries) and the exponential windows as position-yoked.", "questions": "If revising this paper, it will be important to provide rigorous comparisons of different model fits, and to test the approach on a more complete range of linguistic material and a more carefully articulated set of structures. ", "limitations": "The author(s) acknowledge that the relationship between natural language and the integration windows learned by the LLMs was not explored in depth. As mentioned under \"Weaknesses\", they do not appear to have a complete grasp of what is needed to justify the functional forms they selected.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper raises a good question about what patterns of attention are actually acquired during the training of large language models. It undertakes to characterize the patterns in terms of functional forms. This sets up an important point of potential contact between machine learning and the study of scaling laws in physics. ", "weaknesses": "The author(s) assert that the integration windows are surprisingly well fit by a convex combination of an exponential and a power law. They do not rigorously evaluate any alternative fits. They do not appear to be aware of the substantial research literature on power laws, and appear to have overlooked the following points.\n1) Power laws can themselves be generated as mixtures of exponentials.\n2) To statistically distinguish power laws from other similar-looking distributions, it would be necessary to explore a very much greater range of time scales than appear in this study.\nHere are a few of the very large number of references bearing on this issue:\nMEJ Newman (2005) Power laws, Pareto distributions, and Zipfs’s law. Contemporary Physics\nM Mitzenmacher (2004) A brief history of generative models for power law and lognormal distributions. \nRD Malmgren et al. (2008) A Poissonian explanation for heavy tails in email communications. PNAS 105(47)\nS Arbesman et al (2009) Superlinear scaling for innovation in cities. Phys. Rev. E 79, 016115\nAltmann et al. (2009) Bursts, Lulls, and Scaling in the Temporal Distributions of words. PLoS One 4(11\n\nBy equating structure-yoking with sentence boundaries, the author(s) disregard all other types of linguistic structures. These range from smaller components of syntactic structures (such as the structure of noun phrases) to larger structures that control discourse coherence. There seems to be no justification for thinking that all these other structures are \"position yoked\". \n\nFinally the decision to report results only the 1979 Brown corpus is puzzling (the paper states that similar results were obtained using the BookCorpus, but provides no details about that). The Brown corpus contains only 1 million words, hence the vocabulary is only the more frequent words of English. It does not contain examples of various genres that figured in the training sets for the LLMs. It would be important to understand how LLMs work on the range and variety of material they were trained on. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive critique. \n\nWe have addressed your critique about the need for more extensive quantitative comparisons in our general response since other reviewers raised similar points. We tested a much wider range of different parametric forms motivated in part by the papers cited in your response. We find that the exponential-power form outperforms all other forms tested across multiple metrics. If there are any other specific functional forms that you would like us to test, we would be happy to do so. \n\nWe are aware that power laws can be approximated as mixtures of exponentials, and will clarify this point in the manuscript. In the model comparisons described above we included a model that contains a mixture of two exponentials and thus has the same number of parameters as our exponential-power form. This alternative model performs substantially worse than our exponential-power function. \n\nOur original submission tested both sentences and noun phrases, and we find that the results are very similar across both. To address your comment and that of other reviewers we repeated our analyses using paragraphs, and again find very similar results (see general response). If there are other structures that you would consider it important to test, we would be happy to do so. \n\nTo address your comment, we will include the results of BookCorpus in our revised manuscript. If there is another corpus you would like us to test, please let us know. We note that we have run many experiments investigating different architectures, swap procedures, linguistic structures, structure durations, and corpora, all of which consistently yield the same core set of results.\n"}, {"review_id": "A9ahedbTbW", "reviewer": "Reviewer_AsxG", "summary": "The authors investigate the temporal integration window of several transformer language models (but focus on gpt2) by evaluating the effect of word swaps on the activations of individual units as a function of their distance from the swapped word. They then characterize these mean integration curves of the units in each layer as a convex combination of an exponential and power law function and highlight an evolving motif as longer context is utilized at later layers. They also explore specifically whether these windows are tied statically to only token proximity or whether they dynamically capture the boundaries of sentences (and noun phrases), and find that the integration windows of later layers are sensitive to these boundaries.", "questions": "I would have liked to see a more principled approach starting from the integration window curves and arriving at the formulation of this convex combination of exponential and power law functions. What space of functional forms and combinations was explored? And how was this form deemed to be the best fit? Some notion of variance explained as a function of number of free parameters would be a useful datapoint to include. Just a few sentences elaborating on and clarifying this should be sufficient.\n\nEDIT: this has now been addressed.", "limitations": "The limitations are appropriately addressed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Excellent problem presentation within context of related work. The core “word swap” approach is well designed and appropriately motivated as is the analysis of “structure-yoked integration.” All experiments appear carefully constructed and controlled. The submission is clearly written and well organized. The results are important, presented clearly, and discussed appropriately.", "weaknesses": "The motivation of the proposed functional form is underspecified. Please include more detail as to how it was decided that the integration windows should be modeled as such. More specific question below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review. \n\nWe have addressed your comment about needing to better motivate the chosen functional forms in our general response. We show that the exponential-power law function provides substantially better prediction accuracy (measured using three different metrics) than a wide range of other functional forms including exponential and power law forms in isolation. \n"}]

Author rebuttal:
We were pleased the reviewers overall felt that our work addressed a timely question, that the methods were well-motivated and described, and that the results were interesting and robust across multiple experiments. We thank the reviewers for their constructive critiques. Below, we address comments shared across multiple reviewers. 

Functional form of integration windows

Multiple reviewers requested additional motivation/quantification of the functional form used to model integration windows. To address this issue, we performed several analyses, making the following changes:

1.	Tested additional functional forms motivated by prior literature (including papers noted by reviewer NDTr): (1) exponential (2) power (3) exponential-power (4) exponential-exponential (5) Zipf-Alekseev (6) log-normal, (7) log-Cauchy.
2.	Quantified goodness-of-fit with multiple metrics: (1) cross-validated mean-squared error (MSE), (2) cross-validated Kolmogorov-Smirnov (KS) test statistic (3) Bayesian Information Criterion (BIC). 
3.	Extended the sequence length from 40 words to 150 words (we plan to extend it further, e.g. 1000 words).
4.	We plot integration windows on a log-log scale to better visualize the tail.

We found that the exponential-power form provides the best fit across all GPT-2 layers using all 3 metrics (see Figure 1) (we will repeat this analysis for the other tested models).  

Interestingly, when plotted on a log-log scale, integration windows appear to exhibit piecewise-linear decay with a single knot (Figure 1). We are testing whether a piece-wise linear form, corresponding to a transition between two power laws, shows even better fits. The results of this analysis will not change our finding that LLM integration windows can be approximated using a simple (3-parameter) functional form whose timescale substantially expands across layers. It also has no impact on our structure yoking findings or methodological contributions (word swap procedure, structure yoking paradigm, and boundary metric). We will revise the manuscript to incorporate all of these results.

Larger-scale hierarchical structure

Several reviewers (3ifa, hKex, NDTr) asked if our structure-yoking results would occur at supra-sentential scales and/or exhibit hierarchical organization, as well as whether structure yoking would be evident at natural boundaries between sentences (as opposed to between randomly selected sentences). To investigate these questions, we repeated our analyses using paragraphs composed of three 6-word sentences (the longest sentence length for which we could find enough paragraphs) (Fig 2). We observe structure yoking at the boundary between sentences, demonstrating that our effects generalize to natural boundaries. We also observe even stronger yoking to the boundary between paragraphs, suggesting hierarchical integration. We are working on scaling up these analyses by using ChatGPT (GPT-4) to craft larger-scale structures with stereotyped durations (e.g., 3 paragraphs each with 12-word sentences), as well as repeating these analyses with larger LLMs (e.g., LLaMA, LongT5). We will revise the manuscript to include these new results.  

Significance to the NeurIPS community

Multiple reviewers asked us to better articulate the significance of our work for the NeurIPS community:

Relevance to neuroscience. Understanding how the human brain integrates linguistic information is an important question of active interest to the NeurIPS community (e.g., Jain et al. (2020) Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech). The brain must have mechanisms for integrating flexibly across multiple timescales and linguistic structures. Yet, it is largely unknown what functional form best describes human cortical integration windows and whether/how these windows vary with structural boundaries, in large part due to methodological limitations. LLMs are state-of-art in terms of predicting human brain responses to natural language, and there is considerable interest in whether the computations of these systems resemble those in the brain and utilizing these systems to generate new scientific insights (Caucheteux et al (2023) Evidence of a predictive coding hierarchy in the human brain listening to speech, Tang et al. (2023) Semantic reconstruction of continuous language from non-invasive brain recordings). Because our methods are model-agnostic, they are directly applicable to measuring and modeling integration windows in biological neural systems. Our findings provide clear, testable predictions for how neural integration windows in the brain will be structured if LLMs integrate information in a brain-like manner. 

Relevance to language modeling. The goal of our work was to understand existing language models, not advance the state-of-the-art. We agree that our paper does not provide specific guidance on how to improve language models, which we will note as a limitation. The impact of empirical insights on applied research is often difficult to predict, and we believe there are many potential applications of our work. For example,  LLM integration windows show a stereotyped functional form that differs substantially from untrained networks and is robust across different architectures. Thus, a potentially interesting research direction would be to investigate weight initialization schemes (or architectural improvements) that impose this functional form so that the network only needs to learn variations on this form (e.g., structure-yoked integration), which might improve speed or performance. Our metrics might provide useful tools for diagnosing model limitations, such as an inability to yoke to larger-scale structures. We will briefly note these potential research directions/applications in the revised manuscript, flagging them as speculative. 


Meta-review: The work presents a method for measuring and characterizing integration behavior in LLMs, focusing on temporal integration windows and structure-yoking. 

The reviewers collectively acknowledge several strengths of the paper:
- The paper is well-motivated theoretically, and the procedures are clearly explained, making it suitable for measuring the effects of interest in LLMs.
- The experiments, particularly the structure-yoking analysis, are cleverly designed and explore an intriguing property of LLMs, providing valuable insights.
- The paper offers an exhaustive evaluation, testing multiple networks and variations to ensure robustness in its findings.

However, reviewers express some concerns and suggest areas for improvement:
- The paper tests a relatively small set of LLMs, which limits the breadth of the analysis. Reviewers suggest including more diverse LLM architectures, such as encoder-decoder transformers.
- The fixed sequence length of 40 tokens may not capture the full picture of integration behavior. Expanding experiments with longer sequence lengths could provide valuable insights.
- Some reviewers seek a clearer explanation of the practical significance of the research beyond theoretical insights. They suggest exploring potential applications and improvements for LLMs.
- There are concerns about the synthetic or toy-like nature of the data used in experiments. Reviewers propose experimenting with more naturally occurring data, like language modeling datasets, for broader applicability.
- The choice of the exponential-power functional form for modeling integration windows requires more detailed motivation and quantification of its appropriateness.

The authors' response effectively addresses these concerns by conducting additional analyses with various functional forms, quantifying goodness-of-fit, extending sequence lengths, and exploring hierarchical structure yoking. They also articulate the significance of their work for the NeurIPS community, emphasizing its relevance to both neuroscience and language modeling. These rebuttal details should be included in the final paper draft.
Decision: Accept (poster)