SlowGuess/NeurIPS_2023_OCR huggingface dataset:
integration window ( $N$  was set equal to one less than the structure duration, here  $N = 12 - 1$ ; results were robust to this parameter):
$$
\delta [ k ] = \sum_ {j = k - N} ^ {k - 1} \Theta [ j, k ] - \Theta [ j - 1, k - 1 ] \tag {5}
$$
This equation measures the average change in the integration window at a given swap position (averaged across model position) in a manner that controls for the effect of absolute distance, since the distance between the swap and model position is the same for the two quantities being subtracted. Visually, this equation corresponds to subtracting two offset, off-diagonal columns from the integration matrix (see Fig. 3C for schematic).
We found that the value of our integration change metric  $(\delta [k])$  was close to 0, except at the boundary between sentences at later network layers (Fig. 3D). This boundary-specific change demonstrates that there is a drop in the integration window at the juncture between sentences in these layers, but not at other points in the sequence. This pattern suggests that the average value of our integration change metric  $(\delta [k])$  at structural boundaries provides a useful measure of structure-yoked integration:
$$
\eta = \sum_ {k = 1} ^ {B - 1} \delta [ k S + 1 ] \tag {6}
$$
where  $S$  equals the length of each structure (here  $S = 12$  words) and  $B$  equals the number of structures (here  $B = 5$  sentences). When we plotted this boundary metric across layers (Fig. 3E), we observed a clear increase in structure-yoked integration from early to late network layers (the median value of  $\delta[k]$  increased from 0.003 (layer 1) to 0.074 (layer 12);  $p \ll 0.001$ ; Mann-Whitney U test). This finding suggests a transition from position-yoked to structure-yoked integration across the layers of GPT-2. A similar increase in structure-yoked integration was observed for both LLaMA (Fig. 3G) and roBERTa (Appendix fig. 2B), suggesting that the emergence of structure-yoked integration is a common feature of LLMs.
4.1 Generalization across different structural lengths, types, and natural contexts
We next sought to examine the generality and robustness of our effects across different types of linguistic structures and contexts.
We examined whether the degree of structure-yoked integration depends on the structure's temporal extent by varying the length of the component sentences (using our base GPT-2 model). We found that our boundary metric (6) was similar when measured for short (8 words) and long (36 words) sentences (Fig. 4A), demonstrating generalization across structural extent. We also tested whether a similar boundary effect could be observed for a different type of linguistic structure, by repeating our analyses using sequences composed of noun phrases (6 words) rather than sentences (specifically, noun "chunks" as computed by spaCy, which do not allow nested noun phrases within them; Honnibal and Montani [2020]). We observed a similar increase in our boundary yoking metric across network layers, demonstrating that structure yoking is not specific to sentences (Fig. 4B).
In our original paradigm, we concatenated randomly selected 12-word sentences and the resulting sequences therefore lack coherent linguistic content across sentence boundaries. To test whether structure-yoked integration was present at natural structural boundaries, we repeated our analyses using 84-word sequences, excerpted from paragraphs (Brown Corpus) such that each sequence had a 12-word sentence aligned at the middle of the sequence (Fig. 4E). We observed a clear boundary effect at the start and end of this 12-word sentence that increased across network layers (Fig. 4D), demonstrating that our findings generalize to natural structural boundaries. The magnitude of the boundary change was somewhat smaller than that observed for randomly selected 12-word sentences, plausibly due to greater across-sentence integration for natural sequences that have coherent across-sentence linguistic structure.
Finally, we attempted to investigate hierarchical integration of linguistic structures. Specifically, we used ChatGPT-4 to generate sequences composed of exactly 3 paragraphs, each with 3, 12-word sentences. We then calculated our boundary strength metric at paragraph and sentence boundaries
A
Sentence length
B Noun phrases
C Example aligned natural sequences
Boundary change
Figure 4: Generalization across different structural lengths, types, and natural contexts. A, A similar increase in structure-yoked integration was observed for sentences composed of 8 and 36 words (same format as Fig. 3C). Results in all panels are based on GPT-2. B, A similar increase in structure-yoked integration was observed for sequences composed of noun phrases instead of sentences. C, Examples of two sequences of words that contain a 12-word sentence aligned to the middle of the sequence (indicated by color). D, Boundary change metric calculated from natural sequences with centrally-aligned sentences, demonstrating the emergence of structure yoking for fully naturalistic contexts. E, Boundary change metric computed for hierarchically structured sequences composed of 3 paragraphs, each with 3, 12-word sentences (generated by ChatGPT-4). The boundary change metric was computed separately for paragraph boundaries and sentence boundaries (excluding paragraph boundaries). We did not observe a greater change at paragraph vs. sentence boundaries.
(excluding paragraph boundaries). We did not observe greater structure yoking at paragraph boundaries (Fig. 4E), in contrast with what might be expected from hierarchical integration. However, the sequences generated by ChatGPT-4 did not always have a clear topic change at paragraph boundaries, which might have limited our ability to detect hierarchical integration.
5 Dependence on training
We tested whether the integration windows observed in trained LLMs were learned from the structure of natural language by repeating our analyses for an untrained GPT-2 model (Fig. 5; using the default settings from Huggingface; Wolf et al. [2020]). We found that integration windows from the model were flat across all network layers, consistent with the idea that attention-based architectures are capable of integrating uniformly over their input (Fig. 5A). The integration value for  $\Delta = 0$  was higher than for other distances, likely due to residual connections whose integration window is effectively a spike at 0. When we repeated our structure-yoking analyses, we found that our boundary metric was close to 0 for all network layers (Fig. 5B), providing further validation that our boundary metric provides a meaningful measure of structure-dependent processing in LLMs.
6 Conclusions, limitations, and broader significance
We have introduced a simple word-swap procedure for estimating integration windows in black-box neural language models and their dependence on linguistic structure. Using this procedure, we found that LLMs have stereotyped integration windows that substantially constrain the influence of an input word and follow a simple functional form, being well-approximated by a convex combination of an exponential and a power-law function. Across the layers of the network, we found there was a substantial increase in integration times driven partly by a transition from exponential to power-law dynamics. Finally, we found that integration windows at late network layers began to partially contour to the boundaries of linguistic structures, and we developed a metric to quantify this boundary-specific change. All of our findings were robust across different model architectures, the procedure used to swap words, and the length and type of linguistic structures manipulated, and were absent from an untrained model, demonstrating that they were learned from the structure of natural language. Thus our findings help to reveal how LLMs learn to flexibly integrate across the multiscale structure of natural language, and provide a flexible toolkit to study temporal integration in black-box language models.
Given finite time, there were limits on the number of different model architectures and linguistic structures we were able to test. Future work, for example, could examine how temporal integration windows differ between transformer models and recurrent neural networks, as well as examine a broader range of linguistic structures. While, we characterized how structure-yoked integration varies across the layers of an LLM, we did not investigate how units within a layer vary. For example, units or heads within a layer might specialize for integrating across particular types of linguistic structure, as suggested by prior work (Lakretz et al. [2019], Manning et al. [2020]).
The goal of our work was to understand existing language models, not advance the state-of-the-art. However, we believe there are many potential applications of our work. For example, one could investigate weight initialization schemes (or architectural improvements) that impose the functional forms demonstrated here at the start of training so that the network only needs to learn variations on this form (e.g.,
structure-yoked integration), which might improve speed or performance. Our metrics might also provide useful tools for diagnosing model limitations, such as an inability to yoke to larger-scale structures.
Our methods and findings have relevance to understanding neural integration windows in biological neural systems. Currently, little is known about what functional form best describes neural integration windows in the brain and whether/how these windows vary with structural boundaries (Chien and Honey [2020], Norman-Haignere et al. [2022], in part due to methodological limitations. LLMs are state-of-art in terms of predicting human brain responses to natural language ( Schrimpf et al. [2021]), and there is considerable interest in whether the computations of these systems resemble those in the brain as well as utilizing these systems to generate new scientific insights (Anderson et al. [2021], Caucheteux et al. [2023], Tang et al. [2023]). Because our methods are model-agnostic, they are potentially applicable to measuring and modeling integration windows in biological neural systems, and our findings provide testable predictions for how neural integration windows in the brain will be structured if they are similar to those in LLMs.
Results for GPT-2 with random weights A Integration Window
Figure 5: Dependence on training. Key findings are absent from an untrained GPT-2 model demonstrating they are learned from the structure of language. A, An untrained model shows flat integration windows with the exception of  $\Delta = 0$ . B An untrained model shows no evidence of structure-yoked integration.
The research reported in this publication was supported by the National Institute of Environmental Health Sciences (NIEHS) and the National Institute of Deafness and Communication Disorders (NIDCD) of the National Institutes of Health (NIH) (T32ES007271 support of D.S., R00DC018051 grant to S.N.H.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17: 261-272, 2020. doi: 10.1038/s41592-019-0686-2.
T. Wolf, V. Sanh, J. Chaumont, C. Delangue, A. Liu, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, T. Xu, Y. Le Scao, S. Gouws, B. Ginsburg, and N. Shazeer. Transformers: State-of-the-art natural language processing. https://huggingface.co/transformers, 2020. Accessed: [Insert Date].
M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences, 2021.
Equations for all functional forms tested is below:
- Log-Normal:  $f(x|\mu, \sigma) = \exp \left( -\frac{(\ln(x) - \mu)^2}{2\sigma^2} \right)$
- Log-Cauchy:  $f(x|\sigma, \mu) = \frac{\sigma}{\sigma^2 + (\ln(x) - \mu)^2}$
- Exponential:  $f(x|\lambda) = \exp(-\lambda x)$
- Power Law:  $f(x|\alpha) = (x + 1)^{-\alpha}$
Zipf-Alekseev:  $f(x|\alpha, \beta) = (x + 1)^{-\alpha - \beta \ln (x + 1)}$
- Exponential + Exponential:  $f(x|\lambda_1, \lambda_2, c) = c \cdot \exp(-\lambda_1x) + (1 - c) \cdot \exp(-\lambda_2x)$
- Exponential + Power:  $f(x|\lambda, \alpha, c) = c \cdot (x + 1)^{-\alpha} + (1 - c) \cdot e^{-\lambda x}$
A LLaMA intwindow
C LLaMA: example unit fits
B LLaMA: functional forms
Appendix Figure 1. Additional plots showing integration windows for LLaMA and BERT. A, Integration windows for individual units from several layers of LLaMA with the median across units overlaid. The influence of a word swap decreases sharply with distance (note logarithmic scale). B, Quantification of the goodness-of-fit for a variety of functional forms for LLaMA. The exponential-power fits were better at predicting integration windows across all layers, as with GPT-2. C, Integration windows for several example units from LLaMA fit using an exponential, power-law, or convex combination of both (see eq. 3). D, Integration windows for individual units from several layers of roBERTa. Negative distances indicate integration across future tokens and positive distances indicate integration across past tokens. E, Violin plots showing the distribution of fit parameter values across all units from different layers of roBERTa. Results were very similar to those for GPT-2.
A roBERTa: Magnitude differences
B roBERTa: boundary change
Appendix Figure 2. Emergence of structure-yoked integration in roBERTa. A, The average magnitude of response changes induced by word swaps as a function of both the model (rows) and swap (columns) position with sentence boundaries overlaid (gray lines). Results are similar to those for GPT-2, except the effects extend to both past and future tokens. B, Violin plots showing the distribution of the integration change metric at sentence boundaries across units for layers of roBERTa showing a substantial increase across layers, similar to GPT-2.