SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1f shows the performance of each layer of every Whisper and WavLM model. For legibility, HuBERT results are omitted from this plot and are included in the supplement (Figure B.2). The upper-middle and uppermost layers of each model tend to have the best performance, aligning with previous results on acoustic encoding models [29, 31]. In contrast with WavLM, the Whisper models increase in performance with layer depth; this can likely be attributed to our choice of only using the encoder module from the network.
Voxelwise scaling laws, examining the relationships described in Figure 1 on a per-voxel basis, can be found in Section A of the supplement.
3.3 Large-scale encoding models
After characterizing these scaling laws, we next visualized the performance of one of the top-performing semantic encoding models<sup>1</sup>.
Figure 2 shows the encoding performance of the best OPT model, which uses the 33rd layer of OPT-30B, as measured on the test story with 10 repeats. For several voxels from different areas of cortex we show the encoding model predicted timecourse and ground truth BOLD response. We see strong prediction performance across cortex, with "classical" language regions like Broca's area and auditory cortex being well explained, as well as areas that are typically considered to be more "amodal" in nature, like prefrontal cortex. Voxelwise correlations for this subject are as high as  $r = 0.82$ . A similar map showing the change in encoding performance from OPT-125M (comparable to GPT models used in earlier papers) to OPT-30B is given in the supplemental material (see Figure C.1).
3.4 Noise ceiling analysis
We further investigated the degree to which encoding models can be improved past this point. To do this, we performed a noise ceiling analysis whereby for each voxel, we estimated its  $CC_{max}$  (see Section 2.5). This gave us an approximation of the degree to which an ideal encoding model could explain the response variance in each voxel. We then renormalized the correlations from Figure 2 to compute a normalized correlation coefficient  $CC_{norm}$ .
Figure 3a shows the room for improvement, or the difference between the correlation coefficients measured in Figure 2 and their  $CC_{max}$ . Voxels are yellow if there is significant room for improvement, and purple if the model for that voxel is already close to optimal. Regions that are typically believed to contain high-level representations of language such as angular gyrus (AG) [50-52] still have the potential for substantial modeling improvement, while some areas in temporal cortex (near AC), prefrontal cortex (PFC), and the precuneus (PrCu) are nearly optimal. Figure 3b shows a histogram of absolute correlation coefficients ( $CC_{abs}$ ), and Figure 3c shows the normalized correlations  $CC_{norm}$ .
<sup>1</sup>Keeping with the scaling results from Section 1, we chose to demonstrate this using the best model from the OPT family, however it should be noted that the best model from the LLaMA family is about  $5\%$  more performant as measured by correlation. This LLaMA model is further explored in Section 3.6.
Figure 3: Noise Ceiling Analysis - Figure 3a: A two channel flatmap showing which ROIs remain poorly explained by an encoding model built from the 33rd layer of OPT30B. Voxels are less transparent if they have a higher idealized encoding performance ( $CC_{max}$ ). Voxels are more yellow if they have high room for improvement, defined as the difference between the best possible encoding model and this model. Angular gyrus and some parts of prefrontal cortex are still poorly explained, while precuneus and higher auditory cortex are close to optimal. Figure 3b: A histogram of voxel correlations ( $CC_{abs}$ ). Figure 3c: A histogram of normalized voxel correlations ( $CC_{norm}$ ). (PFC = prefrontal cortex,  $\mathrm{PrCu} =$  precuneus,  $\mathrm{AC} =$  auditory cortex,  $\mathrm{AG} =$  angular gyrus)
3.5 Long context artifacts
Granting encoding models access to contexts as long as 512 tokens implicitly gives them access to the information that the fMRI scan has started recently. For instance, if the input context has only 64 tokens, this implies that the context is occurring at the 64th token in the story. In parallel, responses in some voxels tend to rise or fall gradually over the first minute of each scan (potentially due to underconstrained detrending at scan edges, MRI magnetization reaching steady state, or neural adaptation). The combination of these two effects can have unintended effects on the fair evaluation of these models by artificially inflating measured performance, as encoding models are adept at capturing this early slow drift. We found that long context effects exist up to roughly 100 seconds into a story, so to mitigate this issue we simply exclude the first 100 seconds of predicted and actual responses from each test story when measuring encoding model prediction performance. Figure D.1 in the supplement gives a map of the effect of long-context artifacts on measured encoding model performance. Long-context artifacts can inflate measured performance by up to  $20\%$ , but the effects are mostly localized to areas typically associated with low-level speech processing such as early auditory cortex. This effect is most prominent for encoding models using early LM layers and speech models, and tends to not be as significant for later LM layers.
3.6 Unifying semantic and speech encoding models with stacked regression
We used stacked regression (see Section 2.4) to augment our best semantic model with the Whisper speech model representations. Figure 4a shows the regions that benefit from this augmentation, blended with a flatmap showing the overall semantic encoding model performance. We observe that these benefits are highly localized to auditory cortex and mouth motor cortex. The butterfly plot in Figure 4b shows the effect on voxels modified by this augmentation. We see that the auditory cortex voxels that are best predicted by the semantic model are also those that are most improved by this augmentation. Figure 4c plots the center of mass of the attribution weights  $\alpha^{v,s}$ . For voxels where the attribution weights favored the later layers of the Whisper model, the voxel is plotted in a brighter hue. We see that this attribution plot demonstrates a clear progression of auditory information from primary AC to secondary AC coinciding with layer depth. Figure 4d shows the benefits of this stacked regression augmentation. We see that the lion's share of the improvements happen in primary AC and early secondary AC.
4 Discussion & conclusions
These results suggest the existence of two major effects on the capacity of encoding models to predict BOLD response given finite brain data. First, LM changes that correspond to downstream task
Figure 4: Stacked Regression - Figure 4a: A flatmap shows which regions of cortex improve when augmenting a semantic encoding model built from the 18th layer of LLaMA-33B with the layers of Whisper using stacked regression. Voxels used the stacked regression if the stacked regression performed better on a validation set. The effect is highly localized to auditory cortex. Figure 4b: A butterfly plot comparing the voxelwise encoding performance of the stacked regression encoding model to the baseline semantic model. Figure 4c: The center-of-mass of the stacked regression attributions,  $\mathcal{C}(\alpha^{v,s})$  are visualized in auditory cortex. Figure 4d: The improvement in encoding performance of the stacked regression model over the baseline is visualized in auditory cortex.
performance improvement tend to also improve encoding performance, such as when moving from a LM trained on little data to one trained on more data. Second, increasing hidden state size while keeping other metrics fixed tends to lower encoding performance, as it worsens the conditioning of the encoding model regression problem without a corresponding benefit to model effectiveness. The conflict between these two effects has led to a scenario where the largest model is not necessarily the best for predicting BOLD responses, as we have seen for both the OPT and LLaMA LMs where encoding model performance peaks at about 30B parameters. Rather, a careful balance must be struck between model size and model efficacy in order to maximize encoding performance. Audio models, on the other hand, have not yet seemed to reach this plateau.
What are the use cases for better encoding models? One promising application is the use of encoding models to supplement more classical experimentation techniques, as suggested by Jain et al. [20]. Higher encoding performance leads to more trustworthy model predictions and more accurate conclusions. Another use case of effective encoding models is language decoding, or predicting the language stimulus from the BOLD response. Recent work has shown that effective language decoding models can be built from encoding models by applying Bayesian techniques [45, 53], so it is likely that the performance of such decoders will improve along with the performance of encoding models [33, 44]. Finally, improved encoding performance could enable fine-grained control over voxel activation through stimulus generation, as demonstrated by Tuckute et al. [54].
Given our results, what can computational neuroscientists do to improve the performance of their own encoding models? One potential observation is that deep datasets [43, 55-57] — those that focus on collecting many samples from a few subjects, rather than a little data from many subjects — are more useful for modelling brain activity. Encoding performance improvements scale well with both model size and dataset size, and large datasets will no doubt be necessary in producing useful encoding models. Another straightforward adjustment that can be performed is to simply use larger, more performant LMs for building encoding models. To the authors' knowledge, no other natural language encoding models paper at the time of this writing has used models larger than GPT-2 XL, which is a 1.5B parameter model with performance far below the best 30B parameter models. This
could be due to valid concerns that the amount of natural language brain data available is insufficient to train effective encoding models on such a scale. However, we found that even in low data cases, such as with as little as an hour's worth of data, encoding models built from larger models tend to outperform their smaller counterparts, as seen in Figure E.1 of the supplement. We have released code as well as selected precomputed features, model weights, and model predictions generated for this paper. We hope this data release will encourage the use of more performant encoding models in natural language computational neuroscience.
The authors acknowledge and thank the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources that have significantly contributed to the research results reported within this paper. This work was funded by grants from the NIDCD and NSF (1R01DC020088-001), the Burroughs-Wellcome Foundation, and a gift from Intel Inc. We thank Ruogu Lin, Leila Wehbe, and Javier Turek for their aid and thoughtful suggestions in assisting with this work.
[1] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023.
[2] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, 2022.
[3] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Deep language algorithms predict semantic comprehension from brain activity. Scientific Reports, 12(1):16327, 2022.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[7] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?, 2023.
[8] Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. *PloS one*, 9(11):e112575, 2014.
[9] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-458, 2016.
[10] Wendy A de Heer, Alexander G Huth, Thomas L Griffiths, Jack L Gallant, and Frédéric E Theunissen. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27):6539-6557, 2017.
[11] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
2These data are available at https://github.com/HuthLab/encoding-model-scaling-laws.
[44] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, pages 1-9, 2023.
[45] Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641-1646, 2011.
[46] Ruogu Lin, Thomas Naselaris, Kendrick Kay, and Leila Wehbe. Stacked regressions and structured variance partitioning for interpretable brain maps. bioRxiv, pages 2023-04, 2023.
[47] Lieven Vandenberghe. The cvxopt linear and quadratic cone program solvers. Online: http://cvxopt.org/documentation/coneprog.pdf, 2010.
[48] Oliver Schoppe, Nicol S Harper, Ben DB Willmore, Andrew J King, and Jan WH Schnupp. Measuring the performance of neural models. Frontiers in computational neuroscience, 10:10, 2016.
[49] Richard Antonello and Alexander Huth. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiology of Language, pages 1-16, 2022.
[50] Helene Van Ettinger-Veenstra, Anita McAllister, Peter Lundberg, Thomas Karlsson, and Maria Engström. Higher language ability is related to angular gyrus activation increase during semantic processing, independent of sentence incongruency. Frontiers in human neuroscience, 10:110, 2016.
[51] Amy R Price, Michael F Bonner, Jonathan E Peelle, and Murray Grossman. Converging evidence for the neuroanatomic basis of combinatorial semantics in the angular gyrus. Journal of Neuroscience, 35(7):3276-3284, 2015.
[52] Francesca M Branzi, Gorana Pobric, JeYoung Jung, and Matthew A Lambon Ralph. The left angular gyrus is causally involved in context-dependent integration and associative encoding during narrative reading. Journal of cognitive neuroscience, 33(6):1082-1095, 2021.
[53] Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011.
[54] Greta Tuckute, Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. bioRxiv, 2023.
[55] Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The "narratives" fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):250, 2021.
[56] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[57] Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. Bold5000, a public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49, 2019.