TopoLM: brain-like spatio-functional organization in a topographic language model

## Abstract

Abstract Neurons in the brain are spatially organized such that neighbors on tissue often exhibit similar response profiles. In the human language system, experimental studies have observed clusters for syntactic and semantic categories, but the mechanisms underlying this functional organization remain unclear. Here, building on work from the vision literature, we develop TopoLM, a transformer language model with an explicit two-dimensional spatial representation of model units. By combining a next-token prediction objective with a spatial smoothness loss, representations in this model assemble into clusters that correspond to semantically interpretable groupings of text and closely match the functional organization in the brain’s language system. TopoLM successfully predicts the emergence of the spatio-functional organization of a cortical language system as well as the organization of functional clusters selective for fine-grained linguistic features empirically observed in human cortex. Our results suggest that the functional organization of the human language system is driven by a unified spatial objective, and provide a functionally and spatially aligned model of language processing in the brain.

## 1 Introduction

Artificial neural network (ANN) models of language have recently been shown to accurately predict neural activity in the human language system (Schrimpf et al., ; Caucheteux & King, ; Goldstein et al., ). When presented with the same text input, the unit activity at internal layers of especially transformer-based models (Vaswani et al., ; Radford et al., ) is strikingly similar to the internal activity measured experimentally in human cortex. The most powerful models predict even close to 100% of the explainable variance of neural responses to sentences in some brain datasets (Schrimpf et al., ). However, while there is a strong alignment to the brain’s *functional responses*, a crucial element of cortex is entirely lacking from today’s language models: the *spatial arrangement* of neurons on the cortical surface.

In recent models of the visual system, the introduction of *topography* has led to ANNs that begin to match brain activity functionally as well as spatially (Lee et al., ; Margalit et al., ; Keller et al., ; Blauch et al., ; Lu et al., ). These models provide a principle for understanding the development of spatial organization in the brain, in the form of minimizing wiring cost, such that neurons with similar response profiles tend to cluster together. These clusters resemble the spatio-functional organization in the early cortex with orientation preferences such as pinwheels (Hubel & Wiesel, ; ; Maunsell & Newsome, ; Felleman & Van Essen, ), and in higher-level visual regions with category-selective regions such as face patches (Kanwisher et al., ; Haxby et al., ; Tsao et al., ; ; ; Freiwald et al., ).

The topography of the human language system on the other hand lacks a comprehensive computational explanation. Neuroscience experiments suggest both a macro-organization at the level of a distributed cortical network that selectively responds to linguistic processing (Fedorenko et al., ; ; ; Blank et al., ), as well as a micro-organization into clusters that correspond to syntactic and semantic categories such as verbs, nouns, and concrete words (Shapiro et al., ; Moseley & Pulvermüller, ; Hauptman et al., ). What are the mechanisms underlying this spatio-functional organization of the language system in the brain?

Here, we develop TopoLM, a neural network model for brain-like topographic language processing. TopoLM is based on the transformer architecture but incorporates an explicit spatial arrangement of units. We train the model via a combined task and spatial loss, which optimizes the model to perform autoregressive language modeling while encouraging local correlation, similarly to a recent approach used in vision (Lee et al., ; Margalit et al., ). The spatio-functional organization that emerges in this model is semantically interpretable and aligned with clusters that have been observed experimentally in brain recordings. Comparing TopoLM with a non-topographic baseline model (i.e. one trained without spatial loss) on a series benchmarks, we show that while TopoLM achieves slightly lower scores on some behavioral tasks (BLiMP), its performance on other downstream tasks (GLUE) and on brain alignment benchmarks (using the Brain-Score platform) is on par with the non-topographic control.

Importantly, this spatio-functional organization arises purely as a result of the combined task and spatial loss, as the model is trained solely on naturalistic text without fitting to brain data. This work thus extends the principle of cortical response smoothness proposed in vision (Margalit et al., ) into the language system, providing a unified explanation for understanding the functional organization of cortex.

Figure: Figure 1: Building a topographic language model with a brain-like spatio-functional organization. (a) TopoLM modifies the Transformer architecture with a two-dimensional spatial encoding at the output of each attention and MLP layer. This representation enables the propagation of a spatial correlation loss that encourages smooth response profiles in adjacent units. This spatial loss is jointly optimized with the task loss (autoregressive cross-entropy) during training. (b) At each forward pass, we randomly select five neighborhoods in each layer (of which we here only show 3 for clearity) and compute the pairwise correlation of unit activations within each neighborhood. The spatial loss is computed by comparing these correlations to the inverse distances between associated unit pairs, with the final loss averaged across units pairs and neighborhoods. Computing the loss on cortical neighborhoods is an efficient approximation of the spatial loss. (c) We use a FWHM filter to simulate the fMRI sampling process such that a simulated voxel’s response (‘fMRI-like signal’) is composed of a combination of responses from neighboring units (Kriegeskorte et al., ). We simulate the response as a Gaussian random variable, with FWHM $2.0$ mm, assuming unit distances of $1.0$ mm.
Refer to caption: x1.png

## 2 Related Work

#### Topographic Vision Models.

In contrast to the core human language system, the primate visual cortex shows a clear hierarchy of interconnected regions starting at the primary visual cortex (V1), passing V2 and V4, and reaching inferior temporal cortex (IT) which is thought to underlie representations of complex visual objects such as faces and scenes. Within V1, orientation-selective cortical patches (’hypercolumns’) are spatially arranged in circular ’pinwheels,’ where the preferred orientation of neurons rotates smoothly around a central point, covering all possible orientations (0 to 180 degrees). This structure is observed across species, including humans and non-human primates (Kaschube et al., ). On a more global level, early visual areas (V1, V2) show strong retinotopic organization where nearby stimuli in the visual field activate nearby locations in early visual regions (Engel et al., ; ; Tootell et al., ). While the strength of retinotopic organization strongly decreases, but remains detectable going to higher-level regions of the visual cortex (Larsson & Heeger, ; Schwarzlose et al., ; Kravitz et al., ; Groen et al., ), the final stage of the ventral visual pathway, IT, shows clear categorical clustering into e.g. regions selective for faces or scenes (Kanwisher et al., ; Haxby et al., ).

This spatial organization of the primate visual cortex has prompted work on topographic artificial neural networks for vision. First approaches focused on the organization of inferotemporal cortex, restricting topographic organization to later model layers (TDANN, ; ITN, ; DNN-SOM, , ). Recent models are designed such that all layers are topographic and thus mimic the topographic features across the visual cortex—for example, smoothly varying orientation preference maps forming pinwheels in model V1 and category-selective regions in model IT (All-TNNs, ; new version of TDANN, ).

Our topographic language model belongs to the Topographic Deep Artificial Neural Network (TDANN) family of models (Lee et al., ; Margalit et al., ). Herein, a central claim is that inducing a preference towards smoothness of cortical responses in the model provides a unifying principle for the development of topography in the brain. This smoothness optimization is applied to all layers in the model and replicates functional organization in early (e.g. V1) and later (e.g. IT) regions of the visual cortex. The TDANN’s spatial smoothness, as implemented with an additional loss term, is an indirect but efficient approach to minimizing local wiring-length, and can additionally help to minimize long-range connectivity which, in neuroscience terms, corresponds to brain size and power consumption (Margalit et al., ).

#### Topographic Language Models.

Comparatively little work has explored the idea of inducing topography in language models. In particular, the only topographic language model we are aware of is BinHuraib et al. ()’s Topoformer, which induces spatial organization onto a single-headed attention Transformer architecture using local connectivity constraints. This model arranges keys and queries on 2D grids, combined with a locally connected layer in the attention mechanism as opposed to full connectivity.
Our approach primarily differs from Topoformer in that we use a spatial smoothness loss term to drive the emergence of local correlations, similarly to Lee et al. () and Margalit et al. ()’s TDANN vision models. In this sense, our model extends Margalit et al. ()’s unifying principle of functional organization from the visual cortex into the language system. TopoLM is thus able to benefit from full connectivity, rather than requiring local connectivity to develop clustering. Because we apply this loss to the output of entire attention mechanism at each layer (as well as to the MLP), TopoLM can also benefit from multi-head attention, which was not explored in Topoformer. Finally, our model also uses an autoregressive task loss, rather than a masked autoencoder objective (as used in BinHuraib et al. ()), which has been shown to have higher performance on neural alignment benchmarks (e.g. Schrimpf et al., ).

## 3 Model Design and Visualization

Contrary to the convolutional neural network architecture used in topographic vision models (Margalit et al., ), we use the Transformer architecture (Vaswani et al., ) which is dominant in language modeling. We augment the objective function with a spatial correlation loss, in addition to the cross-entropy task loss. This loss function measures spatial smoothness, which serves as an efficiently computable proxy for neural wiring length: neurons located close to one another should have similar response profiles—i.e. their activations should be correlated (Lee et al., ).

To introduce a notion of ‘space’ in the model, we encode the units of each attention and MLP layer in a square grid with a one-to-one mapping. We randomly permute these positions for each layer such that each layer has a unique spatial encoding. On each forward pass, we first compute the pairwise Pearson’s correlation vector $\mathbf{r}_{k}$ between unit activations on the input batch for each layer $k$ (see Figure 1B). If a layer has $N$ units, $\mathbf{r}$ is of dimension $\mathrm{nCr}(N,2)$. Then, the spatial loss for layer $k$ is given by

$$ $\mathrm{SL}_{k}=\frac{1}{2}\left(1-\mathrm{corr}(\mathbf{r}_{k},\mathbf{d}_{k} )\right),$ (1) $$

where $\mathbf{d}_{k}$ is a vector of pairwise inverse distances between units, based on their spatial encoding, and $\mathrm{corr}$ is Pearson’s $r$. This means that nearby units (those with high pairwise inverse distance) should have highly correlated activations on the same inputs, and that distant units should have low correlation; this gives us a notion of spatial smoothness. Note that we scale by a factor of $0.5$ to ensure that $\mathrm{SL}\in[0,1]$.

We compute this spatial loss for every attention and MLP layer in a Transformer, prior to normalization and addition into the residual stream. For each batch of inputs, the model is then optimized subject to the loss criterion

$$ $\ell=\mathrm{TL}+\sum_{k\in\mathrm{layers}}\alpha_{k}\mathrm{SL}_{k},$ (2) $$

where $\mathrm{TL}$ is the task loss and $\alpha_{k}$ is the relative weight of the spatial loss associated with layer $k$. This combined loss metric encourages the model to learn representations that are both spatially organized and useful (and, in the case of self-supervised cross-entropy task loss, task-general).

#### Model Specification and Training.

In the below experiments, we utilize an adapted /-2-small style architecture (Radford et al., ). We use hidden dimension 784 such that we can evenly embed units in a $28\times 28$ grid. The model has 12 Transformer blocks, each with 16 attention heads and a GELU activation function. We train our models on a randomly sampled 10B-token subset of the FineWeb-Edu dataset. The task loss is cross-entropy on next-word prediction. We use batch size 48 and block size 1024. For spatial loss, we set $\alpha_{k}=2.5$ across all layers(^1^11For spatial loss, we set $\alpha_{k}=2.5$ across all layers. We chose this value of $\alpha$ after extensive hyperparameter search. In particular, lower values of $\alpha$ do not adequately encourage the development of topography, while greater values impede task performance and the development of meaningful representations.) and operationalize the inverse distance vector $\mathbf{d}$ with the $\ell^{\infty}$ norm. Rather than computing the spatial loss for the entire layer, which is computationally expensive, as in Margalit et al. () we approximate the loss at every batch using small neighborhoods. For each batch, we average the spatial loss across $5$ randomly selected neighborhoods, each of $\ell^{\infty}$ radius $5$. This allows us to compute loss more efficiently without significant performance drops.

We train both a topographic model and a non-topographic baseline, where $\alpha_{k}=0$ and all other hyperparameters remain the same.(^2^22After hyperparameter tuning, we optimize using AdamW with $\beta_{1}=0.9,\beta_{2}=0.95$ and learning rate $6\times 10^{-4}$, scheduled with warmup and cosine decay. We use weight decay $0.1$, gradient clipping at $1.0$, and do not use dropout. Neighborhood size and number of neighborhoods were also determined via hyperparameter search; note, however, that we empirically observed little effect of neighborhood size and number of neighborhoods on task performance or on our topographic metrics.) We trained both models with early stopping after three consecutive increases on validation loss. At the end of training, the topographic model achieved a validation task loss of 3.075 and spatial loss of 0.108 (summed across layers), while the non-topographic model achieved validation loss 2.966. Models trained for 5 days on 4xNVIDIA 80GB A100s.

#### Readout Sampling.

Due to the coarse spatial sampling in fMRI neuroimaging work, voxels contain the aggregated response of a large population of neurons (Kriegeskorte et al., , Figure 1C). In all following analyses, we thus apply a simulated version of fMRI readout sampling to model activations, consisting of smoothing with a Gaussian kernel, to imitate the locally aggregated responses of fMRI voxels. Importantly, we do so before computing selectivity based on these activations, and thus do not apply readout sampling to the functional selectivity maps directly. We set unit distance $1.0$ mm and FWHM $2.0$ mm.

Figure: Figure 2: Brain-like response profiles across the core language system. (a) Applying a functional localizer (Fedorenko et al., ) we isolate the core language system of TopoLM, and find clear brain-like spatial organization (for brevity, we only show Transformer blocks 11-14 here). Response profile across individual language-selective clusters in TopoLM are similar to one another, consistent with (b) the language system in human cortex (Fedorenko et al., ). (c) Across the entire core language system, TopoLM (blue) mostly matches the neural data (green), but not exactly; however, the non-topographic baseline model (orange) fails to capture neural patterns as well.
Refer to caption: x2.png

## 4 Spatio-Functional Organization of the Core Language System

Language processing in the brain engages a set of left-lateralized frontal and temporal brain regions. These areas are typically referred to as the ’core language system’ (Fedorenko et al., ) and respond selectively to linguistic input in contrast to non-linguistic stimuli (see Fedorenko et al., , for an overview). Due to anatomical differences between individuals, the language system is defined via a functional localizer that contrasts syntactically and semantically valid sentences against a perceptually matched control, such as strings of nonwords (Fedorenko et al., ).

Within individuals, the core language system shows clear spatio-functional organization, wherein language selective neurons cluster together across multiple cortical lobes. Anatomically distinct subregions of this system exhibit highly consistent response profiles to stimuli, suggesting that the system operates as a network (e.g. Fedorenko et al., ; Tuckute et al., ) (Figure 2B).

Prior work on neural alignment in language models typically compares neural responses across the *entire* core language system of the brain to model activations. The topographic organization of our model enables us to test for the emergence of a brain-like spatially organized system core language system in silico. A successful spatio-functional alignment between brain and model would mean that (1) distinct language-selective clusters emerge in the model, (2) these clusters all have consistent response profiles, and (3) the response profiles match the activity profiles in the brain (sentences $>$ {unconnected words, jabberwocky} $>$ nonwords; AlKhamissi et al., ).

#### Methods.

To isolate the core language system in TopoLM, we use the same localization stimuli as AlKhamissi et al. (); Fedorenko et al. (), which consists of a set of 160 sentences and 160 strings of non-words, all 12 words each. After passing these through the model, at each attention and MLP layer we run a $t$-test across the activations of all layer units. We then define the core language system as all units that are significantly language-selective ($p<0.05$ after correction for multiple comparison across all layers using the false-discovery-rate (Benjamini & Hochberg, )).

We then define language-selective clusters using an evolutionary clustering algorithm applied to each contrast map. In each layer, we begin with the most selective unit by $t$-value, and then repeatedly add the most selective neighboring unit to the cluster, until we hit a pre-determined $p$-value threshold associated with the unit’s selectivity (here, $p>0.05$). We repeat this process, searching for new clusters until all units in the layer are exhausted. We discard clusters with fewer than $10$ units.

Within each cluster, we measure responses to the same stimuli used in neuroscience experiments (Figure 2B, Fedorenko et al., ):
*sentences* (indexing syntactic and lexical information),
*unconnected (scrambled) words* (lexical but not syntactic),
*Jabberwocky sentences* which are well-formed sentences where content words are replaced by phonotactically plausible non-words (syntactic but not lexical), and
*unconnected (scrambled) non-words* (neither syntactic nor lexical).
Note that these stimuli are distinct from those used for localization. We measure the model ‘response’ as the mean absolute activation across all units in a cluster.

#### Results.

TopoLM exhibits clear brain-like spatial organization of the language network, such that
(1) multiple language-selective clusters emerge across the topographic tissue (Figure 2A and Figure 7), (2) across most clusters, the response profiles are consistent with one another (Figure 8), and
(3) response profiles *mostly* match the ones in the brain.
The response profiles are not a perfect match to the brain data—while sentences have higher activations than Jabberwocky and nonword stimuli, they do not have higher activation than unconnected words as in brain data.
However, looking across the entire language selective network, the response profile of the non-topographic baseline model similarly fails to capture the neural response profile (Figure 2C), suggesting a general shortcoming of the base transformer model, rather than a weakness of topography.

## 5 Spatio-Functional Organization Of Syntactic and Semantic Clusters

Beyond selectivity for language in general, experimental evidence supports the existence of cortical noun- and verb-selective clusters across human subjects during processing of verbal and nominal stimuli in auditory (Elli et al., ; Hauptman et al., ), visual (Moseley & Pulvermüller, ), and production (Shapiro et al., ) tasks. Here, we compare spatial activation patterns predicted by TopoLM to two groups of fMRI studies: Elli et al. () / Hauptman et al. (), who use the same set of stimuli and the same experimental setup (Figure 3 and Figures 6 and 9), and Moseley & Pulvermüller () (Figure 4). We evaluate TopoLM predictions quantitatively using the fMRI data from (Hauptman et al., ) and perform qualitative evaluations where no neural data was available (Elli et al., ; Moseley & Pulvermüller, ).

### 5.1 Clusters selective for verbs and nouns

#### Neuroimaging Study.

Using fMRI data from Hauptman et al. () (Appendix B), we find verb- and noun-selective clusters in the left hemisphere (Figure 3A and Figure 6), thus replicating their results. To quantify the ‘degree’ of clustering in these maps, we use Moran’s $I$ with Queen contiguity (i.e. $\ell^{\infty}$ radius, following the distance metric used to train TopoLM), a common measure of spatial autocorrelation, ranging from $-1$ to $1$ (see Appendix A for details). The group level effects map yields $I=0.96$ ($p<0.001$), indicating strong clustering.

Figure: Figure 3: Brain-like verb- and noun-selective clusters in TopoLM. (a) fMRI data from Hauptman et al. () points to verb- and noun-selective regions in the human cortex with strong clustering (Moran’s $I=0.96$). (b) Quantification of clustering. Relative to high clustering in the brain (green dashed line), the non-topographic baseline shows limited clustering (orange). The topographic model shows moderate clustering at the unit level (light blue) and strong clustering when simulating fMRI sampling (dark blue). On stimuli from Moseley & Pulvermüller (), another neuroimaging study on verb/noun-selectivity (fMRI data not available) we find qualitatively similar results. (c) Exemplary model maps (last MLP layer) showing the verb-/noun contrast (red-blue) in response to stimuli from (Hauptman et al., ). The non-topographic baseline shows no clustering while the topographic model develops verb- and noun-selective clusters.
Refer to caption: x3.png

Figure: Figure 4: Verb- and noun-selectivity in response to concrete and abstract stimuli. (a) Using stimuli from Moseley & Pulvermüller (), we find verb-/noun-selective clusters (verb: red / noun: blue) emerging in TopoLM for concrete (solid lines), but not for abstract words (dashed lines), thus replicating their results. (b) We obtain strong verb-/noun-clustering when concrete words are used to compute the verb-/noun-contrast (light blue, solid lines), $I=0.80$), but substantially lower clustering for abstract words ($I=0.23$, light blue, dashed lines, $t$-test: $p<0.001$). However, we do not find evidence for such a difference in verb-/noun-clustering when using the non-topographic control ($I=0.11$ vs. $0.12$, $t$-test: $p>0.05$, orange). Results do not change qualitatively when fMRI readout sampling is performed before computing contrasts and clustering (dark blue). In all the presented cases, spatial autocorrelation is computed on un-thresholded maps (for all layers, see Figure 10). Following neuroimaging conventions on defining category-selective cortical clusters, we show model maps thresholded at $p(\text{FDR})<0.05$.
Refer to caption: x4.png

#### Clustering in TopoLM.

To investigate whether similar verb- and noun-selective clusters emerge in TopoLM, we extracted the model activations in response to the same stimuli as Elli et al. () and Hauptman et al. (). Contrasting model activations to verb and noun stimuli yields clear verb- and noun-selective clusters across layers in our topographic model (verb: red / noun: blue, Figure 3A, Figure 9).
Similar to our analysis of the neuroimaging data, we quantified this observation using Moran’s $I$.
The non-topographic baseline model yields a low degree of clustering ($I=0.11$).
Contrast maps in TopoLM show a strong degree of clustering ($I=0.48$) which is further increased when applying fMRI-like readout sampling ($I=0.81$). Applying the same sampling to the non-topographic model also increases clustering, but it remains substantially less brain-like than the topographic model ($I=0.60$, Figure 3B,C).

### 5.2 Clusters selective for concrete, but not abstract verb-noun contrasts

#### Neuroimaging Study.

Moseley & Pulvermüller () focus on how cortical noun-verb selectivity relates to semantics, in particular focusing on concreteness.
Examining specific anatomically defined brain regions in fMRI, this study finds evidence for selectivity between concrete verbs and concrete nouns; yet, critically, there is no evidence for responses to abstract words from the same categories. Here, we investigate whether TopoLM replicates these findings—both the existence of spatially organized noun-verb selectivity in response to concrete words and the nonexistence of this selectivity in response to abstract words (Figure 4).

Clustering in TopoLM. We presented the original experimental stimuli to TopoLM and computed contrasts between abstract and concrete verbs and nouns (see Appendix C for stimulus details). Rather than comparing response profiles in anatomically defined subregions as in Moseley & Pulvermüller () (since anatomical regions are not well-defined in TopoLM), we instead explore whether lexical class-selective clustering emerges between concrete and abstract words. Consistent with brain data, we find clustering of verb- and noun-selective model units when concrete words were used, but find only very weak or no clustering for the same contrast using abstract words. We quantified this impression using again Moran’s $I$: concrete words yield substantially higher verb-/noun-clustering than abstract words ($I=0.8$ vs. $I=0.23$ in unthresholded maps for TopoLM, $t$-test: $p<0.001$, Figure 4, light blue). We obtain qualitatively similar results when simulating the fMRI sampling process before computing contrasts (Figure 4, dark blue). Additionally, we observed overall low degrees of clustering in non-topographic baseline models and no evidence for a difference in the degree of clustering when concrete vs. abstract words were used for computing the verb-/noun-contrast (Figure 4, orange).

## 6 Downstream Performance and Brain Alignment

Some topographic vision models sacrifice task performance for the sake of spatial organization, which leads to diminished utility as a model of brain function (Lee et al., ). As a control, we thus evaluate TopoLM’s performance on linguistic knowledge (BLiMP), downstream task (GLUE), and brain alignment benchmarks (Brain-Score), and compare it to the non-topographic baseline model.

#### Benchmark Description and Methods.

BLiMP (Warstadt et al., ) is a multi-task linguistic benchmark consisting of minimal pairs (acceptable / unacceptable) that probe specific knowledge of linguistic phenomena; we measure overall accuracy as the proportion of pairs to which the model assigns higher probability to the grammatically acceptable sentence. Models are evaluated without fine-tuning.
Brain-Score Language (Schrimpf et al., ; ) is a set of benchmarks that measure how well models align with the human language system. Following prior work, we train a ridge regression model to predict brain activity from model representations, using the same stimuli as in neuroimaging studies. Representations are taken from each transformer block where spatial loss is applied, including both attention and MLP layers. For both models, we select the output layer with the best alignment across datasets. The final score is the Pearson correlation between actual and predicted brain activity, averaged over 10 cross-validation folds and over four brain-recording datasets (Blank et al., ; Fedorenko et al., ; Pereira et al., ; Tuckute et al., , Appendix D).
GLUE (Wang et al., ) is a multi-task benchmark for downstream performance on tasks like entailment and sentiment analysis. Following common practice, we fine-tune models for each task. During fine-tuning, we continue to use the same weighted combination of task loss (cross-entropy or MCC) and spatial correlation loss with $\alpha=2.5$. We set batch size 64, dropout 0.1, and learning rate $2\times 10^{-5}$; all other hyperparameters are the same as pretraining. We use early stopping after 3 successive increases in validation loss, keeping the checkpoint before this increase.

**Table 1: Results from evaluation of TopoLM and the non-topographic baseline model (‘non-topo’) on BLiMP, GLUE, and Brain-Score. The non-topographic model outperforms TopoLM on BLiMP (by 5 points) and Brain-Score (by 2 points) out-of-the-box, but TopoLM outperforms the non-topographic control on GLUE (by 3 points) after fine-tuning.**
| Model | BLiMP | GLUE | Brain-Score |
| --- | --- | --- | --- |
| TopoLM | 0.71 | 0.68 | 0.78 |
| Non-topo | 0.76 | 0.65 | 0.80 |

#### Results.

We find that introducing spatial correlation loss does not strongly affect task performance or brain alignment (Figure 1). On BLiMP, we find slight decreases in performance between TopoLM and the non-topographic baseline across all subtasks, with a 5 point average decrease overall. On Brain-Score, we find a 2 point average decrease overall, with TopoLM notably outperforming the non-topographic baseline on some subtasks (see Appendix D). On GLUE, across most subtasks we see an increase in performance from TopoLM compared to the baseline model, with a 3 point average increase overall. We hypothesize this increase might be due the spatial loss term serving as an additional regularization against overfitting.

## 7 Discussion

Here, we present a new topographic Transformer language model based on a spatial smoothness constraint, and showed that it predicts key patterns of spatio-functional organization from the neuroimaging literature. In particular, we demonstrated that TopoLM has a spatially organized core language system where it exhibits brain-like uniformity of response profiles across clusters. TopoLM also exhibits brain-like noun-verb selective clusters, which are specific to concrete over abstract words. This additional spatial correspondence to the human brain seems to come at virtually no cost to performance and functional brain alignment.

#### Spatial smoothness principle.

TopoLM is in the TDANN family of models. Margalit et al. () originally posited the loss term underlying TDANN models as a unifying account of the development of spatio-functional organization in the visual cortex. TopoLM successfully extends the corresponding spatial loss to the domain of language neuroscience, and thus provides evidence that this principle of spatial smoothness indeed generalizes across cortex.

#### Limitations.

TopoLM is a feed-forward model of linguistic processing with multiple layers. Since the topographic maps are specific to each layer, there is as such no coherent tissue across the entire system as is the case for the brain. Transitions and space between clusters is thus likely not well accounted for. This also limits eventual modeling of ‘BMI’ tissue perturbations, such as micro-stimulation, since the spread of current ends at one topographic map (out of several).

While we evaluated our model on three distinct human brain datasets related to topographic linguistic processing, these are far from capturing the topography of language in the brain. New neuroimaging experiments are needed to this effect, which might further validate or also falsify the model—we see this as an opportunity for future experiments.

#### Model-guided experiments.

Since TopoLM is stimulus-computable, it can be used to discover new candidates for spatial clustering in linguistic processing. For instance, the automatic identification of clusters emerging from model activations in response to extensive textual input could inform the selection of optimal stimuli for further neuroimaging experiments. Given that the model successfully predicted spatio-functional organization in three existing brain datasets, it is likely that additional predictions derived from this approach would be informative.

Taken together, our results suggest that the spatial smoothness principle leads to topographic organization that is consistent with the spatio-functional organization of linguistic processing in the brain.

## 8 Acknowledgments

We thank Miriam Hauptman for providing fMRI data and for useful discussion. We also thank Greta Tuckute and members of the EPFL NeuroAI lab for helpful feedback. Neil Rathi was supported by a Summer@EPFL Fellowship.

## Appendix A Moran’s I 𝐼 I italic_I and Spatial Autocorrelation

We use Moran’s $I$ to estimate the degree of clustering in TopoLM layer maps and in surface-based fMRI data. Moran’s $I$ is given by

$$ $I=\frac{N}{W}\cdot\frac{\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}(x_{i}-\bar{x})(x_{j }-\bar{x})}{\sum_{i=1}^{N}(x_{i}-\bar{x})^{2}},$ $$

where $N$ is the number of units (e.g. fMRI vertices or model units), $x_{i}$ is the value at position $i$, $\bar{x}$ is the mean of all $x_{i}$, and $w_{ij}$ is a weight indicating the spatial relationship between units at positions $i$ and $j$ such that $w_{ij}=1$ if two points are ‘neighbors’ and $w_{ij}=0$ otherwise (see below on how this is defined).

Moran’s $I$ ranges from $-1$ to $1$. Positive values indicate spatial clustering, where similar map values (e.g., high-high or low-low) tend to group together; negative values indicate spatial dispersion, where dissimilar map values (e.g., high-low) are neighbors; and a value close to 0 0 indicates no autocorrelation, i.e. a random distribution. In our data, we do not see significantly negative values of $I$, as spatial dispersion is a systematic non-random pattern not encouraged by topography.

#### Application to TopoLM layer maps.

To quantify the degree of clustering in TopoLM maps (figures 3, 4), we apply Moran’s $I$ using ‘Queen’ contiguity (i.e. an $\ell^{\infty}$ radius). Queen contiguity is a measure commonly used in geospatial statistics, where each model unit is considered to be a neighbor if it shares any boundary (edges or corners) with another model unit of the same layer. This method allows for a more inclusive neighborhood structure than ‘Rook’ contiguity, which only considers edge-sharing neighbors.

#### Application to fMRI vertices.

We re-analyzed fMRI data from the left hemisphere of 22 sighted subjects from Hauptman et al. (). The functional and structural data were pre-processed by Hauptman et al. () and provided as GIfTI files. These files contain the cortical surface representation with 32,492 vertices (GIfTI format, Human Connectome Project benchmark) spanning the left hemisphere. To quantify the spatial autocorrelation in the fMRI data (fig. 3), we define $w_{ij}$ such that each vertex is considered to have six direct neighbors, reflecting the surface mesh structure where each vertex connects to six surrounding vertices based on their adjacency on the cortical surface.

Independent of the input modality (model layer units, fMRI vertices) Moran’s $I$ was always applied to un-thresholded maps. This is because thresholding a map before applying Moran’s $I$ would artificially inflate the estimate of spatial autocorrelation, as it results in contiguous patches of 0 0 values.

## Appendix B Hauptman et al. ( 2024 ) : design and fMRI processing

#### fMRI Processing.

Both Elli et al. () and Hauptman et al. () are based on fMRI recordings in response to the same task and stimuli—similarity judgments of auditorily presented words—and both investigate the relationship between sensory input and conceptual representations in the human cortex. Elli et al. () ($n=13$) provide evidence suggesting that lexico-semantic information about verbs and nouns is represented in partially non-overlapping cortical networks (Figure 1A in (Elli et al., )).

Hauptman et al. () presented the same auditory stimuli to 21 congenitally blind and 22 sighted subjects and investigated how neural representations of sensory concepts develop independent of sensory experience; here, we only include data from the 22 sighted subjects in our analysis. Similarly to Elli et al. (), Hauptman et al. () find distinct verb- and noun-selective regions (e.g. Figure 3A in Hauptman et al. (); Figure 6 below). For scanning parameters, and details on the fMRI preprocessing pipeline, see Hauptman et al. (). Here, we investigate whether similar non-overlapping cortical verb- and noun-selective regions emerge in TopoLM.

#### Experimental Design.

In each 2-second trial, subjects assessed the similarity of two auditorily presented words from the same verb or noun subcategory (e.g., bird or mammal for nouns, light or sound for verbs). Four trials, all using stimuli from the same subcategory, form a mini-block, followed by a 10-second baseline condition. This yielded a total of $288$ pairs of nouns and $288$ pairs of verbs.(^3^33Each pair was presented with verbs in the infinitive (with ‘to’) and nouns in the definite (with ‘the’). Removing these determiners when feeding stimuli through the model results in visually more distinct selectivity clusters; see Figure 9.)

To evaluate each vertex’s selectivity for a mini-block, we ran a general linear model with one predictor per mini-block. Contrasting verb and noun mini-blocks produced a $t$-value for each vertex, indicating selectivity toward verbs vs. nouns. As in Hauptman et al. (), we based our analysis on data from the left hemisphere (Frost, ; Szaflarski et al., ), which is more involved in language processing; this is standard in work on language neuroscience.

## Appendix C Moseley & Pulvermüller ( 2014 ) : experimental design.

In each trial, 18 participants from Moseley & Pulvermüller () silently read a single word from one of four categories: abstract nouns (e.g. clue, guide), concrete nouns (e.g. rice, goose), abstract verbs (e.g. heal, dwell), and concrete verbs (e.g. knit, poke). In total, there were $40$ stimuli in each category for a total of $160$ words. For additional information on the experimental design and neuroimaging parameters, please see Moseley & Pulvermüller ()).

## Appendix D Brain-Score Language Datasets and Performance

Figure: Figure 5: Brain-Score brain alignment performance. We tested TopoLM and the non-topographic control on Brain-Score language. TopoLM outperforms the control in Blank2014 and Tuckete2024, but performs slightly worse at Pereira2018 and Fedorenko2016 ($t$-test performed for each benchmark separately: $p<0.05$). For each benchmark, we sampled from 10 cross-validation loops to compute the bootstrapped 95% confidence intervals (black bars). When results are averaged across the 4 benchmarks (last panel ‘Brain-Score’), we don’t find evidence for a significant difference between TopoLM and its control ($t$-test: $p>0.05$).
Refer to caption: x5.png

#### Blank et al. ( 2014 )

This dataset consists of fMRI signals recorded from 12 functional regions of interest (fROIs), offering a lower resolution than the dataset used by Pereira et al. (). Five participants listened to eight naturalistic stories, adapted from fairy tales and short stories (Futrell et al., ). Each story lasted approximately five minutes, containing around 165 sentences, providing significantly longer context compared to other neuroimaging datasets.

#### Fedorenko et al. ( 2016 )

This dataset captures ECoG signals from five participants as they read sentences consisting of eight words, presented one word at a time for either 450 or 700 ms. Following Schrimpf et al. (), we focus on 52 out of 80 sentences presented to all participants.

#### Pereira et al. ( 2018 )

This dataset contains fMRI activations (BOLD responses) recorded as participants read short passages, with each sentence displayed for four seconds. It consists of two experiments: the first with nine participants reading 384 sentences, and the second with six participants reading 243 sentences. Each experiment covered 24 topics. Results are reported as the average alignment across both experiments, normalized using cross-subject consistency estimates.

#### Tuckute et al. ( 2024 )

This dataset is used to assess the distributional robustness of language models. Five participants read 1,000 six-word sentences, each presented for two seconds. BOLD responses from language network voxels were averaged both within and across participants, providing an overall response for each sentence. The stimuli span a wide linguistic range, allowing for diverse model-brain comparisons. Randomized sentence order and averaging across participants reduce temporal autocorrelation effects, making this dataset particularly challenging for model evaluation due to the linguistic diversity.

## Appendix E Supplementary Figures

Figure: Figure 6: Verb/noun-clusters across the human cortex. Re-analyzing data from Hauptman et al. (), we replicated their main findings in that we found verb-/noun-selective clusters across the left hemisphere. We here show group-level results of all sighted subjects at $p(\text{FDR})<0.05$.
Refer to caption: x6.png

Figure: Figure 7: Core language system. Using stimuli from Fedorenko et al. (), we create contrast maps reflecting language-selectivity for each unit of a given layer. We here show maps for TopoLM with (right column, dark blue) or without (center column, light blue) simulating the fMRI readout sampling process. Using a cluster growing algorithm we identify language-selective clusters. For comparison, we also show the language-selectivity maps from the non-topographic control (left column, orange).
Refer to caption: x7.png

Figure: Figure 8: Consistent response profiles across language-selective clusters in TopoLM. Using stimuli from Fedorenko et al. (), we extract response profiles to 4 linguistic categories across language-selective clusters. Importantly, the observed response profiles are consistent across language-selective clusters. For brevity, here, we only show clusters with more than 100 units.
Refer to caption: x8.png

Figure: Figure 9: Verb-noun clusters across TopoLM layers in response to stimuli from Hauptman et al. 2024. Complementing Figure 3, here, we show autocorrelation estimates (Moran’s $I$) for both TopoLM and its non-topographic control with and without FWHM filtering to simulate the fMRI sampling process (upper panel) and the associated verb-noun contrast maps for 5 layers (columns 1-4). When the stimuli are presented without ’to’ or ’the’ to accompany verbs and nouns, respectively, the verb-noun clustering (verb: red / noun: blue) becomes even more prominent (columns 5-6).
Refer to caption: x9.png

Figure: Figure 10: Verb-noun clusters across TopoLM layers in response to stimuli from Moseley et al. 2014. Complementing Figure 4, here, we show autocorrelation estimates (Moran’s $I$) for both TopoLM and its non-topographic control with and without FWHM filtering to simulate the fMRI sampling process (upper panel) and the associated verb-noun contrast maps for 5 layers. Autocorrelation estimates are based on the verb-noun contrast maps (verb: red / noun: blue).
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: Some further experiments and discussions could further strengthen the work:
- How does the choice of randomly permuting positions at each layer influence the representations? It would be extremely interesting to consider different types of mapping between spatial positions at successive layers.
- The authors stress the role of considering architectures with multi-head self-attention. Is this type of architecture expected to better model human neural activity? 
- In the NLP community, it has been observed a hierarchical organization of syntactic and semantic information in hidden layers of deep learning models (e.g. Belinkov, ACL, 2017; Manning, PNAS, 2019). Do the TopoLM clusters of activation exhibit some level of hierarchy through the layers of the model?
- In case it is feasible, comparing TopoLM representations/clusters also to Topoformer would further strengthen the quality of experiments.
soundness: 3
strengths: The authors contextualized and described with clarity the strategy introduced to incorporate spatial information of activations in transformers for natural language processing. Both the training details and experiments are very well explained and organized.

The work, inspired by previous efforts in vision models, introduces for the first time (to the reviewer knowledge) explicit spatial information in the loss of transformers trained on natural language.
summary: This work introduces TopoLM, a transformer language model incorporating information about the spatial arrangement of the units to model brain-like activity in the processing of language. In particular, the spatial information is integrated by adding a spatial smoothing term to the classical autoregressive loss of decoder-only transformers. By performing functional localization experiments, the authors identify clusters of artificial neurons in TopoLM whose activations are language-selective. The authors highlight the similarity between TopoLM clusters of activations and response in the brain by conducting experiments on verb-noun and concrete-abstract selectivity.
weaknesses: Given the strong inspiration from human neural activity studies of the work, the work would benefit from more extended discussion and analyses related to some architectural choices (e.g. random permutation of spatial position between layers and role of multi-head attention).

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Could you elaborate on the specific architectural choices made in TopoLM?  For instance, what motivated the selection of particular layers or activation functions, and how do these choices contribute to the model's performance compared to traditional language models?

Can you provide more insight into the training process for TopoLM?  Specifically, what datasets were used, and what were the criteria for their selection?  Additionally, how did you address potential issues like overfitting during training?

In your discussion of the biological implications of TopoLM, could you provide references or data from neuroimaging studies that support the connection between the topological aspects you incorporate and the way the human brain processes language?
soundness: 4
strengths: TopoLM introduces a novel approach to understanding the spatial organization of language processing in the human brain.  By integrating two-dimensional spatial encoding into a transformer architecture, the model effectively captures the topological structure of language, which has not been extensively explored in existing literature.  This innovative perspective opens new avenues for research in language modeling and cognitive neuroscience.  Meanwhile, this research has significant implications for both artificial intelligence and neuroscience.  By aligning computational models more closely with the biological structures of the brain, TopoLM paves the way for more effective natural language processing applications.  Furthermore, it contributes to a deeper understanding of the neural mechanisms underlying language, potentially informing both theoretical frameworks and practical applications in cognitive science and related fields.
summary: The study introduces TopoLM, a model based on the transformer architecture, aimed at exploring the spatial functional organization of human language systems. This model optimizes autoregressive language modeling by incorporating two-dimensional spatial encoding, along with task and spatial loss, which encourages response smoothness among adjacent units. Experiments demonstrate that the spatial functional organization generated by TopoLM closely aligns with the neuronal clusters found in the brain's language system. Although TopoLM scores slightly lower on certain behavioral tasks compared to non-topological baselines, its performance is comparable on other downstream tasks and brain alignment benchmarks. This indicates that TopoLM successfully captures the topological structure involved in language processing when trained on natural text, providing new insights into the functional organization of language systems.
weaknesses: While the paper presents a promising theoretical framework, the empirical evaluation of TopoLM is somewhat limited.  The experiments primarily focus on a small set of tasks, which may not fully capture the model's robustness across diverse linguistic contexts.  Meanwhile, this paper does not sufficiently explore or compare TopoLM with other existing topological models in language processing.  Including a comparative analysis with relevant models could strengthen the argument for TopoLM's effectiveness and novelty.  Highlighting specific improvements or unique aspects of TopoLM in relation to these models would enhance the discussion around its contributions to the field.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1.  I did not understand the argument about smoothing reducing wiring length. From my understanding, the TDANN network, even after training with the spatial regularization, retains the same dimensionality (weight matrix) as a non-spatial network. What then is the basis for the claim that this minimizes wiring length? (This refers to p. 3: “The TDANN’s spatial smoothness, as implemented with an additional loss term, is an indirect but efficient approach to minimizing local wiring-length.”) Additionally, from a functional perspective, in vision, hierarchical processing may reduce the need for long-range connectivity, as much processing occurs in localized circuits (even though reciprocal connections still exist). But in language processing, long-range connectivity seems essential. For instance, world knowledge can immediately influence lexical processing, and predictions can impact activity in primary and secondary auditory cortices, indicating that long-range connections are necessary in this domain.

2. page 6, the statement that a successful alignment between brain and model is accompanied by “these clusters have consistent response profiles”; the meaning of this phrase was unclear.

3. The analysis is focusing on univariate methods, where preference for sentence or word class is identified by differences in single-unit activation. But it is well known that the brain also represents information using multivariate (population) codes. What if a set of units, each of which shows similar activation for two classes of linguistic stimuli, can still be used to decode stimulus class through a population code?  From the perspective of the current work, these units will be defined as being outside the identified ‘selectivity’ clusters (and hence not considered ‘core’ to language). However, to the extent that a linear probe indicates that this population-code contains relevant information, there is no a priori reason to exclude them as being less relevant, essential or core to language.  For example, the null effect reported for sentences vs. unconnected words (p.6; results section) could still be consistent with the possibility that this set of units can differentiate between the two classes if a classifier trained and tested on their activations using linear probing. In short, the authors should better motivate why the operationalize ‘centrality’ to language based on univariate tests rather than population codes. 

4. For the benchmarks (p. 9) it seemed that fine-tuning involved retraining the entire model, rather than freezing the trained model, using it as feature extractor, and using a linear head for classification.  To evaluate the quality of features produced (rather than the model’s capacity to learn a new task) it would be useful to treat TopoLM and the control architecture as feature extractors, and apply the same procedure for learning a new task.

References

Zhu, J., Evtimova, K., Chen, Y., Shwartz-Ziv, R., & LeCun, Y. (2023). Variance-covariance regularization improves representation learning. arXiv preprint arXiv:2306.13292.
soundness: 3
strengths: The paper addresses a novel and important issue. There has been little work on training LLMs with topography constraints.  Successful training opens the door to interesting analyses, and potentially more flexible approaches for supervised learning of model-to-brain alignment. The writing is clear, and the analyses are interesting in that the authors apply well established statistical approaches used in the human neuroimaging literature to the model itself.
summary: The paper presents a language model whose training objective is based on a joint loss function: the standard next-word prediction loss and an additional spatial regularization term, which enforces a constraint where adjacent neurons exhibit more similar activation profiles than those of more distant neurons. The primary contributions of the paper are: (1) demonstrating that such a model can be successfully trained, resulting in an increase in local spatial correlations, (2) showing that its performance on standard benchmarks is comparable to models trained without the spatial regularization, and (3) providing evidence that the learned representations exhibit some degree of isomorphism with neural representations. For example, certain spatial clusters show stronger activation in response to well-formed sentences compared to nonsensical ones, and there appears to be a distinction between cluster-responses to nouns and verbs.
weaknesses: **P.S  following the discussion period: I believe the more critical perspective that was adopted by the authors during the review process improves the soundness of the paper. I have modified my evaluation.**

1. The spatial loss term is a key element in the work and should be better motivated. 
1a. As phrased in the paper, the loss term constrains neurons that are closer to each other (according to a predefined spatial arrangement) to have more strongly  correlated activation profiles than those farther apart. However, this introduces a global, long-range constraint, because activation profiles of neurons that are far apart are not independent, but constrained to differ. To enforce a strictly local connectivity constraint, the authors could have employed a formulation that emulates Gaussian smoothing or another local averaging kernel, which would only consider the immediate neighborhood of each neuron. Reading the authors' motivation for the spatial regularization term on page 3, I expected a local spatial smoothing constraint rather than the more global constraint that was implemented. They write: "This loss function measures spatial smoothness, which serves as an efficiently computable proxy for neural wiring length: neurons located close to one another should have similar response profiles—i.e., their activations should be correlated (Lee et al., 2020)". In practice, for computational efficiency, the authors do not transform an entire layer to a grid-topology, but select 5 smaller areas; however, the point still holds.

1.b In addition, the spatial-loss term introduces certain computational characteristics that should be considered in greater depth. Given its properties, it may encourage learning of decorrelated features, which could be advantageous for downstream transfer learning and fine-tuning.  In fact, this principle is used in a regularization technique introduced by Zhu et al (2023; https://arxiv.org/abs/2306.13292) to learn a potentially richer feature set. On the other hand, the spatial regularization term also pushes in the opposite direction: it enforces strong correlations between nearby units, potentially reducing the model’s effective capacity. In the extreme case, it can force two adjacent units to have almost identical activation patterns. This could limit the model’s ability to learn distinct features, which is a fundamental goal in neural network training. The authors briefly touch on this issue in the results section (p. 9), where they mention that the spatial regularization may help prevent overfitting, but this point asks for a more thorough treatment. 

2. The effects of the spatial loss on neuron-to-neuron similarity should be quantitatively evaluated
It is not clear how and whether TopoLM training changes the covariance structure of neural firing. It is important to rule out the possibility that non topographic models produce similar unit-to-unit correlation patterns compared to TopoLM. Otherwise, it would mean that training with spatial loss is not necessary for obtaining the resulting covariance structure.  This could be done by computing, for each neuron, a histogram of correlation values and averaging these histograms across all neurons.  If both TopoLM and standard models produce similar correlation distributions, it would suggest that typical neural networks naturally develop a graded connectivity structure similar to what TopoLM enforces, but this structure may manifest in a shuffled form in the absence of an explicit spatial arrangement.

3. Some of the spatial results appear determined or constrained by the training objective and other procedures. 
The finding that training produces stronger spatial smoothness and clustering seems to me to be a direct consequence of the spatial constraint in the training objective, which inherently encourages the formation of locally correlated clusters with weak correlations between remote clusters. Essentially, it implements a result similar to spatial data smoothing. Similarly, the fact that a simulation of fMRI readout-sampling (which applies further local averaging of unit activations) produces an even smoother data map is expected, as this is a direct result of the averaging imposed by the readout kernel.  Consequently, these properties do not emerge organically from the model’s spatial-loss training or fmri-like sampling, but instead reflect the successful implementation of the externally imposed constraints and workflow.  Because of this, the documentation of spatial clusters in statistical analyses is not surprising. For example, the authors write (p. 7) that “Contrasting model activations to verb and noun stimuli yields clear verb- and noun-selective clusters across layers in our topographic model”, but given the smoothing constraint applied, it would be very likely that the results would manifest in the form of clusters, as nearby units are constrained to be non-independent.  If these results are rephrased to be presented as a validation or sanity check that would make sense, but currently they are phrased as if they are a novel result. 

4. Some of the language-activation results appear dependent on the definition of language regions
The authors define a language system in TopoLM as the set of units that respond more strongly to grammatically valid sentences than to strings of non-words, which is analogous to the use of a functional localizer task in fMRI studies.  Then, for the network units identified by this condition, they evaluate responses to well-formed sentences and different types of language and language-like stimuli including strings of scrambled non-words or stings of phonotactically plausible non-words (jabberwocky sentences).  However, the aforementioned test used to define the localizer is not entirely independent of the subsequent evaluations. While it does not fully determine them, the localizer identifies units that are likely sensitive to word meaning and syntactic relationships, which are properties not found in the other stimulus types. Therefore, it is not entirely surprising that in the areas identified by the localizer, the strongest responses were to sentences, as compared to unconnected words, non-words, or Jabberwocky sentences.

It would have been more convincing if these results were found without the use of a localizer, using the spatial distribution alone: if the local clusters formed in the network are functionally meaningful, it would be important to see what information is encoded in those clusters.  If the strongest clusters naturally show a gradient of responses (e.g., from sentences to unconnected non-words), this would suggest that such distinctions are inherently emphasized during learning. Clusters can be identified via regional homogeneity (as done for fMRI) by computing correlation in response profiles across neurons.

The last two points also bear on the author’s ultimate conclusion, where they write, “Our results suggest that the functional organization of the human language system is driven by a unified spatial objective..”  I don’t think this follows from the results,  because I don’t see a basis for the analogy between the explicit spatial objective implemented in the paper (which produces some of the results) and those factors that ultimately lead to local correlations in the brain. Specifically, in the brain it is probably *unlikely* that there exists an objective that constraints nearby areas to fire similarly. This more likely emerges from cytoarchitecture constraints which produce different gradients of correlation (sometimes within the same neural population; https://pmc.ncbi.nlm.nih.gov/articles/PMC3412441/). 

5. General comment: I do not see a general weakness in the fact that benchmarks are not consistently improved by TopoLM. But what is lacking are more clear statements about whether TopLM develops different representations than non-Topo architectures.  This can be quantified by comparing representation-dissimilarity matrices, CCA, CKA or other methods for quantifying similarity of representations.  Of course, a comparison of these against RDMs computed from brain areas could also be helpful.

Meta-review: This paper modifies the standard autoregressive setting by adding a regularizing term that encourages nearby parts the model to behave similarly, with the motivation that this mimics known brain function. All reviewers agree that the idea is interesting and that the paper is ably executed.

Decision: Accept (Oral)