TopoNets: High performing vision and language models with brain-like topography
Authors: Mayukh Deb
Mainak Deb
Apurva Ratan Murty
Venue: ICLR 2025 Spotlight (2025)
Keywords: topography
neuro-inspired
convolutional neural networks
Transformers
visual cortex
neuroscience
Primary area: applications to neuroscience & cognitive science
TL;DR: A generalizable framework for inducing brain-like topography in neural networks without compromising task performance

Abstract: Neurons in the brain are organized such that nearby cells tend to share similar functions. AI models lack this organization, and past efforts to introduce topography have often led to trade-offs between topography and task performance. In this work, we present *TopoLoss*, a new loss function that promotes spatially organized topographic representations in AI models without significantly sacrificing task performance. TopoLoss is highly adaptable and can be seamlessly integrated into the training of leading model architectures. We validate our method on both vision (ResNet-18, ResNet-50, ViT) and language models (GPT-Neo-125M, NanoGPT), collectively *TopoNets*. TopoNets are the highest performing supervised topographic models to date, exhibiting brain-like properties such as localized feature processing, lower dimensionality, and increased efficiency. TopoNets also predict responses in the brain and replicate the key topographic signatures observed in the brain’s visual and language cortices, further bridging the gap between biological and artificial systems. This work establishes a robust and generalizable framework for integrating topography into AI, advancing the development of high performing models that more closely emulate the computational strategies of the human brain. Our project page: https://toponets.github.io

Published as a conference paper at ICLR 2025
TOPONETS: HIGH PERFORMING VISION AND LAN- GUAGE MODELS WITH BRAIN-LIKE TOPOGRAPHY
Mayukh Deb Mainak Deb², N. Apurva Ratan Murty Coonitioon and Brain Science, School of Psychology. Georgia Tech "Center for Excellence in Computational Cognition, Georgia Tech IIndependent Contributor
fmayukh, ratan @ataneegatechhco edu
ABSTRACT
Neurons in the brain organized such that nearby cells tend share similar functions. AI models lack this organization, and past efforts to introduce topog- raphy have often led to trade-off's between topography and task performance. In this work, we present TopoLoss, new loss function that promotes spatially or- ganized topographic representations in AI models without significantly sacrific- ing task performance. Topol loss is highly adaptable and can be seamlessly inte- grated into the training of leading model architectures. We validate our method on both vision (ResNet-18 18, ResNet-50, ViT) and language models (GPT-Neo-122) 125M, NanoGPT), collectively TopoNets. TopoNets are the highest performing super- vised topographic models to date, exhibiting brain-like properties such as local- ized feature processing, lower dimensionality, and increased efficiency. TopoNets also predict responses in the brain and replicate the key topographic signatures ob- served in the brain's 's visual and language cortices, further bridging the gap between biological and artificial systems. This work establishes robust and generalizable framework for integrating topography into AI, advancing the development of high performing models that more closely emulate the computational strategies of the human brain. Our project page: bttps://toponet: github. 10
INTRODUCTION AND RELATED WORK
Neurons in the brain are not tossed around haphazardly; they re spatially organized such that nearby cells perform similar functions (Barlow 1986 Rakie 1988//Eichhor et al. 2018 Krubitzer (2099) Topographic organization is core feature of brains (Geschwind & Rakic 2013; Arcaro & Liv- ingstone 2024). In visual cortex, this organization is evident in micro-scale pinwheel patterns for orientation selectivity (allinnddo category-selective regions for faces (Kanwisher & Yovel, 2006; Kanwisher et al. (002). bodies (Downing al., 2001), , scenes (Epstein al. 1999) etc and in large-scale organizational biases for real-world shape, size and animacy (Konkle & Caramazza 2013, Konkle & Oliva 201T). Beyond vision, in the language cortex, recent studies have also identified neurons with distinct temporal in- tegration windows (Hasson et al.) 20088 Lerner et al.) 2011 Regev et al. 2024). Unlike the brain, most artificial neural network (ANN) models lack any systematic organization of units. In this work, we introduce a new brain-inspired inductive bias, TopoLoss, that can be integrated into the training of most current ANN architectures, including convolutional networks and transformers. The re- sulting models, TopoNets, bring brain-like topography to AI, yielding high-performing models with localized, low-dimensional, and efficient representations much like the brain (Figure 1).
1997 Bonhoeffer & Grinvald 1991). in macro-scale al.
Inducing topography into artificial neural networks (ANNs) has proven to be challenging and two main strategies have emerged. The first, post-hoc topography, involves re-organizing units in pre- trained models using methods like self-organizing maps (Doshi & Konkle 2023 Zhang al. 2021 Kohonen 1997). The resulting models exhibit topographic signatures, but the underlying repre- sentations remain unchanged from the original model. Consequently, the functional advantages of topography, such as reduced dimensionality and increased efficiency, are not realized. The second strategy, joimily- topography, incorporates an additional topographic loss during model

---

Published as a conference paper at ICLR 2025
Baseline models (unstructured)
TopoNets: models trained with TopoLoss brain-like topographic organization
Topographic Vision Models (conv-nets and transformers)
ResNet18s 8s
ResNet50s VIT-b-32s
ResNeet
no organization
topographically organized units
Topographic Language
Models (transformers)
GPT-Neo-125M
NanoGPTs
TopoNec-GPT-Noo-1222
Figure Towards high performing topographic vision and language models (TopoNets). Schematic shows transformation from unstructured baseline models (left) organized topographic representations (right) for vision (top) and language models (right). The stacked maps are 3 repre- sentative layers (early, mid, and late) of the model.
training. These models induce topography by (1) explicitly matching the brain's spatial correla- tion structure (Lee al. 2020 Margalit et al. 2024. (2) imposing distance-dependen constraints (Blauch et al., 2022 Qian et al. 2024)) or (3) encouraging information redundancy (Keller et al., 2021). These approaches suffer significant tradeoff:: the ability of the model learn task-relevant representations is often compromised. They perform poorly on engineering metrics (like perfor- mance on ImageNet) and/or show diminished capacity predict brain data. Also most prior work in this space has focused exclusively vision models and the one attempt ((Binhuraib et al.) 2024) imparting topography to language models focused on the self-attention maps of BERT, which resulted in only modest topographic organization in the output space. To summarize, no unified strategy currently exists for applying topography across ANN algorithms (i.c., convolutional nets and transformers) and domains (vision and language) that can deliver high-performing models to- gether with the functional benefits of topography.
Here, set out to design general inductive model bias to recapitulate signatures of brain-like topography and its computational benefits without sacrificing model accuracy. achieve this, was important understand why and how topography arises in the first place. Theoretical work neuroscience suggests that one of the primary evolutionary pressures on the brain metabolic efficiency only terms of minimizing wiring length of neurons (Kaas 1997). but also in managing the vast network of potential neural connections (Katz & Shatz 1996 Chklovskii et al.,
2002 Chklovskii & Koulakov 2004). The brain addresses this challenge through synaptic pruning.
Early in development, the brain forms an excess of synaptic connections, which are then systemat- ically reduced over time based on activity-dependen mechanisms (Kaas) 1997; Faust et al. 202T] Riccomagno & Kolodkin, 2015| Schulz. & Reggia 2005). This pruning process retains only the most necessary connections, optimizing for the efficiency of the neural network. Our topographic loss incorporates ideas about synaptic pruning into its design.
Our study makes the following contributions: A) We introduce TopoLoss, a new inductive bias that generalizes across model architectures (convolutional networks and transformers) and domains (vision and language). (B) We show that TopoNets, our suite supervised topographic models, outperform previous topographic models ImageNet performance and predictions brain data (as on BrainScore) while maintaining similar levels of topography.
(C) TopoNets provide clear evi- dence resolving theoretical claims about the role of topography in creating dow-dimensional feature
2

---

Published as a conference paper at ICLR 2025
representations. (D) We show that TopoNets demonstrate improved model efficiency. (E) TopoNets replicate topographic signatures observed in vision and language cortex in the brain.
METHODS
2.1 DEFINING THE CORTICAL SHEET
Our first task was to define 2D sheet (See Appendix A.I for a discussion of topography in 2D versus 3D spaces) where we could apply our topographic loss (TopoLoss). To demonstrate that TopoLoss generalizes across different domains, applied it to both language and vision models. For language, we trained GPT-Neo- 125M models (Black et al. 202I) on the Wikipedia Dataset Wiikieedia Foundation) and NanoGPT models/Karpattie (2022) on billio tokens from Fine Edu (Lozhkov et al.) 2024). For vision, we trained topographic ResNet-18 to allow comparisons with previous topographic models, and ResNet-50 (He et al., (016) and ViT-b32 Dosovitskiy (2020) models to further evaluate the generalization of LopoLoss to larger models and architectures. All vision models were trained on a supervised 1000-way classification task ImageNet (Deng al. 2009). Together these language and vision models allowed us to robustly evaluate TopoLoss across varied model architectures and domains.
Cortical sheet in Transformers (language models and ViTs): For linear layer with input units and output units, we reshape its weight matrix W E Roxi to cortical sheet C Axx In this setup, the area of the sheet (h x w) corresponds to the number of output units (o) and the depth (d) corresponds to the number of imput units (t). To maximize the number of neighbors for each element in the cortical sheet, we chose h and to be as close to each other as possible, thereby minimizing the perimeter. Each elementt in this cortical sheet now represents the weights associated with a single output unit (or "neuron") in the original linear layer.
Cortical sheet in Convolutional Models: For a convolutional layer with Cinput input channels and Couppu output channels, and a kernel-size of x k. we project its weight tensor W E R Copp x C x onto a cortical sheet C e gxxuxd, where the area corresponds to the number of output channels (h w) and the depth is defined d Capput X k x k. As in previous work (Qian et al., 2024), arranged the model units (convolutional kernels) on 2D cortical sheet. more detailed explanation provided in the appendix A.2.
2.2 INDUCING TOPOGRAPHI (ToPoLoss)
The second step introduces the TopoLoss to the reshaped cortical sheet, promoting topographic organization. We achieve this by maximizing the cosine similarity between the original cortical sheet and its blurred version. This suppresses the high-frequency noise, leaving behind only the most important and meaningful information. This idea was motivated by synaptic pruning in the brain, which eliminates noisy (high frequency) neural connections, refining the biological network's structure (although note that we do not explicitly remove any weights here). The blurring of a 2D signal X h can be defined using a downsampling function fdown and upsampling function fup as follows:
Blur(X, on vn.) = Sup [aown
X, Oh
11,
Here dr and ow are the downsampling factors along height and width dimensions (both set to 3). To encourage smoothness in the cortical sheet Chxuxd we maximize the cosine similarity between C' and its blurred version C' across cortical sheet layers maps. This process smoothens the representations and encourages topographic organization. The TopoLoss is defined as:
1
C.
Crooo
N 1-1 |CC|||III
This TopoLoss integrated with the original training loss Lusining as follows:
Lisua Lariniigg + T (Linpo)
Here T is a scaling factor that controls the strength of the topographic effect: higher values encourage stronger topographic organization in the model.
3

---

Published as a conference paper at ICLR 2025
Vision Models: We applied TopoLoss to every convolutional layer in the residual blocks (as (Qian et al. 2024)). ). All vision models were trained on a supervised 1000-way classification task using Im- ageNet. 1. ResNet-11 We trained 8 distinct ResNet-18 88 (He et al.] 2016) models from scratch on the
ImageNet (Deng et al. 2009) dataset across various topographic configurations: one baseline model
(no topography), six TopoNets with different topographic scaling factors: 0.5, 1, 5, 10, 20, 50. Models were trained using the ffov (Lecierc et al., (023) training recipe. ffov (Fast Forward Computer Vision) significantly accelerates model training by replacing traditional data loaders with an efficient binary format and leveraging multiprocessing and GPU-accelerated data augmentation to optimize data pipelines.
We selected ResNet- 18 to compare the performance of TopoNets with previous topo- graphic approaches like TDANNs and L.LCN.. However it has been demonstrated that ResNet-50 offers richer visual representational basis for predicting brain responses (see: |aaan Murty et al.
ResNet-50:
(2021 Lahner et al. (2024); McNeal et al. (024)); Khosla et al. (2022). ). Hence we trained 3 addi- tional ResNet-50 (He et al. 2016) models from scratch on ImageNet (Deng et al. 2209):: one base-
line model (no topography), two TopoNets with different topographic scaling factors (T = 10).
ViT-b-32 To demonstrate further generalizability beyond convolutional architectures for vision, we trained a Vision Transformer (Dosovitskiy 2020) on the ImageNet dataset. We followed the recipe provided by Vision maintainers & contributors 2016) and applied TopoLoss with 10 on the last ML. module i.e the mlp.3 module in each transformer block.
PPT-Neo-125M. We trained 5 GPT-Neo- 125M (Black al. 2021) models on the Wikipedia dataset (Wikimedia Foundation) with different scales of the topographic loss (baseline and 7=1, 5, 10 and 50 respectively). We applied TopoLoss to the c_fc layer of GPT-Neo. This choice was based on prior work by (Geva et al., 2020 2022) that has suggested that the feed-forward modules in GPTs act as key-value memory modules storing world knowledge. The elfe modules encode the persistent representations (in contrast to transient representations in the attention matrix) making it the theoretically grounded target for inducing topography.
NanoGPT-125M We trained 4 NanoGPT (Karpathy (022) models on 10 Billion tokens sampled randomly from the Wen-EebbEdu dataset (Lozhkov al. 2024) with different scales of the topo- graphic loss (baseline and = 0.5, 1, 50 respectively). TopoLoss was applied to the e_fc modules in each block (as explained above).
2.3
OTHER METRICS
Effective Dimensionality: Effective dimensionality was measured as described previously in MMrr-
galit et al. 2024, Del Giudice, 202T Elmoznino Bonner 2024).
(E A1) Σ X?
Effective Dimensionality
A indicates the eigenvalues and n the number of eigenvalues. This metric measures the spread of the cigenspectrum. For ResNets. we followed the procedure outlined in (Margalit et al. (022)). We chose 20,000 images from the ImageNet validation set calculated the effective dimensionality of the features for all the convolutional layers. For language models, we chose 8192 samples from the openwebtext dataset and measured dimensionality of the representations from the topographic (c_fc) layers.
Smoothness We measured topography, using smoothness score (as before, (Margalit et al., 2024) Smoothness was defined as the difference between the highest and lowest correlation values from pairwise correlation versus distance plots.
L1 unstructured pruning:
We impose sparsity by pruning percentage of the mmallest-magnitue weights. Specifically, we sort the weights in ascending order of their absolute magnitude and set the smallest n% of them to zero. To reduce the number of weights by factor of we prune (100- 100 ) of the smallest weights. Downsampling: We downsample the topographic layers n
by first projecting them into the cortical space and then performing downsample operation along the height and width dimensions of the cortical sheet. A detailed explanation of the downsampling operation and inference such models can be found appendix A.5.
4

---

Published as a conference paper at ICLR 2025
LI unstructured pruning and downsampling were applied progressively increase the degree of sparsity (ensuring that each sparsity level corresponds to the same effective parameter count) and evaluate the effect model performance. For each sparsity level, we evaluated the model's per- formance (classification accuracy or perplexity) and reported the resulting performance difference from the baseline model (Figure 4).
Estimating selectivity: We collected layer-wise features in response to stimuli. Selectivity is then calculated using a standard method for estimating selectivity from these representations (e.g., (Mar- galit et al. 2024 ):
Pe
Ho
(2)
Where J4, o, N denote the mean, standard deviation and the number of layerwise representations for the target category and other categories o. We used stimuli from previously published studies to identify the category-selective regions SStiliani et al. (2015) and regions with biases for real-world size and animacy (Konkle & Caramazza 2013).
Temporal window analyses: We estimated the temporal integration window of every umit within
GPT-Neo- 125M using recently developed method (Skrill & Norman- Haignere 2024). Briefly, this approach employs a word-swapping paradigm, measuring the difference in response magnitude for the swapped sequences. The integration window defined as the distance- dependent change in response magnitude across multiple sequences and word swaps. For detailed methodology, we refer readers to this important study. The key equation relevant to our work is as follows.
0uamn x c(A + 1) (1 - c)e 64
Ouorm [A] is the normalized temporal integration window and is the convex combination parame- ter. Intuitively is the balancing knob that controls the relative influence of two different integration window shapes. a represents the power-law component of the integration window. Higher values would indicate relatively slower decline. b is the exponential component of the integration win- dow. Higher numbers would indicate a much more rapid decline. We followed exactly the same procedures outlined in the previous study to estimate these values.
3 RESULTS
3.1 TOPONETS ACHIEVE IIGII MODEL PERFORMANCE WITH COMPARABLE SPATIAL TOPOGRAPHY
How do TopoNets stack up against baseline models and other snoography-induchi methods? We first tested vision models, specifically ResNet18 trained in supervised manner ImageNet. This architecture allows for direct comparison with previous work (Margalit et al.) 2024] Qian et al.) 2024]. (Note: data for ITN (Blauch et al., 2022) and All-TNNs (Lu et al.) 2023) are unavailable, as these architectures haven't been scaled to ImageNet). Model performance is presented against the amount of topography (smoothness, see (Margalit al. 2024)) in Figure 2A. We find that (I) TopoNet-ResNet models achieved significantly higher accuracy on Imagenet (red dots) than LL- CNN (Imagenet trained supervised) and TDANN. TDANNs were trained using self-supervised SimCLR objective, while TopoNets were trained in supervised manner. To ensure fairer com- parison with TDANNs (see A.3 for extended discussion), we trained an additional TopoNet with topography induced in similar locations as in TDANN (in 8 locations) and compared the change in performance from baseline toporrapgg.. TDANNs exhibited 6% drop in performance, whereas TopoNets showed only 3% drop. (2) TopoNets achieved comparable levels of topography (dashed gray vertical lines) as previous approaches. (3) TopoNets were trained for fewer training epochs (12% fewer than LLCCN-G) ). Even the worst performing PopoNet-ResNetll 8 (r = 50) was significantly better than the previous best topographic model (25% drop for TopoNet- Net compared to 41% drop for LLCNN-G, from baseline). The model performance Pareto curve for TopoNet- TopoNet-ResNet11 model is shown as a black dashed line across levels of topography. Together these results indicate that models trained in supervised manner with our model inductive bias: TopoLoss (TopoNets) achieve substantially higher task performance than prior topographic models. TTopoNets new standard of performance for supervised topographic ResNet18s.
5

---

Published as a conference paper at ICLR 2025
TopoNet Vision Models
TopoNet Language Models
Rconct-00
Sewoon wall talssn accuracy
T-08
TopoNet
TopoNett
0.8
TopaNees
(lighhu is better)
Aeamat-18
Basaline
W7-b522
T-1.0
5o000
VIT-b32
10
foeanee BUMP
1olk
OP- Topefomar)
Tel
models
lo epocks
TopoNet GPT
C-
50
r-
Imsgen
LENNNG 100 specta
Provious best topographic models
Baseline
TDANN
models
45.
200 apouto
TF's Topaflmmer
De powviss iopogrghhic mode based on BrA7)
0.4
0.2
Smoothness
1.0
0.2
Smoothness
1.0
topagraphy
Degree of topography
Nigh topography
no topography
Degree of topography
high topography
Figure 2: TopoNets achieve higher model performance with comparable topography. A. Esti- mated model topography (smoothness, x-axis) versus model performance (y-axis) for vision models (ResNet-18 18, ResNett-50, ViTs). The black filled dots with the dashed gray crosshairs indicate prior models. The dashed black lines indicate the pareto-curves for 18 and ResNet-50 models. B. Same as A, but for Language models (GPT-Nee-1255) 125M and NanoGPT). The y-axis here denotes the language model evaluation score on BLiMP. The dashed gray line indicates the reported topography from a prior study.
Next, we investigated whether we could develop even higher-performing TopoNets and extend the approach to transformer architectures. We trained 2 ResNet-50 models and one ViT-b32 model with TopoLoss. The model performance and topography measurements are shown for TopoNct-Reshe ResNect- 50s and TopoNet-ViT ViT-b32 as blue and green dots respectively in Figure 2. TopoNet-ReeNNet-000 and TopoNet-ViTi VIT both outperformed TopoNet-ResNeds
at similar levels of topography. Notably, our TopoNet-ResNet500 achieved comparable performance the baseline ResNet-18 18 model, while exhibiting comparable levels of measured topography as prior topographic models.
Does TopoLoss generalize to language models? To investigate this question, trained GPT-Neo- 125M (with and without TopoLoss)) on Wikipedia. To demonstrate further scalability, we also trained NanoGPT models on 10 billion tokens from the FineWeb-Bee dataset. All models were evaluated on common evaluation measure: BLiMP (Warstadt et al., 2020). Our findings revealed that (I) TopoNets were comparable to the baseline (non-topographic model) for GPT-Neo- 125M and were close to baseline for the scaled up NanoGPT models (Figure 2B). (2) Most TopoNets achieved higher levels of topography than Topoformers (BERT trained on IMDB (Binhuraib et al. 2024)) even in the layers where topography was explicitly implemented (attention Q.K.V. Figure 2B). Together these results demonstrate that TopoLoss can generalize across different model architectures (convo- lutional nets and transformers) and domains (vision and language). The resulting models, TopoNets, significantly outperform previous isolated efforts in vision or language alone.
3.2 TOPOGRAPHY, NOT MODEL PERFORMANCE, DRIVES DIMENSIONALITY REDUCTIONS IN TOPONETS. EVIDENCE ACROSS MODEL ARCHITECTURES AND DOMAINS
Prior theoretical work has suggested that the brain's 's topography may affect non-topographic aspects of learned representations, such as the effective dimensionality (Durbin & Mitchison 1990; Swin- dale 1996). Effective dimensionality is lower when neurons are similar to each other and higher when they are independent. Studies show that 1) effective dimensionality increases with model depth and training, and 2) models with lower dimensionality better predict responses in high-level visual cortex (Elmoznino & Bonner 2024). However, recent work (Qian et al., (0224) has can doubts on this observation. Specifically it unclear whether the reduction in dimensionality is driven by lower model performance (Hypothesis I) or because of topography itself (Hypothesis 2). TopoNets finally allow us to test these competing hypotheses more precisely across model architectures and domains.
We measured model dimensionality using standard approach from previous studies (Del Giudice 202T Elmoznino & Bonner, (024) and examined its relationship with model accuracy (Hypothesis 1) and topography (Hypothesis across both vision (ResNet- 18, ResNet-50) and language mod-

---

Published as a conference paper at ICLR 2025
Topography, model performance, explains reductions dimensionality across architectures
Vision
iP 0.66 Pu0.00
40
Dimerrsoonale
20
F-od
Fast
T-
20AMM
15
T-403
1g
Vislon
Twxte Terl
21:00
43%
Model performance
75%
Degrus of topography limoothness)
(ImagaNet validation % cornect)
Language
40
R- 0.87, P.d.41
20-
100
To
Eleecive
Language
10
Ru-d 0005
F-5)
2-:0
57.5%
Model performance
77.5%
[BLMP Accurscy)
Degree of topography (emoothness)
sopagraphy
Figure 3: Topography explains reductions in model dimensionality. A. (Left) Model performance (Imagenet accuracy, x-axis) versus effective dimensionality for vision ResNets. (Right) Measured topography (smoothness) versus effective dimensionality for vision ResNets. B. Same as A, but for language transformers
els (GPT-Nco- 125M, NanoGPT). These results are shown in Figure 3A. (Model performance for TDANN is included for comparison, while LLCNN is not reported as it is not yet publicly avail- able.) We found no significant correlation between model performance and dimensionality in either vision language models (both P > 0.05, Figure 3 left column). Notably TopoNets achieved higher model performance despite equivalent levels model dimensionality as TDANNs (Figure 3A). In contrast, dimensionality was significantly correlated with the measured topography (smoothness) in both domains (each P < 0.05). The difference between the linear relationships was statistically significant for both vision and language models (P <0.05, on Fisher's z-transformed correlations).
These results support Hypothesis 2. Spatial topography, rather than model performance, better ac- counts for reduction in the effective dimensionality of the learned representations. This analysis also shows how TopoNets can be used to evaluate theoretical claims regarding the role of topography in shaping representations across various model architectures and domains.
3.3 TOPONETS DELIVER SPARSE, SEEAMETER-RFFICIERE REPRESENTATIONS
We next explored a previously unexamined application of TopoNets: model efficiency. Brain- inspired topography encourages compact representations. In biological systems, topographic or- ganization results in localized and redundant information by minimizing "wiring length" (weight sparseness) and enabling more compressible "metabolically energy-efficient" representations. In- spired by these biological principles, we asked whether TopoNets, which incorporate similar topo- graphic constraints as brains, might exhibit two forms of efficiency: a) weight sparseness, and b) parameter efficiency. It is important to clarify that these measures of efficiency model dimensionality: effective dimensionality measures the complexity of the feature represen- tation, while weight sparseness and parameter efficiency measure the overall resource of the model. One concerns the quality of the learned features, the other the quantity of the resource utilization.
distinct from
We first assessed weight sparseness in TopoNets by evaluating the effect of pruming small weights using [.I unstructured pruning. Specifically, set low-magnitude weights and measured
7

---

Published as a conference paper at ICLR 2025
A.
B.
TopoNets greater reailience unstructured pruning jaxhibiting sparser weights)
TopoNets (language models) are more parameter-efficint
Drop in InogeNet accurary (frm baselns)
100
Vision
Rosvett
Vision
Resnet
Intrease in model perplexity (from baseine))
44
Language
GPT-Neo- 125M
haowiie P
Eoselin
baselin
Topa Resnet 15
Topo GPT Neo- 125M
Topel Ronet
80%
80%
60%
0
Downsampling 80%
Fraction masked weights (L1 unstructured pruning)
125.5M
Topa Layers
Figure 4: Measuring the efficiency of TopoNets against baseline models: A. Fraction of weights masked through L1 unstructured pruning (x-axis) versus the change in model performance (y-axis) for ResNet-18 (left), ResNet-50 (center), and OPT-Nee-125MM (right) models. Colored cireles rep- resent TopoNets, while hollow black circles represent baseline models. The performance of the baseline models shown by the gray line. B. Percentage of model weights after downsampling (X-axis) versus the drop in model performance (y-axis) for GPT-Neo-12557 models.
the impact of this "lesioning" on model performance. We hypothesized that models with inherently sparser weights would be more resilient to LI pruning. The results, shown in Figure 4A, illustrate the relationship between the fraction of weights lesioned (x-axis) and the corresponding drop in model performance (y-axis). As expected, as the fraction of pruned weights increased, model per- formance declined. However, across both vision models (ResNetl 8s and ResNet50s, left and middle subplots) and language models (GPT-Neo-125MS right subplot), we found that TopoNets (colored dots) were more resistant to weight pruning than the baseline non-topographic models (black dots). This indicates that TopoNets produce sparser weight distributions and maintain performance more effectively when subjected to LI unstructured pruning.
However, LI pruning doesn't It directly address the question of parameter efficiency. To test this aspect more directly, we downsampled the weights, thereby directly reducing the model parameter count. Due to architectural limitations, this method works only on transformer models (downsampling con- volutional weights results in complete drop in performance to 0). These results are shown in Figure 4B. We found that TopoNets were remarkably resilient to downsampling. For instance, downsam- pling the weights by 80% lowered the overall parameter count of the model from 125.6M to 102.3M parameters (a 19% overall reduction), while maintaining performance especially at high levels of topography. This shows that TepoNet-GPT models are significantly more parameter-efficien than baseline models. Thus TopoNets offer significant advantages in both weight sparseness and param- eter efficiency. The downsampling across both GPT-Neo- 1 and NanoGPT (see appendix A.10 and figure 10) results particularly suggest that TopoNets might offer promising approach to scal- ing down GPT models without sacrificing task performance. This brain-inspired approach could unlock new methods for compressing large language models, providing a path for more efficient AI systems.
3.4 TOPONETS REPRODUCE BRAIN-LIKE TOPOGRAPHIC SIGNATURES
Here we evaluated the "brain-likeness" of TopoNet representations compared to other models. We evaluated vision models 2 key neural metrics. We first tested unit-to-voxel correlations (as previ- ously reported in Margalit et al. (024)) from the Natural Scenes Dataset (Allen et al. 2022). Model performance reached the noise ceiling for with comparable prediction accuracies TDANN models (R = 0.54 for TDANN vs. 0.60 for TopoNet-ResNNet-1 18 and 0.63 for
normal-
ized to the noise ceiling across 8 subjects). Next, we compared TDANNs and TopoNets on neural metrics from BrainScore (Schrimpf et al. 2020 2018). TopoNets outperformed TDANN at predict- ing responses across all visual regions (see Table for comparisons between TopoNet and TDANNs, and Appendix A.8 for all TopoNets). Taken together TopoNets predict neural responses a num- ber of measures better than TDANN. We further replicated key topographic signatures in the visual cortex, such category selectivity for faces. bodies, and scenes (Kanwisher 2000: Grill-Spector jet al., 2004), Epstein et al. 1999| Downing et al.] (006] 2001). and organizational biases for object size and animacy (Konkle& & Olival 2011 Konkle & Caramazza 2013). In Figure 5A, we show these
8

---

Published as a conference paper at ICLR 2025
A.
Vision ResMee-19
Bodies
Small Big
Animate Inanimate
Besefine
Language GPT 125M
powsrious clattees miydie-layers
Clustering by temporal Integration windows
early
earlylayeers inpomential disterr
noither sigooonnial powes
eoponsnöall
windiw floges ceNees lanedi
integratin widdrr
Figure 5: TopoNets recapitulate topographic signatures observed in the visual and language cortex. A. Topographic signatures in vision TopoNets (ResNet-18). Colormaps show t-values cor- responding to selectivities for faces, bodies, scenes, real-world size, and animacy. Bold connected lines indicate the same regions across different topographic maps. Maps for the same model layer Topographic
for the untrained (U) and baseline (B) models are shown below for comparison. signatures in language TopoNets. Colormaps display the strength of estimated power-law (red), exponential (green), and sentence-yoked coefficients across layers (from left right). These coeffi cients indicate fast, slow and sentence-yoked temporal integration windows
patterns for the TopoNet-ResNet-188 18--- model. We observed that face and body selectivities were yoked together, while scene selectivity was distinct. This pattern mimics the organization observed in the FFA, FBA, and PPA in the ventral visual cortex. We also confirmed this quantitatively. Face and scene selectivity showed a negative correlation (structural similarity = -0.41), whereas face and body selectivity were positively correlated (0.79). Additionally, TopoNets captured similar organi- zational biases for real-world size and animacy (structural similarity: 0.46), as seen in the brain. Relatively little is known about the spatial organization of the language cortex in the brain, but some
TopoNet NesNet18 TDANN
0.7116
0.3038
0.2923
0.5723
0.6932
0.1775
0.2792
0.4259
Table 1: BrainScore values for TopooNet-ResNet-18 and TDANN across visual regions V1, V2, V4, and IT. The scores averaged over benchmarks (detailed in Appendix A.8)
studies using fMRI (Lerner et al.) 2011 Hasson al. 2008) and invasive recordings (Regev al.
2024) have provided evidence for distinct temporal receptive fields. Based on these results, we won- dered neurons in TopoNets were clustered by their temporal integration windows. We used new word-swapping method from a recent study (Skrill & Norman- Haignere 2024) to investigate this in TopoNets. These temporal integration window results are shown in Figure SB. We replicated the expected pattern from the previous study: early layers were dominated by exponential integration dynamics, while mid-layers exhibited power-law dynamics. Interestingly, we identified three types of clusters. A) "Exponential" clusters with neurons dominated by short, exponential windows. B) "Power-law" clusters dominated by longer, power-law windows C) An intriguing cluster not ex- plained by either exponential or power-law integration windows. These findings are illustrated in Figure 5B. Topographic maps are presented for all models (including baseline model) Appendix Figure 8 for comparisons. To our knowledge, this is the first modeling result in topographic lan- guage models that recapitulates the experimental findings regarding clusters of temporal receptive field sizes in the language cortex in topographic-IILMS. Further work is needed to establish more

---

Published as a conference paper at ICLR 2025
precise correspondences between TopoNets and the human language system, making this an exciting direction for future research.
4 DISCUSSION, LIMITATIONS AND CONCLUSION
Here we introduced TopoLoss, a novel inductive bias that enables Al models to achieve high task performance while exhibiting topographic organization (like brains). We trained a number to- pographic, TopoNets, spanning both vision (ResNet-18, ResNet-50, and ViT-b32) and language (GPT-Nco-125M, NanoGPT) domains. TopoNets outperformed prior topographic models on en- gineering benchmarks while exhibiting comparable topography (Section 3.1), addressed theoretical claims about the importance of topographic principles for low-dimensional feature representations (Section 3.2), delivered parameter-efficient representations (Section 3.3), predicted neural responses better than previous topographic models, and reproduced topographic signatures observed in the brain (Section 3.4).
These results address three central questions about topography and its functional role. Q1. How can one incorporate topography in neural networks with minimal drop in task-performance? TopoLosss is a fundamentally novel approach to inducing topography (beyond TDANN and LLCNN) rooted in neuroscientific principles like synaptic pruning. It an inherently versatile framework that can be applied across model architectures (convolutional networks and transformers) and domains making a generalizable system for integrating topography into Al systems. Q2. What is the represen- tational consequence of brain-like topography? We demonstrate that topography (not task perfor- mance) drives representations to be lower dimensional and in turn more brain-like (Section 3.4) representations. This improvement manifests in two ways: (1) improved ability to predict neural data in monkey and human brains (as seen in BrainScore for instance), and (2) in recapitulating key signatures of brain-like processing in the visual and language cortices, such as gategory-sslectivvee maps in the visual cortex and temporal integration windows in the language cortex. Q3. Why is the brain's design topographic? TopoNets demonstrate parameter efficiency through lesioning (L1 pruning) and downsampling. This offers a novel perspective on the functional significance and evo- lutionary advantages of topography. This insight underscores the role of topography in optimizing computational systems, providing evidence from a surrogate computational system artificial neural networks. We also show that TopoLoss integrates seamlessly with fine-tuning methods like LoRA (Appendix 1). This highlights the complementary nature of TopoLoss and LoRA. Topol imposes topography during training while LoRA enables efficient task-specific fine-tuning. Future work should extend these initial findings into a more comprehensive study of task-specific adapta- tions, exploring their interaction across a wider range of tasks and model architectures.
TopoLoss is versatile framework compatible with foundational ANN components (linear and conv layers). But further work required to explore the full range of its scalability particularly important in the AI setting. The model backbones in this work (specifically ResNet- 18 and GPT-Nco-125MM were chosen to enable comparisons with prior research on topographic models. However, future work will need to scale these models more complex tasks (beyond ImageNet) and larger architectures (e.g.. LLaMA). That said we do not anticipate any challenges in scaling up TopoLoss more complex architectures. Our model can incorporated in only 2 to 3 lines of additional code (pip install topoloss). All preliminary tests show a mere 1- 2% performance overhead compared to baseline (non-topographic models). Another limitation is a incomplete understanding of how т, interacts with model performance and dataset complexity (though see A.4 and Figure 6). A trade-off between topography and model performance to be expected: if T is too high, topography may become overly rigid, limiting the model's ability to learn useful representations. This reflects a well-known principle in computational neuroscience: a critical balance between neural constraints (topographic organization in this case) and task performance. Our framework provides an opportunity to directly test these theoretical ideas in models in future work.
Limitations:
Taken together, TopoNets demonstrate that inducing topographic organization can offer competitive task performance while enhancing the efficiency and interpretability of Al models. This work opens further interdisciplinary work in Al and neuroscience, bringing current Al systems closer to the computational strategies of the brain.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Anna Ivanova and members of the Vision, Cognition, and Computational Lab for feed- back on early drafts of the paper. We also appreciate the constructive discussions provided by the four anonymous reviewers. This work was supported by the NIH Pathway to Independence Award (R00EY032603) from the National Eye Institute (NIH) and a startup grant from Georgia Tech (to NARM)
REFERENCES
Emily J Allen, Ghislain Yee, Yihan Wu, Jesse Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmii dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 1 16-126, 2022.
Michael Arcaro and Margaret Livingstone. A whole-brain topographic ontology. Annual Review of Neuroscience, 47, 2024.
Horace Basil Barlow. Why have multiple cortical areas? Vision research, 26(1):81-90 1986.
Taha Osama A Binhuraib, Greta Tuckute, and Nicholas Blauch. Topoformer: brain-like topographic organization in transformer language models through spatial querying and reweighting. In ICLR 2024 Workshop on Representational Alignment, 2024. URL https::/ /
Forum?id= forum?id==3piM2g02233
Sid Black, Leo Gao, Phil Wang, Connor Leaby, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorfloww, March 2021. URL https://doi.onn org/
10. 5081/281/// /ewddoooooo.ooooooooooooooooooo. 5297715
Nicholas M Blauch, Marlene Behrmann, and David C Plaut. A comnetivityycoostrained computa- tional account of topographic organization in primate high-level visual cortex. Proceedings the National Academy of Sciences, 119(3):e21 125661 19, 2022.
Tobias Bonhoeffer and Amiram Grinvald. Iso-orientation domains in cat visual cortex are arranged in pinwheel-like patterns. Nature, 353(6349):429-433 431, 1991.
Dmitri B Chklovskii and Alexei A Koulakov. Maps in the brain: what can we learn from them?
Annu. Rev. Neurosci. 27(1)336 2004.
Dmitri B Chklovskii, Thomas Schikorski, and Charles Stevens. Wiring optimization cortical circuits. Neuron, 34(3):341-347, 2002.
Marco Del Giudice. Effective dimensionality: tutorial. Multivariate behavioral research, 56(3): 527-542, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 109/CVPR.20099 5506048..
Fenil Doshi and Talia Konkle. Cortical topographic motifs emerge in a self-organized map of object space. Science Advances, 9(25))aade8187) 2023. doi: 12.1112//caa URL
https:// http://www.cc.. www.:science.... org doi aas//bss//100 1ttp:/100.111//oee.
ade8187
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv preprint arXiv:2010. 1929, 2020.
Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science, 293(5539) 2001.
Paul E Downing, AW-Y Chan, Marius Vincent Peelen, CM Dodds, and N Kanwisher. Domain specificity in visual cortex. Cerebral cortex, 15110)1443--141, 2006.
Richard Durbin and Graeme Mitchison. A dimension reduction framework for understanding corti- cal maps. Nature, 343(6259):644 644 647, 1990.

---

Published as a conference paper at ICLR 2025
Simon B Eickhoff, R Todd Constable, and BT Thomas Yeo. Topographic organization of the cerebral cortex and brain cartography. Neuroinage, 170:332-3477 2018.
Eric Elmoznino and Michael F Bonner. High- High-performing neural network models of visual cortex benefit from high latent dimensionality. PLOS Computational Biology,
2024.
Russell Epstein, Alison Harris, Damian Stanley, and Nancy Kanwisher. The parahippocampal place
area: recognition, navigation, or encoding? Neuron, 23(1):115-12) 1999.
Travis E Faust, Georgia Gunner, and Dorothy Schafer. Mechanisms governing activity- synaptic pruning in the developing mammalian ens. Nature Reviews Neuroscience, 22(11):657~- 673,20 2021.
Daniel H Geschwind and Pasko Rakic. Cortical evolution: judge the brain by its cover. Neuron, 80 (3):633-644) 647, 2013.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy, Transformer feed-forward layers key-value memories. arXiv preprint arXiv:2012.14913, 2020.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
Kalanit Grill-Spector, Nicholas Knouf, and Nancy Kanwisher. The fusiforrm face area subserves face perception, not generic within-category identification. Nature neuroscience, 7(5):555-562, 2004.
Uri Hasson, Eunice Yang, Ignacio Vallines, David J Heeger, and Nava Rubin. A hierarchy of tem- poral receptive windows in human cortex. Journal of neuroscience, 28(10):539-2550, 2008.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan C Horton and Daniel L Adams. The cortical column: structure without function. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 360(1456) 2005.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09688 2021.
Jon H Kaas. Topographic maps are fundamental sensory processing. Brain Re- doi: https:/doi..rp/ URL httpts:////// https::/www... sc wwwseeeennee www.eeieeeneeeeeec.eeeeeee www.ssciesneeccccoceecc
search Bulletin, 14- 1997. ISSN 0361-9230.
1016/50361
article/p11/0031122222 /ttp///003611222077000000000
Nancy Kanwisher. Domain specificity in face perception. Nature neuroscience, 3(8):759-763, 2000.
Nancy Kanwisher. Functional specificity in the human brain: a window into the functional archi- tecture of the mind. Proceedings of the national academy of sciences, 107(25):11133-11111 2010.
Nancy Kanwisher and Galit Yovel. The fusiform face area: cortical region specialized for the perception faces. Philosophical Transactions of the Royal Society B: Biological Sciences, 361 (1476):2109-2229 2006.
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. 2002.
Andrej Karpathy. NanoGPT.
https://githu..bccm con/ http:///hhhabboooo//aaaaattttaaa
2022.
Larry Katz and Carla J Shatz. Synaptic activity and the construction of cortical circuits. Science, 274(5290): 1333-11311333113 1996.
12

---

Published as a conference paper at ICLR 2025
T Anderson Keller, Qinghe Gao, and Max Welling. Modeling category selective cortical regions with topographic variational autoencoders. arXiv preprint arXiv:21 11..11...191 13911, , 2021.
Meenakshi Khosla, N Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32(19):415-4477, 2022.
Teuvo Kohonen. Exploration of very large databases by self-organizing maps. In Proceedings of international conference on neural networks (ienn '97), volume 1, pp. PL1-PL6. IEEE, 1997.
Talia Konkle and Alfonso Caramazza. Tripartite organization of the ventral stream by animacy and 2013.
object size. Journal of Neuroscience,
Talia Konkle and Aude Oliva. Canonical visual size for real- world objects. Journal of Experimental Psychology: human perception and performance, 37(1):23, 2011.
Leah Krubitzer. In search of a unifying theory of complex brain evolution. Annals of the New York 2009.
Academy of Sciences,
Benjamin Lahner, Kshitij Dwivedi, Polina lamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, Sou Young Jin, N Apurva Ratan Murty, et Modeling short visual events through the bold moments video fmri dataset and metadata. Nature communications, 15(1):6241, 2024.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. Ffev: Accelerating training by removing data bottlenecks. In Computer Vision and Pattern Recognition (CVPR), 2023,
com http:///ggthhuu...ooooc
Hyodong Lee, Eshed Margalit, Kamila M Jozwik, Michael A Cohen, Nancy Kanwisher, Daniel LK Yamins, and James DiCarlo. Topographic deep artificial neural networks reproduce the hall- marks of the primate inferior temporal cortex face processing network. BioRxiv, pp. 2020-07, 2020.
Yulia Lerner, Christopher Honey, Lauren J Silbert, and Uri Hasson. Topographic mapping of hierarchy of temporal receptive windows using a narrated story. Journal of neuroscience, 31) 1916-2 2011.
Anton Lozhkov,
Loubna Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024. http://ggaaaeeeeeeaeeeaaaee
URL http:::ss:ssggggggggttto wtt.aaccc..o.aaaeeeeeeeeeeeeeeee
Zejin Lu, Adrien Doerig, Victoria Bosch, Bas Krahmer, Daniel Kaiser, Radoslaw M Cichy, and Tim C Kietzmann. End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions. arXiv preprint (tt1:2308.09433) 2023.
ForchVision maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https:
//github.com htw.bb..coo/yytorc///iiesee
2016.
Pedro Maldonado, Imke Godecke, Charles Gray, and Tobias Bonhoeffer. Orientation selectivity in pinwheel centers in cat striate cortex. Science, 276(5318):5511-1555 1997.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron, 2024.
Nikolas McNeal, Mainak Deb, and Apurva Ratan Murty. Small-scale adversarial perturbations expose differences between predictive encoding models of human fmri responses. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024.
Nam Pham. tiny-codes (revision c13428e), 2023. URL. httpps://hugginggfaccc
datasets/namaee http:///aaaaaaaaaiaiiiia---e--ee ai/tiny-codes
Xinyu Qian, Amir Ozhan Dehghani, Asa Farahani, and Pouya Bashivan. Local lateral connectivity is sufficient for replicating cortex-like topographical organization deep neural networks. bioRxiv, pp. 2024-08, 2024.
13

---

Published as a conference paper at ICLR 2025
Pasko Rakic. Specification of cerebral cortical areas. Science, 241(4862):170-111 1988.
N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Com- putational models of category-selective brain regions enable high-throughput tests selectivity. Nature communications, 12(1):5540, 2021.
Tamar Regev, Colton Casto, Eghhal A Hosseini, Markus Adamek, Anthony I. Ritaccio, Jon T Willie, Peter Brunner, and Evelina Fedorenko. Neural populations in the language network differ in the size of their temporal receptive windows. Nature Human Behaviour, pp. 1-19, 2024.
Martin M Riccomagno and Alex Kolodkin. Sculpting neural circuits by axon and dendrite pruning. Annual review of cell and developmental biology, (1):):779 805, 2015.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj. Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy. Franziska Geiger, Kailyn Schmidt, Daniel L.. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint, 2018. URL htt.::tpss///t http://///wwwb.oo..voooo///ooo
10. 1111/4070072
Martin Schrimpf. Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James DiCarlo. Integrative benchmarking to advance neurally mechanistic models hu- intelligence. Neuron, 2020. URL https: h/www//wc.cc//e/oo//ooo www..cell.. wwt.coo//ee///ee /eoo///euro//ee wattrroonfullleee
50896-66 6273 (20) 30605-X
Reiner Schulz and James A Reggia. Mirror symmetric topographic maps can arise from activity- dependent synaptic changes. Neural Computation, 17(5):1059-1033, 2005.
David Skrill and Samuel Norman-Haignere. Large language models transition from integrating across position-yoked. exponential windows to structure-yokes power-law windows. Advances in neural information processing systems, 36, 2024.
Anthony Stigliani, Kevin Weiner, and Kalanit Grill-Spector. Temporal processing capacity in high-level visual cortex is domain specific. Journal of Neuroscience, 35(36): 12412-1222221 2015.
NV Swindale. The development of topography in the visual cortex: review of models. Network: Computation in neural systems, 7(2):161-22 1996.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377-392, 2020.
Wikimedia Foundation. Wikimedia downloads. URL http::tps://a/m http://///ppppppp/// woclssswikin wikimedia orq
Yiyuan Zhang, Ke Zhou, Pinglei Bao, and Jia Liu. Principles governing the topological organization of object selectivities in ventral temporal cortex. bioRxiv, pp. 2021-09, 2021.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
A APPENDIX
A.1
WHY CHOOSE 2D (AND NOT 3D) CORTICAL SHEETS?
The decision to use 2D cortical sheets to induce topography was rooted in what is known from neu- roscience. First, the selectivity of neurons in the cortex known to systematically vary in 2D space, reflecting the organization of feature preferences (e.g., orientation, spatial frequency) across the cor- tical surface (Kaas) 1997 Katz & Shatz 1996; [Chllovski et al., 2002 Chklovski & Koulakov] 2004). Within cortical column, pyramidal neurons along the depth (third) dimension share rela- tively similar selectivity, with differences primarily reflecting input/output relationships (Horton &
Adams 2005). As such, topographic organization therefore specifically refers to systematic changes
14

---

Published as a conference paper at ICLR 2025
in selectivity across the 2D sheet. Second, topographic maps are directly based human fMRI data, which unfolds the brain's ' 3D folded structure into a 2D cortical sheet. This approach has re- vealed the organization of the human visual system in great detail (Kanwisher) 20000 2010 Konkle & Oliva 2011 Konkle & Caramazza 2013) and captures the spatial organization of neural activity in biologically realistic and interpretable manner.
While alternative topographical structures could be of interest for more biological realism, 2D maps allow for meaningful comparison with prior methods like TDANN (Margalit al. [024]] Lee al.) 2020). LLCNN (Qian et al. 2024) and provide a neuroscience grounded foundation for studying topography.
A.2
HOW TO IMPLEMENT THE CORTICAL SHEET?
The cortical sheet for linear and convolutional layers is implemented by reshaping the weights of the layer into an tensor of shape (height, width, e) where e refers to the number of input units in linear layers and number of input channels multiplied by kemel size squared in convolutional layers respectively. The components used for implementing the sheet can be found in our source code on github:
Determining the height and width of the cortical sheet: link
Obtaining the cortical sheets for convolutional and linear layers: link
A.3 COMPARING TOPONETS, LLCNNS AND TDANNS
Each prior approach (LLCNN Qian et al. (2024), TDANN Margalit et al. (2024) starts from com- mon base model architecture (ResNet18) and applies an algorithmic modification to induce topogra- phy. While this modification could be seen as a change in architecture, the critical question remains: how do these changes impact task performance (e.g., categorization) compared the unmodified base architecture? Each prior study has highlighted the trade-off between inducing topography and categorization performance, with large reductions in task performance being a recurring concern (despite training for significantly higher number of epochs).
LLCNN and Toponets are more comparable (trained supervised manner on ImageNet). But there key differences between TDANNs and TopoNets, which make direct comparisons particularly challenging. To ensure transparency for readers, we clearly outline these differences and explain the strategies adopted to facilitate meaningful comparison between the two models.
First, TDANNs were pre-trained on SimCLR and Toponets on Imagenet categorization. The model's training regimen will of course change the model's overall performance. Second, TDANN models induce topography on the outputs of the sub-blocks (8 places within the model). In TopoNets we induced topography in every convolutional layer within the ResNet sub-blocks (19 in total). How then do we compare models? We do so in 2 ways.
Version 1. We adopt the integrative benchmarking (from BrainScore, Schrimpf et al., 2020) strategy. That is, we compare models on engineering measures (Imagenet) and neural measures irrespective on what they were trained on. Note that even though the comparison between TDANN and TopoNets does not seem fair by this measure (though note that LLCNN and TopoNets can be compared), this is now considered somewhat accepted measure vision neuroscience. We present the raw performance of models on Imagenet in Section 3.1 and neural predictivity measures (eg. BrainScore (Schrimpf al., 2018: 2020g) broken down by visual cortical regions in Section 3.4.
We report the difference in model-performannce from baseline (non-topographic ver- sions). For TDANN, the reported difference in categorization performance between the baseline model (trained on SimCLR) and IDANN-SimCLP was 5%. The difference between our baseline model (Imagenet trained) and pographic). We trained an additional ResNet- 18 model with topography induced similar layers as TDANN (N=8) for an even more fair comparison. The difference in model performance is 3%. These results are also reported in Section 3.1.
Version 2.
is 7% (for model with all convolutional layers to-
15

---

Published as a conference paper at ICLR 2025
T.0.5
1=10
DLMP Accurney
7:58
Figure 6: Effect of on model performance and dataset size. x-axis indicates the number of tokens from FeneWeb-Edu used to train a given TopoNet-NanoGPP model and the y-axis indicates the accuracy of the model on BLiMP evaluation. Colors indicate the strength of spatial constraint
A.4 EFFECT OF AND TASK DIFFICULTY ON MODEL PERFORMANCE
controls the degree of topography (spatial constraint) within the model. Theoretically, topog- raphy increases, some drop in model performance is to be expected. Intuitively, high value (say resulting in a single cluster) would limit the capacity to learn tasks effectively. The brain appears to balance this trade-off by achieving "sweet spot," optimizing both efficiency and performance. But order to get further insights we trained several TopoNet-NanoGPT models with increasing numbers of tokens and varying levels of 7 to explore how task performance changes as both data scale and topegraphy constraints vary. These results are shown Figure 6. In general we observe small performance drops from the baseline model with increasing т. What is also interesting is that the overall performance drop on the BLiMP dataset does not seem to increase with increasing the number of tokens. Evaluating the effect of topography with increasing model complexity and dataset size is an important area for future research into topographic models. We hope to explore the idea of "Scaling Laws for Topographic Networks" in greater depth in subsequent work.
A.5 DOWNSAMPLING
To downsample the weights of linear layer, we first reshape the weight matrix the cortical sheet (see method for details). The weights are then downsampled along the height and width dimensions by factors da and e... respectively. The downsampled sheet then reshaped back obtain weight Paagi
matrix with reduced dimensions: (Ø) "o,. Rinput) where Riipptt and Doutpat are the number of input and output neurons respectively.
o Similarly for convolutional layers, we obtain downsampled weights of shape ( pa x i, k, k) where i, o and & are the number of input channels, output channels and the kernel size respectively.
The forward pass through this downsampled linear layer proceeds follows:
Perform the matrix multiplication between the input tensor (shape: (batch, Minpu)) and the downsampled weight matrix. The result has shape (batch, &n. 6x
Thanpe
Reshape the result to (batch, o,," Cm)), where en and ca. are the height and width of the original cortical sheet, such that ch X Cw noutput-
Upsample the reshaped output by factors da and éw. producing tensor of shape outpue (batch, h, w). This upsampled tensor is reshaped back to (batch, neuuput).
Finally, the bias is added to obtain the final output of the downsampled layer.
The forward pass through this downsampled Convolutional layer proceeds as follows:
Perform the convolution operation between the input tensor of shape: (batch, height, width) and the downsampled weight matrix. The result has shape (batch, कe xo. height, width)
16

---

Published as a conference paper at ICLR 2025
Dilenc
OPTT 335m
OffWeen 125m
Figure 7: Pairwise Correlation vs. Distance for various models
Reshape the result (batch, Sh Sue height, width), where en and cw are the height and dx du width of the original cortical sheet, such that C₁, x Cv = o.
Upsample the reshaped output by factors ón and Φw along the 2nd and 3rd dimensions respectively, producing tensor of shape (batch, Ch, Cws height, width). The ca and ca dimensions are then merged to obtain the output of the convolution operation o output
channels.
Finally, the bias is added to obtain the final output of the downsampled layer.
To reduce the number of parameters in the weights by a factor of n, we set Φh = vn and @. = vn.
A.6 PAIRWISE DISTANCE vs. CORRELATION
To perform this analysis on the GPT-Neo- 125M models, we selected the first 100,000 samples from the BookCorpus dataset (Zhu et al. 2015) and extracted the activations from the topographic layers. We then computed the Pearson correlation between the activations every pair of units within each layer. Finally, for the x-axis, caleulated the Euclidean distance between each pair of units in the cortical sheet space.
For ResNet- 18 and ResNet-50, we fed the ImageNet validation set images through the model and collected outputs from the topographic layers. For each of these layers, we computed the Pearson correlation between the outputs of single channel and all other channels within the same layer. Next, we projected each layer onto the cortical sheet and calculated the Euclidean distance between the corresponding weights of each output channel. Then we plotted the Pearson correlation (y-axis) against the Euclidean distance (x-axis) for the output channels in each layer.
A.7 TEMPORAL INTEGRATION WINDOWS
We closely followed the source code provided by (Skrill & Norman- Haignerr 2024). The only change that made was that instead of evaluating the outputs of the c_proj layer, we did the same analysis on the topographic layers ie the c fc layers. Figure visualizes the Power Law and the Exponential parameter estimates for our GPT-Neo- 125M models.
17

---

Published as a conference paper at ICLR 2025
A.8 BRAINSCORE BENCHMARKS
We compared TopoNet-Res R models with TDANN in predicting individual neuron responses in the primate visual system using the BrainScore platform. Notably, the 100 model exhibited smooth- ness (topography) values most comparable to the original TDANN. Across all regions, TopoNets outperformed TDANN in predicting neural responses.
Brain Region
VI
Baseline
0.6913
0.3038
0.2346
0.5953
Top
0.6906
0.1989
0.2299
0.6334
TopoNet-71
0.6555
0.2664
0.2369
0.5325
0.7262
0.1826
0.2784
0.5722
TopoNet-710 10
0.7116
0.3038
0.2923
0.5723
TopoNet-720
0.6989
0.2614
0.3523
0.4746
TopoNet-r50
0.6666
0.266
0.3553
0.5369
TDANN 0.6932
0.1775
0.2792
0.4259
Table 2: Comparison of BrainScore performance for Baseline, TopoNet (with different values), and TDANN models across brain regions V1, V2, V4, and IT. The scores equate to the mean of multiple benchmarks.
We utilized all BrainScore benchmarks that are publicly accessible through the BrainScore GitHub and AWS. The complete list of benchmarks used is provided below for reference. Note that these include both macaque neural responses and fMRI data.
V1:
Tong. Mong..Cggan22224FMH V1-rdm
TreemmanZiiemba2033ppbbiiii V1-pls
3. Marques2020 Aurgges00000CCaaaaaagg22200--- summation field
4. Matges:00...caaaugs2202asmmmmmm diameete
ott.rugiii0002aaammmmmssusssssoo suppression index
7.
8. Marques2020 Ringach20022orr bandwidth bandidh
9.
10. Marques2020 Ringach2022-errulaar variance
11. http://22222.iinagah222222222 p ratio
12.
ratio
13. Margnes20000KKinaaa22220---0.p0
14. Marques2020_Riing www.:/02000..ngeeeeoooooonnnnoon
15. hatpuu222220.inngaa22222mmma...
16. Marques2020 Schiller1976-sf bandwidth
17. htaees202000iilllel99777sfffsss
V2:
1. Mong.Coggaa2024.MMRIII V2-mm
2. FremmanZeemb221113pbiic V2-pls
V4:
1. MajajHongg015publlicc S0--puublc....plls
2. Tong.Coggan2024 1MRI. V4-rdm
.T:
1. MajajHonngU.5Spubie..Te
2. Tong. Cogg.goggan222nnnn 1.M...
18

---

Published as a conference paper at ICLR 2025
Parameter Estimate (Power-Law, smoother falloff)
Parameter Estimate (Exponential, sharp falloff)
Seadir
kpp.1
lopu
Iapu
tssu.
18s2.5
kgp.11
lspo..5
0.3
Exwr
07
Lawe2
Lapere
Exper7
Lape
Figure 8: Temporal integration window estimates for all layers all of the GPT-Neo-1225 models.
19

---

Published as a conference paper at ICLR 2025
24
effictic dmorcios
70
100g
12
4.5
Figure 9: Effective dimensionality of representations v/s model performance for GPT-Neo-1253
models trained on Wikipedia.
= 0.0
0.75
0.5
T 1.0
0.70
Blimp Accuracy
0.65
Validation
0.5
0.60
50.0
6-
0.50
4
0.45
0%
20% (118.1M)
50% (109.7M)
B0% (101.5M)
0%
20%
50%
80%
(124.4M)
(118.1M)
(109.7M)
Sparsity
Sparsity
Figure 10: Training NanoGPT with higher significantly improves performance under sparse set- tings (Downsampling: see Section 3.3) compared to the baseline and models trained with lower values.
A.9 EFFECTIVE DIMENSIONALITY VS. PERPLEXITY ON OPENWEBTEXT
When we evaluated the GPT-Neo- 125M models on the OpenWebTert dataset with varying levels of observed that the effective dimensionality of the layer representations initially increased from the baseline to т = 1, but then again followed a more predictable pattern from T = 5 onward (decreasing gradually with increasing perplexity for higher 7 values). Figure 9.
A.10 DOWNSAMPLING TOPOGRAPHIC LAYERS ON NANOGPT
We repeated the Downsampling experiment from Section 3.3 using topographic NanoGPT models, evaluating them on held-out validation set and the BliMP dataset. The model trained with 7 = 50 significantly outperformed (see figure 10) the models trained with smaller 5 values and the baseline model. See Figure 10.
A.11 THE EFFECT OF ToPoLoss LORA-BASEE FINE-TUNING
The purpose of TopoLoss was to shape spatially organized, persistent feature representations within neural networks. In Transformers, this goal was most directly achieved by targeting the mip.c.fe layer in each transformer block. Prior studies indicate that is the mip.c.fc module that encodes "world knowledge" (analogous to what we imagine as the language system in human brains). This makes it the natural and theoretically grounded target for inducing topographic organization. How- ever, this choice raises potential concern that TopoLoss on the mlp.c fc module may not align with large-scale transformer models because they employ Low-Ran Adapters (LoRAs, Hu et al. (202))
20

---

Published as a conference paper at ICLR 2025
LoRA Rank:
LoRA Rank: 4
LoRA Rank:
4.3
baseline
T 1.0
50.0
Validation Loss
4.0
0,400
Á 0.400
A 0.397
0,422
A = 0.419
0,422
3.7
0.362
0.359
4 0.359
30
60
30
60
30
60
3.4
Validation Loss
3.0
2.7
0.719
A 0.725
0.727
0.643
A 0.651
0.653
0.609
A 0.608
0.613
0
10
20
30
10
20
30
0
10
20
30
Epoch
Epoch
Epoch
Figure 11: Effect of LoRA fine-tuning on the fully trained NanoGPT checkpoints (both baseline and topographic). The top row corresponds to the Shakespeare dataset and the bottom row corresponds the Python code dataset.
exclusively on the attention matrix. Here we demonstrate that TopoNets are, in fact, fully aligned and compatible with LoRA fine-tuning.
To validate this compatibility, we fine-tuned our TopoNet-NanoGPT models on two datasets 1. Shakespeare's 's text (300k tokens) and 1M tokens of Python code (sampled randomly from the tiny-codes dataset Nam Pham (2023)). The experiments were conducted on multiple LoRA ranks (2, 4, and
ensure generalizability. We applied LoRA fine-tuning on our TopoNet-NanoGPT models (r = 1, 50) and compared the LoRA fine-tuning performance with the baseline NanoGPT model (without topography). These validation loss curves exhibit characteristic and consistent drop after LoR A-based fine-tuning. Notably, we observe a consistently larger improvement in model performance (A) for TopoNet-NanoGPP models compared the baseline model. These findings, summarized in Figure 12, highlight that TopoLoss integrates seamlessly with LoRA fine-tuning, further enhancing task-specific adaptations while preserving topographic organization.
Together, the results from this additional experiment show that large-scale transformer models trained with TopoLoss can still be fine-tuned via LoRA on the attention matrix. While LoRA is designed to enable efficient task adaptation through fine-tuning, TopoLoss focuses imposing interpretability and localized representations in the model's persistent representations during pre- training. These objectives complementary rather than conflicting. This combination allows us to leverage the strengths of both methods. LoRA enables efficient and flexible task-specific adapta- tions, while TopoLoss enhances interpretability, parameter efficiency, and localized representations during pre-training. This division of focus highlights the compatibility and synergy between the two approaches, enabling robust and adaptable models for diverse tasks.
21


Reviews:
[{"review_id": "R2icjHklVv", "reviewer": "Reviewer_bq7J", "summary": "This paper introduces a novel topoloss aimed at aligning AI models with the structure of brain neurons to create brain-like topography within these models. The topoloss defines a cortical sheet within AI models by reshaping the weight matrix and maximizes the cosine similarity between this cortical sheet and its blurred version, simulating synaptic pruning in the brain. The authors apply the topoloss to both CNN for vision and transformer for language models. Evaluation results indicate that the resulting toponets achieve a balance between maintaining topographical structure and overall model performance.", "questions": "Consider incorporating ViT and LLaMA as backbones for the framework to enhance relevance and performance. \n\nCompare the application of topological loss on the transformer's attention matrix instead of the fc matrix.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of mapping model weights to a topographical cortical sheet is interesting, with a simple and elegant approach. The reshaping technique appears versatile and could potentially be applied to the weights of various models. The discussion on topographic signatures within toponets highlights promising avenues for brain-inspired AI research.\n\nThe paper is well-written, and the idea is easy to follow. The authors have provided sufficient detail and references to support reproducibility.", "weaknesses": "The selected backbone is somewhat outdated and lacks current relevance, reducing the overall significance of the work. For vision and language models, more meaningful choices would include ViT and LLMs like Llama.\n\nNo performance improvement compared to the original no-topo model, nor is there any discussion on time and data efficiency or interpretability. This makes the topography alignment appear theoretical, aligning with brain structures only on a formulaic level without contributing to human-level intelligence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OI8BlBvK32", "reviewer": "Reviewer_gnkR", "summary": "The paper introduces a novel loss function called TopoLoss, which promotes spatially organized topographic representations in AI models without compromising task performance. The authors present TopoNets, a suite of models that incorporate this loss function into existing architectures, such as ResNet-18, ResNet-50 for vision tasks, and GPT-Neo-125M for language tasks. The key contributions include demonstrating that TopoNets outperform previous topographic models while maintaining high performance and replicating the topographic signatures observed in the human brain's visual and language cortices, thus bridging biological and artificial systems.", "questions": "How does the choice of scaling factor τ affect performance across varied datasets beyond ImageNet?\n\nCan you elaborate on the potential limitations of TopoLoss when applied to larger or more complex architectures?\n\nWhat specific challenges do you encounter while integrating TopoLoss into existing architectures, particularly transformers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper lie in its innovative approach to integrating topographic organization into AI models through the introduction of TopoLoss. This loss function effectively promotes spatially organized representations, which are crucial for achieving localized feature processing and lower dimensionality, like biological neural networks. The experimental validation across diverse architectures, including ResNet-18, ResNet-50, and GPT-Neo-125M, showcases the adaptability of TopoLoss and its ability to enhance model performance without sacrificing accuracy.", "weaknesses": "The weaknesses include a lack of extensive benchmarking against state-of-the-art models beyond those already tested, which may limit the generalizability of the findings. While TopoLoss is shown to improve performance, the paper does not sufficiently address how it scales with larger models or more complex tasks, raising questions about its applicability in high-dimensional settings. Additionally, the discussion surrounding the choice of scaling factor τ could be more detailed, particularly regarding its impact on various architectures and datasets. In addition, the inspiration from the brain's vision and language processing is superficial. The authors could strengthen their claims by providing additional insights into potential trade-offs between topographic organization and task-specific performance metrics!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLlVTefg5P", "reviewer": "Reviewer_3kin", "summary": "The paper addresses the development of a novel loss function designed to create layer-wise 2D topographical organization in NLP and vision models, demonstrating improvements in performance over previous topographic models while maintaining spatial smoothness. This advancement has potential applications in achieving more biologically plausible neural networks with enhanced weight sparsity.", "questions": "1.\tAre there alternative topographical structures to 2D maps that could be more theoretically justified, and could the cortical sheet be better motivated?\n2.\tIs there a specific rationale behind comparing this model against LLCNN-G, given the architectural differences?\n3.\tWhy is perplexity lower for tau = 5.0 compared to tau = 1.0, as shown in Figure 4B (Left)? Could adding a margin of error clarify this?\n4.\tBeyond effective dimensionality and smoothness, what additional metrics could quantify topography?\n5.\tWould adding a Brain Score comparison with TDANN provide valuable insight into brain-like topographic signatures?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed model improves both performance and topographic organization compared to previous models.\n\nThe core claim is well-supported, particularly in vision models, with evidence showing better performance and smoothness.\n\nThis approach opens up avenues for more biologically grounded AI models and potential advancements in weight sparsity.", "weaknesses": "The motivation for choosing a 2D topographic map as the structure is somewhat unclear. Other possible topographical structures might need consideration.\n\nComparisons lack consistency across architectures, raising questions about fairness (e.g., LLCNN-G has a different architecture).\n\nClaims are primarily substantiated in vision models without equivalent evidence in NLP or other areas. Are there any models to compare in NLP?\n\nIn figure 5, the differences in performance between parameter-efficient TopoNets and baseline models only become significant when there are substantial drops in performance. \n \nSpecific areas for improvement include: \n(1) finding tau values such that TopoNets have an equal smoothness or effective dimensionality to previous models (i.e. showing that with a similar strength of topographic effect, this method works better)\n(2) including margin of error in Figure 4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KkBuNLxnBX", "reviewer": "Reviewer_FUA2", "summary": "This work explores cortical topography from a representational standpoint, aiming to integrate such in both vision and language model classes to observe a higher performance than previous models that have attempted to do so, while reproducing certain aspects of topographic signatures within the learned feature representations and model parameters. Specifically, TopoNets show category-selectivity for faces, scenes, and bodies. Representations match neural responses in higher visual cortex under one-to-one mapping. Language TopoNets contained clusters of units with distinct temporal integration windows.", "questions": "- Line 85: “… show diminished capacity to predict brain data.” (a) \"brain data\" is a highly vague term - are you implying neural predictivity (and if so, V1? VTC?), image-by-image human behavior prediction, or something else? (b) Margalit et al. (2024) show that TDANNs have a higher NSD voxel correlation to the VTC than purely categorization-driven models under one-to-one matching. Can the authors please clarify?\n- Line 161 - “We trained 8 distinct ResNet-18 models …”. The authors mention that one is the baseline, and 6 others are TopoNets. What is the 8th model?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Modeling cortical topography is an important line of research for both neuroscience and AI, and I appreciate the authors’ work on the matter.\n- The authors synthesize the application of a single loss term—namely, TopoLoss—to both residual vision networks and GPT. The motivation behind the loss seems well founded.\n- Models trained using TopoLoss incur minimal drop in task performance, which is an important result, since previous models have struggled with this.\n- The emergence of specific topographic signatures in sparse networks is again an important finding, given that the brain is optimized to work in an energy-efficient way.", "weaknesses": "- Comparisons have been made in the manuscript (such as in result 3.1) where TopoNets are compared to the TDANN. TDANN was trained in a self-supervised way (precisely, using SimCLR), but I did not find (and I apologize if I may have missed it) any mention of the training/finetuning objective (categorization versus self-supervision) that the authors used. If TopoNets were trained through self-supervision, what objective was used? If supervised, is it justified to compare TopoNets with the TDANN? What contribution, if any, do the authors speculate the choice of the training objective plays alongside TopoLoss?\n- Figure 6 - I appreciate the analysis around topographic signatures. However, I would prefer seeing the same plots replicated for the baseline vision and language models—(a) without being trained jointly on TopoLoss, (b) unoptimized (control) on both the task and topographic losses. This would really emphasize that what is being shown is not artifactual and attributable to the use of TopoLoss. I hope this is easy enough to check since it does not require any additional training.\n\n**Things (manuscript writing) that can be improved but did not impact my score**:\n- In-text citations that are parenthetical and not narrative should be enclosed in parentheses. For example, lines 30-31 should be written as (Barlow, 1986; Rakic, 1988, …). Use the \\citep{} LaTeX feature to do so.\n- Line 182 - there should be at least a single sentence explanation of what FFCV is.\n- “Resnet\" -> “ResNet\"\n- Lines 213+ - the authors explain \"L1 unstructured pruning” and “downsampling” as policies but not how they are being used as metrics. This should be made clear at this point in the manuscript.\n- The first paragraph of Result 3.1 talks extensively about \"model performance” without being explicit about what the task is (I am presuming object categorization) and what dataset it is being evaluated on (i.e., ImageNet). It is only in the figure on the next page that this is made apparent. I would encourage the authors to be more direct in their writing.\n- Double quotes, such as on line 398, should be implemented using the ``...’’ LaTeX feature to show both opening and closing quotes.\n- There should be a paragraph in the introduction talking about what topographic signatures look like for both vision and language—such as the emergence of pinwheel-like smooth orientation preference maps in V1 (Blasdel and Salama, 1986; Nauhaus et al., 2012), category selective maps in IT/VTC (Desimone et al., 1984), etc. This has been done later in the manuscript, but needs to be brought up earlier.\n- Line 450 - \"B. opographic\" (typo)\n- Line 483 - the authors claim that they have created “a broad suite of topographic AI models”. The use of the term “broad suite” is, in my opinion, an overstatement, given that the only model classes that were evaluated were residual networks and GPT. I would rephrase this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)