Credit-based self organizing maps: training deep topographic networks with minimal performance degradation
Authors: Amirozhan Dehghani
Xinyu Qian
Asa Farahani
Pouya Bashivan
Venue: ICLR 2025 Poster (2025)
Keywords: Computer vision
Neuroscience
Convolutional Networks
topographical organization
self-organizing maps
functional organization
Primary area: applications to neuroscience & cognitive science
TL;DR: We developed a new topographical neural network model that replicates the functional organization of the visual ventral stream while retaining high object recognition performance

Abstract: In the primate neocortex, neurons with similar function are often found to be spatially close. Kohonen's self-organizing map (SOM) has been one of the most influential approaches for simulating brain-like topographical organization in artificial neural network models. However, integrating these maps into deep neural networks with multitude of layers has been challenging, with self-organized deep neural networks suffering from substantially diminished capacity to perform visual recognition. We identified a key factor leading to the performance degradation in self-organized topographical neural network models: the discord between predominantly bottom-up learning updates in the self-organizing maps, and those derived from top-down, credit-based learning approaches. To address this, we propose an alternative self organization algorithm, tailored to align with the top-down learning processes in deep neural networks. This model not only emulates critical aspects of cortical topography but also significantly narrows the performance gap between non-topographical and topographical models. This advancement underscores the substantial importance of top-down assigned credits in shaping topographical organization. Our findings are a step in reconciling topographical modeling with the functional efficacy of neural network models, paving the way for more brain-like neural architectures.

Published as a conference paper at ICLR 2025
CREDIT-BASED SELF ORGANIZING MAPS: TRAINING DEEP TOPOGRAPHIC NETWORKS WITH MINIMAL PER-
FORMANCE DEGRADATION
Amir Ozhan Dehghani 3.3
Xinyu Qian²
Asa Farahani
Pouya Bashivan
Department of Physiology, McGill University
Department of Computer Science, McGill University
3 Mila. Université de Montréal
Montreal Neurological Institute, McGill University
ABSTRACT
In the primate neocortex, neurons with similar function are often found to be spa- tially close. Kohonen's 's self-organizing map (SOM) has been one of the most influential approaches for simulating brain-like topographical organization in ar- tificial neural network models. However, integrating these maps into deep neu- ral networks with multitude of layers has been challenging, with self-organized deep neural networks suffering from substantially diminished capacity to per- form visual recognition. We identified a key factor leading to the performance degradation in self-organized topographical neural network models: the discord between predominantly bottom-up learning updates in the oeffoorgaiizin maps, and those derived from top-down, credit-based learning approaches. To address this, propose alternative self organization algorithm, tailored align with the top-down learning processes in deep neural networks. This model not only emulates critical aspects of cortical topography but also significantly narrows the performance gap between non-topographical and topographical models. This ad- vancement underscores the substantial importance of top-down assigned credits in shaping topographical organization. Our findings are a step in reconciling topographical modeling with the functional efficacy of neural network models, paving the way for more brain-like neural architectures. The code is available at http:///aaaaa.aaoo/aaaaaaaaaaaaa
INTRODUCTION
The neocortex is a highly structured biological system, with many of its functional capabilities tied to distinct anatomical regions. Processes such visual, auditory, and somatosensory perception, as well as motor planning, are localized to specific cortical areas (Hubel & Wiesel, 1962] Humphries al.) 22100) WWog al. 1978 Foerster 1936). This structured organization extends to more gran- ular scales within each cortical area, where neurons are grouped based on specific preferences. For example, in the visual cortex, neurons organized according to the visual features they process (Hubel & Wiesel 1962). while in the motor cortex, they are arranged according to the body parts they control (Foerster 1936).
In the past decade, the reemergence of deep neural networks as tools for modeling neural compu- tations has sparked a new wave of popularity computational neuroscience (Yamins et al. 2014 Khaligh-Razavi & Kriegeskorte| 2014). Deep neural networks such as convolutional networks were shown to capture functional aspects of the neural code across the hierarchy of visual cortex in hu- man (Khaligh- Razavi & Kriegeskorte) 2014 Ratan Murty al. (022 and non-human primates Yamins et al. 2014: Bashivan et al. (019)) as well as rodents (Nayebi et al., 2021). However, despite this remarkable representational similarity, it is widely observed that the arrangement of units (fil- ters in convolutional networks) in these models is arbitrary and therefore they lack any topographical
organization, unlike those in many animal brains (Lee et al. 2020 Bashivan al. 2022).

---

Published as a conference paper at ICLR 2025
It is widely believed that the multi-scale cortical organization is the result of mapping high dimen- sional neural representations onto the two-dimensional cortex through self coff-organizatinn of neurons (Kohonen 1982. Doshi & Konkle 2023). Topographical organization is primarily thought to mini- mize neural wiring across cortical circuits, optimizing physical size and boosting capacity (Jacobs & Jordan T992), though recent work suggests it may also aid in learning robust representations (Qian et al. (004)). Beyond their functional roles, topographical models are valuable for therapeutic appli- cations, such as guiding epilepsy surgeries by predicting cortical organization to help avoid resecting critical patches, like those involved in face recognition [Schrimpf' et al. (2024). In AI, inducing lo- cal structure in networks could enhance interpretability and aid in mitigating harmful behaviors by targeting specific clusters linked to undesired outputs.
During the past several years, a number of studies have proposed different approaches towards to- pographical deep neural networks that not only mimic the brain's 's activity patterns but also its to- pographical organization (Blauch et al., 2022, Margalit et al.] 2024; Finzi et al.] 2023 Doshi &
Konkle 2023). These approaches make use of various topography- -inrupiiy-inuiingggggg learning objectives
(Blauch et al. 2022, Margalit et al. 2024) and learning rules (Doshi & Konkle 2023) to impose
structure across the DNN units' weight parameters in order to encourage topographically organized unit selectivity in each layer. In turn, while these models qualitatively capture facets of cortical topography, they invariably suffer from substantial blows their lask performances (i.e. object recognition). It is therefore natural to question whether topography can emerge without a significant decline in performance.
We hypothesized that the source of such blows to task performance in topographical neural network models is the conflict between top-down gradient based learning with local topography -ondruphy-inducinn induuiigg objective functions or learning rules used in prior approaches (Kohonen 1982; Von der Malsburg, 1973). Hence, we reasoned that aligning the topography according top-down measures of credit could potentially reduce such conftict and lead to topographical neural network models with min- imal reduction in task performance. To this end, we propose new variation of the well known Self Organizing Map (SOM) (Kohonen 1982), where the competitive learning rule is guided by the credit assigned to each unit based on its contribution to minimizing an polloiccally-reeevae loss (Figure [T].. We adapted this method for use in deep neural networks (particularly convolu- tional networks) and parameter training with gradient descent and analyzed the resulting network's representations and topographical organization.
task
Our main contributions are as follows:
We introduce new algorithm for learning topographic deep neural networks called Credit- Based Self Organizing Maps (CB-SOM), Using this algorithm. we train deep convolu- tional network in which filters in all layers are topographically organized.
Despite remaining performance gap between topographical and non-topographical mod- els, CB-SOM significantly improves object recognition compared to previous topograph- ical models like Kohonen's SOM, while achieving strong topographical alignment with macaque and human ventral visual cortices across primary and high-level regions.
We demonstrate enhanced alignment between CB-SOM representations and the neural ac- tivity in the macaque and human brains.
2 RELATED W WORK
Topographical models based on Hebbian learning. Early attempts to replicate cortex-like to- pographical organization employed shallow neural networks, typically consisting of just a single hidden layer, with symbolic inputs such as bars of varying orientations (Kohonen) 1982 Von der Malsburg 1973 Linsker [986] [Swidaae [996).. These models were inspired by the lateral con- nections between neurons on the cortical surface and their parameters were trained unsupervised using different variations of the Hebbian learning rule. More recent work extended self-organizing approaches to additionally consider task-relevant information such as object labels during learning (Hartono et al. 2015 Hartono 2020). While many of these models could replicate the organization of the neurons in the primary visual cortex with high fidelity, their utility was largely confined to early sensory cortices. This limitation was likely due to the inefficiency of local learning rules in constructing the representational hierarchies found in modern deep networks trained with backprop- agation. Despite these challenges, recent work (Doshi & Konkle 2023) has shown that applying some of these local learning rules, such Kohonen's self-organizing maps, to pre-learned fea-
2

---

Published as a conference paper at ICLR 2025
Loss(y: 9)
b
o
1/2
BMU
Dee Neur Retiver.
iteration
Figure 1: A schematic of AB-SOM and CB-SOM models and their respective winner unit/filter selection algorithm. Filters are arranged on 2D map where each iteration, one filter chosen as the Best-Matching Unit (BMU). a) In AB-SOM, the filter that produces the highest response to given input is chosen as the BMU (averaged across batch and spatial dimensions), whereas in the CB-SOM, the filter with the highest norm of gradient (i.e. ac du is chosen as BMU; b) (top) All parameters are updated according to a Gaussian neighborhood function. The neighborhood function weighs the parameter update in each filter function of the distance between the respective filter's distance to the BMU (star) on the 2D map. The Gaussian neighborhood function exponentially decays the effective weight updates for filters that are farther from the BMU. (bottom) The slope of the neighborhood function and the SOM learning rate are both follow an exponentially decaying schedule during training.
ture representations can yield topographical organizations that closely resemble those in the human brain. However, this approach necessitates a division of the learning process into two distinct and sequential phases first learning -on-topographical representations, followed by a topographical restructuring phase.
Deep topographical neural networks. In recent years, several approaches have been developed for training deep neural networks with topographical organization across multiple layers. Most of these methods rely auxiliary learning objectives designed to promote structured similarity between units/filters based on their physical distance within a predetermined layout (Lee et al., 2020; Blauch
al. 2022 Lu al. 2023; Margalit al. 2024). (Blauch et al., 2022) draws inspiration from the
wiring cost optimization theory of cortical topography (Jacobs & Jordan 1992). formulating a loss function that captures the cumulative connection strength between units across different layers as function of their distance. By minimizing this loss during training, they reduce the strength of connections between distant units, resulting clusters of model units that are selective for different object categories. Lee et al.) (2020) and Margalit et al. (2024) introduce a loss function that encour- ages the pairwise correlation of activity between units follow an exponentially decaying pattern, mirroring what is observed in the high-level visual cortex of primates. In contrast, Lu et al. (2023) adopts a comparable approach but enforces similarity in the weights of the units rather than their ac- tivations. Another recent study showed that incorporating lateral connections between filters in volutional layers can replicate various aspects of cortical topography within these networks (Qian) al. 2024). Despite these advances, these models typically exhibit large drops in performance compared to non-topographical models (15-37% relative drop in object recognition accuracy).
METHODS
3.1 ACTIVATION-BASED SELF-ORGANIZING MAPS (AB-SOM)
AB-SE SOM is an adaptation of Kohonen's 's self self-organizing maps for deep convolutional networks. involves three steps:
3

---

Published as a conference paper at ICLR 2025
1. Cortical Sheet Arrangement: Filters within each layer are positioned on 2- 2-dimensional simulated cortical sheet with predefined 2D coordinates (i,j).
Best Matching Unit (BMU) Selection: Filter activities (i.e. feature maps) are averaged across a mini-batch of input images x and spatial dimensions. The filter Wij with the highest average activation is selected as the BMU:
BMU arg max [Wax
3. Weight Updates:
Filter weights updated based on their distance to the BMU the cortical sheet, scaled by a learning rate (t)) and a neighborhood function Te,s((t) (t), which is centered at the BMU position.
Wiy (t + (t) + a(t) (t,((()) (we(t) wiy(t)))
(2)
Here, we(() is the BMU's weight, and w.y(() is the weight of the filter at position (i,j)- The neighborhood function Ye.sy (t) is defined as:
|lpe P 20
Tegyy Te,a((t) = exp
(3)
where P. and Pij are the positions of the BMU and the filter at (i,j), and (t) is the time. The neighborhood function was proposed
variance. Both att) and (t) decay as an approximate model of the functional interaction between nearby cortical neurons whose connectivity strengths decay exponentially as a function of cortical distance (Tuevo Kohonen 1990) (see more details in section A.T
3.2 CREDIT- BASED SELF-ORGANIZING MAPS (CB-SOM)
This a variation self-organizing maps where the competitive learning process is guided by the utility credit assigned to different units/filters as opposed activity level which is used in AB-SOX ABM. The weight update procedure consists of the following steps:
Cortical Sheet Arrungement: Similar to AB-SOM,filters within each layer are positioned on a 2-dimensional simulated cortical sheet.
2. Best Matching Unit (BMU) Selection: order to compute the model output. The model output in conjunction with the object label corresponding to the input pattern are used to compute the object recognition loss L. The gradient of loss function with respect each weight parameter is calculated to estimate each weight's 's importance on reducing the loss. Within each layer, the filter with the highest magnitude of gradient (L2 norm the cross-entropy loss gradient for each filter computed per batch of images) is selected as the winning unit/filter.
The input is passed through all model layers in
Oc Duty
BMJ = arg max if
4))
3. Weight Updates: All weights are first updated with gradient descent to reduce the task loss. After this phase, the errors between each layer's winning filter's weight and other filters' weights within that layer are calculated and all weights are updated according to Eq. 2 similar to AB-SOM.
3.3 MODEL TRAINING
Each of the two self-organizing rules were incorporated into the ResNet-18 convolutional neural network architecture (He et al. 2016). Self-organization in each model was performed accord- ing to the corresponding rule (AB- or CB-SOM) at every layer of the network and every training step. The self orgaaniaazzininn procedure was performed in tandem with the top-down gradient-basse based learning using stochastic gradient descent and an object recognition loss (cross-entropy loss) on the ILSVRC-2012 (ImageNet Large-Scale Visual Recognition Challenge) dataset (Deng et al.) 2009) for 90 epochs with batch size of 256. The initial learning rate was set to 0.1 and were decrease
4

---

Published as a conference paper at ICLR 2025
al
b
C
x10-
Top Accuracy %
ImageNar Val. Acc.
80
10
CTA Index
40
10
CB SCM
S0M
20-
CB SOM
AB SOM
IDANN
HeeNNet1
P-baup 0101
P-btluo 0
ResNat18
P-Vllo 0.05
10-1
0003
10000
'5000
20000
25003
5000
10000
16030
20000
25000
lteration
Iteration
Figure 2: Credit-based BMU selection aligns topographical and recognition updates. a) Ima- geNet validation top-1 accuracy during the first few epochs of training for CB-SOM, AB-SOM and ResNet18 models; b) ImageNet validation top-1 accuracy for alternative models; e) Misalignment between task loss and topographical learning loss (refer to methods) for two models, CB-SOM and AB-SOM, color-coded in blue and red, respectively. Higher numbers indicate greater alignment. The black bar indicates the significance level of the difference in mean between AB-SOM and CB- SOM category-task alignment (CTA) indices at each step.
by a factor of 10 at epochs 60 and 80. The learning rate a were initialized to 0.08 for the first three blocks of ResNet-18, and 0.1 for the subsequent blocks and layers. The neighborhood variance parameter a was initialized to the square root of the number of units. Both a and were exponen- tially decayed during training with a factor of -3, although other decay rates were also tested (Figure A4 Importantly, as the BMU selection process is only used during training, both SOM model variatinns operate similar to the non-topographical ResNet- 18 model test time, with no additional computations. A comparison of training times across models is shown in Figure A5
4 RESULTS
4.1 CREDIT-BASED SELF-ORGANIZATION IMPROVES THE ALIGNMENT WITH GRADIENT- BRADDENN-BASED REPRESENTATION LEARNING
We first explored whether traditional SOM (AB-SOM) could facilitate learning topographically or- ganized representations within deep networks. We jointly trained the network parameters to both minimize the object recognition loss with gradient descent and induce topography via the AB-SOM local learning rule. We observed that concurrent weight updates with AB- AB-SOM and gradient de- scent significantly slows down learning of the object recognition task (Figure P.. Consequently, the AB-SOM model achieved a significantly lower level of performance on the object recognition task compared the non-topographical model (Figure Pp). We attributed these outcomes to two factors in the AB-SOM algorithm: I) the bottom-up selection of best matching units (BMUs) based on filter activations, and weight updates designed better match the observed input. Consequently, these factors may bias the learning trajectory towards developing filters that reflect the frequency of observed input patterns rather than their utility. Thus, the network model suffers from large blow to its object recognition performance.
To quantify this effect, we measured the effect of AB-SOM updates on object recognition loss dur- ing training using the category-task alignment (CTA) index that measures the alignment between apporaphy-inducing updates and the object recognition loss (see Extended methods B.I. CTA in- dices were consistently negative during training of AB-SOM, suggesting that the AB-SOM updates conflicted with gradients necessary for learning the object recognition task (Figure 2
We next performed the same analysis on the CB-SOM model in which each BMUs are selected according to their assigned credit (see Extended Methods section). Compared to the AB-SOM model, CB-SOM CTA indices were consistently more positive and closer to 0, indicating lesser conflict with the maudient-based updates for learning the object recognition task (Figure 2b). In addition, CB-SOM learned substantially faster than AB-SOM model (Figure 2
We then compared the object recognition performance of the CB-SOM model against AB-SOM and another recent topographical model, the Topographical Deep Artificial Neural Network (TDANN)
5

---

Published as a conference paper at ICLR 2025
(Margalit al. 2024) using the ImageNet dataset (Deng et al. 2009) While all topographical mod- els showed reduced performance compared to their non-topographical counterpart with a matched architecture (ResNet- 18), the CB-SOM model exhibited a significantly smaller decline in perfor- mance (11% for CB-SOM vs. 27% and 26% for AB-SOM and TDANN respectively; Figure 2b). These results empirically confirmed that selecting BMUs based their assigned credit (i.e. CB-SOM) is significantly more aligned with the inherent dynamics of representations learning via gradient-based algorithms, reducing the conflict that typically accompanies topographical modifica- tions (Figure 2
4.2 LAYERWISE SELF-ORGANIZATION YIELDS VI-LIKE TOPOGRAPHY IN EARLY MODEL. LAYERS
Neurons in the primate's primary visual cortex (V1) are primarily tuned to the orientation, direc- tion of motion, and spatial and temporal frequencies specific parts of the visual field (Nauhaus al. 2012). The arrangement of neurons across the cortical surface of VI follows a systematic organization where 1) neurons with similar selectivity are arranged into "columns" perpendicular to the cortical surface (Hubel & Wiesel, 1962); 2) columns with more similar selectivity are located at closer parts of the cortex. This structure also gives rise to singularities such "pinwheels" within which neurons arrange radially around a center point according to their orientation selectivity (Bosking et al. 1997).
further arranged such that neurons
filter weights
crientation
spatial freq.
zone"
cord
AB 9CM
sponill KOCK
00/91
Ranwoo Dusance
Frcuenos
Figure 3: V1-like organization early model layers. a) (left) Visualization of the weight param- eters in the first layer of each model. Filters in each layer arranged 2D simulated cortical sheet (here 8x8). Orientation (middle) and spatial frequency (right) selectivity is color coded for each filter in model layers (convl and Block. 1.0. 1.0.covvv) for CB-SOM .BSOMM and ResNet-18 18. b) Pairwise correlations for alternative models as a function of distance on the 2D map: e) Change in selectivity function of distance on the 2D map. plotted for alternative models and two layers. d) Smoothness score for the orientation and of two layers. e) The distribution of the spatial frequency selectivity in each layer and each model.
Deep neural networks trained on object categorization task have been shown to develop filters with V1-like properties in their early layers (Cadena et al. 2019). However, the selectivity in these models

---

Published as a conference paper at ICLR 2025
does not demonstrate any regularity function of position (Figurc3ia bottom). In other words, the arrangement of these filters appear random when they are placed on a simulated cortical sheet. We first investigated how incorporating self-organizing update rules in training of DNN parameters, affects the arrangement of filters in early layers of these models.
Inspecting the learned parameters of the first layer of the models, noticed that most first-layer filters in AB-SOM were selective for colored blobs while most filters in CB-SOM consisted of Gabor-like patterns with varying orientations and spatial frequencies (FigureBa top and middle). This provided additional evidence for our guiding hypothesis discussed in the outset, that the bottom- up selection algorithm of AB- AB-SOM biases the learned filters towards more common features in natural images (such as color blobs) rather than more useful ones (such as edges).
We then quantified the filters' selectivity to orientation and spatial frequency in first two convolu- tional layers of each model (conv1 and block 1. 10-conv1) using an image dataset of grating patterns of different spatial frequencies (Margalit et al. 2024). In both models, the orientation and spatial frequency selectivity of filters smoothly changed along both simulated physical axes (Fi We computed the pairwise correlations between filter activities as a function of their distance on the simulated cortical sheet which showed smooth shift in pairwise response correlations as function of distance for both layers in CB-SOM model but only in AB-SOM's convl layer (Fi Sim- ilarly, the difference in orientation selectivity between pairs of filters increased linearly function of distance (FigurdBp). Both AB-SOM and CB-SOM models demonstrated a significantly higher smoothness in selectivity compared to the non-topographical ResNet-18 model (Figurd3i).
Moreover, selectivity in the first and second layers of CB-SOM were reminiscent of the linear zones and pinwheel structure (Blasdel & Salama 1986 Blasdel 1992) which have commonly been ob- served in the retinotopic organization of the visual cortex. In contrast, while the selectivity in the AB- SOM model also transitions smoothly as function of distance, the emerging organization lacked a clear depiction of signature pinwheel singularity similar to those appearing in the CB-SOM model. In addition, we observed that filters in the AB-SOM model were typically biased toward higher spec- trum of spatial frequencies compared to CB-SOM and the con-topographical ResNet- 18 architecture that were highly skewed towards lower spatial frequencies (Figure3p). This is important consider- ing the link between selectivity to lower spatial frequencies and object recognition generalization (Li et al. 2023). Similar trends were observed in other layers throughout the model (Figure A2) and an alternative neural network architecture (CORnet-S (Kubilius et al. 2019); Figure A6).
worth noting that since all hyperparameters were matched between the two models, the higher smoothness of AB-SOM model could not be attributed to the larger neighborhood size in AB-SOM. Further investigation of the source of this difference revealed that the during AB-SOM training, BMUs are consistently selected from small subset of filters, which leads to over-representation of a few dominant patterns in its layers (Figure A1).
4.3 OBJECT-SELECTIVE CLUSTERS EMERGE IN DEEP CB-SOM LAYERS
Neurons in the Inferotemporal cortex (IT) of human and non-human primates are selective to visual patterns containing particular object categories, most prominently including face, body parts, and scenes (Kanwisher et al.) 1997] Epstein & Kanwisher 1998. Downing et al. 2001). Importantly, neuronal responses in IT cortex are typically invariant pressrit-pressrving transformations like rotation, scaling, and translation (Rust & DiCarlo 2010). Similar to earlier visual areas such as V1, IT neurons are also arranged topographically but instead of orientation and spatial frequency, neu- rons in this area are organized according to their selectivity to different object categories. While deep neural networks possess remarkable ability to predict the neuronal response patterns in category- selective brain regions (Ratan Murty et al. 2021b), they do not account for the spatial organization of these category-selective neurons.
We next investigated whether the same organizational principle leads to the formation of filter clus- ters according to category selectivity as were seen in primate IT cortex. In each model, we examined across three blocks 2,3,4 (see Methods) and calculated the
the organization of category-selective selectivity of each filter to faces, scenes, body parts, and other non-living objects. Visualizing the resulting selectivity maps for the best IT-like layer in each model on their corresponding 2D physi- cal maps revealed significant clustering of filters according to their category -eleetr-ssleectvitty in the deep layers of CB-SOM and to lesser degree in the AB-SOM model (Figure In particular, category-
7

---

Published as a conference paper at ICLR 2025
selective patches in CB-SOM were more varied, demonstrated stronger category-selectiin (i.e. higher face, body, and place selectivity), and the category-sclectiitty transitioned more smoothly across the simulated cortical dimensions (Figure 4 A3).
Moreover, similar to carly visual areas, previous studies on non-human primates have also reported exponentially decaying neural response similarity as function of distance in the macaque IT cortex (Lee et al.] 2020). We observed similar pattern of decay in pairwise filter response correla- tions across all layers of the CB-SOM model including the layer which best matched the macaque IT cortex (see Figure A2.) Additionally, similar results were obtained when experimenting with an alternative neural network architecture (CORnet-S (Kubilius al. 2019); Figure A6).
a
Face
Body
Place
Object
Size
Animacy
Realente
188 Chance)
salechie
Smicttreet
Figure 4: Category-selective clusters in deep CB-SOM layers. a) Category-Selectio maps for each object category from the fLoc dataset (i.e., Face, Body, Place, and Object), including selec- tivity for size and animacy. Maps are derived by computing the d-prime value between the target versus other categories: b) Pairwise differences in selectivity as function of distance the 2D map. analyzed for category selectivity across three models: CB-SOM, AB-SOM, and ResNet-18. Smoothness measure evaluates the continuity of transition in selectivity within each category- selectivity map and across spatial dimensions of the 2D map. CB-SOM selectivity maps are signifi- cantly smoother than both the non-topographical ResNet-18 and the AB-SOM model
4.4 IMPROVED BRAIN-LIKE REPRESENTATIONS THROUGH CREDIT-BASED SELF-ORGANIZATION
In the previous sections, we explored the topographical alignment CB-SOM with that in the human and non-human primates. We next examined whether the filter responses in the CB-SOM model also aligned with the neuronal response patterns across different visual areas. For this, compared the filter activity in different models with electrophysiological recordings from macaque monkeys as well as fMRI recordings from human subjects.
We began by using the Brain-Score platform (Schrimpi et al. 2018) to quantify the representational similarity between each model and neural activity recorded from various regions of the macaque monkey's ventral visual cortex, as well as to human behavior. Our results indicate that CB-SOM achieved representational similarity scores comparable to those of the non-topographical ResNet- 18 model across most benchmarks (Figure 54). significantly outperforming other baselines such as AB-SOM and TDANN (Margalit et al.] 2024). Importantly, the difference between CB-SOM and other topographical models especially pronounced in the behavioral benchmark, where topographical models typically show a larger gap compared to non-topographical ones. CB-SOM markedly narrowed this gap, highlighting its improved alignment with human behavior.
8

---

Published as a conference paper at ICLR 2025
Next, we used fMRI data from the NSD dataset (Allen et al. activity within ters) to the corresponding voxel-averaged activity in human subjects' sntegory-selectitn patches. For each subject, we utilized their available functional masks to identify the voxels associated with these category-selective areas. We then computed model-brain similarity for each category-selectivy patch by comparing average responses in the corresponding model patches with the average responses across 3 NSD subjects. The results were compared against alternative topographical models, includ- ing ITN (Blauch et al. 2022), TDANN (Margalit et al. 2024), and non-topographical ResNet- 18 model, whose filter activations were linearly combined to predict brain activity in each category- selective patch (Ratan Murty et al. 2021b).
compare each model's average clusters (i.e. average response across all filters within those clus-
2022)
We found that CB-SOM filter clusters predicted fMRI responses in all three human category- selective patches significantly better than all alternative models, even surpassing the ResNet-18 18 baseline, which used linear combination of activations to map onto each brain patch (Figure (p))
a
b 0
Brain- Brain-Score
Comeelatio
22
Rggh
Behavior
FFA-1
PPA
EBA
Figure 5: Representational similarity between models and primates. a) Assessing representa- tions similarity between models and macaque visual cortex using the BrainScore platform (Schrimpl et al. 2018). b) Pearson correlation between best-matching patch in each model and the correspond- ing cortical patches of Fusiform face area (FFA-1), Parahippocampal place area (PPA), and extras- triate body area (EBA) in each hemisphere using the NSD dataset (Allen et al., 2022). ResNet-18 model was mapped onto each area using cross-validated ridge regression. Mean Pearson correlation is reported for each brain region and error bars indicate the standard deviation across subjects.
5 DISCUSSIONS AND LIMITATIONS
In this study, we introduced Credit-Based Self-Organizing Maps (CB-SOM), an adaptation of Koho- nen's self-organizing maps that replicates cortical topography in deep convolutional neural networks. Unlike Kohonen's 's SOM, CB-SOM achieves topographical organization at much lower cost on fea- ture utility for tasks like object recognition. Our results show that CB-SOM enhances topographical alignment of early and late CNN layers with the early and high-level human ventral visual cortex, respectively, while also improving representational alignment with macaque recordings and human imaging data.
Bottom-up versus top-down. CB-SOM, like Kohonen's 's SOM, is competitive learning algorithm; however, its key distinction lies in selecting winner units based on their contribution to reducing an ethologically relevant objective function. This top-down approach aligns CB-SOM more closely with gradient descent and significantly influences the network's 's learned representations.
In Kohonen's SOM, winner units ("Best Matching Units") are selected based on activity levels in response to inputs. As shown in our work, this method often overrepresents frequent but task- irrelevant patterns, leading to less effective representations for tasks like visual object recognition and poorer alignment with macaque and human visual cortex responses. In contrast, CB-SOM's credit-based selection emphasizes task-relevant features, resulting in improved visual recognition performance and stronger representational alignment with the primate brain.
These findings highlight the critical role of task-related top-down signals in shaping topographical organization in the visual cortex and potentially across the neocortex. While this study focuses on visual object recognition, we believe the approach could generalize to other tasks and sensory modalities. Recent studies have shown the feasibility of applying topography-inducing methods

---

Published as a conference paper at ICLR 2025
from vision models to language models, where attention and fully connected units in transformer architectures are topographically organized (Binburaib et al. 2024 Rathi et al. 2024).
Performance and topography trade-off. By selecting filters based on the highest gradient of the objective function with respect to parameters, CB-SOM identifies neurons that have the greatest impact on object categorization, prioritizing features like orientation and object selectivity that im- prove task performance. In contrast, traditional activation- based methods, which select neurons with the highest responses across batch of images, tend to overrepresent common but less informative features, such as colors (Figure 3). Unlike AB-SOM, CB-SOM emphasizes filters that contribute significantly to improving the objective function, even if they are not highly responsive to images.
We propose that topographical organization in the visual cortex emerges not solely through wiring cost minimization but via self-organizing processes. This is supported by the superior quality of learned representations in CB-SOM, which achieves better representational alignment with neural responses in human and non-human primate brains (Figure [5). These findings suggest that inducing topography in deep networks through overrepresentation of units with maximal task relevance, as in CB-SOM, may yield representations more closely aligned with the brain.
Towards globally topographic models of visual cortex. Our approach is distinguished by its abil- ity to achieve universal topography across layers of the network with minimal impact on task performance. While numerous topographical neural network models have been proposed, most are constrained in the range of brain areas they simulate. Early models of cortical topography were often tailored mechanisms specific to the primary visual cortex and restricted to that region (Von der Malsburg 1973 Linsker 1986). Recent models leveraging deep neural networks, such as those fo- cusing on calegory-selective neural clusters in later stages of visual processing (Blauch al. 2022 Doshi & Konkle, 2023 Lee et al. 2020), also exhibit limited applicability. A notable exception the TDANN model (Margalit et al., (024). which extended topography-inducing objectives to all network layers, resulting in a universally topographic model. However, this came at the cost of significant performance losses in visual recognition tasks. In contrast, our method provides a biologically plaausbbe hypothesis for spontaneous topography through self-organization, centered on the idea that Matching Units can be selected via a top-down process whether by minimiz- ing task error (CB-SOM) or through higher-level cognitive processes like attention.
The universality of CB-SOM topography combined with its high task performance makes it an ideal candidate for predicting the topographical organization of other visual areas, such as V2, V3, and VO. Notably, artificial neural networks are powerful predictors of areas such as V4 (Yamins et al.) 2014), enabling neuroscientists to predict neuronal responses and control their activity (Bashivan et al. 2019 Ratan Murty et al., 2021a a Ren & Bashivan 2024). CB-SOM and other universally to- pographic neural network models combined with synthesis techniques (Bashivan et al.) 2019 Walker
et al. 2019 Ponce et al.) 2019) provide bypothesis-generation wawpiii-ggeeeeatitttttttne -testing platform for revealing the to-
pographical organization across the visual cortex, similar recent attempts in rodents (Tong al.
2023).
I Limitations. Our work has the following limitations:
Lack of formal convergence guarantees. A key limitation of CB-SOM is the lack of formal the- oretical convergence guarantees for its update rule. This challenge not unique to our work; to our knowledge, no formal proofs of convergence exist for the original SOMs in more than one di- mension. CB-SOM introduces a non-local BMU selection mechanism based on a network wide loss function, which aligns the learning process with top-down objectives. While this innova- tion improves task performance and representational alignment with biological systems, it adds complexity to proving convergence theoretically. Despite this, all our empirical experiments con- sistently showed stable training dynamics and smooth topographical organization under standard scheduling of the learning rate and neighborhood function. Future work could explore deriving CB-SOM updates from an explicit energy-based function (Heskes) 1999), to strengthen its theo- retical foundation while maintaining its empirical advantages.
Remaining performance gap. Our proposed CB-SOM significantly narrows the performance gap between topographical and non-topographical models in object recognition. However, noticeable disparity still persists. These findings underscore the potential of task gradients as a promising direction for developing more effective topographical models in the future.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
This research was supported by the Fecathy-Brains-Healllee NSERC Discovery grant RGPIN-2021-030335 and CIHR Project Grant PJT-191957. P.B. was sup- ported by FRQ-S Research Scholars Junior I grant 3 310924, and the William Dawson Scholar award. All analyses were executed using resources provided by the Digital Research Alliance of Canada (Compute Canada) and funding from Canada Foundation for Innovation project number 42730.
startup supplement grant, the
REFERENCES
Emily J Allen, Ghislain Yte,,es, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 116-):166-12,,,, 2022.
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6443))cuu 2019.
Pouya Bashivan, Adam Ibrahim, Amirozhan Dehghani, and Yifei Ren. Learning robust kernel ensembles with kernel average pooling. arXiv, 2022.
Taha Osama A. Binhuraib, Greta Tuckute, and Nicholas Blauch. Topoformer: brain-like topo-
graphic organization in Transformer language models through spatial querying and reweighting. March 2024. URL
Nb6xQ2goTE
Gary G Blasdel. Orientation selectivity, preference, and continuity in monkey striate cortex, Journal of Neuroscience, 12(8):3139-311 1992.
Gary G Blasdel and Guy Salama. Voltage-sensitive dyes reveal modular organization in monkey striate cortex. Nature, 321(6070) 1986.
Nicholas M Blauch, Marlene Behrmann, and David C Plaut. A commetivity-constriinee computa- tional account of topographic organization in primate high-level visual cortex. Proceedings of the National Academy of Sciences, 119(3)e21 12566119, 19, 2022.
William H Bosking, Ying Zhang. Brett Schofield, and David Fitzpatrick. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex. Journal of neuroscience, 176))2112-2127, 1997.
Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS Computational Biology, 15(4):1-16, 2019. ISSN 15537358, doi: 10.
1006897.
Number: 4.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Fenil R. Doshi and Talia Konkle. Cortical topographic motifs emerge in a self-organized map of object space. Science Advances, 9(25):eade8187 187. 2023. doi: 10.1 10.112//scddvvadess 187. Pub-
URL htttps://www.d.. www.ccinnc....ccc org/ doi full/10.... 1t.11.112//scte 1126/sciadv ade818 887)
lisher: American Association for the Advancement Science.
Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science, 293 2001.
Russell Epstein and Nancy Kanwisher. cortical representation of the local visual environment.
Nature, (9776))))588-66) 1998.
M Ester, H P Kriegel, J Sander, and Xu Xiaowei. A density- based algorithm for discovering clusters in large spatial databases with noise. Second International Conference on Knowledge Discovery and Data Mining, 12 1996. URL URI https: //wp://wwwww///////////////.. oati http:/ata..o/oobiooo///2//001//

---

Published as a conference paper at ICLR 2025
Dawn Finzi, Eshed Margalit, Kendrick Kay, Daniel L. K. Yamins, and Kalanit Grill-Spector. A single computational objective drives specialization of streams in visual cortex. bioRxiv, 2023.
doi: 10.1 2023 22. 1 19. 572460
Otfrid Foerster. The motor cortex in man in the light of hughlings jackson's doctrines. Brain, 59(2): 135-159, 1936.
Pitoyo Hartono. Mixing Autoencoder With Classifier: Conceptual Data Visualization. JEEE Ac- ceas, 8:1053011005300, 2020. ISSN 2169-3536. doi: 10. 10.1109/ACEESS 2020. URL https://ieeexpie
ieee org/
Pitoyo Hartono, Paul Hollensen, and Thomas Trappenberg. Geaming-Regulatee Context Relevant Topographical Map. IEEE Transactions on Neural Networks and Learning Systems, 26(10):2323- 2335, October 2015. ISSN 2162-2388, doi: 10.1109/TNNL 20114..3777.
URL https
//ieeexplore ieee.
abt 6998077
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tom Heskes. Energy functions for self-organizing maps. In Erkki Oja and Samuel Kaski (eds.), Kohonen Maps, pp. 303-315. Elsevier Science V. Amsterdam, Jamuary 1999. ISBN
URL /estts://wwwt sciencedirect. wwl.ccomscccc htt.://cwc.............o..oo..a.
978-0-444-5020-44 doi: 10. 10.10000997880444400774440000440
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
Colin Humphries, Einat Liebenthal, and Jeffrey R Binder. Tonotopic organization of human auditory cortex. Neuroimage, 10(3):1202-122) 222122222221111 1211, 2010.
Robert A Jacobs and Michael Jordan. Computational consequences of a bias toward short connec- tions. Journal of cognitive neuroscience, 4(4):323-336, 1992.
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 1997.
Seyed Mahdi Khaligh-Razay and Nikolaus Kriegeskorte. Deep supervised. but not unsuper- vised, models may explain IT cortical representation. PLoS Computational Biology. 10(1)), 2014. ISSN 15537358. doi: 10. 10.1377//oumallppebb.o 1003915. Number: 11 ISBN: 1553-7358
(Electronie))11553-7747X (Linking).
Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Biological cyber- netics, 43(1):59 69, 1982.
Jonas Kubilius, Martin Schrimpf, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Like Object Recognition with High- Biigh-Performig Shal- low Recurrent ANNs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. D' Alché-Buc. E. Fox, and R. Gamett (eds.), Neural Information Processing Systems (NeurIPS), pp. 12785 12796. Curran Associates, Inc., 2019.
Hyodong Lee, Eshed Margalit, Kamila M Jozwik, Michael A Cohen, Nancy Kanwisher, Daniel LK Yamins, and James DiCarlo. Topographic deep artificial neural networks reproduce the hall- marks of the primate inferior temporal cortex face processing network. BioRxív, pp. 2020-07, 2020.
Zhe Li, Josue Ortega Caro, Evgenia Rusak, Wieland Brendel, Matthias Bethge, Fabio Anselmi, Ankit B Patel, Andreas S Tolias, and Xaq Pitkow. Robust deep learning object recognition mod- els rely on low frequency information natural images. PLOS Computational Biology, 19(3): e1010932, 2023.
12

---

Published as a conference paper at ICLR 2025
R Linsker. From basic network principles to neural architecture: emergence of orientation columns.
Proceedings of the National Academy of Sciences, 83(22):8779-87883 1986. doi: 10. 10. 10.3//ass..
83.22.8779. URLhttps: //tttps::/wwtt http//pwwwwp.ppppppttt..t..p..wp
www.pnaa............ otg/ dol/ attpa//..../........... 83. 22..222.22 ....22.7799 Publisher: Proceedings of the National Academy of Sciences.
Zejin Lu, Adrien Doerig, Victoria Bosch, Bas Krahmer, Daniel Kaiser, Radoslaw M Cichy, and Tim C Kietzmann. End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions. arXiv preprint arXiv:2308.09431, 2023.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying principle for the functional organization of visual cortex. bioRxiv, 2023.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron, 2024.
Ian Nauhaus, Kristina J Nielsen, Anita A Disney, and Edward M Callaway. Orthogonal micro- organization of orientation and spatial frequency in primate primary visual cortex. Nature neuro- science, 15(12):1683-169) 2012.
Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L Gardner, Anthony M Norcia, and Daniel LK Yamins. Unsupervised models of mouse visual cortex. bioRxiv, 2021.
Carlos R. Ponce, Will Xiao, Peter F. Schade, Till S. Hartmann, Gabriel Kreiman, and Margaret S. Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, (094)))99999009.... 2019. ISSN 10974172. doi: 10.1006fj.cell.2019... 9.04.005. URL http:://aae http:///2a..aa..or.//0/........d Number: 4 Publisher: Elsevier.
04.
005
Xinyu Qian, Amir Ozhan Dehghani, Asa Farahani, and Pouya Bashivan. Local lateral connectivity sufficient for replicating cortex-like topographical organization in deep neural networks. BioRxiv, 2024.
Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition be- havior of humans, monkeys, and state-of-the-art deep artificial neural networks. The Jour- nal of Neuroscience, 38(33):7255-7265 2018. ISSN 0270-6474, 1529-2401. doi: 10.1523/ wttp::osci...g/oooooooooo.oo
INEUROSCL.038- 18.2018.. URL https: //wp::://www.
1523 0388 18. 2018
N. Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J. DiCarlo, and Nancy Kan- wisher. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature Communications, 12(1):5540, September 2021a. ISSN 2041- 1723. doi: 10. 1011/0011477777 167-021-22509--6 URLhttps: h/wp://wwwwwe. www.:./.auu.... Publisher: Nature Publishing Group.
s41 4 02 --99--8
Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher. Com- putational models of category-selectoo brain regions enable high-throughput tests of selectivity. Nature communications, 12(150044 202
Neil Rathi, Johannes Mehrer, Badr AlKhamissi, Taha Binhuraib, Nicholas M. Blauch, and Martin Schrimpf. TopoLM: brain-like spatio-fiunetional organization in topographic language model, October 2024. /
2410 11516 arXiv:2410.11516.
Yifei Ren and Pouya Bashivan. How well do models of visual cortex generalize out of distribution samples? PLOS Computational Biology, 2065):e1011145, May 2024. ISSN 1553-7358. doi: URL http:/ttts://oorrr
popploo os .rgg hplpposcoooo www.mpbicooooooooooooooo Publisher: Public Library of Science.
article?id=10 117./1311111 1ww.oooraa...a..aaa.a.....aaa... pcbi 1011145
Nicole C Rust and James J DiCarlo. Selectivity and tolerance ("invariance") both increase visual information propagates from cortical area v4 to it. Journal of Neuroscience, 30(39): 2010.
13

---

Published as a conference paper at ICLR 2025
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J Majaj, Rishi Rajalingham, Elias B Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Martin Schrimpf, Paul McGrath, Eshed Margalit, and James J. DiCarlo. Do Topographic Deep ANN Models of the Primate Ventral Stream Predict the Perceptual Effects of Direct IT Cortical Interventions?, January 2024. URL https: /www. biorxiv org/ content 110.
2024.01.0 09. 572970v1
Pages: 2024.01.09.572970 Section: New Results.
Anthony Stigliani, Kevin S. Weiner, and Kalanit Grill-Spector. Temporal processing capacity in high-level visual cortex domain specific. Society for Neuroscience, 33(3)):1241-2122222 2015. doi: (0.1523//EEROOSC14422 14.2015.
NV Swindale. The development of topography in the visual cortex: review of models. Network: Computation in neural systems, 7(2):161-247, 1996.
Rudi Tong, Ronan da Silva, Dongyan Lin, Arna Ghosh, James Wilsenach, Erica Cianfarano, Pouya Bashivan, Blake Richards, and Stuart Trenholm. The feature landscape of visual cortex. bioRxiv, pp. 2023-11, 2023.
Tuevo Kohonen. The self-organizing map. Proceedings the IEEE, 78(9):1464- 14800 1990. URL 58325
Chr Von der Malsburg. Self-organization of orientation sensitive cells the striate cortex. Kyber- netik, 14(2):85-100, 1973.
Edgar Y. Walker, Fabian H. Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G. Fahey, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 2019. ISSN 15461726. doi: 10. 10.1/48//4 593-019--511111111
Uttpttp:: Publisher: Springer US.
orgg/1..1......0 1039/
s41593- - 0517-
YC Wong, HC Kwan, WA MacKay, and JT Murphy, Spatial organization of precentral cortex in awake primates. i. somatosensory inputs. Journal of Neurophysiology 41): (5) 107-1119, 1978.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performance-optiminn hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111((3):8619 8624, 2014.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RELATION TO LATERAL CONNECTIONS IN THE BRAIN
Local lateral connections in the brain are known to connect neurons exponentially decaying function physical distance, a phenomenon thought to be critical for shaping topographical orga- nization. This concept is explicitly embedded in Kohonen's original Self-Organizing Map (SOM) model [Kohonen] 1982), although is implemented in a modified form via neighborhood function in later variations (Tuevo Kohonen 1990). In Kohonen's original formulation (Kohonen 1982). the
activity of each unit 77 is computed as follows:
1/: =
Σ RES,
(5)
where S; denotes the set of neighboring units around the Best Matching Unit (BMU), oit represents the weighted average of inputs (i.e., the feed-forward computation performed by each unit), and the coefficients 7/k are determined based on the distance between each neighboring unit and the BMU. This formulation reflects form of lateral influence, where increasing the match between the BMU and the input pattern (via Hebbian learning rule) also leads to updates in neighboring units, with each unit's update weighted according to its distance from the BMU (i.e., modulated by the coefficients).
In later implementations of SOM. the explicit role of lateral connections in calculating unit activ- ity was often omitted, while their influence on the update rules was retained through the use of a distance-dependent neighborhood function (similar to 3.. This simplified formulation retains the impact of lateral connectivity on learning, even though the forward computations no longer directly involves lateral connections.
Our proposed Credit- Basid Self-Organizing Map (CB-SOM) utilizes the same update rule as the Kohonen's SOM (Tuevo Kohonen 1990), preserving this connection to lateral connectivity in the brain. The primary difference lies in how the BMUs are selected; in CB-SOM, BMUs are chosen based the gradient magnitude, rather than activation levels. This modification aligns CB-SOM with gradient-base learning methods commonly used deep neural networks, while still incorpo- rating the influence of lateral connectivity principles observed in the brain through the neighborhood function applied during weight updates.
al
CB SOM
AB SOM
Late
Model Layer Hierarchy
3
Entropy
2
Early
5000
10000
15000
20000
25000
Iteration
Figure A1: Diversity of BMUs during training. We collected the distribution of BMU indices every 500 iterations during training and measured the diversity of selected BMUs by calculating the entropy of BMU distribution for each model and each convolutional layer. Color intensity indicates the depth. Higher entropy corresponds to more diversity in the selected BMUs.
15

---

Published as a conference paper at ICLR 2025
151 Corttiming
Figure A2: Visualization of topographical properties for all convolutional layers of CB-SOM AB-SOM and ResNet-18. a) Orientation selectivity for each filter is calculated and plotted for all convolutional layers. b) Similar to a but for Spatial Frequency selectivity. c) Spatial Frequency selectivity distribution for models plotted for all convolutional layers. d) Change in Pairwise orientation selectivity function of Euclidean distance is plotted for models. e) Pairwise Pearson correlation is calculated and plotted as function of Euclidean distance. f) Smoothness is calculated for each model from d using Eq. 7
Curveture
a
CB SOM
AB SOM
Face&Body
Face
Text
Place
Figure A3: Most exciting images for the IT layer in each model. a) For each filter in CB-SOM model, the image with highest elicited activation from the ImageNet validation set plotted on grid that matches the arrangement of model filters on the 2D simulated cortical sheet. b) similar to a but for AB-SOM model.
B EXTENDED METHODS
B. CALCULATING THE CATEGORY-TASK ALIGNMENT (CTA)
To quantify the impact of SOM updates on model performance in object recognition tasks, we calculated the category-task alignment index (CTA) for each iteration. In order to calculate the CTA index, we calculated the difference in object categorization loss before and after each SOM update. This difference plotted every 500 iterations.
16

---

Published as a conference paper at ICLR 2025
a
Decay Rate Magnitude (CB SOM Models)
Paiir (odds)
b
0.3
3
filter weights
Apr
Sigmt
con1
conv1
d
Loaadha
Spoots
Cony!
Rlock (.oooee
Orientation
Blook1.0 .0 convt
Planriee Lidet
Fallito Disonce
spatial freq
convt
Mcd00
Blook1.0
convt
Figure A4: Training CB-SOM model with various decay rates magnitude. a) visualiation of the first layer filters, orientation selectivity and spatial frequency maps for CB-SOM models trained with different decay rates. b, visualization of reduction of the a and a parameter values during training with different decay rates magnitude. d, Pairwise filter activation correlations function of distance on the simulated cortical sheet for models trained with various decay rates. f Topp1 ImageNet accuracy for models trained with different decay rates.
Training Time Per Batch
0.6
Time (ms)
0.4-
0.2-
CBS0M
AB SOM
FN18
0
Figure A5: Training time for CB-SOM, AB-SOM and ResNet18 models per Batch. Training time for Batch Size 256 for three models. Mean and Standard deviation is calculated across 5000 iterations
CTAms L(0) - C( + A0soM)
where and AASOM denote the loss and the SOM update to the model parameters respectively.
B.2 BMU DISTRIBUTION
We collected the BMU indices every 500 iterations during training and calculated the probability distribution of those indices within each layer. We then quantified the entropy the BMU dis- tributions for each model and layer. The entropy value captured the spread of the distribution by assigning higher numbers to more uniform distributions with more diverse set of filters selected as BMUs.
17

---

Published as a conference paper at ICLR 2025
a.
wnights
Vi.Can
Vi Lerd V2.0nrt irou V2.
Wildend
VA Wi.Com, iepu:
Vel Bhip
Vi.Comal
modom Inpu:
FiCCom
Conteet
Face
Body
Place
Objecr
Antrayy
Oorhie C2 5014
Figure A6: Generalization of CB-SOM to CORnet-S. Orientation selectivity in convolutional layers of CORnet-s. Each map corresponds to one convolutional layer and each item on the map corresponds to one filter that layer. Colors correspond to orientation selectivity of different filters in each layer. b) Spatial frequency selectivity in layers of CORnet. Colors correspond to the spatial frequency tuning each filter. e) Pairwise correlation as a function of pairwise distance between filters. d) Category selectivity for the model's IT layer.
B.3 VI ANALYSIS
We assessed the orientation tuning and spatial frequency of filters by using sine grating images from a recent publication (Margalit al. 2024). The imageset contained eight orientations, ranging from O 180 degrees and eight spatial frequencies from 0.5 to 12 cycles per degree. Using similar approach to this work, we assessed the orientation and spatial frequency by constructing the tuning curves and quantifying the strength of the tuning curves using circular variance (CV):
En 20 Σn Tk
CV
where 0g the k-thh orientation radians) and r& the response of the filter that orientation. The orientation tuning curves are additionally fit with von Mises function, and the peak of this function is taken as the preferred orientation.
For plots in Figure 3c (ASelectivity),
assessed the change in selectivity tuning (ASelectivity by calculating the difference in orientation selectivity as a function of pairwise distance between the filters. Similarly, for plots in Figure 31 (Smoothness), and spatial frequency maps each model using the approach from Margalit et al. (2023).
assessed the smoothness of orientation
1. We constructed vector of pairwise filter activation correlations against their cortical dis- tance for each layer. This vector represents the correlation responses between model filters based on their spatial proximity in the cortical structure.
2. Using this vector, we quantified smoothness of the selectivity maps by comparing the selectivity of the closest model filter pairs with those of the least similar pairs. This allows us to assess how tuning similarities vary with cortical distance.
3. Let æ represent a vector of pairwise tuning similarity values arranged in ascending order of cortical distance. The smoothness score ((x) defined as:
5(x) max (x:) To
Fg
where max(æ) is the maximum similarity value among the closest pairs, and xo is the minimum similarity value among the least similar pairs.
4. A higher smoothness score indicates that closely located filters exhibit more similar func- tional characteristics, suggesting smoother transition in the selectivity maps. Conversely, a lower score reflects greater variability selectivity, implying abrupt changes as the dis- tance increases.
18

---

Published as a conference paper at ICLR 2025
B.4 FINDING CATEGORY-SELECTO PATCHES
We used the fLoc datasets (Stigliani et al. 2015) to find the category-selective patches in models containing four image categories: Face, Body, Place, and Objects, where there 144 images per category. We calculated the sensitivity of each filter for each category by measuring the difference between the mean responses of each category's images versus all other categories.
d'
P
Mo
(8)
where e and denote the target and other categories respectively.
B.5 TRAINING CORNET WITH CB-SOM
We demonstrate the applicability of CB-SOM training alternative architecture by applying it to CORnet-S across all convolutional layers (Kubilius et al. 2019). For the CorNet-S architecture, an initial alpha of 0.08 the V4 and IT blocks. The initial sigma parameter was determined based on the square root of the number of units in each layer.
used for the V1 and V2 blocks, while alpha of 0.1 was selected for
B.6 ALIGNING NEURAL NETWORK MODEL LAYERS AND BRAIN AREAS
We quantified neural network model and brain alignment in two ways:
1. BrainScore similarity. We used the publicly available BrainScore platform (Schrimpf) et al., 2018) to measure the similarity of each models with neurophysiological and behav- ioral data. Briefly, BrainScore uses principal component regression to find combination of units (i.c. filter activations) in each layer of the model that can predict the responses in recorded neural sites. The normalized image-level behavioral consistency measure (12n) (Rajalingham et al. 2018) is used to assess the behavioral consistency with human subjects.
2. Model-brain patch similarity. We analyzed the human fMRI data from Natural Scene Dataset (Allen al.) (022), which contains the fMRI responses for various cortical regions including, the Fusiform face area (FFA), Parahippocampal place area (PPA), and extrastri- ate body area (EBA) of 8 subjects the category-selective model patches using the fLoc stimulus set (Stigliani et al.) 2015). we collected the 2D coordinates of filters that were selective for faces, body parts, and places. (OBSCAN) et al. 1996)
total of 73,000 natural images. After identifying
We then used the density-based spatial clustering algorithm label filters into defined patches. We considered a filter cluster as a patch it contained at least three filters with physical distance of one. For the TDANN model, we used the method described in (Margalit al. (023) to identify the category- then averaged the responses of all neurons within each cluster and computed the Pearson correlation between the model patch activity and the corresponding cortical patch. All comparisons were performed using 2-fold cross-vais validation where correlations on the train set were used to select the best layer and patch in each model, and only the test values were reported. For ResNet-18, 18, performed ridge regression the activation of the model to the brain areas. We then performed cross-validation five times, and the average correlation and standard division for the testing set were reported. For each subject, we calculated the correlation for each brain area across all models using the method described above. We then reported the average testing correlation and standard deviation across subjects for all models.
patches. We
We identified layers to be IT-like by measuring the correlation between category-selectiv patches in model and brain (see Model to Brain patch similarity above), and selecting layer with highest correlation.
19


Reviews:
[{"review_id": "sruFSlFWAr", "reviewer": "Reviewer_jkYR", "summary": "The paper introduces Credit-Based Self-Organizing Maps (CB-SOM), a novel algorithm designed to integrate topographical organization into deep neural networks by aligning with top-down learning processes in DNNs. This new method modifies the traditional Kohonen’s Self-Organizing Map (SOM) to assign credit based on each unit's contribution to minimizing task loss. CB-SOM significantly improves object recognition performance compared to prior topographical models while maintaining alignment with the ventral visual cortex of macaques and humans. The model enhances representational alignment with neural activity in both early and high-level visual cortices, displaying substantial improvements over previous approaches.", "questions": "Minor: \n\n1. It would be interesting to discuss implications of this claim ‘resulting CB-SOM model displays substantial improvements in representational alignment with recordings from macaque visual cortex and imaging data from human visual cortex’\n\n2. I am curious if you can provide some concrete examples for this claim ‘we believe our results are not limited to this task and could potentially be generalized to other tasks and contexts that may include multiple sensory modalities.’\n\n3. A few typos e.g. ‘of of’ in line 510 and ‘th’ in line 205", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses the intriguing challenge of integrating topographical organization into deep neural networks, a problem with significant implications for both artificial intelligence and neuroscience.\n\n2. The authors clearly articulate the motivation behind their work, highlighting the need to reconcile the performance of self-organized neural networks with the functional efficacy observed in biological systems.\n\n3.  A comprehensive literature review provides context and demonstrates the paper's grounding in existing research.\n\n4. The paper offers a thorough explanation of the proposed CB-SOM algorithm, including a detailed summary of the parameters for the experiments.\n\n5. The authors effectively link their findings to supporting papers, relating the observed behaviors of their model to biological phenomena, thereby strengthening their claims.\n\n6. The results showcase the superior quality of learned representations in CB-SOM and its enhanced alignment with neural responses in both human and non-human primate brains, demonstrating significant improvements over previous models.", "weaknesses": "1. The paper does not adequately assess / estimate the efficiency and computational time required for the CB-SOM learning algorithm, which could be a critical factor for its practical application.\n\n2. The 'main contributions' statement in the introduction claims substantial improvements in object recognition performance vs prior topographical NNs, but does not comment on performance degradation compared to non-topographical models like ResNet. Including a comment on this trade-off would clarify the performance implications.\n\n3. The trade-off in performance (as seen in Figure 2A) may affect the algorithm's appeal for studies outside neuroscience. Additional commentary from the authors on this would be beneficial.\n\n4. The \"Deep topographical neural network\" section would benefit from a high-level summary of how the previous approaches in the literature impact overall performance, providing clearer context for readers.\n\n5. It would be essential to provide a link to the code to ensure reproducibility \n\n6. Figure 2A should include comparison with the other baselines shown in Figure 2B,for completeness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mQRKvZpq1Y", "reviewer": "Reviewer_WLW9", "summary": "This study proposes an intriguing way to force topography in ResNet18. Earlier topographical models had exhibited significant drops in accuracy, but this study provides an effective way to alleviate the accuracy drop.", "questions": "I do have two questions. \n\nWhen the authors mention “units” of ResNet18, are they referring to ResNet18’s channels? If so, some clarification is needed. If not, a detailed explanation is needed. \n\nIn general, lateral connections in the brain have been believed to play a role in shaping topography. My second question is, could the authors provide insights into how their proposed mechanisms are related to lateral connections in the brain?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors extensively compared the model’s responses and neural responses recorded from Nonhuman primates and humans, which will greatly interest readers of brain science research. Their study may also be of interest to the deep learning research community because it proposes a way to impose local structure to deep learning models, which may aid us build domain-specific models with specific local structure. The authors evaluated a single architecture only. It would be more compelling if they tested more architectures.", "weaknesses": "The authors evaluated a single architecture only. It would be more compelling if they tested more architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sOt6TQYzea", "reviewer": "Reviewer_nVAR", "summary": "This paper proposed a method for seamlessly incorporating topographical learning with top-down supervised learning.\nThis paper's primary contribution is the proposal of an Action-Based Self-Organizing Map (CB-SOM) that utilizes the gradient of a loss function as the criterion for selecting the Best Matching Unit (BMU) for topographical learning.\n\nThe paper is generally very well written and supported with interesting and valid experiments.\n\nThe clarity of the paper was significantly improved after the revision.", "questions": "1. As the selection of a BMU always depends on the gradient of a loss function, it is unclear how the network chooses a BMU without a loss function, for example, in the neural network's running stage. It is outside the scope of the paper, but biological neural networks also deal with reinforcement learning. Hence, the authors need to give at least their comments on how to deal with this matter.\n\n2. It is unclear whether a reconciliation between AB-SOM and CB-SOM occurs, at least in the latter stage of the learning rule. In short, does the CB competition converge into the AB competition in the latter stage of the learning process? It would be insightful if the authors could prove or disprove that CB competition is the transient state of AB competition.\n\n3. There is a model that seamlessly combines topographical learning and top-down reinforcement learning as follows:\na) P. Hartono, P. Hollensen and T. Trappenberg, \"Learning-Regulated Context Relevant Topographical Map,\" in IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 10, pp. 2323-2335, Oct. 2015, doi: 10.1109/TNNLS.2014.2379275.\nb) P. Hartono, \"Mixing Autoencoder With Classifier: Conceptual Data Visualization,\" in IEEE Access, vol. 8, pp. 105301-105310, 2020, doi: 10.1109/ACCESS.2020.2999155.\n\nThese two papers share many similarities with the proposed paper. They also derive a new modification rule for the reference vectors associated with the topographical neurons. They demonstrate that the conventional SOM may be a particular case for general topographical representations in supervised models.\n\nThe authors must argue their proposal's novelties and advantages compared with these past works.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method makes sense, mainly when a top-down loss function can be defined, in that the neural network should select a BMU based on the contribution of the topographical hidden neurons to the loss function. The reviewer agrees with the authors that the integration of the conventional SOM (AB-SOM) suffers from over-representations of some hidden neurons that do not necessarily contribute to the learning performance of the whole network. The proposal clearly alleviates this discrepancy.\n\nThis paper also offers interesting arguments that compare the resulting CB-based SOM with primates, which is essential to argue about the biological validity of the proposed model.\n\nThe paper is also very clearly written and easy to understand.", "weaknesses": "There are no essential weaknesses in this paper. But there are some unclarities, listed in the Questions parts of this review, that should be clarified before this paper is ready for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62TPl49ZLf", "reviewer": "Reviewer_aFS5", "summary": "The paper contributes an original idea for training supervised deep neural networks models whose layers show topographical organization due to self-organization. It is shown that object recognition performance is reduced less by this than by a naive approach of introducing self-organization. It is further shown that DNNs trained by this CB-SOM algorithm exhibit orientation selectivity in lower layers and category selectivity in higher ones. Comparisons to human FMRI studies are performed, indicating as good match.", "questions": "The observation of category/orientation selectivity would more compelling if you compared these to plots for a ResNet that was trained in supervised fashion only, without the CB-SOM \"add-on\". Because it is not clear that the selectivity you observe does not come from the supervised part already. Do you have any insights on this?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper presents an original idea to tackle the long-standing problem of generative classification. It is well-organized and easy to read, with some minor glitches here and there. The technical quality of conducted experiments is good. It shows that self-organization can be shown when training CNNs on complex benchmarks like ImageNet while degrading classification performance significantly less than other models.", "weaknesses": "The main weak point is that the advantage of topographical organization is somehow assumed to be self-evident. Is there a functional advantage to having this property? As long as this is unclear, it is hard to accept that performance degradations as you show them are acceptable\n\nAnother issue is that the CB-SOM update rule is derived ad hoc, without reference to minimizing a loss. So there are no convergence guarantees or anything, which is bad from a conceptual perspective. It would be strongly desirable to derive this update rule from a loss function that is added to the supervised loss. A possibility is the energy-based SOM model of Heskes [1]\n\nFurthermore, some technical aspects of the model are not clearly described, see comments below\n\nLastly, orientation/category-selectivity must be compared to a purely supervised model to assess whether the effect would not have occurred without CB-SOM. It is well-known that filters in lower CNN layers develop orientation selectivity as well.\n\n[1] Heskes, Tom. \"Energy functions for self-organizing maps.\" Kohonen maps. Elsevier Science BV, 1999. 303-315.\n\nComments:\nRelated work could include works on topologically organized GMMs [2,3] as well as energy-based SOM [1].\n3.1 can be more concise, the formula is well known\n3.2 is unclear. What are \"the errors between each layer’s winning unit’s weight and other units’ weights within that layer ...\" ? Just the same as $w_c(t)-w_{ij}(t)$ for AB-SOM? A formula would help here!\n3.2 should be derived from minimizing a loss, e.g., Heskes? Otherwise, what is justification for post-hoc weight adjustment?\n3.3 How does a 3D structure (neurons are organized in width/height/channel dimensions) like a convLayer fit with the 2D organization assumed in CB-SOM? This must be explained as it is critical to the understanding of the approach\n4.1 It should be better described how training AB-SOM works in this setting\n4.1 from line 243: CTA is unclear. Either explain fully, here, or not at all and leave to appendix\nFig.3a: what are we seeing in the orientation-selectivity plots? All filters arranged on a 2x2 grid? Not terribly clear to me...\nAll of Sec.4: this analogy to human brains is interesting, but does it serve a purpose? Why would we want this, given that object recognition is impaired by it? This question needs to be answered, imho, to assess the value of these contributions\nAll of section 4: The observation of category/orientation selectivity would more compelling if you compared these to plots for a ResNet that was trained in supervised fashion only, without the CB-SOM \"add-on\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)