Anatomically inspired digital twins capture hierarchical object representations in visual cortex
Authors: Emanuele Luconi
Dario Liscai
Carlo Baldassi
Alessandro Marin Vargas
Alessandro Sanzeni
Venue: NeurIPS 2025 poster ()
Keywords: neuroscience
representation learning
population responses
cortical hierarchy
computational biology
visual perception
Primary area: neuroscience_and_cognitive_science

Abstract: Invariant object recognition-the ability to identify objects despite changes in appearance-is a hallmark of visual processing in the brain, yet its understanding remains a central challenge in systems neuroscience. Artificial neural networks trained to predict neural responses to visual stimuli (“digital twins”) could provide a powerful framework for studying such complex computations in silico. However, while current models accurately capture single-neuron responses within individual visual areas, their ability to reproduce how populations of neurons represent object identity, and how these representations transform across the cortical hierarchy, remains largely unexplored. Here we examine key functional signatures observed experimentally and find that current models account for hierarchical changes in basic single-neuron properties, such as receptive field size, but fail to capture more complex population-level phenomena, particularly invariant object representations. To address this gap, we introduce a biologically inspired hierarchical readout scheme that mirrors cortical anatomy, modeling each visual area as a projection from a distinct depth within a shared core network. This approach significantly improves the prediction of population-level  representational transformations, outperforming standard models that use only the final layer, as well as alternatives with modified architecture, regularization, and loss function. Our results suggest that incorporating anatomical information provides a strong inductive bias in digital twin models, enabling them to better capture general principles of brain function.

Anatomically inspired digital twins capture hierarchical object representations in visual cortex
Emanuele Luconit
Dario Liscai
Carlo Baldassi
Alessandro Marin Vargas
Alessandro Sanzeni
*Department of Computing Sciences and BIDSA, Bocconi University, Milan, Italy Stanford University, USA "Equal contribution
Abstract
Invariant object recognition-the ability to identify objects despite changes appearance-is hallmark of visual processing in the brain, yet its understand- ing remains central challenge in systems neuroscience. Artificial neural networks trained to predict neural responses visual stimuli ('digital twins") could provide a powerful framework for studying such complex computations in silico. However, while current models accurately capture single-neuron responses within individual visual areas, their ability to reproduce how populations of neurons represent object identity, and how these representations transform across the cortical hierarchy, remains largely unexplored. Here we examine key functional signatures observed experimentally and find that current models account for hierarchical changes basic single-neurg properties, such as receptive field size, but fail to capture more complex population-level phenomena, particularly invariant object representations. To address this gap, introduce biologically inspired hierarchical readout scheme that mirrors cortical anatomy, modeling each visual area as a projection from a distinct depth within shared core network. This approach significantly improves the prediction of population-leve representational transformations, out- performing standard models that only the final layer, well alternatives with modified architecture, regularization, and loss function. Our results suggest that incorporating anatomical information provides strong inductive bias in digital twin models, enabling them to better capture general principles of brain function.
Introduction
Invariant object recognition refers to the ability to recognize objects despite changes in position, shape, scale, or luminance. Despite its importance, understanding the neural basis of this computation, and other complex visual functions, remains central challenge in systems neuroscience (Kar & DiCarlo, 2024). Anatomical studies have revealed clear hierarchical organization of visually responsive areas the brain across species (DiCarlo et al., 2012: Harris et al., 2019). In mice, this hierarchy extends from the retina through the thalamus the primary visual cortex (V1). the lateromedial area (LM), and other higher visual areas (including RL and AL), as inferred from anatomical connectivity (Fig. 1A). Functional studies have shown that this anatomical hierarchy accompanied by systematic changes in single-neuron response properties (Siegle et al., 2021: Glickfeld & Olsen, 2017) and by increasingly linearly decodable object representations (Froudarakis et al., 2020; Hoeller et al., 2024).
Despite significant progress, fundamental questions remain about the neural mechanisms underlying invariant object recognition. What specific transformations neural representations enable this
"Corresponding author: al alessandro. wwwzeni manzzeni@uiibocooii..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

B Receptive field estimation
Sustomical Eicneryy soore
Neural Respansss
200
Slegle es al. (2021)
Foeibrradd
200
x
Broquhve fedl
500
Austonical locardo
Arras
C Hoeller Task
Neural Respones
decoder
Hoeller (2024)
READOUTS
Coniiratio Aecuney
1.1
0.8
0.8
0,0
Atatonieal Jeeamay soom
D
Froudarakis Task
Neural Reeponses
Linear decoder Froudarakis et al. (2020)
Naranch
Muraa mittintion
obj.
dlservable
0
O
Abj. 3
standerd
okj
0.0
-0.5
0.0
meolonscal Strrd
Ausicnirel Liseandiy scues
Figure 1: Probing cortical hierarchy in the mouse visual system. (A) Hierarchy scores for four visual areas (VI, LM, RL, AL) derived from anatomical connectivity (Harris et al., 2019). The method analyzes laminar termination patterns of inter-areal axonal projections to assign each area continuous position in a global cortical hierarchy. While the original analysis included over ten areas, we focus on those present in the MICrONS dataset (Consortium et al., 2021). Scores follow the trend VI<LM <RL<AL. (B) Receptive field (RF) sizes estimated from neural responses to visual stimuli. RF size increases approximately monotonically with anatomical hierarchy (Vi<RLŞLL<<A) (Siegle et al., 2021). (C) Rotation-invarian object recognition (Hoeller et al., 2024) assessed using static images at different rotations, recording neural responses, training support vector machine (SVM) and testing it on unseen rotations. Classification accuracy increases monotonically with hierarchy (VI<LM<RL<AL). (D) Invariant object recognition (Froudarakis et al., 2020) assessed by presenting four moving objects undergoing identity-preserving preserving transformations. Discriminability increases with hierarchy, except for RL, which performs poorly (RL VI < LM < AL). (E) Schematic of standard network generates multi-layer stimulus embedding and the
digital twin models (top), where final layer is used to predict neural responses. In our anatomically inspired architecture (bottom), each area is predicted from different layer of the core, selected based on its anatomical hierarchy score. (F) While both architectures capture RF size increase across areas (as in panel B), only the hierarchical architecture reproduces the increase in invariant object recognition along the hierarchy (panels C-D).
process, and how do they vary across cortical areas? Experimentally answering these questions would require dense sampling of neural responses to nigh-dimensional sets of object transformations, a task currently infeasible due the limited recording time available in behaving animals. A promising alternative is offered by digital fwins: artificial neural networks trained to predict neural responses to visual stimuli (Klindt et al., 2017; Bcker et al., 2018: Sinz et al., 2018; Lurz et al., 2020; Wang et al., 2025). These models have recently emerged as powerful tools for characterizing the functional properties of the visual system and have been shown to replicate diverse single-neuron response features (Pogonchefff et al., 2023; Walker et al., 2019; Bashiri et al., 2025; Tong et al., 2023; Wang et al., 2025; Ustyuzhaninov et al., 2022: Ding et al., 2023). However, most existing models focus on predicting single-ncuron responses without explicitly assessing whether they capture population-level properties, such as differences in representational geometry across visual areas which is critical for invariant object recognition. Without this capability, their ability to reveal how neural circuits progressively construct more abstract visual representations is significantly constrained.
Here, we address this gap by evaluating the ability of digital twins of the mouse visual cortex to reproduce experimental findings on how object representations evolve across the visual cortical hierarchy. We focus on three key experimental results, each probing different aspects of the trans-

---

formation from low-level to high-level representations. First, we test whether models capture the increase in receptive field size across visual areas, well-established signature of increasing spatial integration that supports invariance to object position and scale (Siegle et al., 2021) (Fig. 1B). Second, we evaluate whether models capture the increase in rotation-invariant object decoding across the visual hierarchy, as demonstrated by Hoeller et al. (2024). In their study, linear support vector machines (SVMs) were trained on neural activity from different areas to classify object identity under unseen rotations (Fig. IC). Third, we assess whether the models capture the progression of object discriminability across areas, as discovered by Froudarakis et al. (2020). this task, neural responses were recorded mice viewed four objects undergoing Srenntttyppeeevinng transformations, including rotation, scaling, translation, and variations in illumination conditions (Fig. ID).
We find that current digital twins are able to replicate low-level functional properties, such as receptive field size gradients across areas, but fail to account for differences in higher-level representational invariances (Fig. IE,F). In particular, they do not reproduce the experimental findings of Hoeller et al. (2024) or Froudarakis et al. (2020) when tested under analogous conditions. To address this limitation, we introduce biologically inspired modification to the model architecture. Conventional approaches predict responses across all brain areas using a shared set visual features, referred to as the "core network" (Fig. IE). However, this design overlooks the distinct computational transformations applied within different visual regions. Instead, we propose a hierarchical readout (Fig. IE), where neural responses corresponding different brain areas mapped to distinct layers within the core network rather than extracted from a single shared final layer. This architectural change introduces an inductive bias that improves the alignment between in-silico and apperimentally-oosereeee population-leved representations across the hierarchy (Fig. 1F) while also improving single-neuron predictions in higher visual areas. We show that mimicking the anatomical organization is key achieve this alignment, whereas other alternatives modifications to the baseline architecture fail to capture the experimental phenomenology. Our work provides the first demonstration that digital twins can be adapted to reproduce representational transformation across mouse visual areas, and offers a framework for using such models to investigate the mechanisms underlying behaviorally relevant visual computations.
We summarize main contributions as:
We simulated different experiments to probe hierarchical representations digital twins
We show that current models fail to replicate functional hierarchies across visual areas in multiple experimental datasets (Froudarakis al., 2020; Hoeller et al., 2024)
We investigate different architectures and training framework
We propose a new biologically-inppin hierarchical readout mechanism that captures experimental results.
2
Related work
Hierarchical processing in the visual cortex. The visual system exhibits hicrarchical organization where information flows through a network of interconnected areas (Scabrook et al., 2017; Siegle al., 2021). In mice, experimental evidence (de Vries al., 2018; Consortium al., 2021; Siegle et al., 2021) has revealed substantial differences between primary visual cortex (V1) and higher-order areas (LM, RL, AL) in receptive field properties (Siegle et al., 2021; Glickfeld Olsen, 2017), spatiotemporal characteristics (Murakami et al., 2017; Piasini et al., 2021). and critically, invariant object recognition capabilities (Froudarakis et al., 2020; Hoeller et al., 2024). These hierarchical transformations provide critical baseline for evaluating computational models of visual processing and motivate our investigation into whether digital twins can not only predict neural responses but also capture the fundamental representational changes across cortical areas.
Modeling hierarchical representations with digital twins. Deep neural networks have emerged as powerful "digital twins" for modeling neural responses to visual stimuli, following two main approaches: task-driven models optimized for visual tasks (Yamins et al., 2014; Bakhtiari et al., 2021; Kubilius et al., 2019; Zhuang et al., 2021; Pogoncheff et al., 2023), and data-driven models trained directly on neural recordings (Klindt et al., 2017; Cadena et al., 2019; Lurz et al., 2020; Safarani et al., 2021; Willeke et al., 2022; Li et al., 2023; Wang et al., 2025). Despite their success in predicting single-neuron responses (Turishcheva et al., 2024a; Xu et al., 2024), these models struggle to capture the hierarchical organization of representations across the visual system Syey et al., 2023; Dyballa
o
3

---

et al.. 2024; Liscai et al., 2025). In primates, task-driven networks trained on object recognition revealed a clear hierarchy along the ventral stream, with early layers predicting V1 and deeper layers aligning with V4 and IT (Yamins et al., 2014). In mice, however, no such progression was found: ImageNet-trained and even randomly initialized networks performed similarly across V1, LM, AL, and RL (Cadena et al., 2019), and although unsupervised objectives improved predictions and mapped V1 to early layers and higher areas to intermediate layers, they still failed to separate higher-order regions (Nayebi et al., 2023). This limitation highlights the need for architectural innovations that better align with the biological organization of the visual cortex.
Single-neuron and population-level analysis. Traditionally, neural network models in visual neuroscience have focused on single-neuron responses, investigating how individual neurons en- code sensory stimuli. Studies have examined feature selectivity in V1 by finding the most exciting inputs (Walker et al., 2019), phase-invariant and spatial encoding V1 (Ding et al., 2023; Ustyuzhani- nov et al., 2022; Bashiri et al., 2025), and the columnar organization of selectivity in V4 (Willeke et al., 2023; Burg et al., 2024). Despite these advances, single-neuron analyses fail to capture the broader hierarchical structure of population-level representations across visual areas. Pogoncheff' et al. (2023) investigated how population structure influences VI predictability, while Margalit et al. (2024) examined topological differences between early and higher-order visual regions. Expanding on this, Liscai et al. (2025) analyzed the geometry of neural representations, showing that regu- larization improves the differentiability of representations but still fails to replicate experimentally observed patterns of discriminability across visual areas. To address these limitations, we intro- duce a biologically-insee distinet visual areas, significantly improving the model's ability to capture both single-neuron and population-level responses.
hierarchical readout mechanism that maps different network depths to
3 Digital twins overlook differences in object representations across areas
Digital twin of the mouse visual cortex. We modeled neuronal responses using a compact digital twin with state-of-the-art performance that includes a core module, a readout module, a behavioral module, and a shifter module (see Appendix A for details). The core component consists of four-layer 3D Convolutional Neural Network (CNN), which transforms input videos into a high dimensional embedding. The behavioral module processes behavior data and transforms it into a latent representation which is stacked to the core's output. The readout module, adapted from Lurz et al. (2020), translates this encoded representation into predictions of individual neuronal activity belonging to different brain areas. Its functionality is organized into two key components: a spatial mask, which defines receptive field locations, and a set of feature weights that linearly aggregate information from the final core layer to produce frame-by-frame neural response predictions. Additionally, the model incorporates shifter module, which processes pupil center coordinates to estimate gaze-related shifts in receptive field positioning. This transformation is applied linearly across all neurons and is finalized with an ELU+1 activation function to ensure non-negative outputs. We refer to the configuration with N = 4 layers in the core and standard readout as the baseline model throughout the paper. We trained the baseline model, as well as of all variants discussed in the following sections, by minimizing the Poisson loss between predicted and recorded responses. Training was performed on 83,222 units from V1, 14,817 from LM, 12,599 from RL, and 4,734 from AL. using data from the MICrONS dataset (Consortium et al., 2021), recorded via two-photon calcium imaging in response grayscale natural movies. Further details the model architecture, training procedures, and evaluation metrics are provided in Appendix A.
Our baseline digital twin model achieved strong single-neuron prediction accuracy, comparable to previous work (Wang et al., 2025). Prediction performance was highest in VI, with moderate reductions observed in LM, RL, and AL (Fig. 2A). This trend likely reflects both the inherent variability in higher-order areas (de Vries et al., 2018) and the larger number of VI neurons, which might have biased the model toward optimizing this area compared to the others.
Baseline model fails to capture how object representations evolve across the visual cortical hierarchy. To assess whether digital twins of the mouse visual cortex capture functional distinctions across areas, we evaluated whether our baseline model reproduces key experimental findings summa- rized in Fig. IB-D. Specifically, we replicated in silico the experimental protocols used to generate these measurements and compared preeee-peedicted responses to their biological counterparts. Details of the in silico procedures are provided in Appendix B.

---

A
Model performance per area
basetine layens)
OcNorts
layers)
layers readont
0.5
loss normallzed
VI
1M
RL
B
Receptive field estimation
Hecoptive field area idey")
Hoeller Task Classincation Accuracy
Froudarakis Task
Mutaal information
0.3
0.5
1000
0.2
0.3
0.1
-0.5
0.0
-0.5
0.0
-0.5
0.0
C
Anatomical hierarchy score
Anstomical hierorchy score
Anatomical hierarchy store
Model alegmment
(Kendalll's itsi' rank cor))
0.5
0.5
0.5
0.0
0.0
0.0
-1.0
1.0
1.0
Models
Modals
Models
Figure 2: Digital twin fails to capture hierarchical transformations in object representations across visual areas. (A) Single-neuron prediction performance across areas for the bascline model and its variants. Bars indicate S across training seeds, as in subsequent analogous plots. (B, left right) RF size as a function of anatomical hierarchy score (dots) and best linear fit (dashed line) for each model; decoding accuracy for SVMs trained on neural representations of the digital twin model presented with rotated images from the Hoeller task; and object discriminability based on neural representations of the digital twin model presented with videos from the Froudarakis task. Here and in subsequent analogous plots: markers and errorbars represent mean and SEM across recording sessions and training seeds; the legend indicates if the slope of the linear fit is significantly different from zero (Wald Test): p < 0.001 (***), p 0.01 (**), p < 0.05 (*), p >> 0.05 (n.s.). (C) Agreement between model-pree -oeedi-prede and experimentally observed orderings in (B), as quantified by Kendall's rank correlation coefficient, which measures the correspondence between two rankings based on the number of concordant and discordant pairs (see definition in Appendix A.3.1).
First, we evaluated whether the units of the baseline model exhibit systematic increases in receptive field (RF) size along the cortical hierarchy, as observed in Siegle et al. (2021). To this end, we computed RFs by presenting spatiotemporal white noise and applying standard spike-triggered averaging (See appendix B. To quantify RF size, we fit two-dimensional Gaussian to the RF map of each unit and defined size as the product of the standard deviations along the two principal axes. RF size systematically increased along the model's cortical hierarchy (Fig. 2B), aligning well with experimental observations (Fig. IB). This result is notable: despite all area-specific units being projections from a shared core representation, the model trained on natural stimuli implicitly captured topographic distinctions across areas. We further quantified the agreement using Kendall's т. a rank correlation metric that compares pairwise area relationships (c.g., whether the RF size in LM exceeds that in VI, and so on). Kendall's 7 provides a measure the consistency between the experimentally observed functional hierarchy for the receptive field size (VI<RL Fig. IB) and the one predicted by the model, where a value of +1 indicates perfect agreement, whereas Kendall's 's r of - complete disagreement. The baseline model achieved value of 0.50, indicating moderate consistency (Fig. 2C). The main difference was due to the predicted receptive field size in LM, which, unlike in the data, was not significantly larger than the in RL.
Next, we examined whether the digital twin supports iotation-invarian object recognition and how this capacity evolves across cortical areas. Following Hoeller al. (2024), we implemented
5

---

classification task with 92 static objects at different rotations. A linear SVM was trained on the model-pre preeeiepe neural responses to identify object identity across 92 classes for held-out rotations (details in Appendix B.2). As shown in Fig. 2B, classifiers trained on neural responses performed significantly above chance, demonstrating that object identity information was preserved across rotations. However, classification accuracy did not exhibit a systematic trend with anatomical hierarchy, Kendall's T was negative (Fig. 2C), indicating that the area-to-area relationships predicted by the model deviated from biological data (Fig. IC)-for instance, performance in AL was lower than all other areas. To further challenge the model, we introduced a more complex task inspired by Froudarakis et al. (2020), incorporating variations in object rotation, scale, position, and lighting (details in Appendix B.3). While object identity remained decodable above chance, classification accuracy again failed to correlate with hierarchical level (Fig. 2B). The negative Kendall's (Fig. 2C) confirmed that model area-to-area performance orderings did not match biological observations (Fig. ID).
Taken together, these findings reveal critical shortcoming of the baseline digital twin. While the model successfully captured low level aspects of the cortical hierarchy, such as single neuron responses and the systematic increase in receptive field size across areas, it failed to reproduce the functional transformations necessary for invariant object recognition. In particular, the model did not exhibit the progressive stabilization of object identity across areas that is characteristic of the biological visual system.
4 Architectural and training modifications do not induce hierarchical representations
Rebalancing visual areas during training. Like the mouse brain, the MICIONS dataset includes many more neurons from VI than from higher visual areas. This imbalance could bias the digital twin toward V VI-dominated representations due to the shared core, potentially explaining its failure to capture transformations across areas. Supporting this, the model showed higher single-neuron prediction accuracy in V1 (Fig. 2A). To test whether this imbalance affected hierarchical organization, we trained a model variant with a reweighted loss that scaled each area's 's contribution by the inverse of its neuron count (details in Appendix A.4), ensuring equal influence during training. The reweighted model trained successfully and produced more uniform accuracy across areas, but this was due to reduced performance in V1, with improvement observed in other areas (Fig. 2A). We then tested whether this modification improved the model's ability to capture functional hierarchy. Receptive field sizes remained consistent with experimental observations, indicating that spatial tuning properties were preserved (Fig. 2B.C). However, the hierarchical structure of object representations remained unchanged. On the Hoeller task (Hoeller et al., 2024), the model still failed to reproduce the correct order across areas (Fig. 2B,C). Similarly, on the Froudarakis task (Froudarakis et al., 2020), the hierarchy was again reversed, with AL and LM performing below VI and only minor differences across areas (Fig. 2B,C). These results suggest that reweighting the training loss is not sufficient to induce the transformations in object representations observed experimentally across the visual hierarchy.
Readout with access to all layers features. In the baseline model, all neurons read out from the final core layer, regardless of cortical area. However, both cortical anatomy and physiology suggest that lower (higher) areas rely on simpler (more complex) features, paralleling how feature complexity increases with depth in convolutional networks (Goodfellow et al., 2009; Zeiler & Fergus, 2014; Cohen et al., 2020). Therefore, VI neurons may be better modeled using earlier layers, while higher areas could benefit from deeper representations. To test this hypothesis, we developed a variant of the baseline model in which the readout for each neuron was computed as a projection of kernel activations across multiple layers, rather than from the final layer alone (details in Appendix A.4). Despite its biological plausibility, and the fact that it represents a generalization of the baseline, this model underperformed relative to the baseline, with a substantial decline in single-neuron prediction accuracy (Fig. 2A). This is likely due to the increased model complexity which made optimization more difficult and led to overfitting, despite an L1 regularization term being added to the loss. Receptive field size increased across cortical areas relative to the baseline model (Fig. 2B,C), with changes that were qualitatively similar but larger in magnitude. On both Hoeller et al. (2024) and Froudarakis et al. (2020) tasks, object discriminability decreased across all areas, and differences between areas remained minimal (Fig. 2B,C). These results suggest that, at least given current data

---

constraints, expanding the readout to include features from multiple layers in an unstructured fashion does not enhance the model's ability to capture area-specific functional organization in the visual cortex.
Increasing model depth. Another limitation of the baseline model is its shallow core, only four convolutional layers, which may constrain feature complexity and hinder its ability to capture transformations across cortical areas. To evaluate this, we trained a series of models with increasing depth, keeping all other architectural components fixed (details in Appendix A.4). Increasing the number of layers from four six yielded improvements in singleneeuron predictive accuracy across all areas (Fig. 2A). However, adding layers beyond six did not yield consistent additional improvements (Fig. 3A), suggesting a saturation point in representational benefits under current data constraints. Receptive field properties in this model variant showed a similar degree of agreement with experimental data as observed in the other models discussed above (Fig. 2B,C). For the functional hierarchy, we found that the six-layer model showed improved object-level discriminability relative to the baseline across all areas, along with a mild improvement in capturing inter-area differences (Fig. 2B,C), Although the six-layer model assigned higher discriminability to LM, RL, and AL than to VI. partially recovering the experimentally observed ordering for the Hoeller task, it still misranked AL below LM. In the Froudarakis task, performance remained flat: VI continued to exhibit greater discriminability than higher-orde order areas, contrary to physiological findings. Together, these results suggest that increased depth improves the overall quality of object representations, likely by enabling the learning of higher-order visual features. Yet, as shown in Fig. 2B,C. this improvement is not sufficient to reproduce the known functional hierarchy observed in mouse visual cortex.
Evaluation of a state-of-the-ard large-scale model. The foundation model of Wang et al. (2025) represents the current state of the art in mouse visual cortex modeling, with substantially more parameters (Table 1), neurons, and training data than typical architectures. We evaluated this model in our population-level framework to test whether scaling alone is sufficient to reproduce hierarchical organization (Supp. Fig. S3). At the single-neuron level, the model from Wang et al. achieved strong predictive accuracy (median CCnorm across sessions: [0.58-0.76)) At the population level, it captured the broad distinction between V1 and higher areas but failed to reproduce finer differences across higher-order regions. In both the Hoeller (Hoeller et al., 2024) and Froudarakis (Froudarakis et al., 2020) tasks, AL, RL, and LM were predicted to perform nearly identically, inconsistent with experimental data showing a clear AL advantage (Supp. Fig. S3). These results suggest that larger and more complex architectures, even when trained on more data, do not by themselves recover the functional hierarchy of mouse visual cortex.
5 A biologically inspired hierarchical readout capture differences across visual areas
Hierarchical readout improves performance and functional hierarchy. The baseline model assumes that neural responses across the visual cortex can be described as projections from a shared high-dimensional embedding of visual stimuli, extracted from the final layer of the core. However, this assumption is inconsistent with known anatomical and functional features of the visual system. For instance, V1 is the primary driver of LM (Fig. 1A), suggesting that LM's neural representation is not just an independent projection of shared embedding but transformed version of V1 representations. A similar principle applies other visual areas, like RL and AL, which are positioned differently within the visual hierarchy (Fig. IA). These observations motivate an alternative digital twin architecture, in which the core network progressively transforms visual stimuli through a sequence of layers, and neurons from each area are read out from different depths along this transformation, consistent with their anatomical positions. We refer to this architecture as the hierarchical readout model (details in Appendix A.4). As previously noted, single-neuron prediction accuracy in the baseline model increased with depth up to six layers and then plateaued (Fig. 2A and Fig. 3A). We therefore used an 8-layer core to ensure VI was read out at its optimal depth (layer 6), with readouts for LM and RL. assigned to the same layer (layer 7) due to their similar anatomical positions, and AL to layer 8, consistent with its hierarchical position (Fig. 1A).
The hierarchical readout model trained successfully and achieved single-neuron prediction accuracy that matched or exceeded that of the previously tested architectures (Fig. 2A and Fig. 3A). Notably, unlike the baseline 8-layer model with readouts from the final layer alone, the hierarchical model showed improved performance in predicting responses in AL compared to the 6-layer baseline. This
7

---

A
Model performance per area
OCNort
baseline
0.6
basatine (8 luyers)
Mirrarctucal (anatoauy)
0.5
V1
LM
RL
B
Receptive field estimation
Receptive field area idey")
Hoeller Task Classincation. Accuracy
Froudarakis Task
Mutaal information
0.5
1000
0.4
0.2
0.3
0.1
-0.5
0.0
-0.5
0.0
-0.5
0.0
C
Anatomical hierarchy score
Anotomical hiererchy score
Anatomical hierarchy score
Model alegmment
(Kondall's rank com))
0.5
0.5
0.5
0.0
0.0
0.0
-1.0
1.0
1.0
Models
Modals
Models
Figure 3: The hierarchical model captures gradient in invariant object classification along visual areas. (A) Single-neuron prediction performance across areas for the baseline model (6 or 8 layers) and the hierarchical model. In baseline models (top), neural responses are predicted from the final layer of shared core network. In the hierarchical model (bottom), each area is predicted from different layer, selected based on its anatomical hierarchy score. (B, left to right) RF size, classification accuracy in the Hoeller task, and object discriminability the Froudarakis task, each plotted as function of anatomical hierarchy score (dots) with best linear fits (dashed lines), shown separately for the baseline and hierarchical models. (C) Agreement between preel-predicted and experimentally observed orderings in (B), as quantified by Kendall's 7 rank correlation coefficient.
suggests that distributing readouts across layers in accordance with anatomical hierarchy facilitates the learning of area-specific neural representations. This architectural change also improved the model's 's ability to capture the functional hierarchies observed experimentally in receptive field size (Fig. 3B,C) and object recognition tasks (Fig. 3B,C) In the hierarchical readout model, receptive field size increased monotonically along the cortical hierarchy, in agreement with experimental data. In both the Hoeller and Froudarakis tasks, the hierarchical model not only achieved the highest recognition accuracy across all tested models (Fig. 3B.C) but also exhibited a gradient of decoding performance across areas that mirrored experimental findings (Fig. 3B, C): accuracy increased from V1 to RL and LM, peaking in AL. Interestingly. the model placed RL in an intermediate position (between AL and LM) both discrimination tasks. This result is consistent with Hoeller et 'sss findings, but at odds with Froudarakis et al.'s results. We were unable to reconcile this discrepancy and will address it in the discussion.
Aligning readout structure with anatomical hierarchy improves functional predictions. We next asked whether the specific anatomical ordering of hierarchical readouts was critical for the observed improvements. In principle, the better aligmment could simply arise from distributing the readouts across difffrrnnt layers, rather than placing them all in the final layer, in the baseline 8-layer model, thereby increasing representational flexibility, without requiring alignment to the actual cortical hierarchy. To disentangle these possibilities, we trained set of control models in which we systematically permuted the assignment of the visual areas readouts to the last three layers (details in Appendix A.4). For example, in one such configuration, AL was assigned to layer 6, V1 to layer 7, and LM/RL to layer 8. To quantify the degree of alignment between readout assignment and the known cortical hierarchy (VI<LM ŞRIAAL, Fig. IA), we computed Kendall's J between the readout layer indices and the anatomical order. nenglo-neuron prediction accuracy in AL increased

---

A Receptive field estimation
B Hoeller Task
C Froudarakis Task
1.0
1.0
1.0
Modol-data alignment
[Kendall's 1 rank corr)
2o.41
p=5.9p-27 Me4
p-1 88-32 xiee
Ganosemyy
0.5
0.5
0.5
AL LIURL
VL LMBRL
ALLLMOLLAS
EMAGALAT
0.0
0.0
0.0
EMALAV
-0.5
0.5
-0.5
-1.0
1.0
1.0
1
1
Architecture anatomy allgument (Kendall's rank corr)
Figure 4: Impact of architecture--aaaooy alignment on model performance. Agreement between various model-predicted and experimental observables orderings in different tasks, function of aadoutthheearchy alignment, quantified by Kendall's т between readout layer indices and the anatomical order (V1 < LM/RL= < AL). Each dot corresponds to different permutation of area-to- layer assignments across the last three layers of the network. Dashed lines indicate best linear fits. (A) RF size; (B) decoding accuracy (Hoeller task); (C) object discriminability (Froudarakis task).
systematically with the degree of aligmment (Supp. Fig. S2), as well as in LM that recorded a minor positive trend. For VI and RL, performance remained relatively stable across configurations. At the functional level, models with more anatomically aligned readouts exhibited stronger agreement with experimental data, especially in object recognition tasks (Fig. 4). These findings demonstrate that hierarchical readout is critical for capturing cortical functional organization, and that anatomical alignment of readout positions provides a powerful inductive bias for learning biologically plausible hierarchical representations in digital twin models.
Alternative configurations of RL and LM readout depth. To test whether our results depend on the assumption that LM and RL share the same readout depth, we trained two additional model variants. First, we developed a 9-layer model in which each visual area was assigned to a distinct depth, following the anatomical order V1 < LM < RL < AL. In this design, V1 read out from layer 6, LM from layer 7, RL from layer 8, and AL from layer 9. This variant produced similar overall results, maintaining consistent single-neuron accuracy and a hierarchical structure of receptive fields and object representations (Supp. Fig. S3). Second, motivated by evidence for dorsal/ventral stream separation in mouse visual cortex (Harris et al., 2019; D D'Souza et al., 2022), we implemented a two-stream architecture. After layer 6, the network split into ventral branch with a dedicated LM readout at layer 7, and a dorsal branch with RL and AL readouts at layers 7 and 8, respectively. This variant yielded similar prediction accuracy and maintained the global ordering of inter-area relationships (Supp. Fig. S3). Together, these results confirm that our main findings are robust to alternative architectural choices for LM and RL readout depth, and that grouping them in the main model is reasonable simplification.
6 Discussion and limitations
Our work demonstrates that incorporating biologically inspired hierarchical readout significantly enhances the ability of digital twins to replicate population-level phenomena in the visual cortex, particularly the emergence of invariant object representations. While a sufficiently large model, given enough training data, might achieve similar performance without a hierarchical readout, our architecture accomplishes this more efficiently in terms of both data and parameters. These findings contribute to a growing body of evidence that anatomical organization serves a powerful inductive bias for neural network models of the brain (Kubilius et al., 2019; Margalit et al., 2024).
In our architecture, responses in each visual area are computed from shared convolutional core, with parallel linear readouts from different layers. The core hierarchical and cascaded, with deeper layers (e.g., layer 8 for AL) building on earlier ones (e.g., layer 7 for LM/RL). We do not, however, explicitly model inter-area communication as biological circuits, where the output of one area informs the next. This simplification reflects current data-driven practices: routing activity from one readout to another might be more realistic but is unlikely to substantially change our results, as the shared core already approximates such computations. Moreover, because only subset of

---

neurons is observed in each area, constraining downstream areas to rely solely recorded upstream activity could limit performance. Allowing readouts to access shared internal representations instead yields more complete embedding and improves generalization. Future work could explore modular architectures with explicit inter-area pathways to better capture distributed processing.
Hoeller et al. (2024) showed that invariant object recognition in mouse visual cortex and deeper layers of AlexNet relies on different underlying representations. In both biological and artificial networks, object representations become increasingly invariant to rotation along the hierarchy. However, the mouse visual system also preserves equivariant representations, where geometric transformations in the visual input lead to corresponding transformations of neuronal representations, whereas AlexNet appears achieve invariance by suppressing these equivariant responses. Other studies have further shown that invariance and equivariance reflect not only network architecture but also training objectives and data, with task-driven models exhibiting mirror-symmetric viewpoint tuning (Cheon et al., 2022) and even randomly initialized networks displaying invariance properties (Farzmahdi et al., 2024). Our hierarchical model reproduces the experimentally observed increase in rotational invariance, but it remains unclear whether it also preserves equivariance. Addressing this question requires layer-wise analysis of representational geometry and remains an important direction for future work.
Our findings also raise questions about the role of area RL in visual processing. Anatomically, RL ranks high in the cortical hierarchy- above LM and below AL-and performs well in the object recognition task of Hoeller et al. (2024), yet it ranks lowest in the task of Froudarakis et al. (2020). This discrepancy may reflect differences in experimental conditions, such as the range of transformations or temporal structure. However, our model, which should be able to predict neural responses across conditions, predicts similar performance for RL across both tasks and hence fails to reproduce the result of Froudarakis. This suggests that task-level differences alone may not explain the experimental discrepancy. One possibility is that RL supports distinct processing stream that is selectively engaged depending on task demands. Interestingly, anatomical and physiological studies have reported evidence for two partially segregated pathways in mouse visual cortex separating LM and RL (Wang et al., 2012: D' O'Suuza et al., 2022). We tested this hypothesis in our dual- stream variant, which produced similar RL behavior, indicating that anatomical separation alone is insufficient. Instead, the divergent functional roles of RL may depend on additional factors not captured by current framework, such as feedback from higher-order regions, behavioral state, or temporal integration across stimuli.
To evaluate representational hierarchy, we focused on two established metrics: receptive field size at the single-neuron level and invariant object recognition at the population level. While these are widely used indicators of cortical hierarchy, they capture only part of the full picture. Electrophysiolooical studies have revealed additional systematic variations across areas, such as differences spatial and temporal frequency tuning (Glickfeld & Olsen, 2017), which future work could incorporate. At the population level, further studies might examine how cortical circuits support other computations beyond object recognition, such spatial integration or motion processing. Although experimental data for these metrics remain limited, models like ours can help generate hypotheses and guide future experimental design.
Despite the incorporation of biologically inspired constraints, our model omits several important features of the visual system. It is entirely feedforward, lacking recurrence or feedback mechanisms that are known to shape neural responses, particularly higher-level visual areas. Additionally, the model bypasses early sensory processing stages, such as those performed by the retina and lateral geniculate nucleus. Addressing these limitations will be crucial for closing the gap between digital twins and real cortical circuits, further refining both models and experimental approaches.
10

---

References
Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34: 25164 25164-25178, 25178, 2021.
Mohammad Bashiri, Luca Baroni, Ján Antolik, and Fabian H. Sinz. Learning and aligning single- neuron invariance manifolds in visual cortex. In International Conference on Learning Represen- tations (ICLR), 2025.
Max F Burg, Thomas Zenkel, Michaela Vystrčilová, Jonathan Oesterle, Larissa Höfling, Konstantin Willeke, Jan Lause, Sarah Müller, Paul G Fahey, Zhiwei Ding, et al. Most discriminative stimuli for functional cell type clustering. ArXiv, pp. arXiv-24011 2024.
Santiago Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas $ Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology. 15(4):c1006897, 2019.
Jeonghwan Cheon, Seungdae Back, and Se-Bum Paik. Invariance of object detection in untrained deep neural networks. Frontiers in Computational Neuroscience, 16:1030707, 2022.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
MICrONS Consortium, J Alexander Bac, Mahaly Baptiste, Caitlyn A Bishop, Agnes L Bodor, Derrick Brittain, JoAnn Buchanan, Daniel J Bumbarger, Manuel Castro, Brendan Celii, al. Functional connectomics spanning multiple areas of mouse visual cortex. BioRxiv, pp. 2021-07, 2021.
Saskia EJ de Vries, Jerome Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng. Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. large-scale, standardized physiological survey reveals higher order coding throughout the mouse visual cortex. BioRxiv, pp. 339513, 2018
James J DiCarlo, Davide Zoccolan, and Nicole Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434, 2012.
Zhiwei Ding, Dat T Tran, Kayla Ponder, Erick Cobos, Zhuokun Ding, Paul G Fahey, Eric Wang, Taliah Muhammad, Jiakun Fu, Santiago A Cadena, et al. Bipartite invariance in mouse primary visual cortex. bioRxiv, 2023.
Luciano Dyballa, Andra M Rudzite, Mahmood S Hoseini, Mishek Thapa, Michael Stryker, Greg Field, and Steven W Zucker. Population encoding of stimulus features along the visual hierarchy. Proceedings of the National Academy of Sciences, 221(4):233177712.. 2024.
Rinaldo D D' Souza, Quanxin Wang, Weiqing Andrew M Meier, Henry Kennedy, Kenneth Knoblauch, and Andreas Burkhalter. Hierarchical and nonhierarchical features of the mouse visual cortical network. Nature communications, 13(1):503, 2022.
Alexander S Ecker, Fabian Sinz, Emmanouil Froudarakis, Paul G Fahey, Santiago A Cadena, Edgar Y Walker, Erick Cobos, Jacob Reimer, Andreas S Tolias, and Matthias Bethge. A rotation-equivariant convolutional neural network model primary visual cortex. arXiv preprint arXiv:1809.10500 2018.
Amirhossein Farzmahdi, Wilbert Zarco, Winrich A Freiwald, Nikolaus Kriegeskorte, and Tal Golan. Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks. Elife, 13:e90256, 2024.
Emmanouil Froudarakis, Uri Cohen, Maria Diamantaki, Edgar Y Walker, Jacob Reimer, Philipp Berens, Haim Sompolinsky, and Andreas $ Tolias. Object manifold geometry across the mouse cortical visual hierarchy. BioRxiv, pp. 2020-08. 2020.
Lindsey I. Glickfeld and Shawn Olsen. Higher-order areas of the mouse visual cortex. Annual review of vision science, 3(1):251-273, 2017.

---

Ian Goodfellow, Honglak Lee, Quoe Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. Advances neural information processing systems, 22, 2009.
Julie A Harris, Stefan Mihalas, Karla E Hirokawa, Jennifer D Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, et al. Hierarchical organization of cortical and thalamic connectivity. Nature, 575(7781): 195:)))))199000 2019.
Judith Hoeller, Lin Zhong, Marius Pachitariu, and Sandro Romani. Bridging tuning and invariance with equivariant neuronal representations. bioRxiv, pp. 2024-08, 2024.
Larissa Höffing, Klaudia P Szatko, Christian Behrens, Yuyao Deng, Yongrong Qiu, David Klindt, Zachary Jessen, Gregory W Schwartz, Matthias Bethge, Philipp Berens, et al. A chromatic feature detector in the retina signals visual context changes. bioRxiv, pp. 2022-11, 2022.
Kohitij Kar and James J DiCarlo. The quest for an integrated set of neural mechanisms underlying object recognition in primates. Anmual Review of Vision Science, 10, 2024.
David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating "what" and "where". Advances neural information processing systems, 30, 2017.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong. Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy Roy, Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Bryan M Li, Isabel M Cornacchia, Nathalie L. Rochefort, and Amo Onken. VIt: large-scale mouse v1 response prediction using vision transformer. arXiv preprint arXiv:2302.03003,, 2023.
Dario Liscai, Emanuele Luconi, Alessandro Marin Vargas, and Alessandro Sanzeni. Beyond single neurons: population response geometry in digital twins of mouse visual cortex. In The Thirteenth International Conference on Learning Representations, 2025.
Konstantin KonstantinKKlemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay K Jagadish, Eric Wang, Edgar Y Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas S Tolias, et al. Generalization in data-driven models of primary visual cortex. BioRxiv, pp. 2020-10, 2020.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron, 2024.
Tomonari Murakami, Teppei Matsui, and Kenichi Ohki. Functional segregation and development of mouse higher visual areas. Journal of Neuroscience, 37(3 0401144444449..... 2017.
Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L Gardner, Anthony M Norcia, and Daniel LK Yamins. Mouse visual cortex as a limited resource system that self-learns an ecologically-genee representation. PLOS Computational Biology, 19(10)e1011506, 2023.
Eugenio Piasini, Liviu Soltuzu, Paolo Muratore, Riccardo Caramellino, Kasper Vinken, Hans Op de Beeck, Vijay Balasubramanian, and Davide Zoccolan. Temporal stability of stimulus representation increases along rodent visual cortical hierarchies. Nature communications, 12(1))4448, 2021.
Galen Pogoncheff, Jacob Granley, and Michael Beyeler. Explaining v1 properties with a biologically constrained deep learning architecture. Advances in Neural Information Processing Systems, 36: 13908-13930, 2023.
Shahd Safarani, Arne Nix, Konstantin Willeke, Santiago Cadena, Kelli Restívo, George Denfield, Andreas Tolias, and Fabian Sinz. Towards robust vision by multi-task learning on monkey visual cortex. Advances in Neural Information Processing Systems, 34:739-751, 2021.
Oliver Schoppe, Nicol S Harper, Ben DB Willmore, Andrew J King, and WH Schnupp. Measuring the performance of neural models. Frontiers in computational neuroscience, 10:10, 2016.
12

---

Tania A Seabrook, Timothy J Burbridge, Michael C Crair, and Andrew D Huberman. Architecture, function, and assembly of the mouse visual system. Annual review of neuroscience, 40(1)):99-553) 2017.
Joshua H Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy, Nature. 592(7852):86-922 2021.
Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. Advances in neural information processing systems, 31, 2018.
Ghislain St- Yve,, Emily J Allen, Yihan Wu, Kendrick Kay, and Thomas Naselaris. Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations. Nature communications, 14(1 14)1)33329 2023.
Rudi Tong, Ronan da Silva, Dongyan Lin, Arna Ghosh, James Wilsenach, Erica Cianfarano, Pouya Bashivan, Blake Richards, and Stuart Trenholm. The feature landscape visual cortex. bioRxiv, pp. 2023-11, 2023.
Polina Turishcheva, Paul Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin Willeke, Mohammad Bashiri, Ruslan Baikulov, et al. Retrospective for the dynamic sensorium competition for predicting large-scale mouse primary visual cortex activity from videos. Advances in Neural Information Processing Systems, 37:1 19977-188719999 2024a.
Polina Turishcheva, Paul G Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin F Willeke, Mohammad Bashiri, Eric Wang, et al. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. ArXiv, pp. arXiv-2305, 2024b.
Ivan Ustyuzhaninov, Max F Burg, Santiago Cadena, Jiakun Fu, Taliah Muhammad, Kayla Ponder, Emmanouil Froudarakis, Zhiwei Ding, Matthias Bethge, Andreas S Tolias, et al. Digital twin reveals combinatorial code of non-linear computations in the mouse primary visual cortex. bioRxiv, pp. 2022-022 02, 2022.
Edgar Y Walker, Fabian Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what excites neurons most using deep predictive models, Nature neuroscience, 22(12): 2060-2065, 2019.
Eric Y Wang, Paul G Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, et al. Foundation model of neural activity predicts response new stimulus types. Nature, 64008058):440-477, 2025.
Quanxin Wang, Olaf Spors, and Andreas Burkhalter. Network analysis corticocortical connections reveals ventral and dorsal processing streams in mouse visual cortex. Journal of Neuroscience, 32 (13):4386-4399, 2012.
Konstantin F Willeke, Paul G Fahey, Mohammad Bashiri, Laura Pede, Max F Burg, Christoph Blessing, Santiago A Cadena, Zhiwei Ding, Konstantin-Klenan Lurz, Kayla Ponder, et al. The sensorium competition predicting large-scale mouse primary visual cortex activity. arXiv preprint arXiv:2206.08666 08666,, 2022.
Konstantin F Willeke, Kelli Restivo, Katrin Franke, Ame F Nix, Santiago A Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander S Ecker, et al. Deep learning-driven characterization of single cell tuning in primate visual area v4 unveils topological organization. bioRxiv, pp. 2023-05. 05, 2023.
Aiwen Xu, Yuchen Hou, Cristopher Niell, and Michael Beyeler. Multimodal deep learning model unveils behavioral dynamics of v1 activity in freely moving mice. Advances in Neurul Information Processing Systems, 36, 2024.
13

---

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optiminn hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111(23):8619-8222 2014.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part I 13, pp. 818 18-33.3..5 Springer, 2014.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 2103)9101111 18, 2021.
14

---

Supplementary information
A Model Architecture, Training, and Evaluation
All code used for data processing, model training, and analysis publicly available at https: //github Con//NeuroBLab/// pttllaa/aaattmicalll.... wnwppniddddiieeee diigitalltiinn
A.1 Baseline model architecture
The model architecture consists four primary components: a core component, a behavioral module, readout component, and shifter module. The model takes videos and behavioral variables (pupil size, pupil center coordinates, running speed) as inputs. Videos are processed by the core component, returning embedded representation of the visual stimuli. Behavior is transformed by the behavioral module into a latent representation which is stacked to the output of the last layer of the core. This representation is passed to the "readout", which determines the spatial position for each neuron and linearly combines the representation in the learned position with set of learnable feature weights. The spatial position is further adjusted by the "shifter" module, which takes the pupil center's 's coordinates as input. Predicted responses are then generated frame by frame by applying a final nonlinearity (ELU+1) maintain non-negativity. More details about each element are provided below.
Inputs. The inputs to the neural network model are:
1. Videos (Tensor V): A 5-dimensional tensor with shape (Batch, Channels, Frames, Height, Width), with Channels=1 (gray-screen movies), Height=36 and Width =64 pixels, respec- tively.
2. Behavior (Tensor B): a 3 3-dimensional tensor with shape (Batch, Channels. Frames) where Channels=4 represent pupil size, pupil center coordinates and running speed.
3. Pupil center (Tensor P): A 3-dimensional tensor with shape (Batch, Frames, Coordinates), where Coordinates=2 are the x. y positions of pupil center.
The input resolution (36x64 pixels) represents a trade-off between biological plausibility and com- putational efficiency, aligning with the limited visual acuity of mice and supported by prior work showing that lower-resolution training can improve predictivity (Nayebi et al., 2023). Performance remains comparable higher-resolution models trained on the same dataset (Wang et al., 2025).
Core. The core component, inspired from (Höfling et al., 2022), is composed of 4 layers of 3D Convolutional Network with 16, 32, 64, and 128 channels, respectively. The 3D convolutions are factorized into spatial (2D) and temporal (ID) dimensions. Batch normalization is applied between the two convolutions. Spatial kernels are 11 x 11 in the first layer and 5 x 5 in subsequent layers, while temporal kernels are 11 x in the first layer and x thereafter. Padding was added to both 2D and 1D convolutions keep the original spatial dimensions unchanged and to maintain temporal causality, respectively. Batch normalization and an ELU + non-linearity are applied after each layer. This component is shared across all recording sessions (see Appendix A.2).
This baseline design follows the Sensorium 2023 architecture (Turishcheva et al., 2024b), chosen for its suitability to dynamic visual stimuli and its status the strongest available benchmark for mouse visual cortex. Earlier models used shallower CNNs with fewer channels (Lurz et al., 2020; Sinz et al., 2018; Klindt et al., 2017), making this architecture natural starting point. We later examine deeper variants (6- and 8-layer cores) to assess whether increased expressivity alone can capture hicrarchical transformations (see Section A.4).
Behavioral module. The behavioral module, inspired by (Li et al., 2023) consists of 2-layer (4-128-32 units) multilayer perceptron (MLP), with Tanh and Dropout applied after each layer. A ID convolution with kernel size is applied along the temporal dimension, with pooling applied to enforce causality. The representation generated (a 32 channels vector frame-by-frane is then stacked to the output of the core component. This component is shared across all recording sessions (see Appendix A.2).
15

---

Readout, To compute single responses during stimulus presentations, we adopted the readout described by (Lurz al., 2020). Each neuron associated with learnable spatial position Pn. e [-1,1]², used for bilinear interpolation at the specified positions in the core's 's output. This feature vector then linearly combined with learnable feature weight Wnn E RI60 (128 channels coming from the core and 32 channels coming from the behavioral module) and a bias term to predict neuron activity frame-by-frame. A final (ELU + 1) non-linearity ensures non-negative responses. Each recording session is paired with its corresponding readout module.
We adopted the Gaussian spatial readout (Lurz et al., 2020), which has proven effective in previous large-scale mouse modeling studies. The number of features per neuron is fixed by the dimensionality of the accessed core layer (128 in our baseline). Larger feature counts, used in foundation models (Wang et al., 2025), offer only marginal gains, suggesting our choice is sufficient and not a critical parameter for the results.
Shifter. The shifter is a multilayer perceptron (MLP) with three layers (2-5-5-2 units) (Sinz et al., 2018). It maps the pupil center coordinates to a shift applied to the neuron positions Pa. A Tanh nonlinearity at each layer ensures the output stays within the range E []², Each recording session paired with its corresponding shifter module.
A.2 Training
Training dataset. We trained the model on publicly available functional calcium imaging data from the MICIONS dataset (Consortium et al., 2021), which integrates large-scale, multimodal functional recordings with structural reconstructions of the mouse visual cortex. All acquisition details, stimulus protocols, and preprocessing steps are comprehensively described in the original MICIONS publication (Consortium et al., 2021). Here, we report the key aspects relevant for reproducing our results.
Functional recordings were obtained using two-photon random access mesoscopy, capturing the activity of approximately 75,000 excitatory neurons across four cortical areas: the primary visual cortex (VISp) and three higher-order visual areas (VISIm, VISrI, and VISal). The dataset consists of 14 recording sessions from single transgenic mouse passively viewing visual stimuli. Visual stimuli consisted of diverse gray-scale natural and parametric videos, including cinematic movie clips, naturalistic sports videos, and rendered 3D scenes, designed to probe broad range of neuronal tuning properties. Behavioral data were recorded simultaneously and aligned with neural activity. During imaging, the mouse was head-fixed but free walk on a treadmill. Behavioral variables were precisely tracked and synchronized with neural recordings, including running speed measured via a rotary optical encoder (57-100 Hz) and eye movements recorded at 20 Hz using an infrared camera capturing pupil center coordinates and dilation.
We used all available sessions, which comprise 83,222 neurons from primary visual cortex (V1), 14,817 neurons from lateral medial area (LM), 12,599 from rostrolateral area (RL), and 4,734 from anterolateral area (AL). All videos were isotropically downsampled to resolution of 36 x 64 pixels per frame. Functional and behavioral signals were resampled to 30 Hz by linear spline interpolation. Visual stimuli were normalized by subtracting the mean and dividing by the standard deviation across all recording sessions. Behavioral data, and pupil center coordinates were normalized by subtracting the mean and dividing by the standard deviation of each recording session. Neural responses were standardized to ensure non-negativity.
Loss Function and Training Details. Batches consisted of 150 consecutive frames, randomly sampled from 300-frame trials. Training used batch size of 2. Learning rate was set to 0.005. AdamW (with default hyperparameters) was employed for parameter optimization, minimizing the negative Poisson likeliool loss:
Th
The Pit oo.log(rr.))
Poisson
between the recorded responses o and predicted responses where is the number of frames in one batch and nh. is the number of neurons for recording session 8. Parameter updates were applied after completing one full pass through all sessions, strategy previously found to enhance performance (Li et al., 2023). Training employed a learning rate schedule with línear warm-up during the first 10 epochs. After each epoch, we computed the correlation to average (see below) between predicted and
16

---

observed responses on a validation set averaged across all neurons (excluding the initial 50 frames). The validation set is composed by videos which were repeated 10 times during each recording session and held-out during training. If this score did not improve for eight epochs, training was halted, and the model was reverted to the checkpoint with the highest validation score. The learning rate was then reduced by a factor of 0.3 unless the maximum number of decay steps (n = 4) had been reached, in which case training was terminated. Each model described was trained with three different random seeds.
A.3 Evaluation Metrics
Model evaluation used metries: correlation average and normalized correlation coefficient nomm The former is calculated as follows:
- -
avg. (r, - 0)2
(S1)
where F; = =E Fe,j is the average response across repeats. The latter defined as (Schoppe al., 2016)
CC
CC
CCmux
where abs and C. are defined as:
Cov(r, Ø)
NVr((ō) Var(o) (N-1)) Var(o)
CC
CCmax
Var())
Here, where is the predicted response and o is the observed response to N repeated stimuli; ō and F are the average recorded and predicted responses across all trials. All plots showing single-neuron prediction performance report the median C'Cnarm across neurons for each session averaged over all the sessions, evaluated on the validation set.
A.3.1 Kendall's definition
Kendall's Tau is a rank-based statistic used to assess the ordinal relationship between two variables, specifically quantifying the degree of agreement or disagreement in their relative rankings. It is particularly useful when analyzing data with ordinal scales or when the assumption of linearity in Pearson's 's correlation not valid. The statistic is defined as:
P-Q
T
+Q+U)
where:
P represents the number of concordant pairs, which pairs of observations (æ,, v:) and Mj) where the relative order of both variables is the same, i.e., if x; < &j. then yi < yj
(or if Ij. then y/: yj).
Q represents the number of discordant pairs, where the relative order of the two variables is reversed, i.e., if x; < Tjj then y: > yj. or if x; > Tj. then y: < Mj--
T is the number of tied pairs in the first variable,
U is the number of tied pairs in the second variable.
Kendall's Tau measures the balance between concordant and discordant pairs, with values ranging from - to +1. A value of r = 1 indicates perfect agreement between the two variables' rankings, while -1 indicates perfect disagreement. A value of 0 suggests association between the variables' rankings.
A.4 Model variations
To systematically investigate the influence of architectural choices model performance, particularly concerning the representation of invariant object recognition across visual areas, we developed several model variations:
17

---

Area-balanced loss function. Given the substantial imbalance in neuron counts across visual areas, standard training would bias the model toward optimizing VI predictions at the potential expense of higher visual areas. We addressed this by implementing an area-normalized loss function where each area's contribution scaled by the inverse of its neuron count. This weighting scheme ensures each visual area contributes equally to the loss function regardless of neuron count, preventing the model from optimizing predominantly for VI. This method resulted in reduced loss magnitude which had to be balanced by an increase in the learning rate. We searched the optimal learning rate training models with a single session (5 (__6 to allow extensive experimentation.
All-layer readout model. We explored readout that instead of looking only at the output of the last layer of the core can read from representations extracted at different depths. possibly extending the model's capability to capture visual features at multiple levels of abstraction. In this variant, each neuron assigned learnable feature weights that are applied to output channels from all core layers. In order to encourage sparsity across channels, we explored various configurations of I.1 regularization (5, 10, 25, 50, 200) applied to the learned weights. To allow extensive search, the exploration single recording session (5_6). We found this model variant prone to overfitting suggesting that unconstrained multi-layer access does not provide the appropriate inductive bias for modeling the visual hierarchy.
performed using
Deeper network architectures. To investigate whether the hierarchical processing was limited by model expressivity, we developed variants of the baseline model with increased depth (6 and 8 layers). We extended the baseline 4-layer architecture with two or four additional convolutional layers (with 256 and 128 channels in the 6-layer case and 256, 512, 256 and 128 channels in the 8-layer case). These deeper architectures were trained with the same protocol the baseline model, allowing to assess whether increased depth alone could capture hierarchical transformations without explicit anatomical constraints.
Varying the depth of the convolutional core affects the number of internal parameters, but the number of readout parameters was kept constant across all model variants (including both hierarchical and non-hierarchical readouts). This design choice allows us to isolate the effect of core depth without conflating with decoder complexity. The distribution of parameters across model variants is shown in Table 1.
A A biologically inspired readout scheme that mirrors cortical anatomy. Each visual area is modeled as reading out from a specific depth in the network, reflecting their position in the anatomical hierarchy. We built 8-layer model (since the 6-layer model showed the best performances with no significant improvements with deeper networks) where V1 neurons read out from the 6th layer, LM and RL from the 7th and AL from the last one. This constraint enforces inputs to be processed following the cortical anatomy, enabling each area learn tailored set of filters with more freedom and to benefit from representations generated for lower-level areas. To ensure a fair comparison with other models, each area still read from an output consisting of 128 channels (resulting in 16-32-64- 128-256 128- 128- 128 channels for the corresponding layers).
hierarchical readout. Our key proposed model implements
Control hierarchical readouts. To verify that performance improvements from our hierar- chical model derive from meaningful anatomical constraints rather than merely from the assignment of areas to different layers, we implemented the following control: for each block (V1, LM/RL, AL) we permuted the order with which they were assigned to the last three layers (for example, AL being assigned to the 6th layer, V1 to the 7th layer and LM/RL the 8th layer). These control models help distinguish between benefits arising from anatomically meaningful constraints versus those stemming merely from the flexibility of assigning different areas different layers.
18

---

Table I: Parameter counts across model variants. Core parameters vary with depth, while readout parameters are held constant.
Model Variant
Core Parameters
Readout Parameters
Baseline (4 layers)
382,512
17,396,557
Baseline (6 layers)
2,432,816
17.396,557
Baseline (8 layers)
10,629,424
17.396,557
Hierarchical (8 layers)
3,417,392
17.396,557
Wang et al. (2025)
4,528,640
54,377,262
B Repetition of experiments to probe functional hierarchy
B.1 Receptive field extraction and analysis
Estimation of artificial receptive fields. We estimated artificial receptive fields (RFs) of units in trained models following the procedure introduced by Li et al. (2023). Each trained model was presented with N = 500,000 white noise images sampled from a uniform distribution. The receptive field ORF, of unit unit's 's response to each image:
computed as the weighted sum of all input images, with weights given by the
N F(an): ani on ~21 x 36 64),
ORF,
where F(zn); denotes the activation of unit in response input 2n. and 21(1 36 x 64) indicates sampling from a uniform distribution over the input shape.
Gaussian fitting and receptive field size. To characterize spatial properties of each receptive field, we fitted a 2D Gaussian function to extract its center location (mean) and spatial spread (covariance). smoothed using Gaussian filter, mean-centered, and
Prior to fitting, each receptive field map transformed by taking its absolute value to improve fitting stability. Gaussian fitting was performed og By (where o
using SciPy's s surve_fito function. The receptive field size was defined and are the standard deviations along the two principal axes of the fitted Gaussian).
Conversion from pixels to degrees of visual angle. To convert receptive field sizes from pixels to degrees of visual angle, we applied a conversion factor calculated based on the spatial resolution and viewing conditions of the input stimuli given in (Consortium et al., 2021). Visual stimuli were presented to the left eye on 31.8x56.5cm (heightxwidth) monitor with a resolution of 1,080x ,,920 pixels, positioned 15cm away from the eye. Given this setup, each pixel corresponds approximately 0.88 em on the screen. Assuming the mouse fixates on the center of the screen and the eye is perpendicular the screen, the visual angle subtended by one pixel calculated as:
0.88
0 = arcsin
= 00.056 radians ~ 3.36".
V0.882 152
Because receptive field size is computed the product of standard deviations along two axes (i.e., an area), the size in degrees squared is obtained by multiplying the pixel-based area (3.369)2.
19

---

Figure $1: Example receptive fields estimated by the model. Representative receptive fields obtained by aggregating white noise responses of individual units. Red contours indicate iso-response levels of the best-fitting 2D Gaussian at one and two standard deviations. These fits were used to estimate receptive field centers and sizes.
Selection of units for analysis First of all units were filtered based on their modeling accuracy: neurons with correlation to average below 0.35 were discarded to avoid producing model's artifacts. Furthermore, not all receptive fields yielded reliable Gaussian fits. We therefore excluded units based on two additional criteria: (1) the center of the fitted Gaussian lay outside the stimulus image boundaries, indicating poor localization; (2) either o₂ > 16pixels or ay > 14pixels, which filtered out outliers with implausibly large RFs. This selection ensured inclusion of units with well-defined and biologically plausible receptive fields. Figure S1 presents representative examples of accepted receptive fields.
Summary statistics and visualization For each visual area modeled, the average receptive field size was computed across all accepted units and used for subsequent analyses.
B.2 Hoeller task replication
To evaluate the capacity of our digital twin model to capture hierarchical improvements in rotation- invariant object recognition, we replicated the object classification task of Hoeller et al. (2024) using simulated neural responses generated by our model. While our analysis pipeline followed the structure described by Hoeller al., it was implemented independently.
Stimuli. We constructed a stimulus set analogous to that used by Hoeller et al. It comprised 92 static, grayscale images representing 92 distinct object classes, selected from copyright-fis sources using the COCO dataset classes as a reference. Images underwent preprocessing involving masking with circular aperture and histogram equalization. Each image was then rotated in-plane create 15 versions, spanning 360deg in uniform increments of Aø 24deg. For presentation to the digital twin, which operates on video inputs sampled at 30 fps, each static rotated image was embedded within a short video clip. Each clip consisted of an initial 50 frames (~~ 1.67s) of uniform gray screen, followed by 15 frames (~ 0.5s) where the static rotated image was displayed. This structure allowed for the analysis of responses specifically during the image presentation period, distinct from any preceding baseline or transient activity. Input stimuli were appropriately normalized based on statistics derived from the dataset used for training the digital twin.
20

---

Model responses. Digital twins trained on the MICTONS dataset were used to predict responses the rotated object video clips. For each stimulus presentation (a specific object identity a particular rotation angle), the model processed the corresponding 65-frrmm video clip. The resulting simulated neural activity trace for each neuron was then averaged over the final 15 frames, corresponding to the period of static image presentation only. This produced single scalar value representing the response magnitude of each neuron to that specific stimulus. Responses were generated for simulated neurons located in modeled visual cortical areas VI, LM, AL, and RL.
Reliable neuron selection. From the pool of simulated neurons in each visual area (V1, LM, AL, RL), we selected those exhibiting reliable responses. Reliability was defined based on the neuron' s performance during the original model training phase: we included neurons whose predicted activity on a held-out validation set showed a correlation coefficient greater than 0.35 with the trial-averaged predicted response for that neuron.
Preprocessing, For each selected neuron, the mean simulated response recorded during the initial 50 gray-screen frames was calculated and subtracted from the responses during the image presentation frames for baseline correction. Following this, the responses were standardized. Using only the data corresponding stimuli designated training classes (defined per iteration, see below), the standard deviation of responses for each neuron was computed and used to z-score all responses (from both training and test classes). Finally, the mean response across these training classes was subtracted from all responses to mean-center the data relative to the training set.
Decoding analysis. The preprocessed, high-dimensional population response vectors for each visual area were projected onto a lower-dimensional subspace using Principal Component Analysis (PCA). The PCA was fitted using only the training class data, and the top 100 principal components, capturing the directions of highest variance, were retained for subsequent classification. A linear Support Vector Machine (SVM) classifier was trained to distinguish between the 92 object classes based on the 100- dimensional PCA representations. We used the implementation 'LinearSVC" from the scikit-learn library, configured with a regularization parameter C=1 and number of iterations equal 1000 to ensure convergence. To evaluate the classifier's 's ability to generalize to unseen rotations, we employed a 15-fold cyclic valssaaaio procedure based on the rotation angle, identical to that used by Hoeller et al. (2024). In each fold, the data corresponding to three consecutive rotation angles were held out. The linear SVM was trained using the model responses associated with the remaining 12 rotation angles, encompassing stimuli from all 92 object classes.
Performance and repetitions. The classifier's performance was then evaluated its ability to predict the identity of the stimuli belonging to the 2 designated test classes (see below) presented at the middle rotation angle of the three held-out angles. Classification performance for each fold was measured as the accuracy (fraction of correctly classified test stimuli). The overall accuracy for one iteration of the analysis was the average accuracy across the 15 folds. Since the performance depends on which 2 out of the 92 classes are chosen as test classes, the entire analysis procedure-including the random selection of 2 test classes, subsequent preprocessing, PCA fitting, SVM training, and testing was repeated 50 times. Each repetition used a different random seed to ensure different pairs of test classes were selected from the 92 classes The final reported accuracy for each visual area represents the mean accuracy + standard error of the mean (SEM) calculated across the six MICrONS sessions and three training seed for each model.
15-fold cross-validatie
B.3 Froudarakis task replication
To assess the digital twin's 's ability to support object identity discrimination based on dynamic visual input, mirroring the experimental logic of Froudarakis et al., we replicated their paradigm using simulated neural responses generated by our model in response to videos of objects undergoing continuous
transformations (rotation, translation, scale, illumination condition). Our stimuli and analysis pipeline were designed based on the description provided by Froudarakis et al. but implemented independently.
Stimuli. We generated a stimulus set comprising videos of four distinct chree-dimensional objects using 3D rendering in Blender, aiming to closely resemble those used in the original study by Froudarakis et al.. Each object identity was subjected to continuous, tions involving random variations in position (translation along X, Y, 7. axes), scaling (size variation), rotation (tilt), and environmental lighting conditions (intensity and position). These transformations
prresriy-r transforma-
21

---

resulted in smooth, coherent object motion against uniform gray background. For each of the 4 object identities, we rendered 500 unique video clips. Each clip was 10 seconds in duration, rendered at 30 frames per second (fps), resulting in 300 frames per clip. Input video stimuli were normalized based on statistics derived from the dataset used for training the digital twin, consistent with the procedure used for the Hoeller et al replication described above.
Model responses. We employed the same pre-trained digital twin model used in the previous experiment to predict neural responses. The model processed each 10-second (300-frame) video clip. To account for potential neural adaptation or onset transients. we excluded the simulated responses corresponding to the initial 30 frames second) of each clip from the analysis. The subsequent 270 frames of activity were then divided into 18 contiguous, non-overlapping temporal bins, each spanning 15 frames (500 ms). The simulated neural activity trace for each neuron was averaged within each 500 ms bin. This procedure yielded a sequence of 18 response vectors per video clip for each simulated neuron, where each vector represented the average activity level in a successive 500 ms interval. Responses were generated for simulated neurons located in the modeled visual cortical areas.
Reliable neuron selection. For each visual area (VI, LM, AL, RL), we first identified a pool of reliable simulated neurons. Reliability was assessed based on performance during the model's training phase: neurons were included if the correlation coefficient between their trial-averaged predicted response and the corresponding ground truth neural activity exceeded 0.35 on a held-out validation dataset.
Decoding analysis. Qualifying neurons were pooled together for each visual area. Prior to classi- fication, the neural response vectors underwent preprocessing steps within each fold of the cross- validation procedure described below. First, using only the data designated for training within a given fold, the mean and standard deviation of each feature (i.e., each neuron's 's response) were calculated. All features in both the training and testing sets of that fold were then standardized by subtracting the training set mean and dividing by the training set standard deviation (z-scoring).
For each repetition of the decoding procedure, a subset of 128 neurons was randomly selected without replacement from the pool of reliable neurons available for the target visual area, consistent with the population size sampled by Froudarakis et al. To classify the object identity based on the 128 selected neural responses, we employed multi-class logistic regression approach. Specifically, set of binary logistic regression classifiers were trained using a one-vs-rest strategy, where each classifier learned to distinguish one object class from the remaining three. These classifiers were optimized using an iterative numerical method to maximize the likelihood of the observed data under the logistic model. The optimization procedure did not include regularization penalty term on the classifier weights, but did fit an intercept term for each classifier.
A 10-fold stratified cross- vols-valiittion scheme was used to evaluate the generalization performance of the classifiers. The full dataset of 500 ms response vectors (pooled across all clips and objects) was randomly partitioned into 10 equally sized folds. Stratification ensured that the relative proportion of samples belonging to each of the four object classes was maintained within each fold. For each iteration of the vross-vali,,tion, one fold served as the test set, while the remaining nine folds were used for training the logistic regression classifiers (including the calculation of standardization parameters). This process was repeated 10 times, with each fold serving as the test exactly once.
Performance metric. The primary measure for object discriminability was the Mutual Information (MI), quantified in bits, between the true object labels (c) and the labels predicted by the classifier (7) on the held-out test sets across the 10 folds. MI was calculated from the aggregated confusion matrix C' (summed across folds) using the formula:
())))))) MI(c, 2) = ΣΣ 2(c), p(c:)p((,))
4 4
==1
where p(c,,C) represents the joint probability of true class and predicted class j (derived from the normalized confusion matrix), and p(c;) and p(č,) are the marginal probabilities (derived from the row and column sums of the normalized confusion matrix, respectively).
Repetitions and reporting. To ensure robust estimation of performance, the entire analysis proce- dure encompasnng the random subsampling of 128 neurons and the subsequent 10-fold stratified cross-validation with MI calculation-wii repeated 50 times for each visual area. Each repetition
22

---

utilized distinct random seed to ensure different neuron subsets and data partitions were used. The final reported discriminability value for each visual area is the mean MI computed across these 50 independent repetitions, presented as mean # standard error of the mean (SEM) calculated across the six MICrONS sessions and three training seed for each model.
C Implementation and implications
C.1
Computational resources
All experiments were run on two different infrastructures based on their computational demands. Each model (including baseline and hicrarchical variants) was trained on single NVIDIA A100 80GB PCIc GPU, paired with a dual-socket Intel Xeon Gold 5317 CPU (24 cores total). Training took approximately 3.8 days per model to reach convergence. Memory usage during training remained well within the 80GB VRAM limit. The receptive field estimation analysis was conducted using the same hardware model training (A100 GPU and Xeon CPU) and required approximately 12 hours per session with settings described above. Both the Hoeller and Froudarakis task replications were performed Xeon Gold 6248R CPU (48 threads) with 128GM RAM associated. Each run took approximately 2.5 hours, and most of the computation times with different random seeds and all experimental tasks were repeated for each trained model. Execution was parallelized where feasible.
machine with NVIDIA RTX A4000 (16GB) GPU and single-socket Intel
CPU-bound. All model trainings were repeated three
C.2 Broader impact
Our work represents a step toward developing more biologically informed digital twin models, enabling the study of invariant object recognition in silico. As these models improve, they can provide a powerful framework for conducting experiments that are not feasible in animal studies and for reducing the need for invasive animal experiments. Eventually, understanding of brain function deepens, they may become valuable tools for investigating the neural basis of neurological disorders and exploring potential treatments and interventions.
these models advance and
23

---

D Additional supplementary figures
A
B
AL
0.7
0.7
0.7
0.7
Pes Te-02
00-01
pe A wep
GCNom
A. LAGRE VI
LMEAL VI
p-B.2e
0.5
0.5
0.5
0.5
-1
0
-1
-1
-1
Architecture anatomy aligament (Kendall's rank
Figure S2: Prediction performance across areas function of readout-hierarchh alignment. Single-neuron prediction performance in each area plotted against the alignment between readout layer indices and anatomical area order (V1 < LM/RL < AL), quantified by Kendall's т. Each dot represents model defined by distinct permutation of area-to- area-to-layer assignments across the last three layers of the network (as in all models of Fig. 4). Dashed lines indicate best linear fits.
24

---

A
Model performance per area
hterarchical (anatomy)
CCNorm
hierarchical (two streares)
hierarchical (9 layer)
Wang (2025)
V1
LM
RL
AL Hoeller Task Classification Accuracy
Receptive field estimation Receptive field area (deg')
Froudarakis Task Mutual Information
B
0.3
0.5
1000
0.4
0.2
500
0.3
0.1
-0.5
0.0
0.5
0.0
0.5
0.0
C
Anatomical hierarchy score
Anatomical hierarchy score
Anatomical hierarchy score
1.0
1.0
1.0
Model-dala alignment
(Kendall's t rank corr)
0.5
0.5
1
0.5
0.0
0.0
0.0
-0.5
0.5
0.5
-1.0
1.0
1.0
Models
Models
Models
Figure S3: Comparison of alternative large-scale and hierarchical model architectures. (A) Single-neuron prediction performance across areas for the four tested architectures: the main hierar- chical model (8 layers), the 9-layer variant with distinct readout depths for each area, the two-stream variant separating ventral (LM) and dorsal (RI-AL.) branches, and the large-scale foundation model of Wang al. (2025). Note that the performance of the Wang et (2025) model may be influenced by how we entered behavioral variables into its publicly available implementation, which we used directly. Importantly, this does not affect the conclusions in-silico experiments were conducted without behavioral inputs. (B, left to right) Receptive- fielld size, classification accuracy the Hoeller task, and object discriminability plotted as a function of anatomical hierarchy score (dots) with best linear fits (dashed lines). (C) Agreement between model preeictee and experimentally observed orderings in (B), quantified by Kendall's rank-correlation coefficient. All alternative hierarchical variants reproduced the gradient in invariance and maintained consistent single-neuron accuracy across areas, whereas the Wang et al. (2025) model failed to recover the fine-scale hierarchical distinctions among higher-order regions.
the following analyses, as all other
the Froudarakis task, each
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The introduction clearly outlines the key contributions of the paper, which properly reflected the abstract. All main claims are substantiated by the experimental results presented, and their scope and limitations appropriately discussed in dedicated sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, limitations are discussed in the final section, Discussion and Limitations. We acknowledge several important constraints of our work, including the lack of recurrent or feedback connections in the model, the exclusion of early sensory processing stages, and the inability to fully capture experimental discrepancies such as the role of area RI. We also reflect on the dependence of our findings on specific datasets and experimental paradigms, the trade-off's imposed by architectural choices, and the need for future analyses (c.g.. of equivariance) clarify the mechanisms underlying disucssed results. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
26

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the appendix, we provide a detailed description of the model and training pipeline used, along with any modifications made to it obtain the different configurations discussed in the main text. Moreover, we describe all procedures necessary to replicate the experiments, including dataset processing and evaluation metries. While we do not release specific neuron IDs for visualization tasks, we follow standard practices and show that our results generalize across multiple sessions and random seeds. The code will be made available upon publication. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The experiments are based on the publicly available MICIONS dataset. Other data used are either collected from copyright-free sources (eg., https://ussplash.com generated by us and will be realeased upon publication. We will release also the full codebase upon publication, including instructions for data access and the reproduction of all main results. The appendix describes all necessary procedures to replicate the experiments, including model architecture, training configurations, evaluation metrics, and experimental pipelines.
or
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/gu Coaas//dSSSuubmsiiiicc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //iips.co.c wwtpc/cpwpwp.p.cceee/ee///c/ee/e wwwpc///gueeeeeeeeoeooecee hwtp///ooeeeesiiiieeee www.ssoliccyiosii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and testing details are provided. The appendix specifies the dataset used (specific recording sessions), data splits. preprocessing steps, model architecture. optimizer, batch size, training loss, and other hyperparameters choices. We also describe how model variants were constructed and evaluated, including reproducible pipelines for both receptive field estimation and classification tasks. These details are sufficient to understand and contextualize the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard error of the mean (SEM) across multiple independent training seeds and recording sessions for all key performance metries that support the main claims (receptive field size, object classification accuracy and discriminability, and Kendall's 7). The source of variability (across seeds and sessions) clearly stated. The methods section and appendix detail how error bars are computed and clarify what aspects of variability they reflect. Error bars are visually presented in all figures containing main results. Statistical significance is indicated in figures and criteria for p-value are clearly stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:W. We report detailed information about the computational resources required for all experiments in the appendix (Section "Computational Resources"). This includes GPU and CPU specifications, relevant memory specifications, and the time required for model training and each type of experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
29

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: wwweurrpss.cppuuliiee...c.. httie//teihcccdddeiieeeee
Answer: [Yes]
Justification: Our research adheres fully to the NeurlPS code of ethics. It uses publicly available datasets and introduces synthetic assets that do not involve human subjects. poses no foreseeable safety, security, or discrininatory risks. We clearly document our models and procedures to ensure full transparency.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: impact").
We discuss the broader impact of our work in the appendix (Section "Broader
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
30

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use the publicly available MICrONS dataset, properly cited, and clearly state its open access status in the methods. All code used in our work is either custom adapted from previously published open-source implementations, which all cited appropriately. Upon publication, we will release our code under an open-source license. Dataset licenses and terms of respected in full accordance with their publications.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce two new assets: an anatomically inspired hierarchical model architecture, and set of four synthetic video stimuli of 3D-rendered objects undergoing identity-preservve transformations, created to replicate the Froudarakis et al. experiment. Both assets are descrihed in detail in the main text and appendix. Upon publication, will release the code and rendered videos.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
31

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https / ///ereurip...// cttpps.ccc///oo fcc/oonferencee/222////////////e for what should or should not be described.
32


Reviews:
[{"review_id": "kuJnuiwpEs", "reviewer": "Reviewer_JMCZ", "summary": "This work examines whether data-driven models of neural responses in mouse visual cortex reproduce observations from the brain regarding invariance of object representations. A baseline model predicting all units from the same layer achieves good single-unit correlations, but fails to capture the hierarchy of invariant object representations across brain areas. This shortcoming persists under several changes of the backbone architecture. Finally, reading out each brain area from its own model layer, according to its position in the hierarchy along the visual pathway, leads to a model that is more consistent with brain data at the population level while retaining single-unit performance.", "questions": "1. Perhaps the anatomical hierarchy score shown in Fig. 2 could be slightly better explained. I found the x-axes there a little confusing.\n2. Small typos in lines 82, 247", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe logical flow of the experiments and conclusions is strong and well-presented. The experiments themselves properly support the claims made in the manuscript. Some further ways of characterizing global representational properties are given in the limitations section, but the ones that are tested in the experiments are convincing in their own right. Further, it is interesting to see that the models differ strongly in their ability to capture global geometries, while single-unit predictivity is very similar. It is an important message to the field to look beyond single-unit correlations.\n\n**Weaknesses**\n\nThe only issue I see lies in the novelty of the approach, as the finding that deeper ANN layers correspond more to higher-level visual areas is well established in the literature on task-driven models of the brain (e.g., [1], [2], [3]). Therefore, a data-driven model with a corresponding structure can be considered a somewhat minor advancement. The discussions in the manuscript focus heavily on data-driven models and gloss over this fact – I would recommend that a small section should be dedicated to exploring how the findings presented here relate to task-driven models. \n\nNevertheless, the work is well done and I recommend it be accepted for publication.\n\n[1] Performance-optimized hierarchical models predict neural responses in higher visual cortex\n\n[2] Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition\n\n[3] Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain’s Ventral Visual Pathway", "was_revised": true, "final_justification": "The thorough discussion of the questions I've raised has reinforced my position that this manuscript is suitable for publication.", "rebuttal": "We thank the reviewer for their thoughtful and encouraging comments. We appreciate the recognition of the importance of moving beyond single-unit correlations to study population-level representations.\n\n**Novelty and Relation to Prior Work**\n\nRegarding the concern about novelty, we would like to clarify a key distinction between findings in primates and mice that underpins the contribution of our work. In primates, Yamins et al. [1] demonstrated that performance-optimized deep networks trained on object recognition exhibit a clear hierarchical correspondence with the ventral visual stream: early layers best predict V1, middle layers align with V4, and deeper layers correspond to IT cortex. This provided strong evidence that task-driven models can approximate the functional hierarchy of primate vision.\n\nHowever, when the same approach was applied to the mouse visual system, Cadena et al. [2] found no evidence of such hierarchical alignment. While CNN features outperformed classical subunit energy models, randomly initialized networks performed comparably to ImageNet-trained models in predicting mouse neural responses. Critically, all visual areas studied (including V1, LM, AL, and RL, which we also examine in our work) mapped similarly across layers, showing no consistent progression from lower to higher areas.\n\nMore recently, Nayebi et al. [3] extended this analysis by systematically varying model architectures, training objectives, and input statistics. They found that certain unsupervised objectives, particularly contrastive learning, outperformed both random and supervised models in predicting mouse neural activity. While these models consistently mapped V1 to early layers and higher visual areas such as LM, AL, and RL to intermediate layers, they failed to differentiate among these higher-order areas. As a result, they cannot account for the empirical findings (e.g., Froudarakis et al., Hoeller et al., both examined in this manuscript) that reveal clear differences in object recognition performance across these regions (see Fig. 1 of our manuscript).\n\nIn this context, we believe our findings are both surprising and novel. Instead of relying on predefined tasks or training objectives, we use a *data-driven approach*, training a model directly on neural responses. The resulting *digital twin* successfully recovers meaningful distinctions between visual areas, particularly in their object recognition performance. By allowing each area to be read out from an individually matched model layer, we reveal a functional hierarchy that aligns with observed behavioral and representational differences, something that task-driven models have not achieved. This suggests that models trained directly on neural data may be better suited to capturing the computational organization of the mouse visual cortex than those relying on external tasks or objectives.\n\nFollowing the reviewer’s suggestion, we will add a brief section to clarify the distinction between task-driven and data-driven approaches and to acknowledge the broader relevance of such comparisons. We hope this helps reinforce the novelty and significance of our findings.\n\n**Clarification of Anatomical Hierarchy Score**\n\nIn Figure 2B, the x-axis shows the anatomical hierarchy score introduced by Harris et al. [4], which assigns a continuous value to each brain area reflecting its relative position in a cortical–thalamic hierarchy. This score is computed by analyzing the laminar termination patterns of axonal projections between areas. Each projection is classified as either feedforward or feedback based on its laminar pattern, and an optimization procedure finds the mapping that yields the most self-consistent global hierarchy.\n\nThe initial score of each area is based on the average directionality of its incoming and outgoing projections, and is refined via iterative updates that account for the positions of source and target areas. This method captures the idea that, for example, feedback projections from higher to lower areas tend to terminate in deep layers, while feedforward projections from lower to higher areas terminate more superficially.\n\nIn our manuscript, we introduce this hierarchy score briefly in Figure 1A and use it throughout the text, including as the independent variable in Figure 2B, where we relate it to receptive field size, object decoding accuracy, and discriminability. We will expand the description of this score in the revised manuscript.\n\n\n**Minor Comments**\n\nTypos on lines 82 and 247 will be corrected in the revised manuscript.\n\n**References**\n\n[1] Yamins, D. and DiCarlo, J. *Using goal-driven deep learning models to understand sensory cortex.* *Nature Neuroscience* (2016)  \n[2] Cadena, S.A., et al. *How well do deep neural networks trained on object recognition characterize the mouse visual system?* *NeurIPS* (2019)  \n[3] Nayebi, A., et al. *Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation.* *PLOS Computational Biology* (2023).  \n[4]: Harris et al. “Hierarchical organization of cortical and thalamic connectivity”. Nature (2019)"}, {"review_id": "w6cyWJDPqa", "reviewer": "Reviewer_MMjt", "summary": "This manuscript demonstrates that while previous artificial neural network models of the mouse visual cortex can predict single-neuron activity, they fail to replicate the hierarchical emergence of invariant object representations across different visual areas. To address this limitation, the authors introduce a biologically-inspired hierarchical readout scheme that maps different visual areas to distinct depths of a shared neural network, mirroring their anatomical organization. This anatomically constrained model is shown to significantly outperform standard architectures by successfully capturing key experimental findings, such as the progressive increase in both receptive field size and invariant object decoding accuracy along the cortical hierarchy. The authors conclude that incorporating anatomical information provides a powerful inductive bias that enables computational models to better capture the population-level transformations underlying complex brain functions.", "questions": "**Major concerns**\n\n1. The proposed hierarchical readout model maps areas V1, LM/RL, and AL to the 6th, 7th, and 8th layers of an 8-layer network, respectively. What is the specific rationale for assigning V1 to a relatively deep layer like the 6th? Given that early visual areas are typically associated with the early layers of deep learning models, how can you disentangle the impact of this design choice—for instance, the performance gains from the increased expressivity of a deeper network—from the effect of the hierarchical readout scheme itself?\n\n2. The core contribution of this study is its ability to capture population-level phenomena that were missed by previous models. However, it remains unclear whether the hierarchical model, in the course of better matching population-level orderings, can still reproduce the fine-grained, single-neuron properties that were the focus of prior research. The claims would be significantly strengthened by demonstrating that the proposed model not only improves the prediction of the population hierarchy but also maintains or even exceeds the precision of existing models in predicting single-neuron response characteristics.\n\n3. The proposed architecture features different brain areas reading out in parallel from different depths of a single, shared core network. This abstracts away from the brain's cascaded processing, where the output of V1, for instance, is the input for intrinsic computations within LM before projecting further. The current model lacks these area-specific, cascaded stages (e.g., Core_V1 → Module_LM → Module_AL). How much biological realism is lost in this simplification?\n\n4. The model is trained on data from the MICrONS dataset, which includes only excitatory neurons. However, inhibitory interneurons are critical for shaping receptive fields, controlling gain, and creating temporal dynamics within the cortex. By excluding this entire class of neurons, the model is inherently incomplete. The observed functional hierarchy might not be a product of feedforward excitatory connections alone, but could be critically sculpted by local inhibitory circuits, especially given that mechanisms like selectivity and invariance are often attributed to inhibition.\n\nLiu et al., \"Intervening inhibition underlies simple-cell receptive field structure in visual cortex,\" Nat. Neurosci., 2009\n\nLarisch et al., \"Sensory coding and contrast invariance emerge from the control of plastic inhibition over emergent selectivity,\" PLoS Comp. Biology, 2021\n\n5. As you note in your discussion, Hoeller et al. (2024) proposed that the brain maintains equivariant representations in addition to invariant ones, whereas a DNN such as AlexNet achieves invariance by suppressing equivariant responses. However, other studies suggest different aspects of object invariance in DNNs. For example, Farzmahdi et al. (2024) suggest that brain-like, mirror-symmetric viewpoint tuning emerges in AlexNet. In addition, Cheon et al. (2022) demonstrate that hierarchically increasing object invariance—alongside specific (equivariant) coding—emerges even in an untrained AlexNet. These studies highlight that brain-like invariance tuning, specifically across a hierarchy, may also emerge in artificial neural network models. Comparing the digital twin's hierarchical representation with these previous models may bridge the gap between data-driven, task-optimized, and structure-inherent approaches to modeling the visual cortex.\n\nHoeller et al., “Bridging tuning and invariance with equivariant neuronal representations,” bioRxiv, 2024\n\nFarzmahdi et al., “Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks,” eLife, 2024\n\nCheon et al., “Invariance of object detection in untrained deep neural networks,” Front. Comput. Neurosci., 2022\n\n**Minor suggestions**\n\n1. A point of confusion arises from the description of the experimental ground truth on page 8. The text states that \"...experimentally RL ranks very slightly above LM in RF size, cf. Fig. 1B (line 276)\" to explain the Kendall's τ score. However, this appears to be inconsistent with the data presented in Figure 1B itself. Both the figure's caption, which specifies a hierarchy of V1<RL<LM<AL, and a visual inspection of the plot suggest that LM has a larger receptive field area than RL. This discrepancy could create confusion for the reader and impacts the interpretation of the model's alignment with the experimental findings. Clarification on this point would be helpful.", "limitations": "The purely feedforward architecture omits the critical roles of feedback and recurrent connections, which are essential for complex visual processing and representation in the biological brain.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe anatomically-inspired hierarchical readout successfully enables the model to capture the emergent hierarchy of invariant object representations across the visual cortex, a feat that standard architectures fail to achieve.\n\n**Weaknesses**\n\nThe model's oversimplified architecture treats functionally distinct areas, such as LM and RL, identically, causing it to fail in reproducing key experimental findings related to the unique role of area RL.", "was_revised": true, "final_justification": "This work provides valuable insights into modeling hierarchical information processing in the visual cortex and understanding how populational information is transformed across its layers. Therefore, I support the acceptance of this paper, as in my original review.", "rebuttal": "We thank the reviewer for the thoughtful and thorough review. Below, we respond point-by-point to each of the reviewer’s comments and clarify the steps taken to address the concerns raised.\n\n---\n\n### **Major Concerns**\n\n---\n\n**1. Rationale for mapping V1 to layer 6, and disentangling hierarchical readout vs. increased expressivity due to depth**\n\nOur assignment of V1 to layer 6 was based on empirical validation: performance improved when moving from layer 4 to 6, with no significant gain beyond that (Fig 3). We therefore selected the simplest effective configuration that yielded high performance. It’s also worth noting that while V1 is anatomically early in the cortical hierarchy, its activity results from multiple preceding stages of nonlinear processing. In fact, several recent studies have shown that retinal circuits alone can be well-modeled by multi-layer deep networks [1]. Additionally, V1 responses to thalamic inputs are shaped by complex recurrent dynamics. Given this, it is not surprising that deeper layers in artificial neural networks are required to best model V1 responses.\n\nWe agree that disentangling the effect of hierarchical readout from increased depth is crucial. This is addressed in Fig 3, which compares models with and without hierarchical readouts at matched depth, and in Fig 4, which controls for depth while varying readout organization. These analyses demonstrate that the performance gains stem from the anatomically inspired hierarchy rather than from depth increase.\n\n[1] Tanaka H, et al. *From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction.* Advances in Neural Information Processing Systems (2019)\n\n---\n\n**2. Does the model preserve single-neuron precision while capturing population-level structure?**\n\nWe agree that strong single-neuron predictive accuracy remains important. To address this, we expanded training to include all 14 MICrONS sessions (rather than only 6 as in our initial submission). With the full dataset  (Table 1), our 8-layer hierarchical readout model matches the performance of depth-matched models with non-hierarchical readouts, in which all areas are predicted from the final layer. Note that, while both models use the same number of parameters in the readout (17,396,557), the non-hierarchical model has a substantially larger core (10,629,424 parameters) compared to the hierarchical variant (3,417,392 parameters), resulting in more total parameters. Despite this added capacity, overall performance remains equivalent across models. This performance is comparable to the current state-of-the-art (Wang et al., 2025). When analyzing performance by area (Table 2), we find that the hierarchical model improves accuracy in AL, the top of the functional hierarchy, compared to a depth-matched baseline. Performance for V1, LM, and RL remain unchanged. These findings confirm that hierarchical readout preserves, and in some cases improves, single-neuron prediction accuracy while enabling better alignment with population-level structure.\n\n**Table 1.** Average single-neuron predictive performance (correlation) per model.  \nValues are reported as the median normalized correlation coefficient (CC_norm) across all neurons (mean ± SEM across three random seeds), along with the range of per-session medians. Wang et al. (2025) is excluded from the average due to the lack of per-neuron data.\n| Model                           | Median CC_norm     | Median CC_norm per session |\n|--------------------------------|---------------------|-----------------------------|\n| Hierarchical (8 layers, hierarchical readout) | 0.648 ± 0.010       | [0.60 – 0.72]               |\n| Baseline (8 layers, last-layer readout)  | 0.648 ± 0.020       | [0.61 – 0.73]               |\n| Wang et al. (2025)             | not available       | [0.58 – 0.76]               |\n\n**Table 2.** Median CC_norm per area for 8-layer models with and without hierarchical readout.  \nValues are reported as mean ± SEM across three random seeds. Data from Wang et al. (2025) are not included due to the unavailability of area-level measurements.\n\n| Model                           | V1            | LM            | RL            | AL            |\n|--------------------------------|---------------|---------------|---------------|---------------|\n| Hierarchical Readout (8-layer) | 0.662 ± 0.002 | 0.619 ± 0.001 | 0.604 ± 0.002 | 0.615 ± 0.003 |\n| Baseline (last-layer readout)  | 0.663 ± 0.002 | 0.616 ± 0.002 | 0.602 ± 0.001 | 0.600 ± 0.002 |\n\n---\n\n**3. Biological realism and the lack of cascaded processing**\n\nIn our current architecture, responses in each visual area are computed from the output of a shared convolutional core, with parallel linear readouts from different layers. The core itself is hierarchical and cascaded in the sense that deeper layers (e.g., layer 8 for AL) build on earlier ones (e.g., layer 7 for LM/RL). However, we do not explicitly model inter-area communication as it occurs in biological circuits, where the output of one area directly informs the next. This is a simplification made to accommodate current data-driven modeling approaches.\n\nExplicitly routing activity from one area’s readout to another could in principle be more biologically realistic. However, we do not expect this to substantially alter the results. The transformations between layers in our shared core already approximate such computations and can be learned directly from the data. Moreover, because we observe only a subset of neurons in each area, constraining downstream areas to use only observed upstream activity might limit performance. In contrast, accessing shared internal representations allows the model to capture a more complete and abstract embedding of the population response, improving generalization.\n\nWe will clarify this modeling choice in the revised manuscript and acknowledge the potential value of future work exploring modular architectures with explicit inter-area pathways.\n\n---\n\n**4. Exclusion of inhibitory neurons**\n\nThe MICrONS dataset used in this study contains functional recordings exclusively from excitatory neurons, and our model is trained and evaluated based solely on that activity. This is consistent with the experimental data we aim to replicate, specifically the studies by Hoeller et al. and Froudarakis et al., which also report responses from excitatory neurons only.\n\nTherefore, the functional properties and hierarchical organization we model arise from excitatory populations, which aligns with the scope of both the available data and the experiments we seek to reproduce. While inhibitory interneurons are known to influence receptive fields, gain control, and temporal dynamics, our model is not intended to mechanistically capture those interactions. Instead, it is a functional model that aims to learn statistical regularities in the observed excitatory responses.\n\nIncluding inhibitory neurons, on the other hand, could potentially improve training by providing additional information about local circuit dynamics. However, because inhibitory neurons are typically less selective, their per-neuron contribution to predictive performance may be limited.\n\nFinally, we emphasize that our goal is not to maximize single-neuron prediction but to reproduce the functional hierarchy observed in population-level data. Our results show that training directly on excitatory responses is sufficient to recover this structure.\n\n---\n\n**5. Brain-like invariance and equivariance in DNNs**\n\nWe agree with the reviewer on the importance of emergent invariance and equivariance in artificial neural networks and appreciate the opportunity to clarify how our study relates to prior findings. Farzmahdi et al. (2024) and Cheon et al. (2022) offer complementary insights, demonstrating that task-driven models, such as AlexNet, can exhibit mirror-symmetric viewpoint tuning, while the same randomly initialized architectures already possess invariance properties. These findings suggest that the emergence of invariance and equivariance is shaped not only by network architecture, but also by the training objective and the dataset.\n\nAlthough our study does not directly probe invariance or equivariance, it builds upon these insights by comparing model outputs to experimental data from Hoeller et al. and Froudarakis et al. Specifically, we assess population-level representational changes across hierarchical cortical areas using digital twin models. Our objective is to examine how invariance to object transformations, such as viewpoint, scale, and position, is distributed across cortical regions, and how this invariance emerges with anatomical hierarchy.\n\nTo address the limitations of standard models, which typically use a shared readout from the final layer and can match single-unit responses but fail to capture hierarchical population geometry, we introduce an anatomically-inspired hierarchical readout. This strategy enables the model to capture higher-order response patterns across brain areas by allowing structured variation in how populations access shared features.\n\nWe recognize that a systematic dissection of invariance and equivariance across architectures, training regimes, and biological constraints represents an important future direction. We will revise the discussion section to reflect these connections more explicitly and emphasize how our findings relate to and complement the cited studies.\n\n---\n\n### **Minor Points**\n\n- **RL vs. LM ranking in receptive field size (Figure 1B)**  \n  Thank you for catching this inconsistency. This was a typo in the original manuscript text. The correct experimental ranking is V1 < RL < LM < AL, as shown in the figure. We will correct this in the revised version."}, {"review_id": "71syZ0VyVW", "reviewer": "Reviewer_vQD3", "summary": "The authors study digital twins, networks trained to predict neural activity in the brain, to understand whether these networks learn invariant object recognition using the same low-level and high-level principles as those used in the real visual cortex. They reference previous studies that identify properties of object recognition in visual cortex and find that only low-level properties are present in digital twins.", "questions": "•\tUse of Readout: Can the readout for RL and LM be treated the same way? I may be misunderstanding, but it seems RL and LM have the same readout depth. \n•\tDifferent Neuron types: The digital twin treats every neuron as an identical readout, ignoring the huge diversity of excitatory vs. inhibitory cell types and the laminar structure within each cortical area. Does this have an effect on the results?", "limitations": "•\tAddressed adequately in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n•\tNovelty: The paper identifies gaps in digital twins and analyzes where digital models fail to accurately reproduce well-known brain phenomena.\n•\tEmpirical gaps: The paper demonstrates that digital twins are unable to reproduce certain phenomena, such as area-to-area relationships or hierarchical representations. \n•\tEvaluation: The paper uses benchmarks and evaluations that are extensive and justified. \nWeaknesses:\n•\tPart of the reason for using a linear readout rather than a hierarchically designed readout is to enforce a simplicity bias, i.e., to ensure that whatever is decoded is not due to the complexity of the decoder. How do the authors propose that their hierarchical readout design avoids complexity bias in this way? \n•\tThe design of the convolutional architecture seems odd to me for drawing conclusions. Even Wang et. al. (2025), a paper they cite, has a much larger and more extensive network architecture. Would every “digital twin” need a hierarchical readout? How wide should the applicability be? \no\tFurthermore, I am slightly confused about the network depth experiments. It would be helpful to know how many parameters were actually added to the network when varying network depth. \n•\tThere are some methodological/design choices that are perhaps justified in other digital twins papers, but in my opinion, change the narrative presented in this paper, and I feel these should be addressed. \no\tBypassing Subcortical Regions: The authors rightfully acknowledge that they do not consider the retina or LGN. Although other studies also omit this region, I believe that ignoring where invariances begin to emerge weakens the claim that their hierarchical readout model is very useful. Perhaps representations in this region contain enough hierarchical information to make their own readout less competitive. I think this could benefit from further discussion. \no\tI am not sure RL and LM can have the same readout; see questions. \no\tUse of Calcium Data: Could the paper discuss how results would change if a neural recording measure other than calcium data was used?", "was_revised": true, "final_justification": "The thorough response from the authors and their answers to the questions I raised have convinced me that this manuscript is suitable for publication.", "rebuttal": "We thank the reviewer for the thoughtful and thorough review. Below, we respond point by point to each concern and will incorporate clarifications and additional analyses in the revised paper.\n\n---\n\n**1. Complexity bias in hierarchical readout**  \nOur hierarchical readout is composed exclusively of simple linear projections followed by a rectification step to ensure non-negative outputs, consistent with biological firing rate constraints. It does not include any additional nonlinear components such as multilayer perceptrons (MLPs), nor does it increase the parameter count relative to standard readouts used in prior work (e.g., [1,2]). This ensures that the function class of the decoder remains unchanged and maintains the simplicity bias that has historically motivated digital twin architectures. The novelty of our approach lies in the anatomical organization of the readout: rather than pooling all brain areas into a shared readout layer, as is standard, we assign each area its own dedicated readout layer based on its hierarchical position in the cortical structure. Importantly, the term \"hierarchical\" in our method refers solely to this anatomical alignment and does not imply added computational complexity.\n\n[1] Sinz, Fabian, et al. NeurIPS (2018)  \n[2] Willeke, Konstantin F., et al. arXiv (2022)\n\n---\n\n**2. Architecture design and applicability**  \nOur choice of a relatively compact convolutional architecture was guided by a balance between computational tractability and the need for sufficient representational power. Importantly, this architecture has been used successfully in prior digital twin work and provides strong baseline performance [1]. To go beyond earlier baselines, we extended the architecture from 4 to 8 layers and incorporated a hierarchical readout. Combined with the full MICrONS dataset (all available sessions), this version achieves performance comparable to the model of Wang et al. (2025), which, to our knowledge, is the current state of the art in modeling mouse visual cortex responses according to standard single neuron metrics. Importantly, our model is significantly smaller (number of parameters: 20,824,610 ours, vs 58,915,491 Wang 2025), is trained on a smaller number of neurons (115k ours, vs 135k Wang 2025) and training set (minutes of video: 692 ours, vs 900 Wang 2025). Despite this, our hierarchical model achieves similar performance on key single-neuron metrics (median CCnorm in session: [0.60–0.72] ours vs [0.58–0.76] Wang 2025).  \nTo directly assess whether hierarchical readouts are essential for success in this context, we adapted the Wang et al. model to our population-level evaluation framework and re-ran the core experiments from Hoeller et al. and Froudarakis et al. In both experiments, the Wang model performs significantly worse than our hierarchical model (see Table 1 and 2 below). Specifically, while it captures the broad difference between V1 and higher areas (i.e., lower discriminability in V1), it fails to reproduce distinctions across higher order areas (i.e., AL vs. RL/LM), predicting nearly identical performance. This is inconsistent with empirical findings that show a clear AL advantage. In contrast, our hierarchical model replicates both the V1-vs-higher-area difference and correctly ranks AL above RL and LM, aligning closely with experimental data.  \nThese findings support our central claim: hierarchical readouts provide a useful inductive bias that helps even small models capture key aspects of the visual hierarchy. Increasing model size or training data alone does not recover these features. While larger models may boost overall performance, our results suggest they still benefit from anatomically structured readouts.  \n\n**Table 1. Difference in classification accuracy between areas (Hoeller)**  \n| Model | AL–V1 | AL–RL | AL–LM |  \n|-------|--------|--------|--------|  \n| Our Hierarchical Model | 0.072 ± 0.0046 (p = 4.77e−15) | 0.027 ± 0.0050 (p = 2.27e−06) | 0.028 ± 0.0049 (p = 1.01e−06) |  \n| Wang et al. (2025) | 0.043 ± 0.0079 (p = 4.09e−05) | –0.013 ± 0.0096 (0.720, n.s.) | 0.003 ± 0.0078 (0.209, n.s.) |  \n\n**Table 2. Difference in object discriminability between areas (Froudarakis)**  \n| Model | AL–V1 | AL–RL | AL–LM |  \n|-------|--------|--------|--------|  \n| Our Hierarchical Model | 0.0585 ± 0.0038 (p = 8.33e−14) | 0.0195 ± 0.0040 (p = 7.93e−06) | 0.0387 ± 0.0036 (p = 6.63e−12) |  \n| Wang et al. (2025) | 0.0234 ± 0.0066 (p = 0.0061) | –0.0084 ± 0.0059 (0.312, n.s.) | 0.0023 ± 0.0068 (0.237, n.s.) |  \n\n[1] Turishcheva et al., arXiv (2024)\n\n---\n\n**3. Network depth and parameter count**  \nVarying the depth of the convolutional core affects the number of internal parameters, but the number of readout parameters is kept constant across all model variants (including those with hierarchical and non-hierarchical readouts). This allows us to isolate the effect of core architecture depth without conflating it with decoder complexity.  \n\n| Model Variant | Core Parameters | Readout Parameters |  \n|---------------|------------------|---------------------|  \n| Baseline (4 layers) | 382,512 | 17,396,557 |  \n| Baseline (6 layers) | 2,432,816 | 17,396,557 |  \n| Baseline (8 layers) | 10,629,424 | 17,396,557 |  \n| Hierarchical (8 layers) | 3,417,392 | 17,396,557 |  \n\n---\n\n**4. Methodological and design choices**  \nIn Supp. Section A we provide a rationale for our architectural and design choices, referencing prior work that supports each of them. Our goal was to follow established practices while introducing the hierarchical readout as the key novel component. We would greatly appreciate it if the reviewer could point out any specific design choices they feel were insufficiently justified or that significantly alter the narrative, so that we can address them more directly and clarify our intentions in the revised manuscript.\n\n---\n\n**5. Bypassing subcortical regions**  \nOur current focus is on explaining the increase in invariance from V1 to higher areas like AL, as shown in prior work (e.g., Froudarakis et al., Hoeller et al.). While the retina and LGN may contribute to early visual processing, digital twins without a hierarchical readout fail to capture the functional gradient across cortical areas, even if some invariance arises earlier.\n\n---\n\n**6. RL and LM readout depth**  \nOur original hierarchical readout design was informed by both performance optimization and anatomical constraints. V1 performance was best captured using a readout from the 6th layer of our convolutional core. To model downstream areas, we extended the core and placed readouts at progressively deeper layers based on anatomical hierarchy scores from [1], as shown in Fig. 1. This hierarchy follows the order V1 < LM < RL < AL. Since the separation between LM and RL is small, we approximated their depth as equivalent and used a shared readout at layer 7. We acknowledge this as a simplification.  \nTo assess the validity of this approximation and directly address the reviewer’s concern, we trained an additional model variant that explicitly separates LM and RL readout paths. Inspired by the dorsal/ventral stream distinction in mouse visual cortex [2], this architecture splits into two branches after layer 6. The dorsal stream targets RL and AL with readouts at layers 7 and 8, respectively. The ventral stream targets LM with a separate readout at layer 7. Due to time constraints, we evaluated this variant only in the original six sessions, not the full set of 14 used in the new analysis referenced elsewhere in the response. The model yields improved performance across all areas (Table 1), and preserves the global structure of inter-area relationships (Table 2, 3). This confirms that the hierarchical readout framework is not sensitive to the specific treatment of LM and RL depth, and that our findings hold under a more anatomically refined design.\n\n**Table 1. CCnorm per area for 8-layer models**  \n| Model | V1 | LM | RL | AL |  \n|--------|------|------|------|------|  \n| Hierarchical Readout (8-layer) | 0.601 ± 0.002 | 0.553 ± 0.001 | 0.554 ± 0.002 | 0.602 ± 0.003 |  \n| Two-streams Hierarchical Readout | 0.675 | 0.622 | 0.625 | 0.636 |  \n\n**Table 2. Classification accuracy and Kendall's Tau (Hoeller)**  \n| Model | V1 | LM | RL | AL | Tau |  \n|--------|------|------|------|------|------|  \n| Hierarchical Readout (8-layer) | 0.183 ± 0.004 | 0.213 ± 0.006 | 0.203 ± 0.006 | 0.231 ± 0.003 | 0.56 ± 0.07 |  \n| Two-streams Hierarchical Readout | 0.180 ± 0.005 | 0.210 ± 0.007 | 0.207 ± 0.006 | 0.238 ± 0.004 | 0.82 ± 0.11 |  \n\n**Table 3. Mutual information and Kendall's Tau (Froudarakis)**  \n| Model | V1 | LM | RL | AL | Tau |  \n|--------|------|------|------|------|------|  \n| Hierarchical Readout (8-layer) | 0.365 ± 0.005 | 0.378 ± 0.004 | 0.387 ± 0.005 | 0.402 ± 0.004 | 0.31 ± 0.07 |  \n| Two-streams Hierarchical Readout | 0.376 ± 0.006 | 0.387 ± 0.004 | 0.430 ± 0.003 | 0.425 ± 0.006 | 0.11 ± 0.11 |  \n\n[1] Harris et al., Nature (2019)  \n[2] D’Souza et al., Nat. Commun. (2022)\n\n---\n\n**7. Neuron types**  \nOur decision to treat all neurons identically reflects the limitations of the MICrONS dataset, which contains only excitatory neuron recordings. Including inhibitory neurons could enhance model training by providing additional information about local circuit dynamics. While prior work shows that model performance improves with more neurons [1], inhibitory neurons are typically less selective, so their per-neuron contribution may be smaller. Importantly, we show that single-neuron performance does not reliably predict a model’s ability to capture the functional hierarchy. Moreover, the experimental data we aimed to reproduce also focused solely on excitatory neurons. Thus, this simplification does not compromise our main findings. Extending the approach to include both excitatory and inhibitory populations is a valuable future direction but beyond the scope of this study.  \n\n[1] Lurz et al., ICLR (2021)"}, {"review_id": "DyUwNCMOkB", "reviewer": "Reviewer_vF7y", "summary": "The paper presents a novel way of training digital twin models of mouse visual cortex. Opposed to established approaches, which often only optimize the prediction performance in the final layer, the paper proposes a 'biologiclally inspired readout' procedure, where prediction performance is optimized along different levels of the network hierarchy in comparison to corresponding neural data. It is demonstrated that training of digital twin models using the novel method improves significantly the reproduction of the variations of different properties along the visual hierarchy, including receptive field size, decoding accuracy, and object discriminability in comparison to special data sets recorded in mouse visual cortex.", "questions": "On major question that might be discussed / reviewed is why people previously strongly defended the idea of an end to end training of neural response reproductions by neural models. Given that these structures include strong redundancies, it seems potentially not surprising that end-to end training to reproduce V1 responses with a deep hierarchical  model should provide a good reproduction of the variation of properties long the visual hierarchy in the cortex. In this sense, the author's approach seems much more reasonable. It would be interesting to discuss the arguments why this was done differently before. \n\nSmall points: \n\nELU+1:  maybe explain briefly \n\nSupplement A2: down-sampling to 36 x 64 pixels of the video in the first place seems to be very coarse in terms of spatial resolution; \nwhy does this make sense, or does not constrain the obtained results?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper is very well written, well structured, and the developed approach is well justified. Variations of the proposed methods are tested, and by  appropriate model comparisons and ablation studies it is convincingly demonstrated that the proposed novel training approach really results in better reproduction of invariance properties of neurons along the visual hierarchy than neural models trained with the classical end-to-end approach. \n\nA little weakness in my view is the failure to quite critical literature, e.g. about hierarchies in visual cortex, before 2011. \n\nIn addition, some assumptions of the model could be more clearly motivated in the text, e.g. why the chosen number of layers, neurons etc of the core unit. How are the the chosen readout model parameters chosen, and in how far are they critical , or not, for the results.", "was_revised": true, "final_justification": "I think technically this is a very solid paper. While still I think that the theoretical interpretation might stimulate some debatable points, however it will make a very useful contribution to the literature. The concerns of the other reviewers, e.g. about details of the biological plausibility, seem have to be addressed largely in a satisfying way. Therefore, I maintain my score, even though I am not an expert for models of the mouse brain.", "rebuttal": "We thank the reviewer for their constructive and encouraging feedback. We are pleased that the biological motivation, structure, and empirical support of our approach were positively received. We appreciate the support for our new approach to modeling neural responses. Below, we address the specific points raised in the review.\n\n---\n\n**1. Contextualizing our approach in prior literature**  \nPrevious studies have explored various strategies to understand the visual cortex, including linear-nonlinear (LN) models (Heeger, 1992; Simoncelli et al., 2004), energy models (Adelson & Bergen, 1985), and others. Deep learning has significantly advanced predictive performance, especially through convolutional neural networks (CNNs) trained on image classification tasks (e.g., Nayebi, 2023) or directly on neural data (e.g., Klindt et al., 2017; Wang et al., 2023). For a comprehensive review, see [1]. Importantly, many prior models incorporate biological constraints. For example, Antolík et al. (2016) proposed a structured model of V1 responses inspired by the feedforward visual hierarchy, where populations of V1 neurons share thalamic inputs and are organized into simple and complex-like layers. While such models have been surpassed in performance by end-to-end CNNs, the latter are often still motivated by biological principles. A key innovation, introduced by Klindt et al. [2] and adopted in our work, is the factorization of feature and spatial position into orthogonal components. Traditional approaches often treat each neuron independently, but the visual system exhibits spatial equivariance: similar computations recur across nearby locations. Leveraging this shared structure enables models to learn a common nonlinear feature space, leading to more powerful and generalizable predictions than those based on isolated neurons.  \nOur introduction of a hierarchical readout continues this biologically inspired modeling tradition. We first recognized that standard models, with a shared readout from the final feature layer, perform well at predicting individual neuron responses, but fail to capture population-level response structure. This motivated us to introduce a hierarchical readout variant, which explicitly models population organization by allowing structured variation in how different neurons access shared features. This approach enables the model to capture higher-order response patterns across the population, going beyond traditional single-neuron metrics.\n\n[1] Turishcheva et al. “The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos.” arXiv (2023)  \n[2] Klindt et al. “Neural system identification for large populations separating ‘what’ and ‘where’.” Advances in Neural Information Processing Systems (2017)\n\n---\n\n**2. On prior literature about cortical hierarchies**  \nRegarding the concern about missing citations to earlier literature on cortical hierarchies, we agree this is an important point. For mice, we will add earlier references [1,2] that demonstrate prior knowledge of visual areas beyond V1. For the anatomical hierarchy specifically, [3] was, to our knowledge, the first to systematically characterize interareal connectivity and propose a hierarchical structure based on anatomical data. Please let us know if there are additional references we may have missed. While the literature on primates is indeed more extensive and older, our study focuses on the mouse visual system; for space reasons, we chose to cite a review article [4] to provide broader context.\n\n[1] Wang & Burkhalter. “Area map of mouse visual cortex.” J. Comp. Neurol. (2007)  \n[2] Marshel et al. “Functional specialization of seven mouse visual cortical areas.” Neuron (2011)  \n[3] Wang et al. “Network analysis of corticocortical connections reveals ventral and dorsal processing streams in mouse visual cortex.” J. Neurosci. (2012)  \n[4] DiCarlo et al. “How does the brain solve visual object recognition?” Neuron (2012)\n\n---\n\n**3. Motivation and impact of architectural choices**  \nFor the core, we built upon the architecture introduced in the Sensorium 2023 challenge ([1]), which employed a four-layer factorized 3D convolutional network with increasing channel depth (16, 32, 64, 128). This choice was motivated by its suitability for processing dynamic stimuli with spatial and temporal resolution comparable to those used in our study. While previous models in the literature have generally used shallower cores, e.g., [2] employed a four-layer 2D CNN with 64 channels per layer; [3] and [4] used three layers with up to 48 channels, the Sensorium architecture represented the best publicly available option for mouse visual modeling and served as a practical benchmark.  \nImportantly, our work is among the first to systematically test how increasing core depth affects neural prediction performance. We extended the core from 4 to 8 layers and observed a marked improvement (8.4%) in single-neuron prediction accuracy. However, this modification alone did not enhance alignment with population-level hierarchical organization. This result suggests that deeper networks improve single-neuron fitting but do not resolve functional mapping across cortical areas, highlighting the necessity of architectural biases like our anatomically-inspired hierarchical readout.  \nFor the readout, we adopted the Gaussian spatial readout proposed by [2], which currently represents the best-performing readout framework for mouse visual cortex. The dimensionality of the features used in the readout is determined directly by the channel count of the core layer being accessed, e.g., 128 features per neuron from the final layer in our case. We did not extensively test this dimensionality, but it appeared sufficient for capturing the target responses. As a point of comparison, [5] used 512 features per neuron, yet reported only marginal gains in prediction accuracy, suggesting limited sensitivity to readout dimensionality beyond a certain threshold.  \nWe will revise the manuscript to explicitly include this rationale and emphasize that our work systematically explores design choices that have previously been adopted without extensive justification. This includes evaluating how architectural decisions, such as core depth and readout configuration, affect both single-neuron accuracy and population-level representational structure.\n\n[1] Turishcheva et al. “The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos.” arXiv (2023)  \n[2] Lurz et al. “Generalization in data-driven models of primary visual cortex.” ICLR (2021)  \n[3] Sinz et al. \"Stimulus domain transfer in recurrent models for large scale cortical population prediction on video.\" NeurIPS (2018)  \n[4] Klindt et al. “Neural system identification for large populations separating “what” and “where”.” NeurIPS (2017)  \n[5] Wang et al. “Foundation model of neural activity predicts response to new stimulus types.” Nature (2025)\n\n---\n\n**4. Clarifications on minor points**  \nWe will briefly clarify the use of the ELU+1 activation function in the main text. This ensures strictly positive activations, which is more biologically realistic given that firing rates are inherently non-negative.  \nRegarding the input video resolution (36 × 64 pixels), this reflects a trade-off between biological plausibility and computational efficiency. The low resolution aligns with the limited visual acuity of mice and is further supported by evidence that lower-resolution training can improve neural response predictivity (e.g., Fig. 3 in [1]). Notably, our model achieves single-neuron prediction performance comparable to that reported in [2], which also modeled responses from the MICrONS dataset but used higher resolution (144×256 pixels), enabled by extensive pretraining across multiple mice. This comparison supports the conclusion that lower resolution is sufficient for accurate modeling. We will clarify this rationale in Supplement A2.\n\n[1] Nayebi et al. “Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation.” PLOS Comp Bio (2023)  \n[2] Wang et al. “Foundation model of neural activity predicts response to new stimulus types.” Nature (2025)\n\n---\n\nOnce again, we thank the reviewer for their valuable feedback and are confident that the proposed revisions will strengthen the manuscript."}]

Author rebuttal:

Meta-review: This work examines digital twin models of the mouse visual system: ANNs trained to predict visual neural responses. The authors specifically investigate whether these digital twin models can reproduce population-level properties of visual responses, extending beyond the models' capability to explain individual neuron responses.

The reviewers unanimously agreed that this paper represents a very interesting and valuable contribution that advances our understanding of how to build end-to-end trained ANNs for modeling visual systems. I share this assessment and am convinced of both the scientific value of this work and its methodological rigor. Therefore, I recommend acceptance.
Decision: Accept (poster)