Title: Dimensionality Mismatch Between Brains and Artificial Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: representational geometry

fMRI

convolutional neural networks

brain-model alignment
Abstract: Biological and artificial vision systems both rely on hierarchical architectures, yet it remains unclear how their representational geometry evolves across processing stages, and what functional consequences may arise from potential differences. In this work, we systematically quantify and compare the linear and nonlinear dimensionality of human brain activity (fMRI) and artificial neural networks (ANNs) during natural image viewing. In the human ventral visual stream, both dimensionality measures increase along the visual hierarchy, supporting the emergence of semantic and abstract representations. For linear dimensionality, most ANNs show a similar increase, but only for pooled features, emphasizing the importance of appropriate feature readouts in brain–model comparisons. In contrast, nonlinear dimensionality shows a collapse in the later layers of ANNs, pointing at a mismatch in representational geometry between the human and artificial visual systems. This mismatch may have functional consequences: while high-dimensional brain representations support flexible generalization to abstract features, ANNs appear to lose this capacity in later layers, where their representations become overly compressed. Overall, our findings propose dimensionality alignment as a benchmark for building more flexible and biologically grounded vision models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study addresses the important question of how information is processed across hierarchical stages in both biological and artificial neural systems. The authors begin by clearly distinguishing between linear (effective) and non-linear (intrinsic) dimensionality. The work is focused on the vision modality. The authors compare different CNN-based networks with the different regions of the brain and compare their representational dimensionality. The results show that there is a striking mismatch between the ANNs and the human brain. In contrast to the brain where intrinsic dimensionality increases in a linear fashion, ANNs intrinsic dimensionality follows the canonical hunchback shape defined by an initial increase followed by a continuous decrease. \n\nThis is a very interesting and well-documented study aimed at understanding how the dimensionality of representations in the human visual cortex, specifically the ventral visual stream, evolves. Is it compression–expansion, continuous expansion similar to other species, or expansion–compression as observed in ANNs? Thus, the objective here is to investigate the commonalities and differences in representational geometry between biological and artificial systems.\n\nOverall, the authors have presented a well-curated study with a primary emphasis on biological systems (neuroscience), while artificial systems receive comparatively limited attention and serve mainly as a secondary point of comparison. I will thoroughly explain this in the Strengths and Weaknesses section.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper conducts an extensive set of experiments on the biological side, specifically examining the human brain and its sub-regions in detail. The results are clean and well-presented for both biological and artificial systems. \n\n2. The paper has a single objective, and it achieves that objective through the experiments presented in the main text. However, I have some concerns regarding the experiments conducted for the ANNs, which I will address below in the 'Weakness' section.\n\nWeakness:\nDue to the following major weaknesses, I have deducted two points. If the authors can provide follow-up experiments addressing some of these concerns, I will increase the score.\n\n1. [MAJOR] Comparing ANNs with the human brain using pre-trained models may not be the most appropriate approach. While ANNs are typically trained on large datasets of static, isolated images (e.g., ImageNet), the human brain is exposed to continuous, smooth, and temporally structured visual input throughout life. This results in a significant mismatch in their respective visual diets (i.e., training data). As a result, it's unclear whether the differences reported in the study arise from this mismatch in training data or from deeper architectural or objective functions.\n\nA potential solution is to train ANNs on egocentric datasets (such as first-person video) or in simulated environments that better mimic the visual diets of humans. While such datasets may not achieve perfect equivalence, they would allow for a closer and more meaningful comparison between biological and artificial systems.\n\n2. [MAJOR] Does intrinsic dimensionality collapse in the deeper layers of all ANNs? The title and much of the language in the paper refer broadly to \"artificial neural networks (ANNs)\"; however, the experiments are conducted exclusively on convolutional neural networks (CNNs). While I have no objection to the exclusive use of CNNs for the current analyses, the framing may be misleading. As it stands, the paper appears to generalize its findings to all ANNs, including architectures such as Transformers, which have markedly different inductive biases and architectural features. Clarifying this distinction, or including additional experiments with non-CNN architectures, would strengthen the paper and ensure that its conclusions are correctly scoped.\n\nTransformers have a fundamentally different architecture compared to CNNs. Unlike CNNs, Transformers do not rely on pooling layers, hierarchical structures, or hardcoded spatial biases. Recent studies have shown that Transformers can mimic key features of the brain. For example, they show representations similar to those of hippocampal cells, demonstrate neuron–astrocyte-like interactions, and are capable of learning object recognition when trained through the eyes of newborn animals.\n\nGiven these findings, it is important to include Transformers in the analyses presented in this paper. Specifically, it would be valuable to test whether the results reported here can be replicated using a Vision Transformer (ViT) model. It is acceptable for the ViTs to be pre-trained, just as the CNNs used in the current experiments were. Furthermore, it is also acceptable and valid if the ViTs produce different results; such differences would still offer meaningful insights into architectural and representational differences between models.\nWithout testing other ANN types, the claims in this paper offer a weak generalization.\n\n3. [MINOR] Why is it that a randomly initialized ResNet-50 shows a similar ID pattern to that observed in Subject 1, while a trained ResNet-50 does not exhibit this pattern? I suggest repeating this experiment with different architecture sizes, such as ResNet-18, ResNet-34, and ResNet-101, just for the untrained case. It would be interesting to see whether the pattern observed in the untrained ResNet-50 is preserved across different architecture sizes.\n\nQUESTIONS:\n1. What image resolution was used when testing Human subjects and ANNs? Was there a difference between the two? If yes, then why?\n2. The authors define dimensionality later in the Methods section (Line 123), despite discussing it extensively earlier in the Introduction. Given that this is a computational neuroscience paper and is likely to attract readers from both machine learning and neuroscience backgrounds, it would benefit the reader if this definition were introduced earlier in the paper.\n3. The authors provide no details about the pre-trained models. They should provide information about their objective functions and other necessary information for readers who are not familiar with the ANNs training objectives.\n4. How did the authors calculate the relative depth of the ANNs as shown in the graphs?\n5. The authors provide no information about the human subjects or their age groups. If the authors believ this information was included in the manuscript or the Appendix section, please refer to those line numbers in the comment.\n\nLIMITATIONS:\nThe authors have not included any limitations in this paper. It is highly recommended to add a Limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically analyzes and compares the linear and nonlinear activations of the human brain and ANN when viewing natural images. They point out that nonlinear dimensionality shows a collapse in later layers of ANNs. They found that this collapse would limit Semantic Generalization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This paper reveals a functional mismatch in representational geometry between brains and ANNs.\n\n2. This paper demonstrates that global average pooled features from ANNs align best with fMRI-derived linear geometry, which is a significant discovery.\n\n**Weaknesses**\n\n1. The conclusions of this paper are similar to those of some other literature [1], which affects its originality.\n\n2. The explanation of the stratification of the cerebral cortex and the levels of ANN is not clear enough.\n\n[1] Wang, A.Y., Kay, K., Naselaris, T. et al. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nat Mach Intell 5, 1415–1426 (2023).\n\nQUESTIONS:\n1. Your work says “We find that in the... between brains and models” (Line 63-66). The layers of the human brain,  as hierarchical pathways, such as primary visual cortex -> intermediate visual cortex -> advanced visual cortex, while the layers of the ANN are the network layers of the model. Do these two layers mean the same thing? How do you prove that a certain layer of the model corresponds to a certain visual layer of the human brain?\n\n2. In recent years, several viewpoints have emerged on the interpretability of ANNs. Some people believe that artificial neural networks trained in physical space are similar to the networks of real brains [1]. Others have shown that \"models with higher performance on benchmark tasks achieve peak prediction accuracy in earlier layers. In contrast, models with lower performance show delayed representations and require deeper layers to achieve similar levels of brain prediction accuracy.\"[2]. This result means that the model is gradually becoming more and more like the brain as the level changes. When this similarity reaches its peak, it turns to generate exclusive processing, which exceeds the target range of current neural recordings, resulting in a decline in the encoding model. So, if you want to compare the levels of two \"brains\" in physical space, should the ANN level be truncated at the most brain-like layer (because it can best explain brain response activities)? This seems to be more objective because the deep layer will lose some interpretability due to the conversion of the target. This may get completely different results.\n\n3. You mentioned that \"ANNs exhibit a collapse in intrinsic dimensionality in deeper layers\". The explanation of intrinsic dimension in the article is nonlinear correlation. If the model has a complete basis, it can well linearly represent the nonlinear components in the model's perceived information, then it will naturally get a lower intrinsic dimension because it can be well linearly approximated. What do you think about this issue?\n\n4. You mentioned that in the brain, higher-dimensional representations are associated with stronger abstract generalization abilities. [3] seems to have made a similar conclusion. Is it necessary to discuss it?\n\nI would be willing to improve my score if the above questions can be answered well.\n\n**Reference**\n\n[1] Achterberg, J., Akarca, D., Strouse, D.J. et al. Spatially embedded recurrent neural networks reveal widespread links between structural and functional neuroscience findings. Nat Mach Intell 5, 1369–1381 (2023).\n\n[2] Mischler, G., Li, Y.A., Bickel, S. et al. Contextual feature extraction hierarchies converge in large language models and the brain. Nat Mach Intell 6, 1467–1477 (2024).\n\n[3] Wang, A.Y., Kay, K., Naselaris, T. et al. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nat Mach Intell 5, 1415–1426 (2023).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors probe the representational geometry of the human visual cortex and convolutional neural nets (CNNs) by measuring their effective dimensionality (ED) and intrinsic dimensionality (ID). ED measures the linear expressivity of the neural code, while ID measures the number of independent parameters needed to describe the data locally. In both human fMRI data and CNNs with global-average pooling (GAP), ED grows monotonically in deeper layers. ID grows monotonically along the human visual hierarchy but shows a hunchback profile for CNNs with GAP, indicating a mismatch. The higher ID observed along the human cortex is interpreted as higher semantic abstraction, whereas the lower ID in CNNs indicates that the final layers of deep networks lose such information. Supporting this claim, the authors show that classification tasks can be solved better with deeper layers both in the human cortex and CNNs, while abstract concepts are decoded inceasingly well only in the human data.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe paper was very fun to read; it is clearly written and easy to follow. Experimental methods are clearly defined with sufficient detail. Representational alignment between ANNs and brains is a hot topic. The proper metrics to measure representational geometry and interpretability are a huge debate in the NeuroAI community, and this work makes a solid contribution by providing a carefully designed and interpretable benchmark to distinguish ANNs from brains. The experiments are thorough and well-thought, e.g. testing dependence on pooling strategy, controlled experiments with randomly initialized networks, two different datasets (THINGS, BOLD), cross-validated analysis, etc.\n\n## Weaknesses\n\n1. I am not familiar with the searchlight approach. I could not completely understand the data pipeline and how exactly ED and ID are calculated. A short description for searchlight would be helpful for general audiance. \n2.  In addition to GAP, ANN layer features are randomly pooled to 50 (out of ~512) features to compute dimensionality, but this assumes the features are uniformly distributed in a layer.\n3. For ED/ID and accuracy plots, the y-axes have completely different scales between brain and ANNs. For ED, ANNs are significantly higher-dimensional and hard to compare with human data. For decoding, ANNs perform an order of magnitude better than fMRI data. The paper should at least comment on this. Also, since the paper does not compare dimensionalities between brains and ANNs, there is no need to restrict ANN dimensionality.\n4. The low ID in deeper layers is explained by task specificity of ANNs. Shouldn't SimCLR have larger ID since it is self-supervised? It is not clear how the training objective affects ANN–brain alignment. Including an unsupervised model might help test the paper’s main claim.\n5. It is not clear how high *non-linear* dimensionality relates to better *linear* decodability of abstract features. The authors cite ref. \\[16] (line 41) for this relation, but ref. \\[16] only considers linear dimensionality. Even then, the regression score cannot be explained solely by dimensionality; alignment with the task also matters ([https://arxiv.org/abs/2309.12821](https://arxiv.org/abs/2309.12821)). The authors should clarify this point.\n6. Some reported values—$R^2$ in Fig. 3 and accuracy in Fig. 4—are negative and should be fixed.\n\nQUESTIONS:\n1. How does high ID relate to disentanglement/abstraction in the sense used in ref. \\[34]? Also, how does the ID defined here compare to SD in ref. \\[34]?\n2. How does changing $k$ affect the ID? Does setting $k = 20$ keep each term in Eq. 2 “local” when there are only 50 points in each searchlight sphere?\n3. What is the importance of super-linearity, and how does it differ from mere linearity? How does this relation change if the distances are the ranks rather than geodesic distance?\n4. Would you consider repeating the experiments for biologically inspired models like cornets?\n\nLIMITATIONS:\nThe main claim holds only for CNNs with specific pooling, thus limiting the applicability of the proposed benchmark to other architectures such as ViTs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission investigates how dimensionality measures (particularly effective dimensionality and intrinsic dimensionality) vary across the fMRI responses measured from different visual regions. They compare the observed dimensionality from the fMRI responses with the dimensionality measured from the hierarchical layers of ANNs.  The paper shows that overall the dimensionality increases for deeper visual cortical regions, and features of ANNs tend to show a trend where the dimensionality begins to decrease for late layers. When using spatially averaged features, the authors observe that the trends in ANN dimensionality across layers begin to look more like the visual regions, however the ID still decreases for some late layers, suggesting a mismatch between the hierarchy of the brain and the tested ANNs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* Investigating the local dimensionality of different cortical regions on a large-scale image dataset is interesting, and seems novel to report. \n* The comparisons with ANNs seem to be well thought out, and there are some critical control experiments presented in the supplement. \n* The paper is well structured, and the writing and logic are clear to follow\n\n**Weaknesses**\n\n* The claim about super-linearity in many places of the paper seems exaggerated, as it is driven primarily by one point (primary visual cortex) and might be dependent on the way that the “cortical distance” was determined.  \n\n* The authors claim that the ID decreases for late layers which diverges from the brain, however the actual depth of the models seems somewhat arbitrary not directly aligned with the neural responses.  It is very possible that the better comparison with these visual brain regions would be only the early/mid layers of the models (where there is a steady increase in dimensionality), and higher-order brain regions not included in the analysis would actually drop in dimensionality if they were included in the fMRI analysis. \n\n* Dimensionality measures can sometimes be dramatically dependent on the data processing, and this is often particularly true for noisy data like fMRI (for instance normalization of signals, or the noise in the data). There are some specific questions about this below.\n\nQUESTIONS:\n1. Is there any normalization (for instance, z-scoring) applied to the features of the neural networks or the measured fMRI activations before the dimensionality measure is computed? This preprocessing should be reported. Additionally, I would like to see the replication for Figure 1 (1) with and without z-scoring the features/fMRI responses across stimuli before dimensionality is computed. (2) with and without the per-feature/voxel mean subtracted across stimuli (but not normalizing the variance as you do in z-scoring).\nI am asking about this specifically because, for the fMRI data in particular, things like the overall amplitude of the response between different voxels can dramatically change the effective dimensionality, and this type of response variability can be dramatically different from one region to another due to the SNR of the fMRI response. It thus seems particularly important to make sure that the main trends of the paper are robust to the preprocessing strategy used. \n\n2. Is there a citation for referring to the effective dimensionality / participation ratio as the “linear dimensionality”, or is this something being used by the authors? I have not encountered it being referred to as such before. \n\n3. The sampling for the fMRI data is performed in a searchlight approach where neighboring voxels are included in the dimensionality, while for the ANN data it is sampled by randomly choosing 50 pooled feature maps. This potentially preserves topographical organization in the fMRI data. The authors present a control experiment for this in A10 and show that the overall pattern of responses is conserved, however it looks like the absolute values of the dimensionality are much higher for the random sampling.  Could the authors explain this substantial difference? \n\n4. Are the super-linear fits significantly better than linear fits? I am surprised that the authors are highlighting that the increase is super-linear because from looking at it, it seems to be that the V1 point (at zero) is driving this “super linear” effect. The fits with 3 points in the supplement A11 and A12 seem particularly concerning. \n \n5. The authors talk about the importance of global-average pooling for the dimensionality alignment (201-213), where the last sentence is about them serving as a better proxy for “neural recordings”, but I think that the authors are specifically talking about alignment with fMRI data where nearby neurons are pooled? This should be clarified, and a discussion about how there might be differences when looking at individual neural responses rather than bold activity should be included somewhere in the paper. \n\n6. The specific layers used for the deep neural network analyses and how this is mapped to the \"relative depth\" 0-1 scale should be more clearly explained (I couldn't find this, but maybe it is in a section of the supplement). \n\n7. Given that the authors main interpretation of the results is that higher-order regions remain higher dimensional because they allow for behavioral generalization, it seems appropriate to discuss work on data diet and training types that encourage the deep neural networks to maintain additional information. Some work that has demonstrated changes in the training environment/task to preserve additional information and lead to better neural alignment are: \n* https://openreview.net/pdf?id=AiMs8GPP5q (adds an equivariant loss and shows that this improves predictions for IT response). \n* https://www.nature.com/articles/s41467-024-53147-y (shows that diversity in the visual training diet is one of the main drivers of good predictivity)\n* https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002366 (shows that training on multiple tasks results in the best overall predictions for audio fMRI data)   \n\n8. Please include references to the numbered supplemental figures / sections when they are referenced in the main text.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}