Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations

Authors: Yudi Xie, Weichen Huang, Esther Alter, Jeremy Schwartz, Joshua B. Tenenbaum, James J. DiCarlo
Primary area: applications to neuroscience & cognitive science
Keywords: Vision, Convolutional Neural Networks, Representation Learning, Neural Data Alignment, Ventral Visual Stream, Computational Neuroscience

Abstract:
Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730668067606

thread_type:
official_review

<thread_object>
creator: reviewer_wqcv

<content>
summary:
The paper assesses the neural predictivity and representational similarity of CNNs trained on a novel large synthetic dataset to decode different combinations of the following latent variables: object category , object identity, object position in 3-D space, and object orientation. The author's make the following observations:
- Training networks using a sufficiently large/realistic/diverse synthetic dataset can produce representations that rival those trained on natural images in terms of predictivity of ventral stream responses.
- Model's trained only to estimate spatial latents (i.e. without any explicit incentive for class invariance) can explain a similar fraction of variance as models trained to do categorization. 
- The ability to decode spatial latents is correlated with predictivity.
- Model's trained on different tasks learn similar internal representations (as measured by CKA), especially at early layers and to a lesser degree deeper in the networks.
- The amount of variability of some latent variable in a training dataset modulates the extent to which said latent is linearly-decodable from a given representations, even when that latent is not used to train the neural network.

strengths:
- The subject (the degree to which object classification is a unique normative explanation for features of neural representations in the ventral stream) is very interesting (and to a substantially large community).

- Investigating this hypothesis using a carefully controlled *but appropriately large* synthetic dataset is a sensible and novel approach.

- For the most parts experiments are clearly described, easy to follow, and sufficiently thorough to justify the conclusions. 

- While I find the results of Section 3.3 the least convincing of those presented, doing these experiments is certainly a step toward addressing why similar representations are observed in the early layers across many artificial representations (which is "above and beyond" what is necessary for supporting the core arguments laid out in the paper).

weaknesses:
- Layer Mapping Details Missing: I think it would be good to report which layers are mapped to each of the considered brain areas/datasets (unless I missed this detail). For instance, if the same layers are mapped to V1 across tasks this would be interesting especially in light of the fact earlier layers have higher CKA similarity. 

- More fine-grained/controlled predictivity-to-decoding correlation analyses: It would be interesting to see if performance on any particular task is more or less correlated with predictivity in a particular region (i.e. by reproducing Fig. 3 but measuring the correlation with only V1 predictivity etc.). Furthermore, it may be easier to interpret this type of result if the layers used to decode the latent parameters were matched to the layers mapped to a particular region (so you can make statements along the lines of "the ability to decode distance from layer3.0.relu was correlated with the ability to predict V4 from the same activations, suggesting a possible functional link between the two").

- Difficult to draw conclusions from results in Section 3.3: 
  - It would be nice to see the linear decoding performance for each latent on the pixels as well as early in the network as a baseline. 
  - In Fig. 5b while all models have marginal increases  in category decoding (though most differences seem insignificant based on the depicted error bars, which are not described, but I assume to be 95% confidence intervals?), they are all still generally unable to decode categories at a meaningful rate. I think as well as the chance-performance floor it would be good to include a "trained for category decoding only" ceiling to help assess the scale of the improvement. Furthermore it would be good to include the performance curve (decoding accuracy as a function of depth in the network) for randomly initialized networks. If this curve is above the others a more accurate description would be "learning to decode a spatial latent decreases the decodability of a non-target latent less when that latent is more variable in the training set." Similar baselines would be useful in 5c.
  - The all vs. none variability setup in these experiments produced mostly small differences in the decoding performance with the notable exceptions of layer4.0relu in 5b bottom row and avgpool/fc in the bottom row of Fig 5c. Given this I think the result could be made more convincing by showing an orderly trend in decoding performance as the variability of non-target latents is parametrically decreased in steps between the Full variability case and the very low variability case explored here.  Finally in this setup it would be good to see how decoding performance varies in terms of the variability. Consider an experiment of this type:
    - You have (in addition to the models at full translation variability and zero translation variability already trained) a model trained using a new 50% translation variability dataset. You may then want to measure the translation decoding performance of all models both the full-variability and half-variability test-sets to get the clearest picture possible. 
  - In sum, I think the language describing these results could be softened to reflect how difficult it is to draw definitive conclusions about internal representations from this type of experiment. This isn't a big issue at all but maybe changing line 430 to: "In summary, most of our observations are in line with the hypothesis that models develop better intermediate representations of nontarget latents in the presence of their variability, though future work is needed to understand both the size of and mechanism behind this effect." 

- Model-Brain Comparisons Leave out Relevant Datasets: the author's only compare model responses to 1/4 of the neural recording datasets in both areas V4 and IT available on the BrainScore platform. I understand that predictivities across these datasets are correlated with each other, but I see no reason not to include them in the analyses. Especially given that the images presented in the Majaj-Hong benchmarks have a striking similarity in construction to the TDW synthetic dataset (object at random pose superimposed on a natural background), it would be good to confirm that the observations in this work generalize to other types of images!

questions:
- About the spike in category decoding with rotation latent target (bottom row Fig. 5b): I wonder if rotation decoding is somewhat entangled with category decoding. I.e. to know some object has been rotated by 90 degrees you must know what its shape is/how it appears when presented at zero degrees of rotation (and such information is correlated with category). It might be interesting to look and see how the performance on category decoding breaks down by class (if category performance is high for objects whose shape varies strongly with viewing angle this might support such a hypothesis). 

- Why are two different similarity metrics employed when comparing two artificial representations and when comparing artificial representations to neural measurements? Since the current work only considers the subset of datasets from BrainScore that are public, couldn't you use CKA to compare the artificial and biological representations as well (or alternatively use partial least squares regression to compare pairs of CNNs)? This doesn't seem likely to be an important issue but I am curious as to the reasoning, as it might be nice to have all comparisons on the same "scale", subject to the same method-induced biases, and tolerant to the same transformations. 

- In the paragraph beginning on line 334 the authors suggest that their results from decoding/CKA experiments suggest that each model with similar predicitivity is also explaining the same portion of variance in the neural responses. While this seems sensible (especially perhaps for earlier layers/V1-V2 predictivity), I don't understand why we cannot just address this hypothesis directly. For example we can: directly examine the residuals produced by each model-to-brain prediction, predict the residual of one predictive model with another, or predict the neural data using ensembled models to get a sense for how overlapping the explained variances are. Again, given that the study is limited to the publicly available benchmarks all of these experiments seem feasible, and I think very interesting! 

- Semi-Nitpick: the paper repeatedly references the number of units provided supervision. I don't think this is an interesting axis, or fair to compare between units that receive a discrete category label and a continuous latent variable. Meaning, you could discretize each spatial latent into 117 bins without changing the training signal but equalizing the number of outputs to the categorization task. I think rather than the dimensionality of supervisory signal the number of bits of supervision is the relevant thing to compare! I.e. since you know the distribution of locations you can measure the entropy of the translation variable and compare this to the entropy of the category variable (this may be trickier because of discrete vs. differential entropy etc.), but I think is more meaningful than comparing dimensionalities. 

- Note: I would prefer to leave a score of 7 for this paper, but as that is unavailable I will sit at a soft 6 for now. I am very willing to raise my score if the authors can address/justify/explain some of the weaknesses or questions listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693015723

thread_type:
official_review

<thread_object>
creator: reviewer_acut

<content>
summary:
The paper examines the effect of training models to estimate spatial latents on neural alignment (the "where"), comparing this approach with object recognition (the "what"). It highlights that to achieve high neural alignment, training with a focus on "where" yields similar benefits compared to the more common focus on "what." Interestingly, with fewer presentations in the "where" case, alignment levels appear comparable, and representations in the models' initial layers are strikingly similar.

strengths:
The paper addresses an interesting problem that has been overlooked for some time. The presentation is highly effective, and the paper is easy to read. It takes an unconventional approach to the problem by focusing on spatial location, which proves to be very effective even in a low data regime. There are several controls in the main text and supplementary material that support the claims.

weaknesses:
I know that in  semantic segmentation and other tasks where location and classification of the object is used for training, has been studied for brain-score but have not really succeeded.  Perhaps worth adding a bit of discussion on this topic, because even when its possible to say that the world is 3D and therefore the tasks seem to align better with vision, is also true that the models used in this work are only receiving 2D images as input. Some specific questions:  

1. How this approach differs from previous attempts using semantic segmentation for brain-score alignment? 
2. Why this  spatial latent estimation tasks succeed where others have not
3. Are there any implications of using  2D inputs to estimate 3D spatial properties, how this  relates to biological vision? 

Since the title starts with  "Vision Models...", I largely missed other models like ViTs  to be studied. But it seems that as it stands the results only cover CNNs?
1. I would suggest  a change of the title if that is the case, or inclusion of other kind of vision models in the results to make sure is align with the title.

questions:
Left in weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730761942423

thread_type:
official_review

<thread_object>
creator: reviewer_nmg4

<content>
summary:
The paper asks whether the ventral stream only codes object category information. By showing that CNN models that are trained for spatial understanding can also align with the neural reponse in the ventral stream, the author provide a critical reflection on the assumption that ventral stream in the brain primarily encode categorical information. The authors also introduce synthetic datasets generated by a 3D graphics engine (ThreeDWorld) containing precise spatial labels, overcoming the scarcity of natural datasets with spatial information. This dataset enabled systematic experiments to evaluate CNNs trained on spatial latents.

strengths:
* The paper asks an important question of the function role of the ventral stream besides image categorization. The discovery suggests that representation that is similar to the ventral stream neural response can be derived from training the network under spatial latent estimation objective as well. This contrasts the traditional belief that the primilary goal of the ventral stream is for object categorization only. 
* To demonstrate the proposal, the paper utilizes a 3D graphics engine to generate large dataset to tackle the shortage of spatial latent variables in previous studies. In Figure C.3 in supplementary, the paper demonstrates the importance of having larger scale dataset. 
* The paper shows that CNN trained on purely synthetic dataset can achieve a similar neural alignment score as these trained with natural images. 
* The paper has clear writing and presentations.

weaknesses:
* It could be more comprehensive if the paper utilizes natural images as training, e.g. one could extract sptial variables from SA-1B segmentation dataset to perform the experiments. 
* The evaluation of the neural alignment score is limited to the Brain-Score. More metrics would be helpful to consolidate the results. 
* The idea that ventral pathway is not responsible for classification alone is not new among cognitive neuroscience community. E.g Connor discussed in his paper that the retinotropic spatial information is transformed into relative positional information across ventral stream [1]. A discussion with related works in neuroscience community would be appropriated. 

[1] Connor, Charles E., and James J. Knierim. "Integration of objects and space in perception and memory." Nature neuroscience 20.11 (2017): 1493-1503.

questions:
* In Figure 5b, the decoding accuracy for classification is quite low for models trained for spatial latents, suggesting that the model trained for spatial latents is not suitable for classification. However, this contrasts with the CKA analysis in Figure 4b where the representation similarity is quite high between models trained for spatial latents and models for object categorization. How would you explain such difference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730902546914

thread_type:
official_review

<thread_object>
creator: reviewer_v5sv

<content>
summary:
Traditionally, neural network based models of the visual cortex have relied on CNN backbones trained for object recognition. Here, the paper investigates if CNNs trained with alternate objectives (specifically, spatial tasks like viewpoint and pose estimation) can be used instead, and how that impacts the learned representations. Their main finding is that models trained to predict such alternate spatial tasks can also be used to learn models of the visual cortex which perform just as well. Digging into it further, the authors find this happens because that the representations learned with these different objectives all end up learning quite similar representations (as measured by CKA). Also, authors find that variability in non-target latents helps models.

strengths:
1. **Very important problem**:  
   The paper broadens our understanding of the ventral stream by exploring if spatial tasks, not just object categorization, can also drive effective visual cortex models. This is impactful for both building better models of the cortex, and also provides a computational feasibility argument for better understanding what the cortex does in the brain.

2. **In-depth, exhaustive analysis**:  The authors provide thorough testing across multiple objectives, using extensive synthetic data and well designed metrics. Largely, the evidence matches the claim, and the findings seem quite reliable.

3. **Interesting findings**:  Discovering that spatial-task-trained models achieve similar neural alignment to category-trained ones suggests surprising flexibility in modeling the visual cortex. It might opens an interesting question---how do self-supervised models of the visual cortex behave in comparison to categories and spatial latents?

4. **Interesting new dataset**:   The synthetic dataset with spatial labels could be of use to future researchers.

5. **Writing and figures are high quality**: Clear writing and well-designed figures make complex ideas accessible and enhance the paper’s readability.

weaknesses:
1. Missing comparison to self-supervised models: The paper mentions them briefly, but does not provide experiments with these models. The best performing modern models used in AI are all increasingly relying on self supervised objectives.

2. Missing comparisons with Vision Language models: The majority of upcoming new models are all multi-modal. It would be interesting to see how similar or dissimilar these perform.

3. Missing some references: 
- The ideas of non-latent diversity are closely related to work in machine learning focusing on Data Diversity. It would be nice to connect this work to these papers [1,2,3,4,5]
- This paper presents results on BrainScore which is in-distribution. It would be interesting to see how these evolves out of the training data distribution, especially since invariances become very important in out of distribution settings [6,7,8,9]

References

1. Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I. and Lee, T., 2021, July. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning (pp. 5637-5664). PMLR.

2. Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H. and Boix, X., 2022. When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations. Nature Machine Intelligence, 4(2), pp.146-153.

3. Gulrajani, I. and Lopez-Paz, D., 2020. In search of lost domain generalization. arXiv preprint arXiv:2007.01434.

4. Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez-Paz, D., 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893.

5. Arjovsky, M., 2020. Out of distribution generalization in machine learning (Doctoral dissertation, New York University).

6. Ren, Y. and Bashivan, P., 2024. How well do models of visual cortex generalize to out of distribution samples?. PLOS Computational Biology, 20(5), p.e1011145.

7. Madan, S., Xiao, W., Cao, M., Pfister, H., Livingstone, M. and Kreiman, G., 2024. Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. arXiv preprint arXiv:2406.16935.

8. Pierzchlewicz, P., Willeke, K., Nix, A., Elumalai, P., Restivo, K., Shinn, T., Nealley, C., Rodriguez, G., Patel, S., Franke, K. and Tolias, A., 2024. Energy guided diffusion for generating neurally exciting images. Advances in Neural Information Processing Systems, 36.

9. Ponce, C.R., Xiao, W., Schade, P.F., Hartmann, T.S., Kreiman, G. and Livingstone, M.S., 2019. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4), pp.999-1009.

questions:
1. Were you able to decode categories from the models trained for spatial latents?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732646229770

thread_type:
official_comment

<thread_object>
creator: area_chair_fus8

<content>
title:
[IMMEDIATE ACTION NEEDED] Respond to author rebuttal

comment:
Dear Reviewers,

We are reaching the end of the discussion period (November 26 at 11:59pm AoE): please check the author rebuttal and post your response at your earliest convenience. Please also update your review accordingly, even if your view of the paper has not changed (to acknowledge that you have read the rebuttal).

Thank you,
--Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies alignment of visual representations between artificial convolutional neural networks (CNNs) and primate visual areas (from neural array recording data of primates observing visual stimuli). The authors study whether CNNs trained with tasks that involve prediction of spatial variables (e.g. object pose) on solely synthetically generated data can give rise to high alignment scores. The paper finds that models trained this way (using just a few predicted latents) achieve comparable alignment to models trained on multi-category image classification (with hundreds of categories).

The reviewers all agree that this is an interesting paper, it studies an important problem, is well-executed, and arrives at an interesting, relevant and novel conclusion. Concerns around experimental evaluation, baselines, and methodology were addressed in the rebuttal.

This paper will make a great addition to the conference. The authors are encouraged to take the reviewer feedback into account when preparing the camera-ready version of the paper.