Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations
Authors: Yudi Xie
Weichen Huang
Esther Alter
Jeremy Schwartz
Joshua B. Tenenbaum
James J. DiCarlo
Venue: ICLR 2025 Poster (2025)
Keywords: Vision
Convolutional Neural Networks
Representation Learning
Neural Data Alignment
Ventral Visual Stream
Computational Neuroscience
Primary area: applications to neuroscience & cognitive science
TL;DR: CNNs trained to estimate spatial latents, such as object position and pose, achieved similar neural alignment with the primate ventral stream as models trained on hundreds of categories.

Abstract: Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.

Published as a conference paper at ICLR 2025
VISION CNNS TRAINED TO ESTIMATE SPATIAL LATENTS LEARNED SIMILAR VENTRAL-STREAM- ALIGNED REPRESENTATIONS
Yudi Xie
Weichen Huang
Esther Alter
Jeremy Schwartz
Joshua B. Tenenbaum
James J. DiCarlo
Department of Brain and Cognitive Sciences, MIT
ABSTRACT
Studies of the functional role of the primate ventral visual stream have tradition- ally focused on object categorization, often ignoring - despite much prior evi- dence its role in estimating "spatial" latents such object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And closely related question: How different - if at all are representations learned from spatial latent estima- tion compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by 3D graphic engine and trained convolutional neural networks (CNNs) estimate different combinations of spatial and category la- tents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and sategory-trained models have very similar - but not identical - internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent vari- ability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many train- ing objectives, such spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is op- timized for object categorization only. As field, we need continue sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.
INTRODUCTION
David Marr famously introduced vision as "knowing what is where by looking" Marr 2010). The core functions of vision involve recognizing "what" an object is (object recognition) and determin- ing "wweere" it is by estimating the spatial latent variables of the object, such as its position and pose. Humans and primates excel at object recognition (DiCarlo et al.) 2012) and are clearly capa- ble of estimating spatial properties like object position or pose from visual input (Hong et al., (016)) Traditionally, has been believed that the "what" and "where" functions of vision are dissociated (Schneider) 1969) and supported by two different cortical pathways. with the ventral stream primar- ily associated with the "what" function of object recognition (Mishkin et al.) 1983). However, the exact nature of the "what" vs. "where" dissociation is debated (Goodale & Milner 1992.. Many previous studies found that the ventral stream encodes not only object identity (" ('what") ") but also rich object spatial information ("where")) (Connor & Knierim 2017), such object position and pose (Hong et al. 2016 Hung et al.] (2005). This indicates that interpreting the ventral stream's 's function as purely object recognition may be an oversimplification The exact functional role of the primate ventral stream remains to be fully understood.
*Corrspondence: wwwiemmiteedi
Code and datasets: htnps///gthabccom///aaiemmaamais

---

Published as a conference paper at ICLR 2025
Despite the known role of the ventral stream in representing object spatial information, leading CNN models of this pathway are mostly optimized for object categorization tasks, leaving spatial objectives underexplored. In this study, we ask: Might the ventral stream be optimized for spatial tasks? We examine how training CNNs to estimate spatial latent variables such as object position and pose affects their alignment with neural responses in the primate ventral visual stream. This area remains less explored largely due to the scarcity of large-scale datasets with comprehensive spatial labels beyond categories. To overcome this, we use a 3D graphic engine to generate large- scale synthetic image sets that contain detailed information on various spatial latent variables (Gan al., 2020). In most analyses, we trained CNNs to predict different spatial latent variables while keeping their architecture and training data constant, ensuring that any differences in their learned representations are attributable to the training objective. After training, we evaluated the learned representations and their alignment with primate ventral stream neural data using various methods, including the Brain-Score open science platform [Schrimpf et al. 2018).
We found that the neural alignment of models trained synthetic images is close to those trained on natural images, although their entire training experience has been purely on synthetic images. To our surprise, found training models spatial latent variables is sufficient to drive up neural alignments with the primate ventral stream, as models trained to estimate just a handful of spa- tial latent variables have neural alignment scores comparable to models trained a much larger number of categories. One might speculate that their similar alignment could be due to either (1) they indeed learn similar representations that support similar neural alignment, (2) or their learned representations are different, but the alignment metric is not sensitive enough to tell the differences between these models. Thus, we asked the question: How different if all are representations learned from estimating spatial latents or categories compared to each other? We used centered kernel alignment (CKA) (Kornblith et al., (019) to analyze the similarity between models trained on different latents. We found that, despite being trained on dramatically different objectives, these models learned very similar but not identical internal representations, especially in their early and middle layers.
This observation raises the question: Why do these models learn similar representations despite being trained to estimate very different targets? One hypothesis is that when models are trained to learn some target latents in a dataset that has variations in other non-target latents, they inadvertently learn to represent those non-target latents well, rather than developing representations that invariant to these non-target latents. This may happen even though the models are not explicitly trained to estimate those non-target latents. We provided evidence supporting this hypothesis by analyzing the learned representations while controlling the non-target latent variability in the training data. This partially explains why models learn similar representations, as learning a subset of the latents will inadvertently learn a converged representation that supports the inference of all other latents.
In summary, our findings are as follows:
CNNs trained on purely synthetic image datasets learned representations with neural align- ment scores close to those trained on natural images.
CNNs trained to estimate only handful of spatial latents achieve neural alignment com- parable to those trained on hundreds of categories. Models' neural alignment correlates strongly with their spatial performance.
The learned representations of CNNs trained to estimate different spatial and category la- tents are similar (but identical), especially in their early and middle layers.
Non-target latent variability helps models learn better representations of non-target latents rather than become invariant to them, providing a partial explanation for why models de- velop similar representations.
2 METHODS
2.1 IMAGE DATASETS GENERATED BY A 3D GRAPHIC ENGINE
Investigations on neural network models trained on spatial latent variables are often hindered by the lack of large-scale image datasets with comprehensive and accurate labels of the spatial la-
2

---

Published as a conference paper at ICLR 2025
b
Images from TDW-117 dataset
R
Apple
R
Z
Category
Translation
Distance
Rotation
Figure 1: Spatial latent variables and training dataset. (a) An illustration of the set of spatial latents that are available in the dataset for training models. In addition to the object category (Apple), the set of spatial latent variables we record are the following: translation (X. Y), distance (Z). and rotation (R"M, RM= "T (This image is for illustration only, not in the synthetic dataset.) (b) Example images in our dataset (TDW-117) for training CNNs. Each image contains one object with varying positions and poses against random background.
tent variables. Most previous studies on the neural alignment of vision models have focused on training models with categorization or various self-supervised training objectives. The effect of training on spatial latent variables remains less understood. To overcome this problem, we gener- ated several synthetie image datasets using ThreeDWorld (TDW) (Gan et al. 2020), a Unity-based 3D graphic engine (Figure b). We generated image datasets that contain up to 100 million im- ages from 117 object categories made up of 548 specific object 3D models (on average, about object models per category). Each image in the dataset contains one object rendered with a ran- dom position and pose against a random background. We record the ground truth information of a set of latent variables for each image (Figure [ a); these include the horizontal position æ, verti- cal position y, distance to the center point of the object 2, and the three Euler angles quantifying the object's rotation related to a pre-defined orientation of each object model {R"V R R All these latent variables are recorded in the spatial reference frame tied the camera. We also record each image's 's broader object category Ceet and the specific object identity Ctd in addition to spatial latent image variables. Together, we call them the latent set for each image the dataset {X, Y, Z, Rey _ R0T C Ctdd,, which contains both spatial and category latents. For more details about the image generation process, see Appendix A
2.2 NEURAL NETWORK TRAINING OBJECTIVES
We used the ResNet architecture family for visual estimation (He et al., 2016). Specifically, we focused on ResNet-50 and ResNet-18. We train models estimate different subsets of the spatial or category latents conditioned on the image using supervised learning. For discrete targets like {Ceat Crid in the latent set, used the cross-entropy loss. For continuous variables in the latent set, like {X, Y, Z, 1,",,," "2y=,R=a R=*), we used the square loss. Distance, translation, rotation, object category, object identity models are trained to estimate {2), {X, ( R R²*), (Creat) ), (CKd) respectively. When these objective are combined, such as "Distance + Translation", we take the arithmetic mean of the loss function of each task in the combination as the total loss. We used mini-batch stochastic gradient decent with Adam optimizer (Kingmal 2014) for training the neural networks. Models are trained until they reached a plateau in test performance in a held out test set. For more details the architecture, training, and evaluation of the neural network models, see Appendix B
2.3 ANALYSIS OF THE LEARNED REPRESENTATIONS
Neural alignment metrics. After training each model to estimate spatial latents or categories, evaluate how well models align with the primate ventral stream neural data at four stages of
3

---

Published as a conference paper at ICLR 2025
the ventral stream hierarchy (VI, V2, V4, IT). We used the Brain-Score open-science platform to calculate models' neural-alignment scores with each of the four brain regions (Schrimpf et al. 2018). The scores are numbers in the range of [0, 1J, with I meaning the closest alignment between between the model and data in the benchmarks. These scores measure how well the internal representations of models can predict neural responses in the corresponding brain regions held-out images. For analyses in Figure 2 and Figure we plotted the model's scores averaged over four different neural regions (VI, V2, V4, IT) as the overall alignment. For more details about the neural alignment measures, see Appendix C
Similarity of neural representations. In addition to measuring the models' alignment with the ventral visual stream, we investigated a closely related question: How different are representations learned from different spatial latent estimation tasks compared to each other and compared to models trained on categorization? To quantify the similarity between the representations of different models, we used centered kernel alignment (CKA) (Kornblihh et al.] 2019), which has gained popularity due to its thoughtful consideration of the invariant properties of similarity metric and ability to reveal similarity between models trained with different random initialization. To compute CKA, we extract models' activation at different layers spanning from early to late layers in response to 2000 held-out images in the TDW-117 17 image dataset. For more details, see Appendix D
3 RESULTS
3.1 LEARNING A SMALL NUMBER OF SPATIAL LATENTS PRODUCES TEURAL-ALIGNE CNN MODELS OF THE VENTRAL STREAM
We optimized the CNNs to estimate specific subsets or the full set of latent image variables for which we have ground-truth supervised targets from the graphic engine. These variables include spatial latents such as distance, translation, rotation, category information such as object category and identity, and combinations thereof (see Figure b). First, we found that models optimized purely on the synthetie image dataset (TDW-117) achieved neural alignment scores very close to the same CNN architecture trained on the natural image dataset, ImageNet. For example, models optimized to classify object categories in the synthetic dataset achieved more than 95% of the neural alignment score of the ImageNet-trained model (Object category classification: 0.412 compared with ImageNet-IKK 0.430). This suggests that synthetic image sets are effective in driving model- brain alignment, even though they do not capture all the nuances of natural images. Similar results were found in another model architecture, 18 (Table
We sorted these models by the number of latent variables they were optimized to predict, which corresponds to the number of output units receiving supervision (see Figure 2 For instance, a model trained to estimate object distance has only one output unit, while object category has 117 output units. We then investigated the neural alignment scores of these models. Intuitively, one might predict the "scaling hypothesis," which suggests that the more latent variables model is trained to predict (and thus the more supervision it receives during training), the more neurally aligned it would become. This is true for models trained to do object categorization the neural alignment scores
trained to estimate
with datasets that have different numbers of categories (TDW-N), of these models scale with the number of categories in the dataset. However, to our surprise, we found that models trained estimate spatial latents do not adhere the "scaling hypothesis." Many CNNs trained to estimate a very small number of latent variables achieved surprisingly high neural alignment scores, comparable models trained on over hundred object categories or more than five hundred object identities (Figure 2 a). For example, CNNs trained to estimate the distance of an object to the camera, which receive supervision from only one output unit, achieved neural alignment score of 0.399 - 97% of the score attained by CNNs supervised on all classification tasks and the full set of latent variables (0.411). Similar results were found in another model architecture (Table C.2
We also investigated the behavioral alignment of spatial latent trained models. We found their be- havioral alignment scores are not as good as category-trainee models (Figure C. Figure C.3 f).
The behavioral benchmark we used focuses on the consistency between models behavioral readout and primate behavior in categorization tasks. Thus, it may be biased toward category- -trinned mod- els. Further studies are needed to quantify whether spatial latent trained models are more behavioral aligned than category- trained models in spatial latent estimation tasks.
4

---

Published as a conference paper at ICLR 2025
b
Distance Translation Rotation
Training task
output
.2. IT public benchmarks)
targets
0.40
Distance regression
Translation regression
Mean alignment
Spatial latent regression (TDW-117)
Distance + Translation
3
Training targets
regression
Untrained
Distance Rotation
Spatil latents
Translation Rotation
Object category (TDW-N)
Object category (TDW-117)
Distance Translation Rotation
Object seebiyy (TDW-117)
Classification (TDW-117)
Object category classiticatio
117
dlassficctioon (TDW-117)
Object identity classification
548
Object category lImagaNe::1K 1K)
All spatial latents classification
674
Untrained
NA
Reference
ImageNet- IK classification
1000
10a
10
10a
10a
Number of supervised output units
Figure 2: Learning small number of spatial latents produces ventral-stream-alignel CNN models. (a) The neural alignment of models (ResNet-50) trained on different objectives (X-axis, number of output units, see panel b). Learning few spatial latent variables produced models that have neural alignment scores comparable to models trained on hundreds categories. Error bars shaded regions show the SD across multiple random seeds (N=5). All spatial + classification all spatial and all classification tasks combined. TDW-117 is our TDW dataset with 117 object cat- egories; TDW-N means datasets with N 2,4,6,8, categories. For breakdown of the individual region alignment scores, see Figure|[..] (b) The training tasks we investigated and their correspond- ing number of output units that receive supervision during training.
Category
Distance
0.4
0.4
Mean neural alignment score (VI, V4, public benchmarks)
10f
Pearson 0.94
0.3
Pearson 0.97
Num. training batches
p-value < 1e-10
p-value < 1e-10
0.2
0.0
1.0
-1.0
-0.5
0.0
10a
Object category classification accuracy
Negutive distance regression loss
Translation
Rotation
0.4
0.4
10
Pearson p-value
p-value
0.1
0.0
0.1
0.0
Negative translation regression loss
Negative rotation regression loss
Figure 3: The neural alignment of models correlates strongly with their spatial task perfor- mance. (a) The neural alignment scores correlate with models' categorization performance for models trained on object categories. This figure shows results from multiple random initializations. Each dot shows a ResNet-50 model colored by the number of training batches. (b-d) Models' neural alignment scores correlate with their spatial latent estimation performance when they are trained to estimate those latents respectively. The figures show models trained on (b) distance regression, (c) translation regression, (d) rotation regression (I outlier out of 60 data points where the loss is larger than 0.2 is excluded). For breakdown of the averaged score into individual scores, see Figure C.7
and Table C.

---

Published as a conference paper at ICLR 2025
Early layers
Late layers
1ayer1.0.relu
layer2.0.reht
1ayer3.0.ree r.rrel
layer4.0..em rorellu
Distance-
Translation
similerity (CKM
Rotation
Dis. Tra. Rot.
0.6
Obj Category
639
4:00
RSP
100
LM
051 R15
0.4
All spokial cla
DAN
521
ImageNet- IK
0.2
Untrained
4.
a28 2M0 6M0
AM
cte 6:6 631 an
co
e: eat net aes aot
0.0
Distance
Translatoon
Rotation Dic Tra, Ror Del, Catepory
spatial ci
Imageteer iK Untrained
pistancc
Translaton Rotation Die tre Rot oni Catopory
All spatia x imag imagehet
Untrained
Distance Transiation Rotation
Dic. Tra, Not Dbl. Category
spatial IK Imagetee Untrained
Dietance Tranelation Rotation Dir. Tra, Rot
onl Cateepary
All spatie x
imagefiet
Untrained
b
Similarity (CKA) to Ob), Categery
Training targets Category
Distance
Translation
0.5
Rotation
spatial
Imagellel-1K
Untrained
0.0
1ayer1.0.relu
1ayer2.0.relu
1ayer330.relu
1ayer4.0.relu
Figure 4: Models trained to estimate different latents learned similar representations. (a) Pair- wise similarity (CKA) between models trained on different targets different layers. Until the last few layers (e.g., 1ayer4.0.relu the representations of models trained on different spatial and category latents remain highly similar. Off-diagonal entries are the averaged pair-wise similarity between models in the two groups. Diagonal entries averaged similarity between differentt randomly initialized models in the same group. (All spatial + cla. means models trained on spatial and classification tasks) (b) The similarity between category classification models and models trained on other targets. Models trained on spatial latents remain similar to category models until the last few layers. Layer names in Table B.I intra Obj. Category shows the averaged distance between categorization models trained with different random initializations.
Previous studies about the functional roles of the primate ventral stream have shown that a CNN's object categorization performance strongly predicts its ventral-stream neural alignment (Yamins) jet al. 2014 Schrimpf et al.) 2018). This result is often interpreted as implying that evolution and/or postnatal development derived the primate ventral stream under the objective of object recognition. Our experiments showed similar results in models trained for categorization (Figure 3). However, found that models' performance spatial latent estimation strongly correlates with their neural alignment well when they are trained to estimate those latents respectively (Figure 3 b-d). These results support the interpretation that the primate ventral stream function is also estimate these spatial latent variables. Thus, we should not simply assume that the ventral stream is optimized for object categorization only. Instead, these results support that the ventral stream representations should be considered a set of general-purpose visual representations that can support both object categorization and spatial estimation tasks.
Finally, we investigated how the neural alignment of these latent trained CNNs scales with the size of the dataset used for training. We found that the neural alignment score CNNs trained on TDW images increases logarithmically with dataset size in low data regime (Figure C.2 However, this trend reaches plateau at around 1-10 million images; further increasing the training dataset size does not increase their neural alignment further.
3.2 MODELS TRAINED TO ESTIMATE DIFFERENT LATENTS LEARNED SIMILAR REPRESENTATIONS
Our results suggest that models trained different spatial latents achieve similar alignment scores with the primate ventral stream. Two possibilities could explain this result: First, the models may learn dissimilar representations, either because the neural alignment metric fails distinguish them,

---

Published as a conference paper at ICLR 2025
or because they explain non-overlapping variance in the neural data, leading to similar overall scores. Second, the models might learn similar representations, naturally resulting in similar alignment scores. To further differentiate these two possibilities, it is important to investigate how different if at all are representations learned from spatial and category latent tasks compared to each other. So, examined the similarity of the learned representations across models trained different tasks while holding the training data and model architecture constant.
We analyzed the representation similarity using centered kernel alignment (CKA) (Korbblihh al, 2019). We found that models trained to estimate different subsets of the latent variables learned very similar representations in early (layer1.0) 1.0) mid-layers (layer2.0, layer3.0) and started diverge in late layers (layer4.0). Models trained to estimate rotation showed the most significant divergence from other models a Figure D.3p. Models trained to estimate spatial latents remain similar to categorization-tailn traineed models until the last few layers (Figure 4 b, Figure D.I). In early to mid- layers, models trained to estimate different latents often are not more different than models trained to estimate the same latents but initialized with different random seeds (Figure D.2).
These results suggest that models trained on different targets can learn surprisingly similar represen- tations, especially in the early and intermediate layers. Meanwhile, our findings in Section show that models trained with combined objectives (e.g., Distance + Translation) do not achieve much higher neural alignment scores than those trained on individual latents (Figure 2 Table C.II If mod- els trained on different latents learned largely distinct representations, explaining non-overlapping variance in neural data, one would expect that combining these latent targets would produce more aligned model. However, this was not observed. Taken together, our results suggest that the sim- ilar neural alignment scores we observed may mainly be attributed these models learned. However, it is worth noting that, despite these similarities, differences still exist between models trained on different targets, especially in late layers. It is likely that the cur- rent brain- model comparison measures are not sensitive to many subtle differences that exist among these models.
the similar representations
3.3 NON-TARGET LATENT VARIABILITY HELPS LEARN BETTER REPRESENTATIONS OF THE JOINT LATENTS
We further asked a closely related question: Why did models trained to estimate different spatial and category latents learn similar representations as assessed by CKA? Although these models are trained to estimate different target latents, they are all exposed to the same dataset, which contains variability across all latent dimensions, most of which are not the training target. For example, distance is the target latent for a model trained to estimate distance. Translation, rotation, and latents other than distance are non-target latents. The variability in the dataset's non-target latents may play an important role in shaping the learned representations. One hypothesis is that when models are trained to estimate some target latents, the non-target latent variability may help the models inadvertently learn to represent those non-target latent variables as well. Consequently, when models are trained on a dataset with variability across all latent factors, they may converge to a representation that supports the inference of the entire set of latents, thereby explaining the similarity in representations across models trained on different tasks.
To test this hypothesis, it is crucial to determine whether non-target latent variability indeed facil- itates the learning of these non-target latents. Using a synthetic dataset, we performed causal ma- nipulations to investigate the impact of non-target latent variability on the learned representations. Our experimental design, illustrated in Figure involves measuring how well a model has learned to represent a latent variable by assessing the linear decoding performance of that latent different layers in a neural network. For instance, even if a model is trained to estimate a specific target latent (e.g., translation), it may still allow for decoding other non-target latents (e.g., category) at various layers. We established a baseline by training models with reduced non-target latent variability (e.g., a dataset with only one category) and compared its non-target latents decode performance to models trained with full non-target latent variability (e.g., dataset with multiple categories). We consider two possible outcomes:
(HI) the added non-target latent variability facilitates the learning of that latent, leading to improved decoding performance even though the model is not explicitly trained to estimate it; or (H2) the model's 's representation becomes more invariant to the non non-target latent, as may be irrelevant to the task of estimating the target latent.
7

---

Published as a conference paper at ICLR 2025
b
Decode non-largel latent:
Decode non-targel
Object Category
Translation
Nordderget bitern decotting pertorman aicee on the test oat
Model target: Distance
Model target: Distance
0.20
10-
0.0
0.00
Goggyyy decods socuraccy
Model Transtation
Model Rotation
0.10
deccode pe perfer
Model reduced lalent labilly
0.05
9.5
Baseline non target decoding perforancc
ma
Model target: Rotation
Model target: Obj, category
0.30
Clancr
Model training set: full non-target latent variability
FORT
0.05
0.5
faciltittes leaming of target
last Reduzee trer
0.00 Layers, Ayeer.g. drrrl...vvly Grualy
0.0
H2: makes rop. invariant to non-target lasent
Layers.f. nutu
Siggnoss
leyort P
layo2 Rniiu
Ryyer firuly Decoding layer
Deggari
layer
Figure 5: Non-target latent variability helps learn better representations of the joint latents in intermediate layers. (a) We compared the decoding performance of non-target latents different layers of the models. Using a model trained to estimate some target latents on a dataset that has reduced non-target latent variability as a baseline, we can expect two different outcomes. First (H1), if a model trained with full non-target latent variability decodes the non-target latents better, that in- dicates that the non-target latent variability facilitates the learning of that non-target latent, although the models are not trained to estimate them. Second (H2), it is also possible that additional non- target latent variability makes models become invariant that non-target latent, thus worse decoding performance for that non-target latent. Our experiments suggested that models learned better rep- resentations of the non-target latent with additional non-target latent variability, supporting HI. (b) Decoding performance for the non-target latent category when the models are trained to estimate target latents - distance (top), translation (middle), rotation (bottom). Non-target latent variability helped models learn better representations of these latents in the intermediate layers. (cat. category variability). (c) Similar results were seen when the non-target latent is translation, and the target latent is distance (top), rotation (middle), and category (bottom). When the target latent is category, models learn better representations of translation with additional translation variability, although they became more translation invariant in the last two layers. (tran. var. translation vari- ability). The results for another translation latent Y is similar (Figure E.2). (Error bars show the SD across 5 gross-validation runs x 6 randomly initialized models. "*" indicates a significant difference between the two groups, Mann-Whitne U test, p value < 0.05). Layer names as in Table B.
In our analysis, we compared models trained on spatial tasks with a dataset of reduced category variation against those trained with full category variation (Figure b). Despite not being trained to estimate the non-target latent (category), the added variability led to improved representations of that non-target latent in the intermediate layers (layers 2, 3, and 4), supporting H1. (For compari- son of the category decoding performance of spatial latent-trained models to that of category-trained and untrained models, see Figure B.I] Similar results were observed when used translation as the non-target latent and manipulated lated its variability (Figure An exception is in models trained estimate object category (Figure 5cc, bottom), where increased translation variability led more invariant representations in the last two layers (avgpool and fc), supporting H2. However, even in these cases, intermediate layers still showed improved representations of translation, supporting H1. Moreover, we found that models trained with full non-target latent variability often learned repre- sentations that are more similar to models trained directly to estimate those non non-targets (FigureD..4). More non-target latent variability may help models learn more similar representations within each group of models trained on the same target but initialized differently (Figure D.5] Figure D.6). In addition, found that models trained with full non- non-target latent variability achieved better neural alignment scores when the non- non-targe targtt latent was category (Figure E.3), though this effect was less pronounced when the non-target latent translation.
8

---

Published as a conference paper at ICLR 2025
In summary, most of our observations are consistent with the hypothesis that the variability of non- target latents helps develop better intermediate representations of them, though future work is needed to understand both the size and mechanism behind this effect. The variabilityy across all latents in the dataset likely aids the models in learning representations that facilitate the inference of joint latents. This partly explains the similarity in intermediate representations across models trained on very different targets, as observed in our previous results. However, multiple factors likely contribute to this similarity. While we have identified non-target latent variability as one such factor, others- such as rendering quality and the statistical properties of image stimuli may also play significant role in shaping model representations and need further investigation.
4 RELATED WORKS
The impact of task choice on neural alignment of vision models. Previous studies found CNNs trained on object categorization tasks strongly predict neural responses in the primate ventral stream Yamiins et al. [0114] Shaligh-Razayy & Kriegeskorte, (2144). Subsequent studies have predomi-
nantly focused on studying how different model architectures impact neural alignment (Schrimpf) et al. 2018 Kubilius et al. 2019, Kar et al.) 2019, Storrs et al. 202T) while keeping the training and evaluation task fixed object classification. More recent studies have begun explore the ef- fects of alternative training objectives on neural aligmment. Zhuang et al. (2021); Konkle & Alvarez [202); Yerxa et al. (2023) investigated models trained with sell-supervised learning objectives and showed that they are comparable to models trained on supervised classifications. Conwell et al. (2022) examined various tasks from the Taskonomy dataset (Zamir et al., (018)) such as semantic segmentation and mepth-map estimation. Our study continues in this trend of investigation but fo- cuses on estimating object-centric 3D spatial latents. These objectives can be framed as intuitive tasks humans can naturally do; how these objectives impact models' neural aligmment is unknown. In addition, many self-supervised learning objectives often push models to learn representations that minimize the differences between different augmentations of the images, such as random cropping and resizing, which may destroy the object-centric spatial latent information.
Representation learning from synthetic datasets. Synthetic datasets have become valuable tool in computer vision for addressing limitations associated with real-world data. Synthetic data has been effectively used train deep neural network models and has shown to benefit performance on semantic segmentation, object detection, pose estimation, depth, and optical flow estimation (Richter et al.] [0116; Ros al. 2016 Mayer et al.) 2016 McCormac et al., [017 Varol al. (017). Models trained with synthetic images from generative models learned visual representations that rival the performance of those trained on large-scale natural image dataset (Azizi al. 2023 Tian et al. 2024b aj. However, how well representations learned from synthetic datasets align with neural processes in the brain remains less understood. Our study showed that models trained exclusively on synthetic images can achieve neural alignment scores close to those trained on natural images.
Similarity of neural representation and convergence. Many previous studies developed various methods to analyze the similarity of representations learned in neural networks (Raghu et al.] 2017]] Morcos et al.] 2018: Kornblith et al. 2019 Dravid et al., 2023 Lene & Vedaldi, (2015) Bansal et al, 2021). Significant similarity can be found between representations of networks trained with different random initialization (Li et al. 2015 Kornblith et al., 2019 Bansal et al., (021). different network architectures trained on the same task (Laakso & Cottrell 2000 Raghu al. [017]] Kornblith al.)
2019), networks trained with different dataset (Lenc& & Vedaldi 2015, Kornblith et al.] 2019 Bansal al. 2021), and between models trained with supervised and self-supervised objectives (Bansal et al. 202T), although these results depend heavily on the specific definition of similarity. CKA has gained popularity partly due to its thoughtful consideration of the invariant properties of the sim- ilarity metric and its effectiveness at revealing the similarity between representations trained with different random initialization (Kornblith et al. 2019).. We used CKA to measure similarity and have shown that models trained to estimate very different spatial latents can also learn similar represen- tations. Earlier layers are more similar than later ones, consistent with previous findings in classifi- cation models trained with different random initialization (Kornblith et al. 2019). Meanwhile, there is a trend that peaherrperforming, wider, and larger networks learned more aligned representations MMrcos et al. 2018) Korbllith et al. 2019 Bansal et al. 2021), even representations of vision and language models could become more similar as performance increases (Huh et al.) 2024). Huh et al. 2024) hypothesized that model representations may converge they lear better representation

---

Published as a conference paper at ICLR 2025
of the reality of the world. Our experiments in Section 3.3 provide evidence that models may be learning the world even if the training task is only partially related to it.
DISCUSSIONS
In this work, we found that models trained to estimate a few spatial latents can achieve high neural alignment scores comparable to models trained on object categorization. Moreover, these models develop similar representations in the early to middle layers despite being trained on different latents. To understand why they converge, we explored the role of non-target latent variability in shaping the learned representations. Our analyses suggest that greater variability in non-target latents enhances the models' ability to represent these latents. This partially explains why models trained on different targets converge similar representations, they inadvertently learn to encode all other non-target latents. Our study on latent variability also connects to the broader literature in machine learning that explores the role of data diversity and augmentation in domain generalization (Arjovsky al., 2019 Gulrajani & Lopez-Paz 2020 Koh et al. 2021). Greater data diversity has been shown to improve out-of-distribution generalization (Tobin et al. Madan et al. 2022).
Our findings on the surprisingly strong neural alignment of spatial latent-trained models, along with the observed correlation between spatial latent performance and neural alignment, suggest that one should not assume the ventral stream is optimized for categorization only. Our findings in Section 3.2 indicate that computational models trained on either the "what" or "where" tasks may learn similar representations, which suggests that the separation of these functions may be an oversimplification. The inherent variability in the world may drive models to simultaneously represent both "what" and "where," " learning function may inevitably lead to learning the other. Mcanwhile, the representations learned from the different tasks we investigated are not identical. As a field, we need to continue to sharpen measures of comparing models to brains, it is likely that subtle differences exist among these models that our current comparison measures are not sensitive to.
note that
One way interpret our work is view the trained CNNs as hypotheses for efficient visual in- ference models, which compress the complex computations involved in visual inference typicaaly modeled through generative processes into an efficient feedforward pass (Yildirim et al. 2020 Dasgupta et al., 2020). We found that models trained on purely discriminative tasks to estimate some latents develop representations for other non-target latents, despite not being explicitly trained on them. This suggests that in order to learn certain objectives effectively, models may need to internalize the underlying generative structure of the world. For instance, object rotation is deñned relative to canonical view, which depends on object identity; thus, learning rotation also leads to learning the object category. However, we also found that models trained on translation tasks developed better representations of object categories with more category variability. The relation- ship between translation and object category is less intuitive, yet the models still learned both. This suggests deeper connection between these functions, which not previously known.
Limitations and future works. There are several limitations of the current work that invite future investigation. First, while the neural alignment of models trained on our synthetic dataset is com- parable to, it still does not fully match, that of models trained on large-scale natural datasets such as ImageNet. Future work could explore whether improvements in rendering quality, greater scene and object diversity, or training on other natural datasets, where feasible, could enable models to match or even surpass those trained on ImageNet. Second, when evaluating our models on other non-public benchmarks in Brain-Score (Figure C.5 and Figure C.6), we found that models trained models in most V1, V2, and
on spatial latent variables performed comparably to category-traine V4 benchmarks. However, among the five IT benchmarks, spatial latent-trained models signifi- cantly underperformed category-trained models in two benchmarks while outperforming them in one (Figure C.5). These three benchmarks are based on natural images that are out of the training distribution of our models. Further studies are needed to evaluate the extent to which our findings, primarily based on public benchmarks, generalize to broader set of datasets and benchmarks. This remains an active area of research (Ren & Bashivan 2024; Madan et al. 2024). Finally, our neural alignment evaluations rely heavily regression- based metrics. Future research could incorporate alternative metrics, such as representational similarity analysis (Nili et al. 2014 Schütt et al.) 2023), to assess whether our results hold under different analytical frameworks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We sincerely thank Kohitij Kar, Martin Schrimpf, Michael Lee, Chengxu Zhuang, Guy Gaziv, Robert Ajemian, and Lynn Sörensen for their valuable discussions and insightful suggestions on this project. This work was supported in part by the Semiconductor Research Corporation (SRC) and DARPA.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXivv2304.08466, 2023.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances neural information processing systems, 34:225-236, 2021.
Charles E Connor and James Knierim. Integration of objects and space in perception and memory. Nature neuroscience, 20(1 10:41333333 2017.
Colin Conwell, Jacob Prince, Kendrick Kay, George A Alvarez, and Talia Konkle. What can 1,8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pp. 2022-03, 2022.
Ishita Dasgupta, Erie Schulz, Joshua B Tenenbaum, and Samuel J Gershman. A theory of learning to infer. Psychological review, 127(3):41, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 434,, 2012.
Amil Dravid, Yossi Gandelsman, Alexei Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in model zoo. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1934-1943, 2023.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954 2020.
Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15()):00-25, 1992.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ha Hong, Daniel L.K Yamins, Najib J Majaj, and James DiCarlo. Explicit information for category- orthogonal object properties increases along the ventral stream. Nature neuroscience, 19(4):613- 622, 2016.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07997, 2024.
Chou Hung, Gabriel Kreiman, Tomaso Poggio, and James DiCarlo. Fast readout of object identity from macaque inferior temporal cortex. Science, 310(5749):863-8666 2005.

---

Published as a conference paper at ICLR 2025
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6):974-983, 2019.
Seyed-Mahdi Khaligh-Razav and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(11)=e003915, 2014.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.696) 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Talia Konkle and George A Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. Nature communications, 13(1):491, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR. 2019.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong. Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, et Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Aarre Laakso and Garrison Cottrell. Content and cluster analysis: assessing representational simi- larity in neural systems. Philosophical psychology, 13(1):47-76, 2000.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 991-999, 2015.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint urXiv:1511.0P1 2015.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho. Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how convolutional neural networks generalize to -iittrribittnnonnnnn category-viewpoit combinations. Nature Machine Intelligence, 4(2):146-153, 2022.
Spandan Madan, Will Xiao, Mingran Cao, Hanspeter Pfister, Margaret Livingstone, and Gabriel Kreiman. Benchmarking -istt--iistiiuutoon generalization capabilities of dnn-based encoding models for the ventral visual cortex. arXiv preprint arXiv:2406. 16935, 2024.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040-4048. 2016.
John McCormae, Ankur Handa, Stefan Leutenegger, and Andrew J Davison. Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In Proceedings of the IEEE International Conference Computer Vision, pp. 2678-2687 2017.
Mortimer Mishkin, Leslie G Ungerleider, and Kathleen A Macko. Object vision and spatial vision: two cortical pathways. Trends neurosciences, 6:414 1983.
12

---

Published as a conference paper at ICLR 2025
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen- Marslen-Wilson, and Nikolaus Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology, 10(4):e 2014.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dicksterd Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Yifei Ren and Pouya Bashivan. How well do models of visual cortex generalize to out of distribution samples? PLOS Computational Biology, (155):100114:5 2024.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision Asion-ECCV&A 2016: 14th European Conference, Amsterdam, The Netherlands, October / 11-14, 2016, Proceedings, Part II 14, pp. 102-118. Springer, 2016.
Genman Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3234- 3243, 2016.
Gerald E Schneider. Two visual systems: Brain mechanisms for localization and discrimination dissociated by tectal and cortical lesions. Science, 163(3870))895 902, 1969.
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J Majaj. Rishi Rajalingham, Elias B Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Heiko H Schütt, Alexander Kipnis, Jömn Diedrichsen, and Nikolaus Kriegeskorte. Statistical inference on representational geometries. Elife, 12:e82566, 2023.
Katherine R Storrs, Tim C Kietzmann, Alexander Walther, Johannes Mchrer, and Nikolaus Kriegeskorte. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. Journal of cognitive neuroscience, 33(10):2044 2064, 2021.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15887-15898, 2024a.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024b.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 109-117. 2017.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-optiiizz hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111(23)86119-622 2014.
Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and SueYeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Informa- fion Processing Systems, 36:24103-24128, 2023.
13

---

Published as a conference paper at ICLR 2025
liker Yildirim, Mario Belledonne, Winrich Freiwald, and Josh Tenenbaum. Efficient inverse graphics in biological face processing. Science advances, 6110).caxx5979 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference on computer vision and paltern recognition, pp. 3712-3722, 2018.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C' Frank, James DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Pro- ceedings of the National Academy of Sciences, 118(3):e2041166 18, 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDICES
A DETAILS OF SYNTHETIC DATASETS GENERATED BY A GRAPHIC ENGINE
A. GENERATION OF THE TDW-117 DATASET
We use ThrleDWorld Gan et al. 2020), a Unity-based 3D graphic engine, to generate the training dataset to train the latent estimation models from scratch. The image dataset we used to train the ResNet- 18 and ResNet-50 models containing 1.3 million images from 548 specific object 3D models belonging 11 117
object categories (used in Figure 2 Figure C.Il and Figure C.3) We called this TDW-117 dataset. To investigate how neural alignment scales with dataset size (Figure C.2) we generated larger dataset that contains 100 million images. For all the datasets, 3D scene spaces including indoor and outdoor scenes. To generate each image, we randomly choose one camera position within the 3D scene space and randomly sample an object position and pose in a spherical space around the camera while making sure the object is not placed far below the camera. We then point the camera toward the object and randomly rotate the camera before taking an image. The above procedure ensures that there are variations in distance, object translation, and rotation related to the camera. We also randomly sample the skyboxes the 3D engine to add additional variations in lighting and background. All objects are scaled to a uniform size so that their longer axis the same.
used 10 different
For each image, we record the object center's position related to the camera's 3D spatial reference frame (I, y, 2), with x and y being the coordinate in the horizontal and vertical axis and x is the coordinate in the axis pointing into the image. We used x and y for the translation regression task and 2 for the distance regression task. Meanwhile, we also record the objects' 3D rotation relative to the camera's s 3D reference frame, characterized by the 3 Euler angles (R*y, R"= R There is only one object model in each image. We also record the object's category Ccaa and the 3D model that has generated the image as the object's identity Cld.
A.2 GENERATION OF DATASETS WITH REDUCED LATENT VARIABILITY
In addition to generating the full dataset with all the categories (TDW-117), we generated multiple datasets that contain the same number of images as TDW-117 (1.3 million images) with reduced number of object categories. We call these TDW-N, where N represents the number of broader object categories used generate the datasets. For each dataset, only object models belonging to the N classes are used to generate the dataset. For example, when N=2, the dataset contains only images from two object categories. For the analysis in Figure 2 Figure C.I and Figure C.3
used datasets where N = 2, 4, 6, 8, 16. For the analysis in Figure 5 Figure E. and Figure B.3 the reduced category variation dataset used only N category, and the full category variation dataset is the TDW-117 dataset.
We generated a dataset with reduced translation variation, and are used in the analysis of Figure 5 Figure D.4] Figure Figure D.6] Fi. and Figure [. The full translation variation dataset the TDW-117 dataset. For the reduced translation variation dataset, we generate a dataset with the same size and setting in 117, except that we set the camera to point at the object center before taking a picture each object. So that the objects always appear at the center of each image, thus having translations very close to O.
B MODEL ARCHITECTURE AND OPTIMIZATION DETAILS
B.1 MODEL ARCHITECTURES
For analyses in Figure 2 Figure 3 Table C.I Figure C.I Figure C.2
Figure C.6
Figure C.7
and Table C.3 we used the ResNet-50 architecture. We found the neural alignment score results
from ResNet-50 is qualitatively similar to results from ResNet-18 (Table C.2 Figure C33
so we
used 18 for subsequent analysis of neural representations in Figure 4
Figure 5 Figure C.4
Figure Appendixxxxxi Daand Appendix E We implemented the model backbone using the PoTorc We changed the last linear layers of
library hhnnss/yyyoccannggooooomnn/nnnnd
15

---

Published as a conference paper at ICLR 2025
the models to accommodate outputs for different tasks that investigated. The architecture of ResNet- 18 from which we extract activation to perform representational similarity analysis is shown Table B.1
Table B.I: The architecture of ResNet-18. Bold font shows the layers from which extract the output activation for analysis. We labeled these layers as "layer Pryeer...relu,, "layer2.0.relu", "'aaer3.0.relu", and "layer4.0.relu" respectively. The activations are extracted from the ReLU layer after the activation from the skip connection is combined.
Block name
Layer architecture
Layer position
Conv (7x7, 64)
BatchNorm ReL.U
1/18
MaxPool
Cony (3x3, 64) BatchNorm ReLU
2/18
layer1.0
Conv (3x3, 64) + BatchNorm + ReLU
3/18
(skip connection combined before last RcLU)
Cony (3x3, 64) BatchNorm ReLU
4/18
layerl.1
Conv (3x3, 64) - BatchNorm + ReLU
5/18
(skip connection combined before last ReLU)
Conv (3x3, 128) -> BatchNorm - ReI.U
6/18
layer2.0
Conv (3x3, 128) + BatchNorm ReLU
7/18
(skip connection combined before last ReLU)
Conv (3x3, 128) + BatchNorm - ReLU
8/18
layer2.1
Conv (3x3, 128) + BatchNorm + ReLU
9/18
(skip connection combined before last ReLU)
Conv (3x3, 256) + BatchNorm
Rel.U
10/18
layer3.0
Conv (3x3, 256) - BatchNorm ReLU
11/18
(skip connection combined before last ReLU)
Conv (3x3, 256) + BatchNorm
ReLU
12/18
layer3.1
Conv (3x3, 256) + BatchNorm - ReLU
13/18
(skip connection combined before last ReLU)
Conv (3x3, 512) - BatchNorm - ReL.U
14/18
layer4.0
Conv (3x3, 512) - BatchNorm ReLU
15/18
(skip connection combined before last ReLU)
Cony (3x3, 512) > BatchNorm > ReL.U
16/18
layer4.1
Conv (3x3, 512) + BatchNorm + ReLU
17/18
(skip connection combined before last ReLU)
AvgPool Linear (fc)
18/18
B.2 MODEL TRAINING OBJECTIVES AND OTHER TRAINING DETAILS
We train models estimate different subsets of the spatial category latents conditioned the image using supervised learning. For discrete targets like (Coat Chidy in the latent set, we used the cross-entropy loss. The loss function for object category classification is given below. Suppose there are N data samples in a mini-batch drawn from the dataset, which we denote as {(Te, Iz. 1/s 2,, r "x,,"" .rye ri" oz ..4% E 1, 2)..N) where denote the image input. We parame- terize the model as (T), the loss function is given by:
1 N
Le
log(s (C =Creat===
j((+) 1(j caat
Where there are M possible class for label Creat and 1fj cfatt is the indicator function that takes value I only when eaat.
16

---

Published as a conference paper at ICLR 2025
Similarly, the loss function for object identity classification is given below.
1
Les
log(Sa(C'd - - -1i cf
For continuous latents like [X., Y, Z, R R we used the square loss. The loss function for distance regression models is given below:
N E (fo(Z))) - 23)²
distance
(3)
N
The loss function for translation regression models is the following:
1 Lz + Iny 2
Liranslation
(4)
N
Lg N
(f(())))) - xz
(5)
1 N _ (f - 96))
Ly
(6)
For rotation regression task, we use the squared error between the model outputs and the sin and cos of the three Euler angles { R R[a} characterizing the rotation of the object in the cam- era's 3D reference frame. For each Euler angle, the model has output units fol (R (4)) and fa Rox [4)) that are trained to match the sin and cos of the Euler angle respectively. The following the loss function:
Lrotation
LR= LR + LR=2
LR
1
(sin (fo (4)) sii(r,"))) + (cos (Re L)) cr,"""")))"" (r)
(8)
2.V
Y=1
Lgas= 1
(R sin(r""))) (R [I:) ))
(9)
2.V
1=1
N
Lggs
Σ
( [T)) sin(r (cos(fo(R [T)) +
cos(r
(10)
2N
When these objectives are combined, such as "Distance + Translation", we take the arithmetic mean of the loss functions of each task as the total loss. Loss function for Distance Translation:
Ldis.
Ldisttnce + Ltranslation
(11)
Loss function for Distance + Traassaatoo Rotation:
Lyjee traitttra.too. Ladistnnce + Iltranslation + Lrotation
(12)
Loss function for All classification + all regression:
L cla. + all 5 Tedistance + Ietranslation + Laratation + Tugeat + Lew
(13)
In addition to these models, we also trained models using the cross-entropy loss on the ImageNet dataset (Deng et al. 2009). which has 1000 classes. The untrained models we used are randomly initialized with different random seeds.
Before training, we normalized the continuous variables (X,Y, Y, Z) to have standard deviation of and centered around We used mini-batch stochastic gradient descent with Adam optimizer Kingma| 2014) with learning rate of 0.001 for training the neural networks. Models are trained until they reach a plateau test performance in a held-out test set. All experiments are trained with a batch size of 64, ResNet-50 models are trained with 1,000,000 batches, except those used for analysis in Figure C.2 are trained with 1,500,000 batches. ResNet- 18 models are trained with 500,000 batches.
17

---

Published as a conference paper at ICLR 2025
C VENTRAL-STREAM ALIGNMENT MEASURES
We use the benchmarks in the Brain-Score open science platform to measure the align- ments of models trained with different tasks with the ventral stream neural and behav- ioral data. For most of our analyses in the main text, we used the set of public bench- marks. Specifically, we used "FreemanZiemba2233pbbii V1-pls" benchmark for V1 alignment, FreemanZiemma201330 alignment, and (pppiiHHon2OSSpuii...IT7 behavioral alignment scores from the representation of the penultimate layers ("avgpool' layer) us- ing the "Rajalingham201 8public-i2n' benchmark. We also evaluated a limited set of models on other non-public benchmarks on Brain-Scou (see Figure C.3 and Figure C.6).
V2-pls' for V2 alignment, MajjajHong2015pubbii V4-pls" for V4 for IT alignment. In addition, we got the models'
The neural benchmarks (VI, V2, V3, V4) measure how well a trained linear readout from the neural network models activation predicts the electrophyssological data in the respective brain regions. The procedure first identifies the best predictive layers for each region from a set candidate layers in the model. The selected layer is then used score against the data. In evalua- tions using public benchmarks, we used the following candidate layers for ResNet-18 models: "relu', 'layer1.0g 1ayer1...reelus .0.relu', 'layer1.1.relu" 'layer2.0.relu' "layer2.1.relle 1.rllu', 'layer3.0.res 0...lu',,, 'layer3. I.rel',, 'layer4.0.relu', 1layer4.I.relu.. 'aggpoo'," 'fc" J. We used the following candidate layers for ResNet- 50 models: {'relu', 'layerl r...u..lu'' 'layerI.I.rllu' 'layer1.2.rele 'layer2.0.relu', 'layer2.1 'layer2.2.reluu Ilay'' 1layer3.4.relu', 'layer3.5.relu', 'layer4.0.relu' 'layer4.1 .relu', 'layer4.2.rell 'avgpool', 'fe' }. In our evaluations using non-public benchmarks, we used the following candidate layers for both ResNet-18 and ResNet-50 models: { I'convl' 'layerl', 'layer2', 'layer3', 'layer4', 'fc' ]. For havioral scores, we used the activation from the penultimate layer 'vvgoool' for scoring. Figure(C.t shows the layer assignments for ResNet-18 18 models in our experiments. For more details of the procedure scoring, see SSchrimpf et al. (018) and https://wwwwbbriinscoortt
r'ree', Mayer3.1.relu 'layer3.2.relu "layer3.3.relu'
Table C.2 and Figure C.3 show the detailed region scores and behavioral scores of ResNet-18 mod- els. Table C and Figure C.1 show the detailed region scores and behavioral scores of ResNet-50 models. Figure C.2 shows how the mean neural alignment scores scale with the number of images used to train those models.
Table C.I: Neural alignment scores of ResNet-50 models on Brain-Score public benchmarks. Each entry is the mean score of N=5 different models trained under the same condition with different random seeds. dis.; distance, tra.: translation, rot.: rotation.
output targets
Neural mean
Training task
V1
V2
V4
IT
Behavior
Distance
0.279
0.267
0.577
0.472
0.399
0.107
Translation
0.283
0.275
0.580
0.480
0.405
0.072
Dis. Tra.
0.273
0.284
0.588
0.483
0.407
0.093
Rotation
6
0.302
0.283
0.568
0.460
0.403
0.171
Dis. Rot.
0.284
0.272
0.571
0.481
0.402
0.212
Tra. Rot.
8
0.276
0.271
0.580
0.484
0.403
0.141
Dis. Tra. + Rot.
0.277
0.274
0.578
0.491
0.405
0.143
Object category
117
0.285
0.280
0.582
0.501
0.412
0.440
Object identity
548
0.271
0.278
0.588
0.509
0.412
0.443
All spatial classification
674
0.262
0.289
0.586
0.506
0.411
0.413
Untrained models
NA
0.212
0.108
0.387
0.246
0.238
0.020
ImageNet-IK,
1000
0.250
0.324
0.590
0.557
0.430
0.416
18

---

Published as a conference paper at ICLR 2025
b
VL. DDD JCANGG
0.30
0.25
0.25
congany mis 9n
0.24
en daaaty TEW 1IT
(magetee11
0.25
0.2
0.10
10"
Lei
10t
12'
101
10t
10"
30'
10
Numbbr of superviaes sutput unis
Number of supersbed euppttrrs
humber of supervase sutuutunts
0.55
0.50
24hurse algemeettttnm score
0.50
0.60
02
0.25
0.40
0.25
10a
307
117
10
12a
307
12'
10
127
307
107
101
supervised oupput WRitS
Number Superyaeee pupput
Namber suservised OUTEAR unis
Figure C.I: Neural and behavioral alignment scores of ResNet-50 models on Brain-Score public benchmarks. (a) Mean neural score. (b) VI alignment score. (c) V2 alignment score. (d) V4
alignment score. (e) IT alignment score. (f) Behavior alignment score. For results on other non- public benchmarks, Figure
(V1, V2. V4, IT public benchmarks.
0.40
Mean alignment
Training task
ImageNet- 1K
Untrained
Translation
Rotation
Obj, category
spatial classification
0.25
10a
10a
10a
107
10a
Dataset size, number images
Figure C.2: Neural alignment (y-axis) of ResNet-50 models trained on different tasks as a function of the number of images in the image dataset (x-axis) used to train them. For all the tasks, the alignment scores initially scale with the dataset size but reach a plateau at around million images. The alignment scores for models trained different tasks very similar when the training images are at or more than million.
19

---

Published as a conference paper at ICLR 2025
Table C.2: Neural alignment scores of ResNet- models on Brain-Score public benchmarks. The score is averaged over different models trained under the same condition with different random seeds (N=5 for untrained models, N=8 for the rest). dis.: distance, tra.: translation, rot.: rotation.
# output targets
Neural mean
Training task
V1
V2
V4
Behavior
Distance
0.264
0.248
0.562
0.448
0.381
0.109
Translation
2
0.266
0.258
0.560
0.455
0.384
0.048
Dis. Tra.
0.260
0.257
0.563
0.458
0.384
0.034
Rotation
6
0.254
0.245
0.539
0.452
0.372
0.118
Dis. Rot.
0.255
0.245
0.561
0.463
0.381
0.155
Tra. Rot.
8
0.265
0.256
0.564
0.470
0.389
0.064
Dis. Tra. + Rot.
9
0.265
0.259
0.566
0.468
0.389
0.078
Object category
117
0.248
0.266
0.583
0.492
0.397
0.352
Object identity
548
0.247
0.274
0.581
0.490
0.398
0.352
All spatial classification
674
0.254
0.269
0.588
0.504
0.404
0.354
Untrained models
NA
0.217
0.114
0.381
0.246
0.239
0.026
ImageNet-1K
1000
0.250
0.307
0.584
0.512
0.413
0.327
b 5.28
0.40
0.30
V2 atatmmentto scoee
6.25
0.24
Oblect 770A
Oblec cnapery
ITAA L.?
Moan
pectal foatin
0.15
Oblec
0.25
0.30
10a
200
102
20"
Kumber of euperviee curpurt
Number of supervieed outpor units
Nunber of supoviied cupputuris
0.60
0.50
0.35
Bohawtr Sitattraaggooott.cooo scote
0.50
0.20
0.65
0.35
0.15
0.50
122
10F
10P
107
10F
16:
107
107
10)
Number of alauparvied sutoutunts
Number of superviedd cutpor Un
Numbar of superviaed cutput
Figure C.3: Neural and behavioral alignment scores of ResNet-18 models on Brain-Scort public benchmarks. (a) Mean neural score. (b) VI alignment score. (c) V2 alignment score. (d) V4 alignment score. (e) IT alignment score. (f) Behavior alignment score. For results on other non- public benchmarks, Figure C.5.
20

---

Published as a conference paper at ICLR 2025
Layer assignment for dej Gategory models
Layer assignment
Behaxtor
loyet Roads
Deeel Dyaiy deeu
ubye? Lonty
loyer ctell
Hymuz Zondg
Ulyed mim
Lyyer Lett
argioos
hamey
carl
Layer assignment for Distance models
Layer assignment for Translation models
Layer assignment for Rotation models
10
10
0.5-
05-
055
Gategry tre-gep
Dittacc
keaaten de
0.0
weu
Inveli rola
spperd rel
rats
layert zela
lpvond mu
relor
augnon
0.0
Invela rale
lower Pals
ipper rels
loyer velr
layert zela
Layent reto
(svord mu
argioo
lover me
rate
layer relo
Jnport Laye Lasto
Lower rit
layent Leeu
loyer
lyper
Layer names from
Layer names from
Layer remey
grement spatial
Layer assignment Imagettet-1K
Layer assignment
1.0
1.0
3.0
0.
5.5
Wisteen ca Galggre CD
Ieapshe Do OR Creper Oi
Untoter OM Celogrr Dct
00
Uvedl u mis
ral
teyer vela
deyee reld
Uvved mia
Uved Uvez mas
ril
aayer vell
layeed Upyer mia
orgiral
byed mu
bydt Lanu
Upei' mid
Gaper rames
carly to M00
names
corly to a0l
carly to lane)
Figure C.4: Layer assignment details of ResNet-18 models trained on different tasks compared to their representational similarity to category- trained models. The dot positions indicate the best predictive layers for the regions on Brain-Score public henchmarks. The behavior readout layers assigned the penultimate layers by hand. (a-g) shows models trained on (a) object category, (b) distance, (c) translation, (d) rotation, (e) all categories and all spatial latents, (f) ImageNet- K classification, and (g) untrained models. Most of the VI, V2, V4, and IT layers of latent-trained models are assigned to layers before their representations diverge significantly from the category- trained models. Compared category- trained models, the layer assignments spatial latent trained models are slightly shifted towards earlier layers.
21

---

Published as a conference paper at ICLR 2025
a
b
19
Training Bargets
Spatial
STDW
spaltel caaseiication (TOW3311)
ect conogery Imageterr 14)
2200
e
Neural benchmarks mean
1.0
Neural benchmarks mean scora
0.8
201
0.6
0.4
0.2
0.0
100
10a
102
10a
Number supervised output units
Figure C.5: Neural alignment scores of ResNet- 18 models in other (non-public) neural benchmarks on Brain-Score. (a-d) show results for (a) V1, (b) V2, (c) V4, and (d) IT benchmarks. The top panel the region. Each point represents the result from
shows the mean score across all benchmarks one model. (e) shows the average of V1, V2. V4, and IT scores. Models trained on spatial latents performed comparably to category-trainee models in most V1, V2, and V4 benchmarks. However, among the five ITT benchmarks, spatial latent-trained models significantly underperformed category- trained models in two benchmarks "SSnghaviMurtyy00001112 "Sanghavilozwik2220.. 2000000. while outperforming them in one aannci2200
22

---

Published as a conference paper at ICLR 2025
a
b
C
Vi mans
Theamror S1embn0113 V7
MeNi lengooa.... Selipe
MAN lingious
Nhe cha seyiue wraduuat
Sanggno Nangowobe2
30'
Training
Untroined
Spattal letents (TONA117)
Sanghee Mangyaao
Diject catogery (1.9A17))
Coject isenthy (T006317)
spatial cissaiicaion (TW
Coject categery 0magetwe-120
Sunghand Now
e
Neural benchmarks mean
1.0
Neural benchmarks mean zcore
0.8
Samber cosceeeee S0ReE
No serieed suA
0.6
WAACOETS anceveentt
0.4
0.2
0.0
10a
101
10a
10a
Number supervised output
Figure C.6: Neural alignment scores of ResNet-50 models in other (non-public) neural benchmarks on Scoie. The layout is the same as Figure C.5
23

---

Published as a conference paper at ICLR 2025
a
b
Model target: Distance
C
30
uva < Ie-10
0.803
0.28
Pearson 0.966
p-value < 1e-10
0.575
p-value 1e-10
algnment
algnment
28
V2 elgnment
0.525
0.24
0.500
02
0.22
0.475
10a
Num. of training batchos
0,450
-1.0
-0.5
0.0
-1.0
-6.5
00
-100
-0.3
0.0
-10"
Nogative distance regression loss
Nogative distance regression lass
Negative distance regression oss
d
100
Pearson
948
Behavior limmmntttton score
Pearson
-0
aiagnment score
0.45
0.12
0.40
0.10
0.08
0.35
ie regression
Nogative distance regression
Figure C.7: breakdown of neural alignment scores plotted against the overall distance regression performance of ResNet-50 models trained to estimate distance. (a-e) The alignment scores for (a) V1, (b) V2, (c) V4, (d) IT, and (e) behavior on Brain-Score public benchmarks. We see strong corre- lations between the distance regression performance with all neural alignment scores. The distance performance does not have positive correlation with the behavioral alignment score. This figure shows results from multiple random initializations. Each dot shows ResNet-50 model colored by the number of training batches. For model trained with other objectives, see Table C.3
Table C.3: Correlations between model performance and neural/behavioral alignment scores of ResNet-50 models. The entries in the table show Pearson correlation coefficients between the model performance (negative losses on held-out test set) and their neural/behavioral alignment scores on Brain-Score public benchmarks. For example, the "Translation" row shows the correlation between the translation regression losses and alignment scores from models trained translation. The "Ro- tation" row shows the correlation between the rotation losses and alignment scores from models trained on rotation. There strong positive correlations between all neural scores and all of the target performance. Distance and translation performance do not positively correlate with behavior scores, while rotation and classification tasks do.
Neural mean
Model targets
V1
V2
V4
IT
Behavior
Distance
0.803
0.866
0.970
0.948
0.975
-0.273
Translation
0.807
0.874
0.931
0.912
0.941
-0.073
Rotation
0.659
0.723
0.706
0.785
0.736
0.744
Object category
0.735
0.947
0.945
0.969
0.958
0.982
Object identity
0.732
0.964
0.948
0.979
0.969
0.984
24

---

Published as a conference paper at ICLR 2025
D SIMILARITY OF NEURAL REPRESENTATIONS
We use centered kernel alignment (CKA) to measure the similarity between the learned representa- tions of different models (Kornblith et al. 2019). For most of our analysis, we extract the internal activations of our models from different layers in ResNet-18 as in Table B. Those are acti- vation of our models in response 2000 held-out test images in the TDW-117 dataset that have full variations in all the latents we investigated. For all of our analysis, we used linear CKA. The implementation follows Korblltth et al. (2019).
For the analysis is Figure 4 we analyzed the pair-wise similarity between models trained with dif- ferent objectives and models trained with the same objectives but initialized with different random seeds. To visualize the relationship between different groups, we used nulti-dimensional scaling (MDS) to embed the model similarity matrix in a 2D space. The distance matrix used for MDS is calculated using the Euclidean distance between the row vectors of the similarity matrix between all of our models.
Similarity (CKA) to ImageNet-1K
Training targets
intra ImageNet-16
Distance
Translation
0.5
Rotation
Dis.
Obj. Category
All spatial cla.
Untrained
0.0
layer1.0.relu
layer2.0.relu
1ayer3.0.relu
layer4.0.relu
Figure D.1: The similarity ResNet-18 models trained with different tasks to ImageNet classifica- tion (ImageNet cla.) models at different layers. intra ImageNet cla. shows the averaged distance between categorization models trained with different random seeds.
25

---

Published as a conference paper at ICLR 2025
layer1.0.rel r.
b
aaer2.0.. r.llllu
1ayer3.0.reie
d
ayer4 0.relu
Distance Others
Distance Others
Distance Others
Distance Others
1.00
1.00
0.75
0.8
0.50
0.50
0.50
0.25
0.25
0.25
Translation vs. Others
Translation vs. Others
Translation vs. Others
Trans Iraoniation vs. Others
1.0
1.00
0.75
0.75
0.25
0.6
0.25
0.00
Rotation Others
Others
Rotation Others
Rotation
1.00
1.00
0.75
0.8
0.50
0.50
0.50
0.25
0.25
Tra. Rot. vs. Others
Di. Tra, Rot. vs. Others
Dis. Tra Rot, vs. Others
Dis. Tra. Ret. vs. Others
1.0
1:00
0.75
0.75
Representational similarity (CKA)
0.50
0.25
0.6
0.25
0.00
Obj. Category Others
Obj. Category Others
Obj. Category Others
0b). Category Others
1.00
1.00
0.75
0.8
0.50
0.50
0.50
0.25
0.6
0.25
spattal Others
spattal
spatial Others
All spetial cla. vs. Others
1.0
0.75
0.75
0.8
0.50
0.25
0.25
0.00
imgeetet 1K vs. Others
Imagelet: 1K Others
ImageNet vs. Others
ImageNet-1K Others
1.0
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.6
0.25
0.00
Untrained vs.
Untrained
Untrained
0.75
0.8
0.50
0.25
Distance
Rotalion
Dia Tha Bot.
08) Catapory
ImageNeo-16
untrained
Distancc
Transiation
Rotalion
Ds. Rot.
00) Canogory
ImageNee-1K
Untrained
Distance
Transtction
Rotalion
Dis, Tra Rot
O0) Catopory
All spatial cla.
Imageeee IK
Untrained
Dis Tra, Rot.
Oa) Category
All spatial da...
ImageWer-)K
Untrained
spatal
spattal
Training tasks
Figure D.2: Representational similarity (CKA) of ResNet-18 models trained on different tasks. Distance vs. Others means the CKA between models trained on distance regression and models trained on other tasks. Intra-group similarity measures the similarity of models within the same group and trained with different random initializations. Between-group similarity measures pair- wise similarity of models in two different groups. In early to middle layers, most models trained to estimate different spatial latents are not much more dissimilar than models trained the same group with different random seeds. Error bars shaded regions show the SD of pair-wise model similarity. (a) early layer, layer1.0. O.. (b, c) middle layers, layer2.0, layer3.0. (d) late layers, layer4.0.
26

---

Published as a conference paper at ICLR 2025
MDS of model similarity matrix
layer1.0.relu
layer2.0.relu
layer3.0.relu
layer4.0.rel
MDS dim.
Sernace
ketamin Da. Do. Io..
041 Calearry spaial C0.
MDS dim.
Tharo aloo
Figure D.3: Multi-dimensional scaling (MDS) visualization of similarity matrix of ResNet- 18 mod- els trained on different tasks. The representations of models trained on different spatial latents are mixed together until the last few layers (.relu...reelu..
1.0
Similarity Transiation (CKA)
Translation var. Full Reduced
Distance
0.5
Rotation
Dis. Rot.
Category
intra Translation
0.0
layer1.0.relu
layer2.0.relu
layer3.0.relu
layer4.0.red
b
1.0
Similarity Category
Category var.
Full Reduced
Distance
0.5
Translation
Rotation
Tra. Rot.
intra Category
0.0
layer1.0.relu
1ayer2.0.relu
1ayer3.0.relu
layer4.0.relu
Figure D.4: Non-target latent variability helps models learn representations that are more similar to models trained directly on that non-target latent. (a) Similarity between wanslation-trained models (ResNet- 18) and models trained on other targets. Although not trained to estimate translation, mod- els trained data with full translation variability learned representations that are more similar to translation-trainet models than models trained with reduced translation variability. (We found this true in most cases in layer3 and layer4, except rotation in layer4, which has large SD) (b) Simi- larity between eetegoorytrriinee models and models trained on other targets. Although not trained to estimate category, models trained data with full category variability learned representations that models than models trained with reduced category variability. (We found this is true in most in layer3 and layer4, except rotation, which has large SD). For both (a) and (b), error bars show the SD over CKA scores of 6x6=36 pairs of models.
are more similar to category-traines
27

---

Published as a conference paper at ICLR 2025
rayer1,0.relu
layer2. r..lluu
layer3.0.ralu
layer4.00 rellll
emiartt (COA)
saraar
canase
awads
B.c.
0.
04
3.4
04
Pul
02-
9.2
02:
Dutance
Rotation
Dic. Aut
Callgooy
Derence
Aotstion
Ois Rut.
Catepory
Ousence
Autation
Dis. Aut
Calogory
Ditence
Assation
Dlr. Rot.
Catogory
Figure D.5: The intra-group similarity between models (ResNet- 18) trained to estimate different tar- gets. The intra-grous similarity is the mean pair-wise similarity between models trained on the same target but with different random initializations. Compared models trained with reduced transla- tion variability, models trained with full translation variability learned more similar representations layer4, except
models, which have large SD.
layer1.0.rg
layer2.0.relu
layer3.0.relu
layer4,0.red r.rrel
Intra smirity (0K4)
EMOR
A27431
awess
0.6
0.
Anduced can USE
Disency
Dantianon
Rotation
Duttnnu Drnniatton
Actstion
Distecc
Trastetin
Rotation
Distane
Vaassaatoon
Aotation
Rot
Hot.
Figure D.6: Similar to Figure D.5 but compared to models (ResNet-18 trained with less category variability. Compared to models trained with reduced category variability, models trained with full category variability learned more similar representations in layer4, except Sration-trainee models, which have large SD.
E ANALYSES OF REPRESENTATION OF NON-TARGET LATENTS
In Figure Figure and Figure E.2 analyzed the effect of non-target latent variability in the training data on the learned representations. Specifically, we investigated how well model represent non-target latents in their internal representations and whether more non-target latent variability helps learn better or invariant representations of them. We extracted activation from internal layers of models in response to the 2000 held-out TDW-117 test dataset that has variability in all latents we investigated. Many layers have very high dimensional activation, so we reduced the dimensions of the layers that higher than 6515 dimensions 6515 using random projection. This number 6515 is determined by the reduced representations train mapping models that map from the activation the specific non- target latents in the dataset. For discrete targets such categories, used linear support vector classification. For continuous targets, we used ridge regression. We run 5-fold cross-validation for each decoding tayer-target pair and aggregate the decoding performance from multiple models initialized with different random seeds. The data shown Figure 5 Figure E.T and Figure E.2 is the mean of the aggregated decoding performance, and the error bars show the standard deviation of each aggregated condition.
lemma. We then use the dimensionally
28

---

Published as a conference paper at ICLR 2025
Category decode accuracy P
Distance
b
Translation
10"
10"
Crazen
Chince
Pous decading
cecsein
(log scale)
Eitaaio yuu aer)
Thane uran Hill
brokucod
10
10
10"2
10-2
C
Rotation
d
Dis. Tran. Rot.
Category decade acturacy
10a
10"
Post decading
Paads decading
outegryy
o2, categary the
10-1
koradio Jucusse cat vi41
10-1
DS for eus midkued
(log
10-2
10
pixets
layer1 0.relu
layer2. layer2.0.relu O.relllu
layer3 D relu
layer4.0.mm relull
avgpnol
plzels
layer1 L..relu
layer2.0 relu
layer3 D. relu
layer4, w.r.4.0.rili r.relu
avgpool
pecoding layer
Decoding layer
Figure E.1: Object category decoding performance at different layers of spatial latent- attent-trained models (ResNet-18) 18) compared with category trained and untrained models. (Orange curves show models trained estimate: (a) distance, (b) translation, (c) rotation, and (d) distance translation + rotation. The category decoding performance of spatial latent trained-models is similar to category trained models in early to middle layers. Then, they become divergent at late layers. Spatial latent-trained models usually have much better category decoding performance than untrained models. (cat. var. category variability. Error bars show the SD across different cross- validation runs from different randomly initialized models.)
b
C
Model target: Distance
Model target: Rotation
Model target: Obj. category
perfomance
1.0
1.
0.5
Full tran var.
Reduced tran var
0.0
0.0 layer2
layerl.0.v
layer2 0.ver2.0.rehi 0.relu
layer3.0. r.r.euu
layer4, layer4.0.ra relu
avgpool
fc
layer1.0.ret)
layer2 1. 0.relu
1.yer3.0.relu 0.relu
layer3
layerd 0.relu
avgpool
fc
layer 1ayerl...eel 1...elu
O.rel 0.relu
layer3.0 layer3.0.relu 0.
layort 0.relu
avgpoo!
Decading layer
Figure E.2: Decoding performance for the non-target latent translation, Y when the models (ResNet-18) are trained to estimate target latents - (a) distance, (b) rotation, (c) category. Additional non-target latent variability helped these models learn better representations of the non-target latent in the intermediate layers. (tran. var. - translation variability. Error bars show the SD across 5 cross- validation runs x randomly initialized models. "*" indicates significant difference between the two groups, Mann-Whimey U test, value 0.05)
29

---

Published as a conference paper at ICLR 2025
0.40
Mean neurral aligmmet score
0.34
0.40
0.33 s
926
0.35
0.35
0.30
Untraieed
0.30
Untrained Ful fran var Reduced tron. var.
Full var
Reduced cat
0.25
Distance
Translation
Rotation
Dis. Tran, Rot
Distance
Rotation
Dis Rot.
Category
Figure E.3: Non-target variability helps models learn more neural-aligned representations. (a) Mod- els (ResNet-18) trained to estimate different target latent with the full non-target (category) varia- tions dataset have higher neural alignment scores than models trained with reduced non-target vari- ation. (b) We see similar results when the non-target latent is translation and the targets are rotation or object category, but the effect is less pronounced.
30


Reviews:
[{"review_id": "rG14t14X6W", "reviewer": "Reviewer_v5sv", "summary": "Traditionally, neural network based models of the visual cortex have relied on CNN backbones trained for object recognition. Here, the paper investigates if CNNs trained with alternate objectives (specifically, spatial tasks like viewpoint and pose estimation) can be used instead, and how that impacts the learned representations. Their main finding is that models trained to predict such alternate spatial tasks can also be used to learn models of the visual cortex which perform just as well. Digging into it further, the authors find this happens because that the representations learned with these different objectives all end up learning quite similar representations (as measured by CKA). Also, authors find that variability in non-target latents helps models.", "questions": "1. Were you able to decode categories from the models trained for spatial latents?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Very important problem**:  \n   The paper broadens our understanding of the ventral stream by exploring if spatial tasks, not just object categorization, can also drive effective visual cortex models. This is impactful for both building better models of the cortex, and also provides a computational feasibility argument for better understanding what the cortex does in the brain.\n\n2. **In-depth, exhaustive analysis**:  The authors provide thorough testing across multiple objectives, using extensive synthetic data and well designed metrics. Largely, the evidence matches the claim, and the findings seem quite reliable.\n\n3. **Interesting findings**:  Discovering that spatial-task-trained models achieve similar neural alignment to category-trained ones suggests surprising flexibility in modeling the visual cortex. It might opens an interesting question---how do self-supervised models of the visual cortex behave in comparison to categories and spatial latents?\n\n4. **Interesting new dataset**:   The synthetic dataset with spatial labels could be of use to future researchers.\n\n5. **Writing and figures are high quality**: Clear writing and well-designed figures make complex ideas accessible and enhance the paper’s readability.", "weaknesses": "1. Missing comparison to self-supervised models: The paper mentions them briefly, but does not provide experiments with these models. The best performing modern models used in AI are all increasingly relying on self supervised objectives.\n\n2. Missing comparisons with Vision Language models: The majority of upcoming new models are all multi-modal. It would be interesting to see how similar or dissimilar these perform.\n\n3. Missing some references: \n- The ideas of non-latent diversity are closely related to work in machine learning focusing on Data Diversity. It would be nice to connect this work to these papers [1,2,3,4,5]\n- This paper presents results on BrainScore which is in-distribution. It would be interesting to see how these evolves out of the training data distribution, especially since invariances become very important in out of distribution settings [6,7,8,9]\n\nReferences\n\n1. Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I. and Lee, T., 2021, July. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning (pp. 5637-5664). PMLR.\n\n2. Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H. and Boix, X., 2022. When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations. Nature Machine Intelligence, 4(2), pp.146-153.\n\n3. Gulrajani, I. and Lopez-Paz, D., 2020. In search of lost domain generalization. arXiv preprint arXiv:2007.01434.\n\n4. Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez-Paz, D., 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893.\n\n5. Arjovsky, M., 2020. Out of distribution generalization in machine learning (Doctoral dissertation, New York University).\n\n6. Ren, Y. and Bashivan, P., 2024. How well do models of visual cortex generalize to out of distribution samples?. PLOS Computational Biology, 20(5), p.e1011145.\n\n7. Madan, S., Xiao, W., Cao, M., Pfister, H., Livingstone, M. and Kreiman, G., 2024. Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. arXiv preprint arXiv:2406.16935.\n\n8. Pierzchlewicz, P., Willeke, K., Nix, A., Elumalai, P., Restivo, K., Shinn, T., Nealley, C., Rodriguez, G., Patel, S., Franke, K. and Tolias, A., 2024. Energy guided diffusion for generating neurally exciting images. Advances in Neural Information Processing Systems, 36.\n\n9. Ponce, C.R., Xiao, W., Schade, P.F., Hartmann, T.S., Kreiman, G. and Livingstone, M.S., 2019. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4), pp.999-1009.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVDUEELpER", "reviewer": "Reviewer_nMg4", "summary": "The paper asks whether the ventral stream only codes object category information. By showing that CNN models that are trained for spatial understanding can also align with the neural reponse in the ventral stream, the author provide a critical reflection on the assumption that ventral stream in the brain primarily encode categorical information. The authors also introduce synthetic datasets generated by a 3D graphics engine (ThreeDWorld) containing precise spatial labels, overcoming the scarcity of natural datasets with spatial information. This dataset enabled systematic experiments to evaluate CNNs trained on spatial latents.", "questions": "* In Figure 5b, the decoding accuracy for classification is quite low for models trained for spatial latents, suggesting that the model trained for spatial latents is not suitable for classification. However, this contrasts with the CKA analysis in Figure 4b where the representation similarity is quite high between models trained for spatial latents and models for object categorization. How would you explain such difference?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper asks an important question of the function role of the ventral stream besides image categorization. The discovery suggests that representation that is similar to the ventral stream neural response can be derived from training the network under spatial latent estimation objective as well. This contrasts the traditional belief that the primilary goal of the ventral stream is for object categorization only. \n* To demonstrate the proposal, the paper utilizes a 3D graphics engine to generate large dataset to tackle the shortage of spatial latent variables in previous studies. In Figure C.3 in supplementary, the paper demonstrates the importance of having larger scale dataset. \n* The paper shows that CNN trained on purely synthetic dataset can achieve a similar neural alignment score as these trained with natural images. \n* The paper has clear writing and presentations.", "weaknesses": "* It could be more comprehensive if the paper utilizes natural images as training, e.g. one could extract sptial variables from SA-1B segmentation dataset to perform the experiments. \n* The evaluation of the neural alignment score is limited to the Brain-Score. More metrics would be helpful to consolidate the results. \n* The idea that ventral pathway is not responsible for classification alone is not new among cognitive neuroscience community. E.g Connor discussed in his paper that the retinotropic spatial information is transformed into relative positional information across ventral stream [1]. A discussion with related works in neuroscience community would be appropriated. \n\n[1] Connor, Charles E., and James J. Knierim. \"Integration of objects and space in perception and memory.\" Nature neuroscience 20.11 (2017): 1493-1503.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "glXOs6JZPG", "reviewer": "Reviewer_acUt", "summary": "The paper examines the effect of training models to estimate spatial latents on neural alignment (the \"where\"), comparing this approach with object recognition (the \"what\"). It highlights that to achieve high neural alignment, training with a focus on \"where\" yields similar benefits compared to the more common focus on \"what.\" Interestingly, with fewer presentations in the \"where\" case, alignment levels appear comparable, and representations in the models' initial layers are strikingly similar.", "questions": "Left in weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an interesting problem that has been overlooked for some time. The presentation is highly effective, and the paper is easy to read. It takes an unconventional approach to the problem by focusing on spatial location, which proves to be very effective even in a low data regime. There are several controls in the main text and supplementary material that support the claims.", "weaknesses": "I know that in  semantic segmentation and other tasks where location and classification of the object is used for training, has been studied for brain-score but have not really succeeded.  Perhaps worth adding a bit of discussion on this topic, because even when its possible to say that the world is 3D and therefore the tasks seem to align better with vision, is also true that the models used in this work are only receiving 2D images as input. Some specific questions:  \n\n1. How this approach differs from previous attempts using semantic segmentation for brain-score alignment? \n2. Why this  spatial latent estimation tasks succeed where others have not\n3. Are there any implications of using  2D inputs to estimate 3D spatial properties, how this  relates to biological vision? \n\nSince the title starts with  \"Vision Models...\", I largely missed other models like ViTs  to be studied. But it seems that as it stands the results only cover CNNs?\n1. I would suggest  a change of the title if that is the case, or inclusion of other kind of vision models in the results to make sure is align with the title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "559na8r0On", "reviewer": "Reviewer_wqcV", "summary": "The paper assesses the neural predictivity and representational similarity of CNNs trained on a novel large synthetic dataset to decode different combinations of the following latent variables: object category , object identity, object position in 3-D space, and object orientation. The author's make the following observations:\n- Training networks using a sufficiently large/realistic/diverse synthetic dataset can produce representations that rival those trained on natural images in terms of predictivity of ventral stream responses.\n- Model's trained only to estimate spatial latents (i.e. without any explicit incentive for class invariance) can explain a similar fraction of variance as models trained to do categorization. \n- The ability to decode spatial latents is correlated with predictivity.\n- Model's trained on different tasks learn similar internal representations (as measured by CKA), especially at early layers and to a lesser degree deeper in the networks.\n- The amount of variability of some latent variable in a training dataset modulates the extent to which said latent is linearly-decodable from a given representations, even when that latent is not used to train the neural network.", "questions": "- About the spike in category decoding with rotation latent target (bottom row Fig. 5b): I wonder if rotation decoding is somewhat entangled with category decoding. I.e. to know some object has been rotated by 90 degrees you must know what its shape is/how it appears when presented at zero degrees of rotation (and such information is correlated with category). It might be interesting to look and see how the performance on category decoding breaks down by class (if category performance is high for objects whose shape varies strongly with viewing angle this might support such a hypothesis). \n\n- Why are two different similarity metrics employed when comparing two artificial representations and when comparing artificial representations to neural measurements? Since the current work only considers the subset of datasets from BrainScore that are public, couldn't you use CKA to compare the artificial and biological representations as well (or alternatively use partial least squares regression to compare pairs of CNNs)? This doesn't seem likely to be an important issue but I am curious as to the reasoning, as it might be nice to have all comparisons on the same \"scale\", subject to the same method-induced biases, and tolerant to the same transformations. \n\n- In the paragraph beginning on line 334 the authors suggest that their results from decoding/CKA experiments suggest that each model with similar predicitivity is also explaining the same portion of variance in the neural responses. While this seems sensible (especially perhaps for earlier layers/V1-V2 predictivity), I don't understand why we cannot just address this hypothesis directly. For example we can: directly examine the residuals produced by each model-to-brain prediction, predict the residual of one predictive model with another, or predict the neural data using ensembled models to get a sense for how overlapping the explained variances are. Again, given that the study is limited to the publicly available benchmarks all of these experiments seem feasible, and I think very interesting! \n\n- Semi-Nitpick: the paper repeatedly references the number of units provided supervision. I don't think this is an interesting axis, or fair to compare between units that receive a discrete category label and a continuous latent variable. Meaning, you could discretize each spatial latent into 117 bins without changing the training signal but equalizing the number of outputs to the categorization task. I think rather than the dimensionality of supervisory signal the number of bits of supervision is the relevant thing to compare! I.e. since you know the distribution of locations you can measure the entropy of the translation variable and compare this to the entropy of the category variable (this may be trickier because of discrete vs. differential entropy etc.), but I think is more meaningful than comparing dimensionalities. \n\n- Note: I would prefer to leave a score of 7 for this paper, but as that is unavailable I will sit at a soft 6 for now. I am very willing to raise my score if the authors can address/justify/explain some of the weaknesses or questions listed above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The subject (the degree to which object classification is a unique normative explanation for features of neural representations in the ventral stream) is very interesting (and to a substantially large community).\n\n- Investigating this hypothesis using a carefully controlled *but appropriately large* synthetic dataset is a sensible and novel approach.\n\n- For the most parts experiments are clearly described, easy to follow, and sufficiently thorough to justify the conclusions. \n\n- While I find the results of Section 3.3 the least convincing of those presented, doing these experiments is certainly a step toward addressing why similar representations are observed in the early layers across many artificial representations (which is \"above and beyond\" what is necessary for supporting the core arguments laid out in the paper).", "weaknesses": "- Layer Mapping Details Missing: I think it would be good to report which layers are mapped to each of the considered brain areas/datasets (unless I missed this detail). For instance, if the same layers are mapped to V1 across tasks this would be interesting especially in light of the fact earlier layers have higher CKA similarity. \n\n- More fine-grained/controlled predictivity-to-decoding correlation analyses: It would be interesting to see if performance on any particular task is more or less correlated with predictivity in a particular region (i.e. by reproducing Fig. 3 but measuring the correlation with only V1 predictivity etc.). Furthermore, it may be easier to interpret this type of result if the layers used to decode the latent parameters were matched to the layers mapped to a particular region (so you can make statements along the lines of \"the ability to decode distance from layer3.0.relu was correlated with the ability to predict V4 from the same activations, suggesting a possible functional link between the two\").\n\n- Difficult to draw conclusions from results in Section 3.3: \n  - It would be nice to see the linear decoding performance for each latent on the pixels as well as early in the network as a baseline. \n  - In Fig. 5b while all models have marginal increases  in category decoding (though most differences seem insignificant based on the depicted error bars, which are not described, but I assume to be 95% confidence intervals?), they are all still generally unable to decode categories at a meaningful rate. I think as well as the chance-performance floor it would be good to include a \"trained for category decoding only\" ceiling to help assess the scale of the improvement. Furthermore it would be good to include the performance curve (decoding accuracy as a function of depth in the network) for randomly initialized networks. If this curve is above the others a more accurate description would be \"learning to decode a spatial latent decreases the decodability of a non-target latent less when that latent is more variable in the training set.\" Similar baselines would be useful in 5c.\n  - The all vs. none variability setup in these experiments produced mostly small differences in the decoding performance with the notable exceptions of layer4.0relu in 5b bottom row and avgpool/fc in the bottom row of Fig 5c. Given this I think the result could be made more convincing by showing an orderly trend in decoding performance as the variability of non-target latents is parametrically decreased in steps between the Full variability case and the very low variability case explored here.  Finally in this setup it would be good to see how decoding performance varies in terms of the variability. Consider an experiment of this type:\n    - You have (in addition to the models at full translation variability and zero translation variability already trained) a model trained using a new 50% translation variability dataset. You may then want to measure the translation decoding performance of all models both the full-variability and half-variability test-sets to get the clearest picture possible. \n  - In sum, I think the language describing these results could be softened to reflect how difficult it is to draw definitive conclusions about internal representations from this type of experiment. This isn't a big issue at all but maybe changing line 430 to: \"In summary, most of our observations are in line with the hypothesis that models develop better intermediate representations of nontarget latents in the presence of their variability, though future work is needed to understand both the size of and mechanism behind this effect.\" \n\n- Model-Brain Comparisons Leave out Relevant Datasets: the author's only compare model responses to 1/4 of the neural recording datasets in both areas V4 and IT available on the BrainScore platform. I understand that predictivities across these datasets are correlated with each other, but I see no reason not to include them in the analyses. Especially given that the images presented in the Majaj-Hong benchmarks have a striking similarity in construction to the TDW synthetic dataset (object at random pose superimposed on a natural background), it would be good to confirm that the observations in this work generalize to other types of images!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)