Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations
Authors: Avi Cooper
Daniel Harari
Tomotake Sasaki
Spandan Madan
Hanspeter Pfister
Pawan Sinha
Xavier Boix
Venue: Accepted by TMLR ()

Abstract: The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the training data distribution is not well understood. We investigate the limitations of DNNs’ generalization capacities by systematically inspecting DNNs' patterns of success and failure across out-of-distribution (OoD) orientations. We present evidence that DNNs (across architecture types, including convolutional neural networks and transformers) are capable of generalizing to objects in novel orientations, and we describe their generalization behaviors. Specifically, generalization strengthens when training the DNN with an increasing number of familiar objects, but only in orientations that involve 2D rotations of familiar orientations. We also hypothesize how this generalization behavior emerges from internal neural mechanisms – that neurons tuned to common features between familiar and unfamiliar objects enable out of distribution generalization – and present supporting data for this theory. The reproducibility of our findings across model architectures, as well as analogous prior studies on the brain, suggests that these orientation generalization behaviors, as well as the neural mechanisms that drive them, may be a feature of neural networks in general.

Published in Transactions on Machine Learning Research (08/2025)
Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations
Avi Cooper
acooper@fujitsu.com com
Fujitsu Reseorch of America
Daniel Harari
The Institute for Artificial Intelligence Weizmann Instibute of Science
Center for Breins, Minds and Machines Massachuseetts Institute of Technology
Tomotake Sasaki
Pojitsu Limited
Spandan Madan
Department of Computer Science Harvard University
Hanspeter Pfister
Department of Computer Science Harvard University
Pawan Sinha
Department of Brain and Cogritive Sciences Massachusetts Institute of Technology
Xavier Boix
Pujitsu Reseorch of America
Reviewed on OpenReview: https: /t openreviewwweeee net/ forum? id= 4WBQTZVSHU
Abstract
The capability Deep Neural Networks (DNNs) to recognize objects in orientations out- side the training data distribution not well understood. We investigate the limitations of DNNs' generalization capacities by systematically inspecting DNNs' patterns of success and failure across out-of-distributio (OoD) orientations. We present evidence that DNNs (across architecture types, including convolutional neural networks and transformers) are capable of generalizing to objects in novel orientations, and we describe their generalization behaviors. Specifically, generalization strengthens when training the DNN with an increas- ing number of familiar objects, but only in orientations that involve 2D rotations familiar orientations. We also hypothesize how this generalization behavior emerges from internal neural mechanisms that neurons tuned common features between familiar and unfamil- iar objects enable out of distribution gencralization and present supporting data for this theory. The reproducibility of our findings across model archilectures, well as analogous prior studies on the brain, suggests that these orientation generalization behaviors, well as the neural mechanisms that drive them, may be a feature of neural networks in general.

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
Recognizing objects in novel orientations lies at the heart of biological and artificial intelligence, as it is a fundamental capacity necessary to understand the visual world (Sinha & Poggio 1996 Ullan 1996).
However, the computational mechanisms underlying this capacity in both brains and machines are not yet
well understood (Gazzaniga et al. 2006 Pinto al. 2008 Li et al. 2021).
In the realm of artificial systems, Deep Neural Networks (DNNs) have recently made large strides in object
recognition (He et al.] 2017 Carion et al., 2020 However recent studies have shown that DNNs perform
poorly when objects are presented in novel orientations, even when learning from large datasets with millions
of examples (Barbu et al.) 2019 Alcorn et al. 2019 Madan et al.) 2022] [bbaa & Deny 2023 Ollikka
ct al. 2025). More broadly, novel orientations are a special case of distribbbbn (OoD) data. DNNs
general gentralization Tization is often limited to the images from the training distribution, known as in-distribution data,
while it remains difficult to give a principled account of their performance OoD scttings.
One promising approach to understand the capabilities of DNNs is to leverage the knowledge gained from studying biological intelligence (Hassabis et al. 2017 Ullman 2019). In natural settings, biological intel- ligent agents observe instances of object categorics from diverse orientations. When encountering a new object instance, these agents often demonstrate the capacity to accuralely identify the object. in different orientations by drawing upon past experiences with similar instances (Booth & Rolls [998 Freiwald & Tsao 2010; Ratan Murty & Arun 2015). Extensive investigations into human and mammalian perception and object recognition in unfamiliar orientations have revealed that recognition accuracy varies across novel orientations, with some orientations exhibiting superior generalization compared to others (Logothetis & Pauls| 1995). Additionally, studies into the neural mechanisms underlying these cognitive abilities have mar- shaled compelling evidence suggesting that ncurons respond to their own specific set of object features when
present in the visual field (Desimone et al. 1984: Kobatake & Tanaka 1994 Gauthier et al. 2002: Fang &
He 2005). This neural tuning has been reported to be invariant to a certain degree from the object's 's orien- tation (Logothetis & Sheinberg 1996). Theoretical frameworks have proposed that such neural invariance to object orientation forms the basis for the ability to recognize objects in novel orientations within biological systems (Poggio & Anselmi 2016).
In this paper we employ these same analytical tools utilized in the study of biological brains in order to understand DNNs' generalization abilities in OoD orientations. Specifically, we study DNNs under conditions akin to the operating regime of biological brains, in which some instances of an object category (e.g., a 'Boeing 777 airliner' is an instance of the 'airplanc' category) are seen from all orientations during training (fully-scen instances), while other instances are only seen in a subset of all orientations (partially-seen instances). During test time, we evaluate the generalization performance of the networks by measuring instance classification performance OoD orientations (f.e., those orientations not included in the training set) of partially- seen instances. This simple paradigm, inspired by (Jang et al.) 2023), facilitates analyzing the impact of several key factors that may influence OoD generalization, such as the number of fully-seen instances and the in-distribution orientations of the partially-seen instances. This paradigm allows us to more precisely characterize performance challenges of DNNs for OoD orientations. Figure summarizes the paradigm that we follow in this work.
A large number of previous works have begun to understand the generalization capacities in DNNs to OoD orientations. For example, (Lenc & Vedaldi 2015 Gruver et al.) 2023) investigated the emergence of invariance, and specifically rotational invariance to affine rotations. However, it remains unclear whether and how DNNs generalize to OoD orientations, such as in the task we outlined above. Our novel analytical approaches yield three important findings: 1) DNNs readily generalize to certain surprising OoD orientations.
2) These generalizable orientations are parameterized by the in-distribution set, and can be quantifiably predicted. 3) Neural representations at the individual-unit level, across orientations and object instances, support theory on brain-like mechanisms that drive the emergence of orientation invariance (Logothetis & Sheinberg 1996 Poggio & Anselmi 2016).
2

---

Published in Transactions on Machine Learning Research (08/2025)
Full Experience
Partial Experience
Dissemination of Orientation Invariance
b
Fully Seen Instances
Partially Seen Instances
Per-Orientation Analysis
Orientation
Instance
In-Distribution
Diitr--Diitributionn
Figure Learning paradigm and network's per-orientation accuracy. (a) The network trained with images of certain airplanes all orientations, constituting 'full experience, and for other airplanes, small subset of orientations, constituting 'partial experience.' The -ittoff-istribbtion, or OoD, gener- alization capacities of the network are evaluated by measuring the classification accuracy for partially-seen airplanes at unseen orientations. Our results suggest that OoD generalization is facilitated by the dis- semination of orientation invariance developed for all orientations for the fully-seen airplanes to the OoD orientations of the partially-seat airplanes. (b) Left: The learning paradigm employed in this work. Each column is sample object instance (here from the airplanc datasct) and each row is a sample orientation. The training set includes all orientations for fully-seen instances, and partial set of orientations (outlined in red) for partially-seen instances (in this example, with the airplanes' nose pointing down). The orientations included in the training set referred to in-distribution orientations (pink shading). Orientations of the partially-seen instances that are not included in the training set are referred to as OoD (yellow shad- ing). Right: visualization of per-orientation-anllyyi The set of all orientations are arranged to capture proximity relationships between orientations. (Further details are provided in Fig 3
2 Methods
In this section detail the experimental setup and analytical tools we employ in this work, which were first introduced in Section [ and depicted in Figure [ In the following, outline the dataset and DNN architecture details in the next two subsections. We then present the analytical tools employed in Sec- tion 3 Specifically, we introduce a visualization of per-orientation accuracy, predictive model of DNN generalization, and finally a neural metric related to generalization.
Code used for this study can be found at: https://githul. wttp////cooggrr///000000000 http:////000000etaatiiooooo Generalization Data generated and analyzed in this study can be found 7910/DVM/M9KPMR
3

---

Published in Transactions on Machine Learning Research (08/2025)
a b
C
Figure 2: Object Datasets. In our experiments used three object categories: (a) Airplanes, (b) Cars, and (c) Shezard&MMetzler objects. The first two were curated from ShapeNet (Chang et al. 2015 and we procedurally generated the last one.
2.1 Datasets
The datasets consist of images object instances, some of which are seen from all orientations during training (fully-seen) and other instances which are only seen from a subset of orientations during training (portially-seen). Each dataset also contains validation set of fully-seen images and partially-seen instances at orientations during training. The distributions of the validation and train overlap, but no specific images overlap. The test set consists of images of the partially-seen instances at OoD orientations, which are not included during training. For each experiment, we construct datasct with an object category, set of seen orientations for the partially-seen instances, and a proportion of fully-seen to partially-seen instances. We describe these elements below.
Object Categories. We used three categories of objects: Airplanes, Cars and Shepard&Meetele ob- jects. For the airplanes and cars we eurated 50 high quality object instances of each category from the ShapeNet et al. 2015) database. Both airplanes and cars have clear axes symmetry, which allow for intuition how networks generalize to OoD orientations. We therefore also experimented with highly asymmetric objects similar to those tested for 3D mental rotations in (Shepard & Metzler 1971) (which denote Meelerd&Metzler objects; Fig. 2). We procodurally generated the Shepard Metzler objects. Images were rendered from the 3D models under fixed lighting conditions, and the models were centered and fully contained within the image frame.
We experimented with datasets where both fully-seen and partially-seen instances are from the same category, and also where they come from different categories. For example, where the fully-seen instances were airplanes, and the partially-seen instances were Shepand&Metaler objects. In this case, high classification accuracy on the test set (i.e., the partially-seen instances in OoD orientations) requires that orientation generalization disseminates to new object categories.
In- in-Diitribbtion Orientations for Partially-Seen Instances. We chose several ranges of orientation to be in-distribution for the portially-seen instances, which refer seed orientations. We the term
4

---

Published in Transactions on Machine Learning Research (08/2025)
a Range
3 Range
y Range
Seed Name
(-πT, m)
(-0.1,
(-0.25, 0.25)
á
(- 0.225, 0.25)
(-5:3) 5)
(-0.25, 0.25)
3
(- 0.25, 0.25)
(-0.1,0.1)
(-T,
(.1.1,0.1
(-1.8,-1.3)
6'
Table 1: In-distribution orientations, in radians, for partially-seen instances, termed seed orientations. The seed name is chosen for the full axis, i.e. & is for the seed where a ranges from -r to x.
seed because any generalization to OoD orientations must stem from these orientations. In choosing the seed orientations, chose small ranges of rotation along two of the three axes, and the full range of rotation along the final axis. This allowed for some variation in the instance appearance along the "full" axis, while still preserving many OoD orientations along the 'bounded' axes. We employed several seeds to ablate any idiosyncrasies within any particular choice.
To express an orientation we use 0 == 8,7,,)), the Euler angles with respect to the orthogonal axes of reference coordinate system R3 (Goldstein et al. 2002), with the convention that and y bounded within 2x radians and 3 is bounded within æ radians. Following this convention, the seeds we employed are enumerated in Table 1 Note that we experiment with four seeds, which are denoted as å, B, A and á' as they refer to the axis that is fully observed.
Proportion of fully-seen instances. In each experiment, there were always 50 instances that were seen. However we vary the number of fully-seen instances N between 10 (20% of the total number of instances) and 40 (80%). The remaining instances are partially-seen. For a fair evaluation of the effect of data diversity, the amount of training examples is kept. constant. as we vary the data diversity.
Dataset Size. Each dataset is 200k images, 4k image for each of the 50 object instances. A training epoch ilerates through every image in the datasel once. Since the generalization abilities of the network may change with the amount of data, we evaluate OoD accuracy under the most favorable scenario for the DNN, in which adding more data does not lead to a higher Ool) accuracy. We repeat the experiments with half the data, see Figures S5 S6 S8
and results are consistent with our findings for the default dataset size, which provide some reassurance that adding more data won't 't change the conclusions of this paper.
2.2 Experimental Setup
For each experiment, we train a DNN from scratch on supervised instance classification task (i..., N-way classification where each dimension of the output layer corresponds to one object instance), using the datasets described above in Sec. 2.1 In this subsection we outline the DNNs experimented with, along with other training parameters.
DNN Architectures. We tested ResNet18 (He et al. 2016 on all permutations of dataset structure (combination of seed, object category, and proportion of fully-seen) as well as for all experimental ablations (pretrained on ImageNet, image augmentations, and using half the training data: see Figs. 54 55 S8). To ensure that our results were not specific to ResNet18, we ran the same experiments with 1en (Huang et al. 2017), ViT-Base (Vaswani et al. 2017 Dosovitskiy et al. 2020) and CORnet-S (Kubilius et al. 2018 The first two were chosen they are representative feed- forward DNNs. The architecture of CORnet is brain-inspired and includes recurrence at higher layers in addition to convolutions in lower layers. We observed the same behavior for all analyses for all the architectures when analyzing the results of airplane datasets (across number of fully-seen, seed orientation etc.) Since we observe results are consistent for all architectures, we only report the full battery with ResNet. For DenseNet and CORnet, we report results with the airplane dataset with all seeds. For ViT-Base, we report results with the airplane dataset on the à seed.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Architecture
Learning Rate
Batch Size
1. R8sNet18
10-3
230
2. DenseNet121
10- 3
64
3. CORnet-S
10-4
128
4. ViT-Base
10-4
256
Table 2: Training Hyperparamters
Figures in the main text are shown only for ResNet, except for Figs. pp,e, which are scatter plots and therefore don't require averaging over architectures. The analysis was repeated for all architectures and these results can be found in the supplement. This includes accuracy heatmap visualizations (Fig. S3)
average OoD accuracy (Fig. (8), and predictive modcling (Fig. (55). The same conclusions drawn in the main paper may be drawn from the results with other architectures well.
Hyperparameters for training. We trained the four deep convolutional neural network archilectures using the Adam Optimizer (Kingma & Ba 2017| with the learning rates and batch sizes listed in Table 2
Batch sizes were chosen to be as large as possible while still fitting the model, the batch of images and forward-pass compulations memory. Learning rales were chosen from 10", x E (-1,,22, -3, -4, 55} to be as large possible while ensuring that validation accuracy, which included fully-seen instances from all orientations and partially-seen instances from in-distribution orientations, remained stable. Each network trained for minimum of 100 epochs, after which training stopped if validation performance lower than some epoch for seven epoch in a row. After this point in-distribution performance was stabilized at 100% and OoD performance reached an asymptote.
Hardware details. Experiments were run with one CPU, 25GB of memory and on several generations of Nvidia GPUs with a minimum 11GB of memory.
Repetition of the experiment. We re-run each experiment (tuple of dataset and model architecture) live times, each time randomly sampling the specific instances which comprise the fully-seen and partially- seen sets. For all figures, data come from all available repetitions, except where otherwise noted. Error bars denote one standard deviation from the mean.
2.3 Per-Orientation Accuracy Visualization
Previous works have typically reported average performance over all orientations. In contrast, we evaluate the network's performance for each orientation across the entire range of orientations. To express an orientation of an object instance we use 0 == (a, Y),, the Euler angles with respect to the orthogonal axes of a reference coordinate system R3 [Goldstein et al. (202).. with the convention that a and 7 are bounded within 2 radians, and 3 is bounded within N radians. We define vr()) E 1] to be the network's average classification accuracy at an orientation = (a, over either the fully-seen or partially-seen instances.
To facilitate intuition of we introduce a visual representation of this function. Since orientations are continuous values and are related spatially we map the range of bounded values of orientations (a, 3,,7) 7) onto Cartesian coordinate system, resulting in cube-tth the basis of our visualization. We discretize the continuous space of orientations into cubelets, which are sub-cubes with width of range of each respective angle. This approach preserves local behavior in aggregate analysis. In addition, we outline the range of orientations which are in-distribution for the partially-seen instances - the rest. are OoD orientations. To illustrate the object orientation at a given cubelet, we sample one representative image and overlay it onto the heatmap the location of the cubelet.
# #Cbelleta of the full
See Fig. 3a, which shows this visual representation scheme, and Figs. 8 and g
for examples.
6

---

Published in Transactions on Machine Learning Research (08/2025)
2.4 Model of DNN Per-Orientation Generalization
We hypothesize set of rules which govern the partitioning of the orientation space into generalizable and non-generalizable orientations. To quantitatively evaluate this hypothesis we formulate a model of the partitioning rules, which can be used to prodict the OoD gencralization patterns of the network, given a seed in-distribution orientations. Briefly, the model, denoted by (w),,, has three components: A(0), which captures small angle rotations around 0; E()); which captures in-plane (2D) rotations; S(0), which captures object "silhouette" projections at the orientation 0. We define (w) as the predictive model for generalization per each orientation. To mcasure the goodness of our prediction, correlation coefficient to measure how closely our model correlates with DNN recognition accuracy, (O))). We choose this metric because it normalizes data with respect to amplitude and variance, and therefore measures patterns of behavior across and relative to other 0, rather than the exact performance for every 0.
employ the Pearson
The model's three components (A(0), (()) and S(0)) easily lend themselves to formalization with Euler's rotation theorem (Goldstein et al., 2002). The theorem states that any rotation can be uniquely described by single axis, represented by a unit vector è R2, and angle of rotation, denoted ø E [0, πr around the axis è. We employ this representation to describe the rotation between an arbitrary orientation of interest, 0, and an orientation in the set of in-distribution, denoted 0, E Ωg We use ča,0, and Φ0.0, to denote the unit vector (axis) and the angle of this rotation, respectively.
Small Angle Rotation, A()). The first component of the model captures orientations that are small angle rotations from the orientations in the training distribution. Visually similar orientations are those that are arrived at by small rotations from in-distribution orientations, small @e.0.- We therefore define the first component A(0) as
Component 1:
60.0. E [0,1].
A(O) == max 0, EQ,
1
(1)
The maxe, eo, operator chooses the in-distribution orientation that is closest to 0 of interest.
Component 2: In-plane Rotation, E(0). The second component of the model captures orientations which appear as in-plane rotations of in-distributtio images. Let c R3 be the unit vector representing the camera axis. In-plene rotations are those for which the axis of rotation is parallel to the camera axis. Thus, an orientation appear as an in-plane rotations of an in-distribution images when с E R and êa.a. E R (d.e,, the vector of object instance rotation) are parallel. Taking their standard inner product yields the proximity to being parallel, which is therefore the degree which the rotation is in-plane.
Thus, we define the second component E(O) follows:
E(O) = max |c To,.o, E 1]. 0.E9.
(2)
where c denotes the transpose of c.
Component 3: Silhouette, (x))), (0). We refer to the object's silhouette as a featureless solid shape with its edges matching the outline of the object seen as a shadow from the camera view. The third component of the model captures orientations in which the object's silhouctte is similar to the silhouctte of the object the in-distribution views for example, the similarity in the appearance of an airplane's silhouette when viewed from above and from below. These orientations are defined as radians rotation around the axis, which results in a silhouette orientation. We transform all the in-distribution orientations, Re, in this way, and we call these silhouette in- mn-distribution orientations Ωg. We then compute S and (e) substituting 2g for Ω2 in A(0) and F(() respectively.
Nonlinearities.
The components described above capture general trend, but do not match the range of values given by (-100% accuracy metric. We therefore fit the components with logistic function. The
7

---

Published in Transactions on Machine Learning Research (08/2025)
-i'l shape of the logistic function allows for the highest and lowest values of E(0), A(O), SA and (g) to be close to the highest and lowest values of (0). In addition, it allows for smooth transition wetw these highest and lowest values. Most importantly, the simplicity of the logistic function allows for fitting while preserving the interpretability of the model components, ensuring that the models remains related to small angle, in-plane and silhouette rotations. We employ the following logistic function:
1
(a, 1 ehc eb(=rta))
where a E (E(0), SA and e spreads out saturated values of the component.
and b translate and scale the values of the predictive components
Fitting the Model with Gradient Descent. The model combines four components A(O), E()), SA(0) and (e) by laking the sum of their respective values after applying the logistic function a:
fw( c(A(0); WA) (B(O): a (SA(0)) A5A) + (0(S WsE;
(4)
where represents the parameters of the logistic functions i.e, w (WA. WE, WSA, WsE). The logistic fitting function is differentiable, and (0), the linear combination of these logistic functions, is also differ- entiable. Further, the Pearson correlation coefficient is also differentiable. Therefore we employ gradient descent to fit w with the Pearson correlation cocfficient as the cost function.
2.5 Neural Analysis
In search of how OoD generalization and dissemination of orientation invariance emerge in DNN's, we turn to analyze the neurons' activation in the trained networks. We focus on neurons in the penultimate layer of the network, which are attuned to the highest level features in the input stimuli, but reflect a consolidated representation of the entire network for inferring the downstream task (instance classification in our simulations).
In this section, we outline the process by which we quantify several different network invariance metrics. We first formalize the notation for neural activations for single orientations and for sets of orientations. We then define the invariance score (Eq. 6 Finally, we average logether many invariance calculations to arrive at the network invariance metric.
We begin by formalizing our approach to neural activations. In Section 2.3 we introduced (00), the net- work's 's average accuracy at a specific orientation. We can similarly define the neural activation al a specific orientation, though we do so with more granularity. Namely, we introduce 4, which is the average activation of a neuron from the set of all pemultimate-layer neurons N (í.e., n E N) across all images of an object instance i from the set of all object instances I (í.e., i E I) for given orientation O. We normalize the activity of each neuron by dividing the activity level of each image by the maximum activity generated by any image We exclude any neurons with a maximum activation of 0 from further analysis.
Having defined we note that is useful to perform analysis not on single orientations only, but sets of orientations. We demonstrated that under our experimental conditions, orientations can be partitioned into coherent subsets in-distribution and OoD orientations. Further, the OoD orientations can be partitioned into generalizable orientations, i.e., those OoD orientations that the network can generalize to, and non- generalizable orientations. We refer to the in-distribution, generalizable and non-generulizable orientation sets as InD, G and -G' respectively. The determination of membership of the generalizable and non-generulizable orientation sets is as follows: We compute 10% of the maximum value of (0), the predictive model, in the experiment with 40 fully-seen instances. All orientations for which f is greater than the 10% threshold are considered generalizable otherwise they are considered non-generalizable. This threshold for generalizable accuracy intentionally low, as our goal was to capture challenging out-o--distribution cases where the network performs just above chance. This allows us to include orientations that are borderline in terms of
8

---

Published in Transactions on Machine Learning Research (08/2025)
generalization. See Fig. S8 for OoD accuracy partitioned between generalizable and non-generalizable this partition captures model behavior well. We can now compute the average activation of a set or orientations. For example, the average activation for a given neuron n and object instance i of the generalizable orientations is defined in the following way:
(C
Σ (7) DEG
(5)
G
The same may be computed for in-distribution and non-generulizable orientations.
To determine how dissemination occurs in the network, we calculate the degree of similarity in a neuron's response to given instance across different orientations. Specifically, given neuron n and instance i, we calculate the similarity between the neuron's response at orientation pair 2; (01), and Φ (02), pair of
sets of orientations (InD), $ (G) for example. We use 8, invariance score, as the similarity metric, which is defined (based on previous work (Madan et al. 2022) in the following way:
8 ((D) ( 1 ( (. (G) (InD)
(6)
We note that under some conditions, 5 reports high, yel trivial, invariance. Namely, if the response of neuron is low or zero for both elements of the pair, the denominator approaches zero and the invariance becomes large. However in this case the neuron is not responding to anything any activity is most likely noise. We therefore calculate threshold of activity for neural response invariances to be considered to contribute to the generalization capability of the network. Otherwise, these invariances are not integrated into the overall network invariance metric. This threshold ensures that we focus on neurons that are clearly active, reinforcing our interpretation of them as meaningful feature detectors, and it avoids relying on marginal noisy activations that may not be functionally relevant. The threshold, т, is the 95th percentile of activity for all neurons across all images. We employ 7 with an indicator function as follows:
1 1 (InD), (G)) (InD),"(GG)
{ if (IaD) T ( 2T
0 otherwise
Finally, we can compute the overall network generalizable and non-generalizable invariance scores. To do so, we compute a triple average: an average activation over the of orientations (Eq. 5) and averaged over the invariance of all neurons and object instances. We say that the generalizable invariance score the invariance between the in-distribution orientations and the generalizable orientations determined as follows:
1
1 (G)) 8(0 (InD), (G)),
L
where L is the quantity of activity pairs above the threshold r, i.c.,
L ( (G)))
(8)
NEN TET
The definition of the network's non non-genenalin generleeeel invariance score is the same, though -G replaces G.
3 Results
Overview. In exploration of the generalization capabilities of DNNs to novel orientations, our com- putational experiments show that the OoD orientation space divided between generalizable and non- generolizable orientations, in terms of the networks behavior. We find this division to be governed by of rules which determine partitioning of the orientation space, given seed of orientations for test instances

---

Published in Transactions on Machine Learning Research (08/2025)
seen by the DNNs at during training. Among several partitioning rules, we identify rather intuitive one: small 3D perturbations around seen orientations will be included in the highly-gemeralizable partition. We find other rules to be more surprising, including that the highhly-generraizable partition also consists of shape and silhouctte preserving rotations, such as in-plane (i.e., 2D) rotations and flips along axes of symmetry of the seen seed orientations. In order lo quantitatively assess this hypothesis, we evaluate the degree of correlation between predicted network behavior induced by these partitioning rules and measured behavior. We find them to be highly correlated under variely of training regimes. We also explore the DNNs' internal representations and identify neuronal mechanisms that allow for the dissemination of erientation-invariin from familiar objects to novel objects and orientations.
3.1 Per-Orientation Accuracy Heatmaps
A detailed inspection of the network's generalization capability for OoD orientations is enabled by introducing per-orientation accuracy healmaps. Recall from Sec. 2.3 that the continuous space of object orientations, represented by Euler angles, is discretized into cubelets (Fig. 3a) - local areas of the rotation space. For each cubelet the network's performance is evaluated in terms of the classification accuracy (()),, where 0 is an orientation of interest. Accuracy heatmaps are 2D projections across a specified dimension of the full accuracy orientation cube (Fig. 3o.c Section 2.3) These heatmaps reveal a structured pattern of generalization the form of increased classification accuracy for OoD (i.e., novel) orientations.
For example, Figure 3 shows that for seed orientations at the center of the heatmap (red box), the network (in this experiment ResNet18 (He et al.} 2016); see Section [2.2] yields the highest accuracy (brightest cubelets) for adjacent orientations around the seed, depicting small 3D perturbations of the seed orientations. Further inspection of the heatmap reveals other orientations, in this example, brighter cubelets forming the figure '8' (stretching the seed sideways and along the heatmap's boundaries, enclosing two darker "holes"), for which the network performs better than for the rest of the OoD orientations. These orientation mainly depiet in-plane rotations of the seed orientations. See also Figure S2 heatmap where the seed is in the "hole" (ă' seed). This heatmap yields an inverse of the figure "g" pattern. This heatmap demonstrates case of "silhouette" generalization, as the airplane is only seen from the front for partially-seen instances, but the model generalizes to views of the airplane from the back.
Figure 3p also shows qualitatively that an increase in number of fully-seen instances leads to stronger OoD generalization the aforementioned orientations (sce also Fig. S2) This phenomenon can be reliable quantified. Figure D reports the OoD accuracy considering the average classification accuracy across all OoD orientations. It shows an increase in OoD accuracy data diversity (i.e., the number of fully-seen instances) increases, under various conditions, including different seed orientations, different image datasets and across datasets. The accuracy heatmaps provide more detailed means of assessment to the overall average accuracy measure, depicting the generalization patterns and indicating which orientations account for the network increased performance (f.e., Fig. 3))
The patterns of increased accuracy depict a partitioning of the orientation space, which reappears for various seed orientations (Figs. 32. SI. various sizes of the training set and different object categories (e.g., Airplane, Car, Shepard & Metzler (SM) objects, see Fig. S2) other architectures (see Fig. [53), and training controls (see Fig. 54.. Next, we model the patterns of generalization observed in these qualitative results in order to gain more insight.
3.2 Modeling Generalization Patterns
We formulate a model of partitioning rules, which can be used to predict the OoD generalization patterns of the network, given a seed of in-distribution orientations. Recall from See. 2.4 that the model, denoted by (0), has three components: A(0), which captures small angle rotations around 0; E(O), which cap- tures in-plane (2D) rotations; S(0), which captures object silhouette projections at the orientation (see details in scction 2.4p. We evaluate the model's performance by mcasuring the Pcarson correlation coef- ficient P between the accuracy of the networks as measured in our experiments and as predicted by the model, i.e., (O)) (o))
10

---

Published in Transactions on Machine Learning Research (08/2025)
1.0
Clssifficttoon
-0,6
0..4
X
X
0.0
20 Fully Seen
40 Fully Soon
R2 -
R 0
Figure 3: Observed generalization patterns in per-orientation accuracy heatmaps. When trained with a combination of fully-seen instances and partially-seen instances, DNNs demonstrate the ability generalize outside of their training distribution. Generalization behaviors demonstrated measuring per- orientation accuracy. (a) All orientations can be described by three Euler axes (a, (,,3, and rotations are periodic around these axes. These properties allow for the visualization of all possible orientations with an orientation cube, shown here. The orientations contained within the colored reclangular prisms are the seed orientations - those orientations of the partially-seen instances included in training (i.e., are in-distribution). The seed orientations differ depending on the experiment. All other orientations are OoD. (b) Increased network generalization for OoD orientations, with increased instance diversity (ú.e., number of fully-seen.) Fach cell in the heatmap the average classification accuracy of the network for given value of B and 7 across all values of a. Chance level is 0.02 (2%), single repetition, seed is å. (c) Different in-distribution parameters (in this case, seed) alfect generalization behaviors. Single repetition.
Figure -op shows the predictive power of the model and its components in experiment with different seed orientations and several object categories. We conducted a large series of experiments under various settings, including different seed orientation distributions, various amounts of training examples, object calegories with different levels of symmetry (Fig. (5).. In all experiments our model highly predicts the network's behavior, indicating that indeed the networks generalization patterns for OoD orientations follow the model's partitioning rules. This is true even across categories, when the seed is taken from one category (e.g., SM) and the fully-seen instances are taken from another (e.g., Airplame).
We now analyze the contribution each component of the model to predicting per-orieeeeee per-orientation OoD accuracy. The model's component A(0) ('small-angle' rotations), the best prodictor for the network's OoD behaviour, for highly articulated objects such as the SM objects. The model's component E(0) ('in-plane' rotations), is better predictor for non-articulated objects with inherent symmetries. Finally, the model's component
11

---

Published in Transactions on Machine Learning Research (08/2025)
Tuly
0OD Accuracy
0.4-
0.3
0.0
Airpiane SM
Car Airplane SM Airplane SM
Car Airplane SM
Airpiane SM
Car Airplane SM Airplane SM
Car Airplane SM
SM Airplane
SM Airplane
SM Airplane
SM Airplane
b
1.0
Component
Random
Peason Correlation Coefficient
0.6
Uniform
0.4
DiD.itttbuuion
0.2
Small Angle
0.0
In-Plane
.2
Small-An Small-Angle
In-Plane
AirplaneCar
SM AMipiiaaane SM AirplaneCar
SMAiipllne
AirplaneCar
SMAirpiane
AirplaneCar
SM Alrplanne SM
All Components
SM Airplane
SM Airplane
SM Airplae
SM Airplane
1.0
0.8
In Plane
0.6
e Tuly Seen
10
0.4
0.2
0.0
CirplaneCar
AMAirplane SM Car
SM AMArrplane SM AirplaneCar
SMAirplane SM Airplane
SM SMAplaane SM
Airplane
Airplane
AMAirpleee
SM Airplane
Figure 4: Modeling generalization patterns for OoD orientations. The bar plots show several trends related to DNN OoD classification patterns. The trends are measured under the various controls, including in-distribution orientations conditions (a,4, 8, and object category, which is either single object in Airplane, SM, Car, transfer across two categories, when the fully-seen instances of different category than the parlially-seen instances as in Airplane + SM and vice versa. These transfer cases are visually separated from the other cases. (a) Network generalization for OoD orientations increases with increasing number of fully seen (blue shading.) This trond holds across in-distribution orientations (seed) and object calegory conditions differ (i.e., Airplane- - SM). (b) Top: We introduce a predictive model for OoD orientation generalization (black "All Components") which is highly predictive of experimental results, with greater than 0.8 Pearson Correlation Coefficient for all experimental controls. (Results are shown for experiments with 40 fully-seen instances.) Null hypothesis predictive models, including "Random Uniform' and "In-Distribution, have very low correlation coefficients. We also ablate our predictive model, including only some sub-components, like only-"Small Angle', only-"In-Plane" or only-"Small Angle + In-Plane." These ablated models have lower correlation coefficients than "All Components," and vary relation to one another depending on the experimental condition. Bottom: We isolate the predictive power of the only-"In-Plane! component for all experiments with range of number of fully-seen. The increasing predictive power of the "In-Plane" component correlates with increasing OoD accuracy the number of fully-seen instances increases. This suggests that generalization to "In-Plane" orientations drives OoD accuracy.
including when fully-seen and partially-seen are the same calegory, and when they
8(0) is predictive of network behavior under special cireumstances, when the silhouette of an object appears similar both in-distribution and OoD, but this component docs not have much of an effect overall model performance. Further analysis of 'in-plane" component in Fig. Ep-bottom illustrates how generalization lo 'in-plane rotations emerges with the increase in data diversity. Thus, the increase in OoD generalization is primarily to 'in-planc' rotations. On the other hand, the "small-angle" and 'all-components' models are negatively correlated and uncorrelated with the number of fully-seen, respoctively. See Fig. [S for further details, and also Fig. S5 for the same analysis with various training controls and other architectures.
12

---

Published in Transactions on Machine Learning Research (08/2025)
3.3 Individual Unit Neuronal Analysis
Figure 5 illustrates activation of individual neurons for stimuli of fully-seen and partially-seen instances. Each group of images depict input stimuli of particular instance for which a particular neuron has the highest activation, along with the neuron's per-orientation activation heatmap for the instance. The pat- terns soen in the ncurons' activation heatmaps resemble the partitioning patterns of the accuracy heatmaps shown in Figure p. Some neurons exhibit similar activation patterns for both fully-seen and partially-seen instances, while others do not.
A quantifiable mcasure of these neuronal responses can help with understanding how generalization occurs in the network- particularly generalization to OoD orientations of partially-see instances, where generalization must stem only from seed orientations seen during training. Previous neural analysis approaches, like t-SNE, fail to capture the shared representations between fully-seen and partially-seen instances, and therefore cannot be used to advance hypotheses related to dissemination (see Fig. S9
Recall from Section 2.5 we define an activation invariance score in the range [0 (Eq. 6 between sets of orientations, in particular between the seed orientations and OoD generalizable orientations or non- generalizable orientations. The invariance score yields higher values when neuron fires for both sets of orientations, and lower values when it fires only for one set. See See 2.5 for details on how to operationalize partitioning the OoD orientations into generalizable and non-generulizable orientations, and see Fig. S8 for results that demonstrate that OoD accuracy is well captured by this partitioning.
We now evaluate whether OoD generalization is predicted by invariance. If that is the case, this would provide evidence that invariance underlies OoD generalization. Thus, we analyze whether the network accuracy is explained by the emergence of invariant representations. Figure 5 depicts a scatter plot of the invariance score against the classification accuracy. Each dot represents an experiment (a tuple of number fully-seen, object and architecture type) and the coloring indicates the respective instance set (fully-seen or partially- seen) and orientation set (generalizable or non-generalizzbbee. There is a clear correlation between increasing levels of classification accuracy and increasing invariance score for the partially-seen instances. Furthermore, the plot shows a clear partition between generalizable and non-generalizable orientations with respect to the invariance score, where significantly higher invariance scores are measured for the generalizable orientations.
Note that for fully-seen instances (Fig. 5p gray dots), all orientations are in-distrübution, including the seed, generalizable and non-generalizable (data comes from the validation sct). Since all orientations are in-distribution, the network achieves accuracy al ceiling levels regardless of the neuronal invariance score. i.e., these orientations fall within the training distribution and the network has learned to associate them with their corresponding object instances. Nevertheless, the fully-seen instances cxhibit the same invariance partitioning between generalizable and non-generulizabbee orientations as the partially- seen instances.
These results provide evidence that OoD generalization is driven by emergent invariant representation. How- ever, the emergence of partially-seen invariance in generalizable orientations is intriguing, as this invariance can not be directly learned since partially-ser instances are only seen in seed orientations during train- ing. We hypothesize that this invariance is disseminated from the invariance that develops for fully-seen instances. To investigate if this may be the case, we analyze the relationship between partially-seen and fully-seen invariance. Figure 5p depicts a direct comparison between the invariance score of the fully-seen and partially-seen sets for both the generalizable and son-generalizable orientations. The partition wet generalizable and non-generalizable orientations cxhibited again - the non-generulizable invariances are in the bottom left. corner, while the generalizable invariances are in the top right corner. Each point in this plot represents the joint invariance of fully-seen and partially-ses instances at a given orientation. The plot shows a tight correlation between the invariance scores of the fully-seen and partially-seen instances, as most of the points lie within a band roughly 0.1 units away from the line of parity, x = y. This correlation suggests that partially-seen invariance emerges due to development of fully-seen invariance. See also Fig. S0 for the same analysis with other training controls.
Taken all together, the correlations between fully-seen invariance and partially-sen seen invariance, and between partially-seen invariance and generalization behavior, provide substantial evidence that OoD generalization
13

---

Published in Transactions on Machine Learning Research (08/2025)
Neuron
Neuron 2
Neuron
Neuron 13
Neuron 16
0.5
Fully Seen
-0.4
normalized Activation
+
-0.3
0.
Partially Seen
1.0
Orientation
0.9
Orientation
Fully Seen,
Generalizabie
Generalizable
Non- Caneralizilleeee
Fully Scen,
Bon-Gennrolizable
Invariance Partial
Parity
Accuracy
0.6
Partially Seen, Genaralzabll
0.7
4/- 0.1 Deviation
Portial
Nor-G Generalizable
Partially Seen.
0.6
0.4
VIT
0.5
1
0.2-
0.4-
0.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.35
0.95
0.4
0.6 0.7
0.9
Invariance
Invariance Full
Figure 5: Neuronal analysis, Invariance and Dissemination. (a) An intuitive visualization of neural activity. Each square is the response of a single neuron to the airplane instance that most highly activates, it portrayed in two ways: 1) the top-8 images that most highly activate the neuron (in no particular order), 2) the heatmap of the per-orientation normalized neural activity for the airplane instance. Neurons tend exhibit patterns of activation related to the patterns of generalization behavior (Figs. 3p for example.) and are invariant to range of orientations that respect the partitioning of OoD orientations. Comparing the neural responses in each column demonstrates that the patterns of activation are similar between the fully-seen instance that most highly activates the neuron and the partially-seen instance that most. highly activates it due to shared visual, part and semantic features between these instances. Several randomly sampled penultimate layer nourons, arranged into columns, demonstrate that these findings apply to many neurons. (b) Each experiment is portrayed by four dots, one for each of the invariance scores (Eq, 6 depicted with different colors. Each experiment category and proportion of fully-seen, see Sec. Averaging the activations in the partitioned regions (seed, generalizable, mon-generalizable) and computing the invariances (defined here: Eq between seed and OoD regions captures overall generalization in the network. Plotting the generalization metrics against accuracy for those regions demonstrates clear correlation between increasing invariance and increasing OoD classificatton accuracy (i.e., purtially-seen instances in OoD orientations.) Visual Transformer results are placed separately to highlight that they follow the same trend, though their invariance is scaled higher. (c) Plotting fully-seen invariance against partially-seen invariance for the same experiment also yiclds a tight correlation, suggesting that dissemination of invariance from fully-seen to partially-seen instances enables increasing generalization in OoD orientations of partially-seen instances.
DNN trained on a specific combination seed, object
is driven by dissemination internal network invariances from fully-seen to partially-seen instances through shared features.
14

---

Published in Transactions on Machine Learning Research (08/2025)
4 Conclusions
In this work we analyze the generalization behaviors of DNN's on rendered images from all orientations and observe dissemination of orientation- invariiann for orientations that appear like 2D rotations (in-plane) of in-distribution orientations, For non-generalizable orientations, the network has not developed orientation- invariance with respect to the seed orientations (demonstrated by the lower invariance score in our results). Our results support the hypothesis that the network disseminates stances to partially-seen instances using brain-like mechanism similar to those reported by (Logothetis & Sheinberg 1996; Poggio & Anselmi 2016). Neurons are feature detectors, and during training neurons are tuned to detect the foatures of fully-seen objects at multiple orientations - i.e., the neurons become selec- tive to the feature, but invariant to the orientation. Some features that neurons are tuned to are shared between fully-seen and partially-seen instances (p). Therefore the invariance that develops for features of fully-seen instances are gained "for free" for partially-seen instances in the same orientations. Our results provide a quantitative assessment of this hypothesis and elucidate the intricate neural processes involved in object recognition, underscoring the critical role of individual neuron; feature-based representations for OoD object recognition.
inrnutaaioo-ivvaiinne
of fully-seen in-
Limitations and future work. This study reveals discernible patterns in the successes and failures of DNNs across diverse orientations which can be effectively characterized and explained through the analysis of neural activity. This underscores the potential for more comprehensive analyses of DNNs that transcend the conventional approach of solely focusing on average accuracy. This study was limited to a supervised classification setting, and to fairly small DNNs. Nonetheless a promising research avenue is to apply the analytical methods introducod in this paper to other newer models.
A key question arising from our results is to explain why DNNs disseminate orientation-iivariince only to in-plane orientations. All object instances are distinguishable at all orientations, as evidenced by the high in-distribution accuracy achieved by the DNNs. Therefore the lack of orientation-invarianee for such non-generuliaable orientations is an outcome of the DNN's learning process. We speculate that this may be because orientations that are not in-plane are affected by self-occlusion, which poses a particular challenge for DNNs (Michalkiewicz et al. 2024)..
Furthermore, various efforts have been made enhance DNNs' generalization capabilities to OoD orienta- tions including leveraging preconceived components for DNNs, such as 3D models of objects (Angtian et al. 2021), sophisticated sensing approaches like omnidirectional imaging (Cohen et al. 2018) or novel archi- tectures like Light Field Networks (O'Connell et al. 2025). Other works, including by Cohen and Welling (Group Equivariant CNN's (Cohen & Welling 2016) and Steerable CNN's (Cohen & Welling 2017 ) induce invariance by construction with modifications to the CNN architecture. These works generalize the transla- tion inductive bias inherent in CNN's to broader mathematical groups, including rolations. However, these approaches focus on architectural improvements that augment the underlying capabilities of these networks, but don't 't investigate whether the vanilla architectures cxhibit emergent orientation generalization. Instead, novel approaches that extend the emergent intentation-invariann further gains of OoD) generalization. Biological agents may overcome the difficulties associated with recog- nizing OoD orientations by leveraging the temporal dimension to associate orientations and learn invariant representations (Ruff, 1982; Johnson & Aslin 1996; Ratan Murty & Arun 2015). The mechanisms that utilize temporal association may hold fundamental significance, given that they have access to a plentiful source of training data that docs not rely on external guidance and task specific labels. This data is readily available prior to any visual task and has the potential to contribute to the emergence of oriintatioon-inaaiian representations beyond in-plane orientations.
inherent within networks might allow for
Previous studies have extensively compared the behavioural and clectrophysiologicall aspects of brains and DNNs (Yamins et al. 2014 Yamins & DiCarlo 2016). However, a direct comparison between these systems alone has limitations in providing insights into the underlying mechanisms of object recognition in DNNs. This is due to the possibility that while certain fundamental mechanisms may be shared across these systems, the manifestation of these fundamental mechanisms can differ at the behavioral and eloctrophystological levels. Our study has provided compelling evidence of brain-like neural mechanisms in DNNs that facilitate
15

---

Published in Transactions on Machine Learning Research (08/2025)
object recognition in novel orientations, even though these mechanisms manifested differently than in biological systems. For instance, while humans and primates can recognize objects in orientations that are not simply 2D rotations, this capability is not fully replicated in DNNs. Thus, we can conclude that the neural mechanisms that have been observed to govern recognition in biological systems largely apply to DNNs, albeit with distinct manifestations across these systems. will be interesting to follow this line of investigation across biological and artificial systems envision a general theory to explain emergent mechanisms in both brains and machines.
References
Amro Abbas and Stéphane Deny. Progress and limitations of deep networks to recognize objects in unusual poses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp- 160-168, 2023.
Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proc of the IEEE/CVF Conf on Computer Vision and Pottern Recognition, pp. 4845 4854,454, 2019.
Wang Angtian, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. In Proc of the Int Conf Learning Representations, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. NejectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, pp. 9448 9458, 2019.
MC Booth and Edmund Rolls. View-invariant representations familiar objects by neurons in the inferior temporal visual cortex. Cerebral cortex (New York, NY: 1991), 8(6):510-522 -23,, 1998.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213 229. Springer, 2020.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repasitory. Technical Report arXiv:1512.03012 [cs.GR), Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
Taco Cohen and Max Welling Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999. PMLR, 2016.
Taco Cohen and Max Welling Steerable CNNs. In International Conference on Learning Representations, 2017. URL
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical CNNs. In Proc of the Int Conf Learning Representations, 2018.
Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce. Stimulus-selectiee properties of inferior temporal neurons in the macaque. Journal of Neuroscience, ((8):2051-2062 1984.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Fang Fang and Sheng He. Viewer-centered object representation in the human visual system revealed by viewpoint aftcreffects. Newron, 45(5):793-800, 2005.
Winrich A Freiwald and Doris Tsao. Functional compartmentalizaation and viewpoint generalization within the macaque face-processing system. Science, 330(6005)::84)) 08155581,151 2010.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Isabel Gauthier, William G Hayward, Michael J Tarr, Adam W Anderson, Pawel Skudlarski, and John C Gore. Bold activity during mental rotation and viewpoint-dependen object recognition. Newron, 34(1): 161 1711 2002.
Michael Gazzaniga, Richard Ivry, and GR Mangun. Cognitive neuroscience. the biology of the (2014) 2006.
Herbert Goldstein, Charles Poole, and John Safko. Classical mechanics. Addison-Wesley, 3rd edition, 2002.
Nate Gruver, Mare Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, 2023. URL http://pparreeeeeeeeeetfrrrraeaa
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience- inspired artificial intelligence. Neuron, 95(2):245-255)) 258, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition, pp. 770-778, 2016. We used the following implementation in our experiments: torchvision porchvision..ndeee.reerrnm models.
htmll
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), pp. 2961 2969, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Denscly connected convo- lutional networks. In Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017. We used the following implementation in our experiments: https://pytordh org/
vision/stable/modeer htm.htmmlttmmmm Seml@torchvissin modelss.denseeet..
Hojin Jang, Syed Suleman Abbas Zaidi, Xavier Boix, Neeraj Prasad, Sharon Gilad-Gutnick, Shlomit Ben- Ami, and Pawan Sinha. Robustness to transformations across categories: Is robustness driven by invariant neural representations? Neural Computation, (5(12) 2023.
Scolt. Johnson and Richard N Aslin. Perception of object unity in young infants: The roles of motion, depth, and orientation. Cognétive Development, 11(2):161 180, 1996.
Diederík P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint, arXiv:1412.6980, 2017.
Eucaly Kobatake and Keiji Tanaka. Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex. Journal of neurophysiolony, 71(3):856 867, 1994.
Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. CORnet: Modeling the neural mechanisms of core object recognition. bioRxiv preprint, 408385, 2018. We used the following implementation in our experiments: https://githut: http:///iicuuu.cooooiccaaa/aa/// wtn.//icaalllaa//oRtee.
Karel Lene and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 991- 991-999,400 2015.
Bin Li, Yuki Todo, and Zheng Tang. The mechanism of orientation detection based on local orientation- selective neuron. In 2021 6th International Conference on Computational Intelligence and Applications (ICCIA), pp. 195 199. IEEE, 2021.
Nikos K Logothetis and Jon Pauls. Psychophysical and physiological evidence for viewer-centered object representations ions in the primate. Cerebral Corler, 5(3):270 288, 1995.
Nikos K Logothetis and David Sheinberg. Visual object recognition. Annual review of neuroscience, (1):577 621, 1996.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how convolutional neural networks generalize to out-of-distribution category category-vievpoint combinations. Nature Machine Intelligence, 4(2):146 153, 2022.
Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, and Guha Balakrishnan. Not all views are created equal: Analyzing viewpoint instabilities in vision foundation models. arXiv preprint arXiv:2412.19999, 2024.
Netta Ollikka, Amro Kamal Mohamed Abbas, Andrea Perin, Markku Kilpeläinen, and Stephane Deny. A comparison between humans and AI at recognizing objects in unusual poses. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://oppenreviory wwt.:eett/ooou httpss//gpaaoorroooooo///o//oo/p
Thomas P. O'Connell, Tyler Bonnen, Yoni Friedman, Ayush Tewari, Vincent Sitzmann, Joshua B. Tenen- baum, and Nancy Kanwisher. Approximating human-level 3d visual inferences with deep neural net- works. Open Mind, 9:305 324, 02 2025. ISSN 2470-2986. doi: 10.1162/opmi a 00189. URL
http::///oi..o////// 2ttp11/6//0mmma_0111__0011 00189
__188
Nicolas Pinto, David D Cox, and James DiCarlo. Why is real-world visual object recognition hard? PLoS computational biology, 4(1):e27, 2008.
Tomaso Poggio and Fabio Anselmi. Visual corter and deep networks: learning invariant representations. MIT Press, 2016.
N Apurva Ratan Murty and Sripati P Arun. Dynamics of 3d view invariance monkey inferotemporal cortex. Journal of Neurophysiology, 21100)2210-22194, 2015.
Holly A Ruff. Effect of object movement on infants' decision of object structure. Developmental Psychology. 18(3):462, 1982.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, (1):21)211-22) 252, 2015.
Roger N. Shepard and Jacqueline Metzler. Mental rotation of threo-dimensional objects. Science, 13172)))): 701-703, 1971.
Pawan Sinha and Tomaso Poggio. Role of learning in threo-dimensional form perception. Nature, 36606)))): 460 460-463, 1996.
Shimon Ullman. High-level vision: Object recognition and visual cogmition. MIT Press, 1996.
Shimon Ullman. Using neuroscience to develop artificial intelligence. Science, (63(844)) 693, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 2016.
Daniel LK Yamins, Ha Hong, Charles Cadien, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optmmized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619 8624, 2014.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Supporting Information
S1: Accuracy heatmaps: alternative seed orientations.
S2: Accuracy heatmaps: elfect of data diversity alternative object categories.
S3: Accuracy heatmaps: alternative backbone architectures.
S4: Accuracy heatmaps: alterative training conditions pretraining and augmentation.
$5:
Alternative architectures and ablation studies: modeling generalization patterns.
S6: Ablation studies: invariance and dissemination.
S7: Other predictive model components, by number of fully-seen
S8: OoD accuracy, split between gemeralizable and non-generolizable orientations
89: tSNE analysis on the penultimate layer of a representative experiment.
19

---

Published in Transactions on Machine Learning Research (08/2025)
b
X
Figure S1: Accuracy heatmaps: alternative seed orientations. (a) 9 seed. (b) &' seed.
20

---

Published in Transactions on Machine Learning Research (08/2025)
b
7
7
7
2 2
?
2
7
C
T
J
>
J
20 Fully Seen
30 Fully Seen
40 Fully Seen
b
20 Fully Seen
30 Fully Seen
40 Fully Seen
Figure S2: Accuracy heatmaps: effect of data diversity - alternative object categories. Increasing
number of fully-seen instances, with different object classes. (a) Shepard-Metzler Objects. (b) Cars.
21

---

Published in Transactions on Machine Learning Research (08/2025)
X X
X +
X
Figure S3: Accuracy heatmaps: alternative backbone architectures. (In place of ResNet-18): (a)
DenseNet121. (b) CORnet-S.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Figure $4: Accuracy heatmaps: alterative training conditions pretraining and augmentation. (a) ResNet-18 pretrained on ImageNet Russakovsky et al. (2015), finetuned learning paradigm with airplancs. Network behavior isn't 't mcaingfully alterod. (b) All data (both from fully-seen and partially-seen instances) were augmented with random 2D image rotations. This effectively expands the in- in-distribution set to include all generalizable orientations, This results in generalizable orientations with high accuracy.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Ablation Studies In order to better understand how certain experimental design choices may effect results, we conducted several ablations with ResNet-18. These ablations include:
Pretrained: On ImageNet Russakovsky et (2015). finetuned on our learning paradigm with airplanes.
Augmented: All data (both from fully-seen and partially-seen instances) were augmented with ran- dom 2D image rotations. This effoctively expands the in-distribution set include all generulizable orientations.
50% of the samples from the full experiment for each subset (i.e., each instance in the appropriate orientations.)
Half-Data:
Peason Correlation Coefficient
Random
Uniform
Distiiibibutin
Small-Angle
DenseNet
CORnet
Pretrain
Augment
Half-Data
DenseNet
CORnet
VIT
Pretrain
Augment
Half-Data
DenseNet
CORnet
VIT
Pretrain
Augment
Half-Data
DenseNet
CORnet
VIT
Pretrain
Augment
Half-Data
In In-Plane
Small Small-Angle In-Plane
Companents
1.0
0..8
In Ponne
50.6-
Cemporent
0.4
10
30
40
0.0
DenseNet
CORnet
ViT-
Pretrain
Augment
Half-Data
DenseNet
CoRnet
ViT
Pretrain
Augment
Half-Data
DenseNet
CORnet
vi7-
Pretrain
Augment
Half-Data
DenseNet
CoRnet
ViT
Pretrain
Augmnett
Data
Figure $5:
Alternative architectures and ablation studies: modeling gencralization patterns.
The same analysis as Fig. is applied to the other architectures and the ablation controls enumerated above.
b
1.0
Instance Orientation Fully Seen,
0.9
Orientation
Generaizable
0.8
Fully Seen.
Invariance Fartial
Nar-Genara izable
Non Generallzabll
Partially Seen.
0.7
Partis
0.6
+/ 0.1 Deviation
Fartially Sean.
Non-Generalizable
0.6
0.5
0.2
0.4
Invariance Partially Seen
Invariance Full
Figure S6: Ablation studies: invariance and dissemination. The same analysis as Figs. 5 is applied to the ablations controls enumerated above.
24

---

Published in Transactions on Machine Learning Research (08/2025)
1.0-
Peason Correlation Coefficient
0.8-
Small-Angle Companents Fully Seen
0.6
0.2-
0.0
-2
CirplaneCar
SM AMArplnne SM CirplaneCar
SM Airplaane SM Airplane Car
SM Airplane SM
AirplaneCar Car
SM Airplane SM
SMiilplanee
Airplaa
SM Airplane
SM Airplane
1.0-
All Components # Fully Sean
0.8-
0.6-
0.0
CirplaneCar
SM Airplane SM AirplaneCar
SM Airpiane SM AirplaneCar Airpiane SM
AirpianeCar Airplane
SM AMArlllnee
AMrilaa
SM Airplane
SM Airplane
Figure S7: Other predictive model components, by number of fully-seen In Fig. 4p we report the correlation coefficient for the In-Plane Component for the number of fully-seen instances. We chose this component specifically, as it is well correlated with the number of fully-seen instances, i.e., most of the OoD generalization is to In-Planc orientations. In this figure we show the same plots but for the Small-Angle component and the All-Components models. The former is negatively correlated with number of fully-seen, i.e., it predicts OoD generalization with few fully-seen, but not with increasing fully-seen. The latter has no change correlation, as linear combination of the Small-Angle and In-Plane components, and changes the weigting between the two to maximize correlation at each number of fully-seen.
25

---

Published in Transactions on Machine Learning Research (08/2025)
OoD Accuracy, ResNet, Base Experiments
0.60-
0.15
Arplane
SM
Car
Al Jlplane
SM
Aimiane
3M
Cer
Arpare
9N
Airbbae
90
Ca'
Airtone
64
Arplate
94
Ger
Arplare
SM
SM
Arplone
SM Arpanne
SM Aipbone
SM Arprre
OoD Accuracy, Various Controls
6
0.60
Accurscy
0.45
0.30
0.15
0.00
densenet122
carnet
VIT
Pretrained
Augmented
HalkData
densenet122
corner
VIT
Pretrained
Augmented
Half-Data
deruenet122
cornet
VI
Pretrained
Augmented
Hal--Data
densene
comeet
VIT
Pratrained
Augmented
Hala
Figure S8: OoD accuracy, split between generalizable and non-gennerllizbble orientations. In Fig, 4 we report the average accuracy across all OoD orientations. As we note, however, accuracy behavior is differentiated between generulizable and gen-geenerlliaale orientations. Here we report the average accuracy for these two orientation groups. Gray horizonial lines indicate chance performance of 2% and 10% (the latter relevant in the case where fully-seen seen and partially- seen instances are two different classes.) Generalizable accuracy is always greater than non-generalizable accuracy. The former always well above chance, while the latter is below or at chance level. (a) The generalizable and non-generulizable average accuracy for the
set of experiments presented in Fig. 4_. (b) The average accuracies for several other conditions. These
other conditions are alternative architectures and training controls (explained above Figs. S5 S6
26

---

Published in Transactions on Machine Learning Research (08/2025)
b
Rully Seen
Partia ly Seen
d
1.0
2.4
Go Pasteee sicabbi Scor
2.2
0.6
2.0
0.4
0.0
Figure 89: tSNE analysis on the penultimate layer of a representative experiment. The 512 dimension activation vectors in the penultimate layer each instance and orientation are recorded. We employ tSNE reduce these 512 dimensions down two dimensions. (a) Object instances are colored (semi-) uniquely. (20 colors are distributed to 50 instances due to the limits of choosing many perceptually different colors.) For the most part, instances cluster together without much overlap between clusters of different instances. This indicates that representations of instances are separable, and that the task is solved by DNN. (b) Each point is colored based on whether the instance it represents is fully-seen partially- seen. Partially-seen clusters are independent of other clusters, both fully-seen and partially-seen. It is therefore difficult to determine the range of behaviors for partially-seen instances namely, why certain OoD orientations are generulizable, while others are not. (c) Points are colored with the degree of generalizabilty, as predicted by the predictive model of DNN generalization behavior. Note that points within each cluster are ordered - they are arranged such that generalizable orientations are far from non-generalzzable orientations with smooth transition between them. (d) Points are colored with the classification accuracy of the network (for the given instance and orientation.) While fully-seen instances have near 100% accuracy across all orientations, portially-seen show differentiation in accuracy between generalizable and non-generulizable orientations.
27


Reviews:
[{"review_id": "XFyfqdG4ro", "reviewer": "Reviewer_hW1z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the study interesting and well executed. The conclusions are also well supported by the results. Some technical aspects of the study could be described more clearly (see requested changes).\n\nAdditional remarks:\n\"A key question arising from our results is to explain why DNNs disseminate orientation-invariance only to in-plane orientations. \"\n=> I think the convolutional architectures (and patchify process in ViT) may provide an inductive bias helpful for generalizing in-plane rotation but not in-depth rotation. Indeed, in-plane rotations translate (but also rotate) features on the image grid, which convolutional architectures can partly exploit. \n\nConsider updating your citations with more recent work investigating the ability of recent and SoTA architectures (including VLMs) to generalize pose invariance, and comparing them to humans:\nAbbas & Deny 2023: https://arxiv.org/abs/2207.08034\nOllikka et al 2025: https://openreview.net/forum?id=yzbAFf8vd5\nMichalkiewicz et al 2024: https://arxiv.org/abs/2412.19920\nO'Connel et al 2025: https://direct.mit.edu/opmi/article/doi/10.1162/opmi_a_00189/128124/Approximating-Human-Level-3D-Visual-Inferences\nKosoy et al 2025: https://www.arxiv.org/abs/2503.03840", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study empirically investigates the ability of various deep network architectures (CNN, transformers) to generalize invariance to 3D object pose. Networks are trained on an object recognition task (distinguishing planes, cars and Shepard&Metzler shapes). Some objects are seen in all views during training, while others only in some views. Network performance is then assessed at test time on the partially viewed objects. Some transfer of pose invariance is observed, both at the level of single neurons in the penultimate layer of the network and when looking at classification accuracy. This transfer of invariance can both be seen within classes and across classes (e.g., only planes are seen in all views during training, and then the models are tested on Metzler shapes seen in only some views). Interestingly, networks are found to generalize much better in-image-plane rotations than in-depth (3D) rotations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify the following aspects (or indicate the relevant passage in the manuscript):\n\n1. I am not completely sure what task the networks were trained on. Is it a three-way classification task (plane/cars/SM)?\n\n2. It is unclear and I don't know whether it is said somewhere whether the partially seen views include all alpha, beta and gamma angles together, or whether these angle ranges are explored in separate experiments.\n\n3. In fig 3a, I am not sure to understand why there is an increase in generality for both alpha and gamma. Shouldn't it be only for alpha (rotation in plane)? Also, what do these different conditions represent exactly? Do they represent different angle *training* ranges, or different angle *testing* ranges? And, what is alpha'?\n\n4. In fig 4b, \"Each dot represents the results of an experiment\". It is unclear to me whether these experiments correspond to networks *trained* on different objects/views or *tested* on different objects/views.\n\n5. Fig 4b and c: labels and ticks are too small. Fig s3: typo in \"meaingfully\". Fig s6: Labels (a. and b.) missing from the figure.\n\n6. I did not fully understand what the \"silhouette\" variable is.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GGiU18FX2U", "reviewer": "Reviewer_uetx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n * A nice experimental approach to attack an important and open question using a limited, well-defined task.\n * Use of three kinds of interesting stimuli, which both differ from one another and were not included in the pre-training of the models, and thus allow for an analysis of fully-observed and partially-observed object angles.\n * The regression model used allows for arguing what kind of out-of-distribution angles get learned, suggesting the main result regarding the contribution of in-plane and nearby angles to generalisation. It would have been even more interesting if you could show more factors which contribute very little to generalisation to unseen angles.\n\n### weaknesses\n * It is not clear if naive models (without training on the datasets included in this study, after the original training on ImageNet, etc) already demonstrate some non-trivial baseline of correct classification for the object at hand. I expect them not to, but this needs to be established as a control. For example, models may exhibit a \"prototype effect,\" where, at a specific angle, classification accuracy is high due to general image recognition training.\n * It is unclear whether the observed generalisation is similar to what was reported in previous studies. It needs to be established that the current approach discusses the same phenomena as out-of-distribution generalisation and that generalisation to unseen orientations does not behave categorically different.\n * The partitioning of the orientations space into generisable and non-generalise orientations is somewhat arbitrary, and you need to demonstrate it does not change dramatically when changing parameters (thresholds, etc.).\n * It is not clear if you claim that different architectures behave similarly in terms of their ability to perform out-of-distribution, and if this behaviour has similar roots (e.g., similar regression coefficients, or similar neural behaviour).\n * The neural analysis contribution is unclear: \n   - It seems to be based on very specific assumptions (something like \"generalisation is high activation to both seen and unseen angles\");\n   - It is using ad-hoc defined statistics \"invariance score yields higher values when a neuron fires for both sets of orientations\" (e.g., if a neuron has constant high activation, will you count it correctly?);\n   - It is limited to \"neurons in the penultimate layer\";\n   - It does not lead to interesting conclusions: the idea that \"neurons are feature detectors\" was assumed by your analysis, as it treats neurons' responses as binary through the use of thresholding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Empirical analysis of the ability of visual DNNs to generalise from observed object angles to unobserved ones. As a concrete case of extrapolation, or out-of-distribution generalisation, gaining understanding of when such generalisation succeeds or fails is an important question. Reports some generalisation, which can be attributed to generalisation from observed angles to nearby ones, as well as in-plane generalisation. Tries to uncover single-neuron responses which are associated with such generalisation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Compare results across different architectures, both quantitively (e.g., compare the mean and variance of some statistic) and qualitatively (e.g., in figures where you pool results across architectures, indicate architecture using different markers). Make it clear if you claim that different architectures are similar or not.\n 2. Perform baseline analysis on naive models, as detailed above.\n 3. Show how the ability to perform out-of-distribution generalisation compares between your task and previous studies, using different tasks (present the data to support the claim that \"average classification accuracy across all OoD orientations, our experiments reproduce previous results\").", "broader_impact_concerns": "Not provided and not needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W6NNwJxYbO", "reviewer": "Reviewer_viQw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\nThis paper studies the generalization of DNNs to novel orientations unseen during training, which should be important to machine learning researchers working with 3D data. \n\n[Weaknesses]\n\n[Clarity]\nI would prefer if the experimental setup were made more explicit. In particular, it would be good to clarify the training task, e.g. is it supervised? How are $\\alpha, \\beta, \\gamma$ sampled, and how are the ranges for sampling chosen? At test-time, are the in-distribution images tested exactly those used during training, or are they just drawn from the same distribution as the training distribution? Which plots correspond to which sampling settings?\n\nI also think the organization of this paper makes the paper hard to understand. I would prefer if the Methods were presented before the Results, which seems to be the standard format for most machine learning papers. \n\nIn Figure 3, the $\\alpha, \\beta, \\gamma, \\alpha’$ labels are not explained, and it's not clear which architecture(s) this plot was generated from. Plots (e.g. Figure 4b, 4c) are not labeled with what dataset and model the plot was generated from.\n\nFor models where there is not one set configuration (e.g. ViT, which frequently comes in tiny, small, base, giant, etc. sizes), it would be good if the authors could report the number of parameters. \n\n[Limitations of the experimental setup]\nThis paper only examines small networks (e.g. ResNet-18), (presumably) trained to do supervised instance image classification.The focus of this paper on supervised training is a major limitation of this paper, as self-supervised models (many of the large models used today) behave differently than supervised models. I also think that ideally, tasks other than classification should be examined, as classification may not require a more limited understanding than, say, segmentation, and may have a different pattern of generlization to OOD orientations. It’s not clear that we can draw conclusions about generalization in such a scenario.\n\n[Potential overclaims]\nAs discussed above, I think the implication that this work may have ramifications for Qwen2-VL and Llama 3-V (last paragraph of the Introduction), which use self-supervised, pre-trained vision encoders, should probably be removed. \n\nIn the first paragraph of the Discussion, it seems to me that his paper is implicitly claiming that the scenario examined in this paper is more realistic or challenging than that considered in equivariant networks. I disagree that this is the case, as a network that achieves invariance to 3D rotations would perform perfectly in this scenario. \n\n[Additional empirical results]\nIdeally, the paper would also include qualitative neuronal activation results for Cars and Shepard-Metzler objects. \n\n[Hyperparameter settings]\nThe choice of some hyperparameter settings, such as the 10% threshold for for generalizable vs non-generalizable, and the 95th percentile threshold, are not explained in the paper. Some are not reported (e.g. parameter count/architecture of ViT models). \n\n[Results of multiple runs]\nWhile the multiple runs were made for each experiment, we don't see the results of this (except possibly Figure S.1, which is not very clear: is this a different experiment, with 5 runs for each of (a) and (b), are these 2 of the 5 runs of the same experiment?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the ability of deep neural networks (DNNs) to generalize to objects in orientations that are not seen during training. This paper studies this question through the following experimental setup: during training, objects of a particular class are either seen in all orientations (denoted fully-seen), whereas other objects are only shown in a subset of the orientations (denoted partially-seen). Generalization is measured by test-time performance on the unseen orientations of objects partially-seen during training. Training was presumably done in the supervised fine-grained image/instance classification setting, and the metric used to evaluate generalization is classification accuracy. The paper finds that DNNs generalize out-of-distribution to shape and silhouette preserving orientations as well as orientations that are small perturbations of orientations seen during training. The paper also examines generalization at the neuron level and presents a logistic regression model that quantitatively evaluates the paper’s hypothesis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify the experimental settings, which experimental settings were used to generate the figures/plots, and undefined notations\n2. Reorganize the paper to put Methods before Results\n3. Address limitations of this work, e.g. limitation to small networks sizes, limitation to only supervised training, limitation to only examining classification. I think this can be addressed by repeating the experiments on larger networks and self-supervised networks as well as another task.\n4. Address potential overclaims (listed above)\n5. Explain some additional hyperparameter choices\n6. Show more clearly the results of multiple runs", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)