A comparison between humans and AI at recognizing objects in unusual poses
Authors: Netta Ollikka
Amro Kamal Mohamed Abbas
Andrea Perin
Markku Kilpeläinen
Stephane Deny
Venue: Accepted by TMLR ()

Abstract: Deep learning is closing the gap with human vision on several object recognition benchmarks. Here we investigate this gap in the context of challenging images where objects are seen in unusual poses. We find that humans excel at recognizing objects in such poses. In contrast, state-of-the-art deep networks for vision (EfficientNet, SWAG, ViT, SWIN, BEiT, ConvNext) and state-of-the-art large vision-language models (Claude 3.5, Gemini 1.5, GPT-4) are systematically brittle on unusual poses, with the exception of Gemini showing excellent robustness to that condition. As we limit image exposure time, human performance degrades to the level of deep networks, suggesting that additional mental processes (requiring additional time) are necessary to identify objects in unusual poses. An analysis of error patterns of humans vs. networks reveals that even time-limited humans are dissimilar to feed-forward deep networks. In conclusion, our comparison reveals that humans are overall more robust than deep networks and that they rely on different mechanisms for recognizing objects in unusual poses. Understanding the nature of the mental processes taking place during extra viewing time may be key to reproduce the robustness of human vision in silico. All code and data is available.

Published in Transactions on Machine Learning Research (01/2025)
A comparison between humans and AI at recognizing objects in unusual poses
Netta Ollikka
netta. Siita.ollikkkooaaltto
Deportment of Neuroscience and Biomedical Engineering Aulto University, Espoo, Finland
Amro Abbas
afagirn(@ainssammiiory
The African Institute for Mathematical Sciences, Mbour-
Abbur-Thicss Thies, Senegal
Andrea Perin
zinandre@@gmiilcomm com
Department of Computer Science
Aalto University, Espoo, Finland
Markku Kilpeläinen'
Deportment of Psychology and Logopedics University of Helsinki, Finland
Stéphane Deny!
stephone. Brateeyyyrr@@maaaeeeeo com com
Deportment of Neuroscience and Biomedical Engineering Department of Computer Science Aolto University, Espoo, Finland
Reviewed OpenReview: https://opeenrevivi net/ ttr/formm? id= XXXX
Abstract
Deep learning is closing the gap with human vision on several object recognition benchmarks. Here we investigate this gap in the context challenging images where objects are seen in unusual poses. We find that humans excel recognizing objects in such poses. In contrast, state-of-the-art deep networks for vision (FfffciienNNet, SWAG, VYT, SWIN, BEIT, ConvNext) and stato-of-the-art large viston-language models (Claude 3.5, Gemini 1.5, GPT-4, SigLIP) are systematically brittle unusual poses, with the exception of Gemini showing excellent robustness to that condition. As we limit image exposure time, human performance dogrades to the level of deop networks, suggesting that additional mental processes (requiring additional time) are necessary to identify objects in unusual poses. An analysis of error patterns of humans vs. networks reveals that even time-limited humans are dissimilar to feed-forward deep networks. In conclusion, our comparison reveals that humans are overall more robust than deep networks and that they rely different mechanisms for recognizing objects in unusual poses. Understanding the nature of the mental processes taking place during extra viewing time may be key to reproduce the robustness of human vision in silico. All code and data is available at lhttt::/ //ggggttbbb.../ c hww.a//aaAaltttuussslllapoolooo
1
Introduction
In an era marked by the rapid advancement of deep lcarning for computer vision, a natural qucstion arises: Can machines meet or even exceed the capabilities of the human visual system? Numerous recent studies have shown that deep networks outperform humans on well-known object recognition benchmarks (e.g., ImageNet: He et al. 2015); Vasudevan et al. 2022); Dehghani et al. (2023)). Deep networks are even said outperform
Tequal contribution as last authors.

---

Published in Transactions on Machine Learning Research (01/2025)
humans nut-of-distributtio recognition tasks (Geirhos et al. 2021 where the images used for evaluation are subjected to distortions that the networks were not specifically trained on.
However, studies that investigate out-of-distribution generalization mostly focus on local distortions (e.g., texture modifications, blur, color modifications) as opposed to transformations that affect the global structure of the image, such as a change viewpoint. A few studies have investigated the generalization capability of deep networks to recognize objects in various poses, both in non-adversarial (Alcorn et al.
2019 Ibrahim
2022 Madan et al., 2022; Abbas & Deny 2023 and adversarial settings (Zeng et al. 2019 Madan et al.
et al. 2021b a/, and they show substantial degradation of network performance for unusual poses. However, direct comparison with human vision has been missing from these studies, leaving open the question of whether humans outperform networks on this task.
Here, we compare human subjects with state-of-the-ate deep networks for vision and state-of-the-at large vision-language models recognizing objects various poses. We show that-while both humans and networks excel on upright poses humans substantially oulperform networks on unusual poses (except. for Google's Gemini showing exceptional robustness in that condition). As we limit viewing time, humans exhibit similar brittleness to deep networks unusual poses, demonstrating the necessity of additional mental processes for analysing these challenging images. Although networks and time-limited humans are both brittle, an analysis their patterns error reveals that time-limited humans make different mistakes than networks. In conclusion, our results show that (1) humans are still much more robust than most networks at recognizing objects unusual poses, (2) time of the essence for such ability to emerge, and (3) even time-limited humans are dissimilar deep neural networks.
2 Methods
2.1
Dataset collection
Pitcher
Pitcher
Pitcher
Folding chair (
Folding chair (
Folding chair X
Vase
Conch
Drum x
Rocking chair
Tray X
Shovel
Lawn mower >
Lawn mower X
Lawn mower 3
Parachute
Parachute
Parachute
Harvester x
Mouse trap X
Projector X
Balloon
X
Lighter x
Swing x
Figure 1: Example images of the dataset, and corresponding answer choices. Four examples of objects and their three different rotations: (left) upright, (middle) rotated and correctly labeled by EfficientNet, and (right) rotated and incorrectly labeled by EfficientNet. Above each image is shown the correct label and below each image is the alternative label that we selected based on Efficient predictions (see Datasel collection 2.1 for details of this selection).
We collected a dataset of objects viewed in different poses (upright and rotated out-of-planc), test the ability of humans to recognize these objects, and compare this ability state-of-the-ae deep networks (Figure D.
2

---

Published in Transactions on Machine Learning Research (01/2025)
a
Mountain tent
Crash helmet
Unlimited time
ms 150 ms
ms
Unlimited time
Crash helmet
Trush bin
+
Mountain tent
Revolver
Unlimited time
Unlimited time
Unlimited time
Unlimited time
Figure 2: Description of the human tests used in this study. a) Task with limited viewing time: First the subject fixates on cross, then image is displayed for either 40 ms 150 ms, followed by dynamic checkerboard mask shown for 500 ms. Then the subject is asked to choose between two labels for the image (i.e., two-forced-choice task), and has an unlimited time to answer. b) Task with unlimited viewing time: similar test-setting, but now the image and the answer choices are displayed together for an unlimited vicwing time and without back-masking.
We chose 51 different object categories from the ImageNet classes (see Appendir D for the list of objects), and obtained corresponding 3D model for each category from Sketchfab (https:/ //ssee/cafaaaboom/ The objects were chosen based on a few criteria:
Objects should look distinct in different rotation angles, e.g., a symmetrical ball wouldn't be an acceptable object.
Objects should have clear upright poses, e.g., a pen wouldn't be an acceptable object.
The object in its upright pose should be correctly labeled by Noisy Student Efficient.Net-2,, pretrained on JFT-300M and finetuned on ImageNet (Xie et al. 2020) (referred to as EfficientNet below). We chose EfficientNet to guide our selection of objects because it. the best open-source network
recognizing objects in unusual poses, according to recent comprchensive study comparing state-of-the-arl networks this task (Abbas & Deny 2023).
Once the objects that satisfy the criteria were chosen, rendered 180 different random rotations of each of them (along axis and 2). These rotations were given to EfficientNet to be labeled. Then, we performed selection of poses. First, we selected unusual poses that Efficient.Net was able to correctly identify. Second, we selected unusual poses that EfficientNet failed to correctly identify. The dataset thus consisted three different types of poses: (1) upright, (2) rotated and correctly identified by EfficientNet (rotated-correct condition), and (3) rotated and incorrectly identified by EfficientNet (rotated-incorect) condition).
Once the different poses for each object were chosen, created two-forced choice questions for each image based EfficientNett's predictions. The choices consisted of the correct label, and an incorrect label. In the cases where EfficientNect was able to correctly identify the object, we chose the second best guess of EfficientNet. to be the incorrect label, as measured by the category corresponding to second most act ivated unit of the output layer EfficientNet T In the cases where EfficientNet incorrectly predicted the label, we
In some cases, had choose 3rd 4th best guess of EfficientNet, the criterion discard 2nd best guess being that it would be similar the lahel for a non-specialist human (e.g., golden retriever vs. labrador).
3

---

Published in Transactions on Machine Learning Research (01/2025)
used the wrong predicted label as the incorrect label. The final dataset contained 147 different images: 51 upright poses, 51 rotated-correct poses, and 45 rotated-incorrect poses (examples in Figure P
Caveat 1: We used Nef to identify challenging object poses and label alternatives. A caveat of this selection method is that it may introduce a bias against networks, since half of the images were selected to fool EfficientNee (which is a neural network). To mitigate this bias, we separately compare networks and humans on the half of images that have not failed EfficientNet (rotated-correct condition in Fig. 4). Essentially, our conclusions remain identical on that subset: humans (nearly 100% accurate) are much more robust than pure-vision deep networks (overall best network of our collection: SWAG: 82% accuracy) and of note that our selection is hardly adversarial,
most rision-language models (except Gemini). Additionally, since EfficientNet failed on a large fraction of random poses (14.5% of poses, cf Abbas & Deny (2023)) ). Our study is thus not akin to studies performing fine pose optimization to find adversarial examples (e.g., Madan et al. 2021aa)
The stimulus set applied here is not entirely ecologically realistic, as all objects are synthetic 3D models and are displayed without background. However, these choices are justified by our objective to create highly controlled stimulus set and experimental setup: performance in canonical view is compared with performance in rotated view of the very same object, both shown without background. Thus, any performance differences between these conditions, be it human or network, can only be caused by rotation. Conversely, any nobustness found across conditions can only be due to rolalion-invariant recognition, not. context-based reasoning.
Caveat 2:
2.2
Psychophysics experiments
2.2.1
Observers
Altogether 24 observers (12 women, 12 men, aged 19-54) participated in the experiments. Our observers comprised Fimnish students from diverse academic backgrounds, including disciplines such as engineering, psychology, business, and medicine. Participants were required to have normal vision (i.e., individuals using vision aids such as glasses to achieve normal vision) or corrected-to-nomee vision. Additionally, the observers did not report any medical condition (epilepsy, migraine, etc.) that could affect the results.
2.2.2 Apparatus and stimuli
The object images were synthetic images of objects belonging to the ImageNet database classes. For information on the process of stimulus creation, see Dataset collection 2.1 above. Noise masks, which were presented after object image presentation, consisted of dynamic white noise chromatic checkerboards (0.67 degrees of visual angle check size, 100 Hz temporal frequency). The diameter of the object images and the noise masks were 13.3 degrees of visual angle. The stimuli were presented using MATLAB Psychophysics Toolbox in 22.5" VIEWPixx display with a resolution of 1200 1920 pixels, frame rate of 100 Hz, and a viewing distance of 54 cm.
2.2.3 Procedure
Prior to the experiments, observers were provided with a small training set, order to familiarize themselves with the experimental procedure. None of the training objects were featured in the real experiment. After this training, each observer participated in a limited viewing time experiment and an unlimited viewing time experiment. In the limited viewing time experiment, each trial proceeded follows (see Figure 2h). First, fixation cross-hair
presented in the centre of the screen. Once the observer was ready, they pressed the space bar. After 500 ms, the object image was presented. The duration of the object image presentation was 40 ms for 12 observers, and 150 ms for the other 12. These stimulus durations (40 and 150 ms) were based on pilot experiments which suggested that those durations would lcad to performance levels between perfect and chance level performance. The object image was immediately followed by 500 ms of dynamic chromatic
2There were fewer incorrectly identified poses, because in cases, FefficientNet was able correctly identify all 180 rotated poses.
4

---

Published in Transactions on Machine Learning Research (01/2025)
white noise. After that, two response alternatives (in Finnish), one of them always the correct one, were presented and the observer had to choose (by pressing one of two keys on the keyboard) which alternative corresponded to the presented object. Each observer performed 49 trials, in which the image was in one of three types of poses: upright in 17 trials, rotated-correct (correctly classified by EfficientNet, see Dataset collection 2.1 17 trials, and rolated-incorreet (incorrectly classified by NeficientNet) 15 trials. The trial types were interlcaved and presented random order. The unlimited viewing time experiment (see Figure 2b) followed the same structure, except that the image remained on the screen until the observer responded (2.0 on average), the labels were present from the start, and no noise was presented after the object image.
Each observer saw each object only in one pose during the limited viewing time experiment. The object-pose combinations were, however, balanced across observers such that each object was shown in a particular pose (e.g., upright) an equal amount of times, The order in which the objects were presented was different for every subject in order to avoid potential order effects. Furthermore, each observer saw exactly the same object-posc combinations they had seen in the limited time experiment for the unlimited time experiment. This allowed more powerful within-subject statistical analyses. In principle, the previous experience with the same images could affect the observers performance in the unlimited viewing time experiment. However, since the second presentation was not time-limited, making the task very casy for the viewer (sce Results), we expect any advantage of having the image flashed previously to be negligible.
2.3 Machine tests
We then implemented the same tests on a collection of 5 stale-oftthe-art deep works for vision (referred below as 'pure vision networks') and 6 large vision- language models (VLM), Since EfficientNet was used to select the problematic images and labels, we did not include EfficientNet this collection nor in our study of network robustness.
We tested the 5 following pure vision networks: SWAG-RegNetY (Singh et al. 2022 BETT-L/16 [Bo et al. 2021), XonyNext-XL (Liu et al. 2022), SWIN-L (Liu et al. 2021 and ViT-L,/1 L/16 Dosovitskiy et al. 2020 C for model descriptions). The network selection was based on previous work from Abbas & Deny 2022)) which thoroughly evaluated the robustness of 37 state-of-the-art networks to objects presented
ssee Appendic
in unusual poses. This collection provides a diverse set of state-of-theeart and very large models trained on large datasets (e.g., Imagenet- 21K, Instagram 3.6B) with various architectures (ConvNet and Transformers) and training objectives (supervised and self-supervised. These networks obtained some of the most robust results to unusual poses according to Abbas & Deny 2023 and all of them were fine-tuned to the categories of Net.
We mirrored the experiment done on humans by submitting each pure vision network to the same two-forced- choice test. Each network was shown the 147 different images of objects, 51 upright poses, 51 rotated-correct poses, and 45 roloredd-ncoorred poses. We compared the activations of the units of the final layer corresponding to each of the two categories given as choices for each image, and selected the category with corresponding highest unit-activation among the two to be the chosen answer.
Our selection of VLMs included Gemini 1.5 Flash, Gemini 1.5 Pro, Claude 3 Opus, Claude 3.5 Sonnet, GPT-4o, CPTT-4-vision-preiiie and SigLIP /Zhai et al. 2023). For the large-language models (all models excluding SigLIP), the experiment was conducted via the API. Each model was shown the 147 images and provided with the following prompt (see examples in Appendix A
What's 's in this image?
A. |label 1]
B. [label 2]
Choose either A or B and answer in one or two words.
If the model chose the correct label, the answer was counted as correct. If the model chose the wrong label or gave response outside of the label options, the answer was counted as incorrect. If the model didn't provide any answer (e.g., Gemini threw a safety error for a few images), the image was disregarded and not included in the analysis.
5

---

Published in Transactions on Machine Learning Research (01/2025)
For SigLIP, state-of-the-art contrastive text-image model (successor of CLIP (Radford et al 2021)), we fed it the image well as the two label options. The label with the highest score of similarity was accepted as the answer of the model.
2.4 Statistical tests
T-tests: We conducted series of t-tests to compare the accuracies different observers (networks and/or humans) and the same observers in different viewing conditions (e.g., upright vs. rotated objects, limited vs. unlimited viewing time). When the data came from the same observers two different viewing conditions, used paired t-lests (i.e., 40 ms upright vs. 40 rotated) and unpaired I-lests when the observers were different (i.e., 40 ms rotated vs. 150 ms rotated). T-tests were run percentages of correct answers
Upright
g
g O
deep mets
VL.Ms unlimited
VL.M&
150 ms ms
100
Rotated
deep nots
Accuracy (% correct)
90
VL.Ms
unlimited
80
40
SWAG
Gemini 1.5 Flash
70
Gemini 1.5 Pro
60
CPT-4o
G GPT-4-vision-per prevveww
50
Claude 3.5 Sonnet
Claude 3 Opus
40
SigLIP
Figure 3: Comparing neural networks and humans at recognizing objects in various poses. Dark grey bars: Average performance of pure vision deep networks (left: upright right: rotated) with 95% confidence intervals (n=5, grey points represent individual network performances). Light grey bars: Average performance of seven large vision-language models (VLMs). Diamonds indicate individual model performances (full diamonds show the best-performing version the different model classes). Blue bars: Average human performance with unlimited viewing time (n=12, grey points represent. individual performances). Orange & Red bors: Average human performance with limited viewing time (150 ms and 40 ms, respectively, n=12, grey points represent individual performances). Chance performance 50%. Three stars (***) indicate highly significant differences (p<0.0001, "n.s." for not significant. With unlimited time, humans excel at recognizing rotated objects, while pure vision networks struggle (best: SWAG at 70.1%). GPT-4, Claude and SigLIP models follow the same pattern as the pure vision networks, with a significant drop in accuracy for rotated images compared to upright ones. However, Gemini 1.5 Flash mirrors human performance with unlimited viewing time, achieving 97.7% accuracy on rotated images compared to human accuracy of 98.9%. Limiting human viewing time (40 ms or 150 ms) impairs their ability to recognize rotated objects, substantially more than upright objects, bringing their performance closer to network levels.
Error consistency: When two obscrvers and j (networks and/or humans) respond the same n trials, can measure how well their responses (correct/incorreel)) align by computing their observed error overlap
We acknowledge that K-accuracy non-linear index of performance and according to signal detection theory (reen et al. when computing
1966 z-scores would be more appropriate measure commonly used in the field of machine learning, used %-accuracy when performing t-tests. We have, however, also conducted all statistical tests with 2-scores (not reported) and the changes results were very minor, with effect the conclusions of this study.
and performing statistical tests. However,
since K-accuracy

---

Published in Transactions on Machine Learning Research (01/2025)
C
Ci.j
(1)
Cobsu.3
72
where Gi, is the number of same responses, either both correct or incorrect. However, considering only the observed error overlap has limitations, because this overlap is expected to depend on overall accuracy of the observers in virtue of the probability of coincidence of two independent binomial variables (e.g., a network and human with 99% performance are expected to have more overlap than network and human with 90% performance). To address this issue, we compare observers and j to theoretical model of independent binomial observers. This model considers two observers making random decisions, and thus we can only expect overlap due to chance. This expected error overlap Ceap,,, is given by:
Cerps, = P.P, + (1-1:)(1111111 - Pp)),
(2)
where piPj is the probability that observes give the same correct response by chance and (1 (1- (1)p pÿ) the probability that they give the same incorted response by chance. To evaluate the consistency between the two observers i and j. and determine whether it goes beyond what could be expected by chance, we use metric called error consistency (Geirhos et al. 2020 2021 (i.e,, Cohen's x) given by:
Cobss,
Cempis
Kjj
Ceapi.j
Error consistency compares observed consistency to expected consistency, and allows us to quantify whether the observed consistency is larger than would have been expected by chance.
3 Results
We compared human and machine vision on a task consisting in recognizing objects in unusual poses. For this, we selected 5 different pure vision deep neural networks (SWAG, ViT, SWIN, BEÏT, ConvNext), chosen for their state-of-the-arl performance in recognizing objects in unusual poses (Abbas & Deny 2023 and state-of-the-ar large vision-languee models (Gerniii 1.5 Flash, Gemini 1.5 Pro, Claude 3 Opus, Claude 3.5 Sonnet, GPT-4o, SPT---vision-prrvveen SigLIP). We performed a wwo-altenattie-forod--hheee object calegorization Lask: an image of an object was presented to the viewer (either deep network human participant), and then the viewer had to choose between two different names (i.e., labels) for the object. The two labels were selected based on the highest activations of Noisy Student Efficient 's output layer (Xie) et al.; 2020), the most robust network to unusual poses according to Abbas & Deny (2023). For pure vision networks, the choice was made by looking at the highest activation of the softmax output layer for these two labels. EfficienttNet was excluded from our analyses comparing humans and networks because images and labels were chosen based on its mistakes (see Methods). For the vision-language models, each VLM was given an image and prompted with the 2 label choices, and the label chosen by the model was selected to be its answer. The performance of VLMs is discussed in the last paragraph of results, while previous paragraphs concentrate on the comparison of pure vision networks and humans.
With unlimited viewing time, humans outperform deep networks at recognizing objects in unusual poses. First, we compared the ability of humans and networks to recognize objects in upright and unusual poses when humans were not given any time limit for the object recognition (Figure 3 In the upright poses, both humans and neural networks performed very well: networks obtained 100% accuracy and one (ViT-L) failed to correctly label only 1 out of 51 objects; humans had slightly lower performance with accuracy of 98.0% + 2.4% (95% confidence intervals). The very few errors that human observers made are likely to be keystroke errors and misunderstandinng of the verbal response alternatives (as reported by some subjects). We conducted an additional test to discern whether network errors arise from changes in viewpoint, or alterations in image statistics between training and testing distributions (natural vs. synthetic). We subjected all five of our networks to a dataset comprising objects rotated at angles of +10" from their canonical poses. All of our networks obtained perfect or near-perfect results (SWAG-RegNatY: 100%, BEIT-L/16: 99.3%, ConvNext- 98.2%, SWIN-L: 91.1%, ViT-L/16: 95.0%). For wnusual poses, human participants with unlimited viewing time clearly oulperformed neural networks. For humans, the performance on rotated
7

---

Published in Transactions on Machine Learning Research (01/2025)
images and upright images was on par (rotated condition: 98.9% + (.0%)(t(11)) -0.79, p= 0.45), whereas for networks, the performance drastically dropped bet upright (99.6% + 1.0%) and rotated objects (67.1% = 4.3%), with a drop of over 30% performance on average (t(4) = 18.7, P 4.8e-05). Even the most robust network of our collection, SWAG trained on IG-3.6B (Siggh et al. 2022 showed a drop in performance of 30% (from 100% on upright poses to 70.1% on rolated poses).
Rotatad-corest
b
20
2 O
deep uets
VLMs
Com:Nest
BEi
VLMMs
D.S.
0.5
7.5
unlimited
1.50 ms
SKAG
SWIN
4.6
100
40
Rolated incorrett
grou
group
VIT
0.4
deep nets
grop 3
Accuracy (% correct)
90
VLMS
group
unlimited
SWA SMIN
vir
gmmp
80
1.50
40 ms
70
Gemini 1.5 Flash
Gemini 1.5 Pro
60
GPT-4o
Move comteann
0.35
0.3
GPT-4-vision-oe preview
0.25
50
Claude 3.5 Sonnet
Clande 3 Opus
40
Sigf.IP
Figure 4: Error patterns are different for neural networks and time-limited humans. a) Comparison of human and network accuracy for the rotated-correct condition (EfficientNel was correct on these rotations) condition (rotations that have failed EfficientNet).. Humans show consistent accuracy between rotated-correct and eetated-incorree, conditions. In contrast, networks, including the seven VLMs (represented by diamonds), exhibit a performance drop. However, for Gemini 1.5 Flash, Gemini 1.5 Pro and Claude Opus, the decrease in performance between the two conditions is not as notable. b) Frror consistency analysis performed on the 5 neural networks (not including VLMs) and 40 ms time-limited human subjects. 12 human subjects were partitioned into 4 groups of 3, so that every group saw each rotated image exactly once. The darker red cluster for networks (dark red = highly consistent errors) indicates that they have similar patterns of error, which are not shared by human subjects, highlighting that EfficientNet crrors transfer better to other networks than to humans. c) Mean error consistencies were calculated by comparing networks with each other and with human subjects (i.e., average computed over the different matrix clusters). A two-tailed unpaired t-test confirms that networks indeed make more consistent errors with each other than with humans (t(28) 4.0, p 3.8c-04).
vs.
Under limited viewing time, humans are faulty on unusual poses, like deep networks. In the unlimited viewing time condition, observers used on average 2 seconds before taking a decision. We next studied the performance of human observers when the viewing time was limited to 40 ms and 150 ms followed by a dynamic white noise chromatic mask (Figure 3) (see Procedure 2.2.3 for the justification for these duration choices). With the upright poses, the accuracy of human observers decreased quite modestly (to 90.2% + 4.8%) when viewing time was 40 ms. The unlimited vs. 40 ms difference is statistically significant (t(22) 3.19, p 0.0042). The performance for 40 ms rotated images (69.8% + 4.3%) drastically decreased compared to 40 ms upright images, with drop of 20% (t(11) 8.5, p 3.7e-06). Additionally, the performance on 40 ms rotated images notably decrcased compared to unlimited viewing time (98.9% + 1.0%))t(22) = 14.3, 1.2e-12). At 150 ms exposure lime, performance was barely allected when the objects were presented upright (98.0% 1.8% for the 150 ms viewing condition compared to 98.0% + 2.4% for the unlimited viewing time condition). Subjects performed substantially better at recognizing objects unusual poses when given 150 ms (88.4% = 4.0%) compared to 40 ms (69.8% + (.32) 6.9, p = 6.7c-07), but not as well as with unlimited viewing time (98.9% + 1.0%)(t(11)) 7.0, p 2.3e-05). We conclude that the process needed to
8

---

Published in Transactions on Machine Learning Research (01/2025)
Networks 3/5
Networks
Networks
Netwurks 5/5
Netwurks Humans 1/4
Networks 3/5
Humans 1/4
Humans 1H
Electric fan Microphune
Tennis shoe Honeycumb
Umbrella Kimuno
Jellyfih Lampshade
Jellytish Lampshade
Fire engine Trailortruck
Networks 0/5
Networks 0/5
Networks 0/5
d Networkss
Networks 1/5
Networkss Humans (14
Humans
Humans
Humans
Forklift Mountain bike Unicycle
Stretcher Matchstick
Racket Ballpolut
Hotcher Carpenta'ss
Bow Be Hammerhoou
Figure 5: Examples of objects in unusual poses, where deep networks for pure vision and 40 ms lime-limited humans made similar and differing errors. Above each image is quantitative score of correct answers and below each image are the given answer choices, correct answer being the first. a) Images where all networks and humans correctly labeled the object. b) Images where all networks failed to correctly label the objects, but where humans mostly chose the correct answer. c) Images, where all the networks correctly labeled the objects, but where most humans failed. d) Imagcs, where both networks and humans were mostly not able to correctly label the object.
recognize objects in unusual poses in the brain is impaired when interrupted by mask after 40 ms viewing time, and starts taking place in a time frame of approximately 150 mss
Patterns of errors differ between humans and deep networks. We next subdivided the unusual poses in two sets, the poses that EfficientNet predicted correctly (rotated-correct) vs. incorrectly (rotateddinccoreett) and analysed the errors of humans and networks on these sets (Figure 4a). We found that errors made by EfficientNet transfer well the 5 networks tested (80.0% 7.4 accuracy on rotated-corred condition vs. 54.2 condition, significant difference: t(4) 6.99, p 0.0022) despite the diversity
% + 5.3 rotated-incoree of archilectures and training procedures but not to humans (difference in average performance between votated-corred and rotated-incorrect non-significant). To investigate further whether networks and humans make similar errors (i.c., whether they classify incorrectly the same images), we computed error consistency (defined in Statistical tests 2.4 across networks, across humans, and between networks and humans. When computing the error consistency across networks and between humans and networks, we found that networks do similar mistakes each other, but that human errors are on average not consistent with network errors (Figure Ab and c). Moreover, there is greater diversity in the patterns of errors made by humans than in the pattern of errors made by networks. Overall, this suggests that the behavior of time-limited humans is different that the one of networks.
We then sought to understand what qualitatively explained the differences in the error patterns between time-limited humans (40 ms) and deep networks. We found that-ere even under acute time limitation-human rely on different strategies than networks for recognition, relying more on object structure than their network counterparts (which rely on object details). In Figure 5 show examples where both networks and humans performed well. In these examples, both the structure and details of the objects are well informative about the identity of the object. In Figure 5p, we show examples of errors that networks made but humans typically didn't. A striking example is the image of stretcher, which networks mislaboled as a matchstick. We speculate that the similarity between the strelcher's legs and matchsticks the cause for this misclassification. Despite the resemblance to matchsticks, humans however correctly identified the object because the overall structure of the object rules out the possibility of matchsticks. Thus, we can reason that in that case, humans tended perceive the overall structure rather than fixating on minute details, unlike worksrk which based their prediction such details. In Figure 5 show examples where identification of

---

Published in Transactions on Machine Learning Research (01/2025)
object is challenging for humans but not for networks. For instance, images of jellyfish posed difficulties for human observers. The short of the exposure time, 40 ms, could potentially limit humans to recognising only the overall shape and color of the object, compatible with both a lampshade and a jellyfish. In contrast, networks are able to process fine details, such as the patterns on the jellyfish's umbrella, allowing for accurate identification regardless of pose. Figure D presents examples that prove to be challenging for both humans and networks. It can be noted that in all these examples, both the structure and details contain only limited information about the object identity. The impairment of both networks and humans on these images is thus consistent with our hypothesis that humans rely structure and networks on details for recognition. These qualitative observations reinforce our conclusion that time-limited humans are not well-modelled by feed-forward networks. More examples of time-limited human failures are shown in App. B
Next, we tested seven large vision-language models (VLMs): GPT-4o, GPT---visson--reeviee, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Flash, Gemini 1.5 Pro and SigLIP. With upright poses, all 6 models performed well (Gemini 1.5 Flash, GPT-4o and SPT--4-vision-previee achieving 100% accuracy, Gemini 1.5 Pro and SigLIP 98.0%, Claude 3.5 Sonnet 94.1% and Claude 3 Opus 92.2.%). However, there were significant differences in performance when recognizing rotated poses (Figure 3). Claude 3 Opus performed below the 50% chance level, achieving only 45.8% accuracy. Claude 3.5 Sonnet's 's accuracy on par with pure vision networks with 62.5% accuracy. GPT-4o had an accuracy of 84.8%, SigLIP 84.0% and -.P----vsioo-preeview 79.7%, performing substantially better than our best performing pure vision model, SWAG (70.1%). Remarkably, Gemini 1.5 Pro achieved an accuracy of 96.6% and Gemini 1.5 Flash an accuracy of 97.7.%, performances on par with unlimited humans (98.9%). Separating the rotated condition into two categories rotated-correct (images correctly labelled by EfficientNet) and rotated-incorrect (images incorrectly labelled by EfficientNet)-revellll conditions, similarly to the five pure vision networks (Figure 4p). Performance differences of GPT-4o (18%), GPT-4vvision- (17%) and Claude 3.5 Sonnet (32%) were the most similar to the 5 pure vision networks, whose average performance gap was 26%. For Gemini models and Claude 3 Opus, the performance differences were much smaller: 4.7% for Gemini 1.5 Flash, 2.7% for Gemini 1.5 Pro and 2.4% for Claude 3 Opus. For both unlimited and time-limited humans, the performance difference was not significant between the two conditions. This suggests that errors from EfficentNet transfer better to most VLMs than to humans.
that the 7 VLMs exhibited performance drop between these
4 Discussion
Our results demonstrate that the human visual system is more robust than deep networks for vision and than most vision-language models at recognizing objects in unusual poses. Indeed, with unlimited viewing time, human performance is not affected by unusual poses, whereas deep networks are fooled by them. Only in the very limited viewing time condition (40 ms) does the performance of humans deteriorate to the level of deep networks.
4.1
The gap still exists
Our finding is in contrast to a recent study /eeirho et al. 2021 which showed that deep networks are closing the performance gap with humans on many out-of-distribution visual tasks. Our study differs from this recent study in two major ways: (I) We are considering a transformation (object pose) which affects the global structure of the image, unlike the distortions studied in (Geirhos et al. 2021 which mostly affected the local texture of images (e.g., blur, color modifications, stylization); (2) Geirhos et al. (2021) compare network performance to time-limited humans only. The human subjects in their study were given 200 ms to view the image, followed by mask. We find that viewing time does affect the robustness of recognition in crucial way. When time-limited to 40 ms or 150 ms (followed by mask), the performance of humans substantially degrades at recognizing objects in unusual poses.
Regarding the deep networks, it is notable that the architecture (e.g., visual transformers vs. convolutional neural networks), loss function (e.g., self-supervised vs. supervised), training set size (e.g.. 3.6 billion images for SWAG) and modality (pure images, or images and text) did not affect their performance on the task (with the exception of Gemini, see discussion below). This observation adds a litcrature showing that scaling alone seems insufficient. to match human internal representations and performance in visual tasks
10

---

Published in Transactions on Machine Learning Research (01/2025)
(e.g., (Shankar al. 2020 Fel et al. 2022 Muttenthaler et al. 2022 Linsley et al. 2023a bj). We also note
that works do not benefit from additional time, as their processing time is fixed by the nature of their ferw-forward architecture.
4.2 Main limitation of the current study
A critical extension of this study would be to systematically explore how viewpoint affects the performance of humans and networks. In Abbas & Deny (2023), the effects of different rotation axes (image-plane rotation vs. 3D rotation) and rotation range are investigated in detail for networks. is found that networks that have been trained with image rotation as a data augmentation scheme perform well on image-plane rotations, but are still impaired on 3D out-of-plane rotations. It also found that networks perform at their worst on objects rotated 90º from their canonical pose, regardless of rotation axis. It would be interesting to study whether time-limited humans have the same different failure modes, this would shed light on their respective mechanisms.
While our current study doesn't 't allow systematic exploration of these factors, we can still comment on some of the typical failure modes dime-limited humans. It appears that many challenging images are such that the major axis of the object is occluded due to out-of-plane rotation, causing one end of the major axis to point towards the observer (Marr Nishihara 1978). Examples include the corkscrew and the jellyfish in the rotated correct image (Figure as well the violin and the crib in the rotated incorrect image (Figure S.. In other cases, it seems that occlusion of signature detail of an object, such the watch face a lawnmower motor, also hinders human recognition performance. This also happens most frequently due to out-of-plane rotations. It is important to notice, however, that there are images that are challenging despite only involving in-plane rotation, such as the jellyfish and the parachute in the rotated incorrect image. Thus, a challenging rotation condition is often quite object specific.
For human participants, it is difficult to obtain quantitative results of recognition as function of rotation for real objects, each individual object cannot be presented to each observer in different rotation conditions (as there is heavy transfer of recognition across rotations). Different objects, the other hand, will be differently alfected by the same rotations. Artificial objects may be beller stimuli for the quantitative characterization ion of the effect of rotation on human and network recognition ability, as have recently proposed (Bonnen et al., 2024..
4.3 Why do humans need the extra time?
Our results are in line with a tradition of psychology studies (Shepard & Metzler 1971 Jolicoeur 1985
Edelman & Bülthoff 1992: Kosslyn et al. 1994 Sofer et al., 2015 Jeurissen et al. 2016 Kallmayer et al., 2023
Mayo et al. 2024) showing that humans need more time to accomplish more challenging visual tasks. Closest. to our work, Jolicoeur (1985) showed that the naming time of sketches of everyday objects was proportional to the difference in orientation with their upright pose. Below, we list some of the possible mental processes that could be responsible for this extra time:
1) Evidence accumulation: The increased time could be needed for evidence accumulation. For example, it is known that different types of information about a visual stimulus (c.g., coarse vs. fine features, low contrast vs. high contrast features) propagate to cortical visual areas with different timings (Van Rullen & Thorpo 2001 VanRullen & Thorpe 2002). It might thus be necessary to see the stimulus for a certain time for all relevant information to arrive cortex. Additionally, stimulus information always involves certain amount of noise and uncertainty, which can be filtered with increased viewing time (Olds & Engel 1998
Perrett et al. 1998
2) Recurrence within the visual system: The visual system is known to be highly recurrent (Feleeman
& Van Essen 1991. Suzuki et al. (202).. Previous evidence suggests that visual information is mainly processed in a feed- forward way during the first 80 ms after exposure (Liu et al. 2009 Wyatte et al. 2014 Cichy et al. 2016 Mohsenzadeh et al. 2018), after which recurrent processes start. taking place within the
11

---

Published in Transactions on Machine Learning Research (01/2025)
visual system (Liu et al. 2009 Thorpe 2009
Wyatte et al. 2014 Cichy et al. 2016 Kar & DiCarlo
2021
e-g., from extrastriate areas to V1. Moreover, other studies have suggested that backward masks, such as
the ones used in this study, disrupt these recurrent processes (Lamme & Roelfserna
2000
Lamme et al.,
2002; Breitmeyer & Ogmen 2006 Fahrenfort ct al. 2007 Macknik & Martinez-Corie
2007 The observed
drop in human performance from 150 ms to 40 ms viewing time could thus be explained by these recurrent
processes taking place in the 150 ms condition but being interrupted by backward masking in the 40 ms
condition. From a computational standpoint, there is mounting evidence that these recurrent circuits provide important functions such as grouping and filling-in of features, and are important for object identification in noisy real-world situations, and particularly when objects are partially occluded (Wyatte et al. 2014). It. is an interesting possibility that incorporating such recurrence in deep networks could bring human-like robustness to recognition of objects in unusual poses. Some efforts to incorporate recurrence in deep learning models of the visual system exist (Wyatte et al. 2012; O'Reilly et al.
2013 Spoerer et al.
2017
Tang et al,
2018 Kictzmann et al., 2019 Rajaci ct al. 2019 Nayebi et al. 2022:
Goctschalckx et al. 2023 but to our
knowledye ledge not in the context of recognizing objects in unusual poses.
We note that evidence accumulation and recurrence are not mutually exclusive and, in fact, recurrent circuits are one possible neural implementation to allow evidence accumulation.
3) Recurrence between the visual system and other systems: The neural substrate is also highly recurrent between the visual system and other brain regions (Felleman & Van Essen 1991), such as the perirhinal cortex in the medial temporal lobe (Bonnen et al. 2023) and many frontoparictal areas, such as the prefrontal cortex (PFC) and the Frontal Eye Fields (FFF)- Feedback processes originating in frontoparietal arcas through reciprocal connections to striate cortex provide attentional support to salient or behaviorally- relevant features (Wyalte et al.] 2014). FEF, in turn, play a crucial role in the initiation and execution of saccadic eye movements. The additional processing time could be needed to perform saccades across the object to encode the objects' subparts with successive fixations. Medial temporal cortex (MTC) has been recently shown to support object perception by integrating over multiple glances or saccades (Bonnen et al.) 2023). However, these recurrent interactions between the visual systems and other areas are thought to take place at least 150 ms after the first glance (Wyatte et al. 2014 Bonnen et al.) 2023). They can only partly explain the performance for unusual poses observe, since we already see great improvement from 40 ms to 150 ms, before these interactions could take place.
4.4 Time-limited humans are not feed-forward deep networks
Both sime-limited humans and networks have similar error rates at recognizing objects in unusual poses. This result could suggest that time-limited humans are well-modelled by feod-forward deep networks, as proposod
by (Yamins et al. 2014 Rajalingaam et al., 2018 Kar et al. 2019 Serre 2019 Dapello el al. 2020 Kar &
DiCarlo 2021: Dapello et al. 2022 Muttenthaler et al., 2022 Bonnen et al. 2023] Dehghani et al. 2023).
Here, our analysis of the error patterns of time-limited humans vs. networks suggests otherwise. While the performance of networks and time-limited humans is similar on unusual poses, there is little consistency in their patterns of error. Networks share consistent. patterns of error with each other, which are inconsistent. with the patterns of error of humans. In addition, humans are overall more unpredictable in their error patterns than networks. Moreover, a qualitative analysis of the problematic images for networks vs. humans suggests that networks make errors when they overlook the overall structure of the object and rely instead on misleading details or textures. Conversely, time-limited humans tend to make errors when the overall structure of the object is not clearly disambiguating the object identity, and where one should instead rely details of the object and textures to make the correct decision. This discrepancy is in line with previous
studies (Geirhos et al. 2018 Rajalingham et al. 2018 Geirhos et al. 2021 Wichmann & Geirhos 2023 which have shown that deep networks do not make the same mistakes as lime-limited humans. In particular, Geirhos et al. (2018) showed that deep networks are prone to a bias towards texture, whereas humans are prone to a bias towards shape of objects in their perceptual decisions. Our results suggest that this discrepancy persists under extreme time limitation, and that food-forward deep networks are not adequate models of limittd humans. However, to fully disentangle the effect of texture vs. structure, one would need to design an experiment where the texture of 3D objects is manipulated independently from their structure.
12

---

Published in Transactions on Machine Learning Research (01/2025)
4.5 On the role of retrospective thinking in humans and vision-language models
In the current study, humans were presented with an image followed by (wwealtenaativeeooreddchii question. This provides humans with an opportunity to utilize retrospective thinking. For instance, subject who initially failed to identify an image of rotated stretcher, could reconsider their interpretation after seeing the answer alternatives 'stretcher' and 'matchstick'. Deep networks for pure vision, on the other hand, are not capable of such retrospective thinking: they classify the object based on the predefined 1000 classes without. the possibility of a retrospective thought process when given the two possible answers. Conversely, large vision-language models are given the opportunity to ponder the content of the image after being fed the two labels, like humans. Their performance is notably better than deep networks for pure vision (except Claude), and even matching human performance in the case of Gemini. The impressive performance of vision-language models suggests that they might be able to exploit similar mechanisms for image recognition as humans, which could for instance involve retro-fitting the labels to the image. However, an alternative explanation is that they could have been fed a different data diet involving more images, or more rotated objects than their pure vision counterparts. Indeed, Abbas & Deny (2023) found that increasing the training datasct size generally resulted in better performance in rotated object recognition. The exceptional performance of Gemini could even be due data contamination issues, since the objects selected were available online. Because of the limited documentation available for these models, it is difficult to come to a firm conclusion regarding the mechanisms underlying their abilities. Finally, the current study suggests that the human visual system improves its accuracy by using more time, possibly by means of recurrent processing. Tt is a highly promising hypothesis that Al systems for vision may benefit from such recurrent processes, we can understand their principle in the brain.
Acknowledgements
This work has been supported by an Aalto Brain Center (ABC) grant to N.O., and an Academy of Finland (AoF) grant to S.D. under the AoF Project: 3357590.
References
Amro Abbas and Stéphane Deny. Progress and limitations of deep networks to recognize objects in unusual poses. AAAL, 2023.
Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the CEEE/CVF Conference on Computer Vision and Pattern Recognation, 2019.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08255 2021.
Tyler Bonnen, Authony D. Wagner, and Daniel LK. Yamins. Medial temporal cortex supports compositional visual inferences. bioRriv, 2023.
Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O' O'Connell, Yoni Friedman, Nancy Kanwisher, Joshua B Tenenbaum, and Alexei A Efros. Evaluating multiview object consistency in humans and image models. NewrIPS 2024, 2024.
Bruno Breitmeyer and Haluk Ogmen. Visual masking: Time slices through conscious and unconscious vision. 2006.
Radoslaw Martin Cichy, Dimitrios Pantazis, and Aude Oliva. Similarity-based fusion of meg and fmri reveals spatio-temporal dynamics in human cortex during visual object recognition. Cerebral Cortez, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition workshops, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J. DiCarlo. Simmlating a primary visual cortex at the front of cmns improves robustness to image perturbations. Advances in Neural Information Processing Systems, 2020.
Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Geary, Michael Ferguson, David D. Cox, and James J. DiCarlo. Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robustness. bioRriv, 2022.
Mostafa Dohghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Tbrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Ma- hendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohna Zhai, Danicl Keysers, Jeremiah Harmson, and Neil Houlsby. Scaling vision translormers to 22 billion parameters. Proceedings of the 40th International Conference on Machine Learning, 2023.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint 1rXi:1811.04886, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Xiv preprinl 0rXiv:2010.11920 2020.
Shimon Edelman and Heinrich H Bülthoff. Orientation dependence in the recognition of familiar and novel views of dinensdimensional objects. Vision research, 1992.
Johannes J. Fahrenfort, H. Steven Scholte, and Victor AF. Lamme. Masking disrupts reentrant processing in human visual cortex. Journal of cognitive neuroscience, 2007.
Thomas Fel, Ivan F Rodriguez Rodriguez, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems, 2022.
Daniel J. Felleman and David C. Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1991.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, Xiv preprint arXie:1811.12233 2018.
Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by-trial bohaviour of cnns and humans by mcasuring error consistency. Advances in Neural Information Processing Systems, 2020.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thicringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. arXiv preprint arXiv:2106.09 2021.
Lore Goetschalckx, Lakshmi Narasimhan Govindarajan, Alekh Karkada Ashok, Aarit Ahuja, David L. Sheinberg, and Thomas Serre. Computing a human-like reaction time metric from stable recurrent vision models. arXiv:2306. 11582, 2023.
David Marvin Green, John A. Swets, et al. Signal detection theory and psychophysics. 1966.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision (TCCV), 2015.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Mark Ibrahim, Quentin Garrido, Ari Morcos, and Diane Bouchacourt. The robustness limits of sota vision models to natural variation. arviv preprinl or arXi::2200 1306044 2022.
Danique Jeurissen, Matthew W. Self, and Pieter R. Roelfsema. Serial grouping of 2d-image regions with object-basod attention in humans. eLife, 2016.
Pierre Jolicoeur. The time to name disoriented natural objects. Memory & Cognition, 1985.
Aylin Kallmayer, Melissa L-H Vo, and Dejan Draschkow. Viewpoint dependence and scene context effects generalize depth rotated three-dimensional objects. Journal of Vision, 2023.
Kohitij Kar and James J. DiCarlo. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition. Neuron, 2021.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's ' execution of core object recognition behavior. Nature Neuroscience, 2019.
Tim C. Kietzmann, Courtney J Spocrer, Lynn KA Sörensen, Radoslaw M Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 2019.
Stephen M Kosslyn, Nathaniel M Alpert, William Thompson, Christopher F Chabris, Scott L Rauch, and Adam K Anderson. Identifying objects seen from different viewpoints a pet investigation. Brain, 1994.
V.A. Lamme and P.R. Roelfsemn. The distinct modes of vision offered by feedforward and recurrent processing. Trends in neurosciences, 2000.
Victor AF Lamme, Karl Zipser, and Henk Spekreijse. Masking interrupts soure-ground signals in vl. Journal of cognitive neuroscience, 2002.
Drew Linsley, Pinyuan Feng, Thibaut Boissin, Alekh Karkada Ashok, Thomas Fel, Stephanie Olaiya, and Thomas Serre. Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance human perception. arXiv preprint arXiv:2306. 000n23 2023a.
Drew Linsley, Ivan F Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, and Thomas Serre. Performaace-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex. arXiv preprint arXiv:2306.003: 03779, 2023b.
Hesheng Liu, Yigal Agam, Joseph R. Madsen, and Gabriel Kreiman. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron, 2009.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet. for the 2020s. arXiv preprint urXin:2201.03545 2022.
Stephen L. Macknik and Susana Martinez-Conde. The role of feedback in visual masking and visual processing. Advances in cognitive psychology, 2007.
Spandan Madan, Tomotake Sasaki, Tzu-Mao Li, Xavier Boix, and Hanspeter Pfister. Small in-distribution changes in 3d perspective and lighting fool both cnns and transformers. arXiv preprint arXiv:2106. 161988 2021a.
Spandan Madan, Tomotake Sasaki, Hanspeter Pfister, Tzu-Mao Li, and Xavier Boix. Adversarial examples within the training distribution: A widespread challenge. arxiv preprint 2XX::2106.16196 2021b.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how convolutional neural networks generalize to out-of-distribution category viewpoint combinations. Nature Machine Intelligence, 2022.
category-viewpoint
David Marr and Herbert Keith Nishthara. Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Sociely of London. Series B. Biological Sciences, 1978.
David Mayo, Jesse Cummings, Xinyu Lin, Dan Gutfround, Boris Katz, and Andrei Barbu. How hard are computer vision datasets? calibrating dataset difficulty to viewing time. Advances in Neural Information Processing Systems, 2024.
Yalda Mohsenzadeh, Sheng Qin, Radoslaw M. Cichy, and Dimitrios Pantazis. Ultra-rapid serial visual presentation reveals dynamics of feedforward and feedback processes in the ventral visual pathway. Blife, 2018.
Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, and Simon Korublith. Human alignment of neural network representations. arXiv preprint arXiv:2211.00 0.11001, 2022.
Aran Nayebi, Javier Sagastuy-Brena, Daniel M. Bear, Kohitij Kar, Jonas Kubilius, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. K. Yamins. Recurrent connections in the primate ventral visual stream mediate a trade-off between task performance and network size during core object recognition. Neural Computation, 2022.
Elizabeth S. Olds and Stephen A. Engel. Linearity across spatial frequency in object recognition. Vision Research, 1998.
Randall C. O'Reill, Dean Wyatte, Seth Herd, Brian Mingus, and David J. Jilk. Recurrent processing during object recognition. Frontiers in psychology, 2013.
D.I. Perrett, M.W. Oram, and E. Ashbridge. Evidence accumulation in cell populations responsive to faces: account of generalisation of recognition without mental transformations. Cogmitionn 1998.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Karim Rajaci, Yalda Mohsenzadeh, Reza Ebrahimpour, and Seyed-Mahdi Khaligh- Beyond core object recognition: Recurrent processes account for object recognition under occlusion. PLoS Comput Biol, 2019.
Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. Journal of Neuroscience, 2018.
Thomas Serre. Deep learning: the good, the bad, and the ugly. Annual review of vision science, 2019.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang. Benjamin Recht, and Ludwig Schmidt. Evalu- ating machine accuracy on imagenet. In International Conference Machine Learning, 2020.
Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. American Assn for the Advancement of Science, 1971.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the FEEE/CVE Conference on Computer Vision and Pallern Recognition, 2022.
Imri Sofer, Sébastien M. Crouzet, and Thomas Serre. Explaining the timing of natural scene understanding with a computational model of perceptual categorivation. PLoS Comput Biol, 2015.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Courtney J. Spocrer, Patrick McClure, and Nikolaus Kriegeskorte. Recurrent convolutional neural networks: better model of biological object recognition. Fronlãers in psychology, 2017.
Mototaka Suzuki, Cyriel MA Pennartz, and Jaan Aru. How deep is the brain? the shallow brain hypothesis. Nature Reviews Neuroscience, 2023.
Hanlin Tang, Martin Schrimpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 2018.
Simon J. Thorpe. The speed of categorization in the human visual system. Newron, 2009.
Rufin Van Rullen and Simon Thorpe. Rate coding versus temporal order coding: what the retinal ganglion cells tell the visual cortex. Newral computation, 2001.
Rufin VanRullen and Simon J. Thorpe. Surfing spike wave down the ventral stream. Vision research, 2002.
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Pridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. Advonces in Neural Information Processing Systems, 2022.
Felix A. Wichmann and Robert Geirhos. Are deep neural networks adequate behavioral models of human visual perception? Annual Review of Vision Science, 2023.
Dean Wyatte, Tim Curran, and Randall O'Reilly. The limits of leedforward vision: Recurrent processing promotes robust object recognition when objects are degraded. Journal of Cognétive Neuroscience, 2012.
Dean Wyatte, David J. Jilk, and Randall C. O'Reilly. Early recurrent feedback facilitates visual object recognition under challenging conditions. Frontiers in psychology, 2014.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Daniel LK Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimmized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 2014.
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial attacks beyond the image space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the TEEE/CVF International Conference on Computer Vision, 2023.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Appendix
All code and data is available at /ttttp/s//g.hhh...oo////laaaaaaa _poses
A Testing Large Vision-Language Models
We tested 6 large vision-languags models (VLMs), Gemini 1.5 Flash, Gemini 1.5 Pro, GPT-4o, GPT-4-vision- preview, Claude 3.5 Sonnet and Claude 3 Opus, via their API. Figure shows an example of the used prompt and models' answers.
Swapping answers A and B did not change the accuracy the VLMs. Indeed, we ran all VLM analyses also with switched choices. That did not have any effect.
Changing the prompt did affect the results, but carcfully selected the prompt that worked well for all VLMs.
For example, we also tried the following prompt with the stated-incorreed images:
The image represents either flabell] or flabel2). Which is it? Answer in one or two words.
For GPT-4o, this prompt led to 78% correct answers, whereas our original prompt had 76% correct answers. However, for Gemini, this prompt led 91% correct answers, against 95% for our original prompt. This illustrates the variability that comes with changing the prompt.
a
C
e
What's e 11 this teage?
what's is in this Inage?
What's in this image?
paractute
A. parachute
a. parachute
5, svdrg
8. salag
B. selng
Choose wither atauer are the words.
anower words.
aither antuer too words.
OPT-40: parechate
Cloude 1:5 Semnet: oiting
oesdnt 1:s Maath: R, parachute
b
d
f
What's image?
aaat's in image?
parechite
perectate
talng
8. saing
Cheese ecther er and vonds.
repures resperse to ventare valie Por bel wow caick contttats..reey.yrttt "ottng" decentire
07--4-isson- previsss B. Sydna
Claide Obse: A. sarachute
Figure 6: Examples of a prompt and models' answers in API mode. a) GPT-4o, b) GPT-4-vision- preview, c) Claude 3.5 Sonnet, d) Claude 3 Opus, e) Gemini 1.5 Flash and f) Gemini 1.5 Pro (example of safety-blocked instance, which was disregarded in our analysis).
B More examples of time-limited (40 ms) human failures
Figure and Figure X show more examples of images that were typically challenging for time-limited humans, respectively in the rotated--orreet. and rotated-incorrect conditions.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Correct
25
Figure 7: Examples of images where time-limited humans were mostly incorrect, for the rotated-
correct condition. First row: humans were <50% correct; Second row: 25%%, Third row: 0%
% Correct
25
Figure 8: Examples of images where time-limited humans were mostly incorrect, for the rotated-
incorrect condition. First row: humans were <50% correct; Second row: <25%, Third row: 0%
C Deep Networks (pure vision) details
C.1 Dataset and Training Objectives
We used different models in this experiment: Noisy Student EfficientNNet-2,, SWAG-RegNetY, VIT-L16, BEiT-L/16, ConvNeXI-XL, and SWIN-L. The models chosen for their demonstrated performance at recognizing objects in unusual poses in a prior study conducted by Abbas & Deny (023).. The models were trained on datasots of varying sizes ranging from 1 million (ImageNet) to 3.6 billion images (TG-3.6B). These models had a variety of architectures (e.g., convolutional architecture, visual transformer) and were trained under a variety of training objectives:
Supervised learning: Visual Transformers architectures (Liu et al. 2022
Dosovitskiy et al.
2020 Liu et al.
2021 and convolutional
Self-supere self-supervised learning: BETT Bao et al. 2021
Semi-supervised learning: Noisy Student Xie el al. 2020
Semi-weakly supervised learning: SWAG Singh et al.
2022
19

---

Published in Transactions on Machine Learning Research (01/2025)
C.2 Model Descriptions
This section describes the networks used in this study in detail.
Standard Vision Transformer: Standard Vision Transformer ViT-L/16 (Dosovitskly pretrained on ImageNet(1M)) with imput size 224x224.
2020 which
BEIT: BFIT-L//16 is a Vision Transformer trained using self-supervised training method introduced with the model [Bao et al. 2021). The method is known as Masked Image Modeling (MIM) and it inspired by Masked Language Modeling (MLM) (Devlin et al.) 2018). The model is pretrained with the self-supervisee MIM task on ImageNet2lk(14M) and fine-tuned on ImageNet. Even with smaller pretraining datasets, BEiT outperforms standard Visual Transformers.
ConvNext: ConvNeXt-XLL <Luu et al. 2022 is pure convolutional architecture which layers are designed carefully by choosing an optimal collection of architecture hyperparameters. The model undergocs training methods established for Visual Transformers, and they are known to enhance ConvNext's performance as well. The model is trained on ImageNet2II((4MM),
SWIN Transformer: SWIN-L Transformer (Liu et al. 2021 is hierarchical Vision Transformer: The self-attention layer of the original Vision Transformer is replaced by Shifted Window Self-attention layer (SWSA). The SWSA layer divides the image into windows, calculating self-attention within each window. This process significantly reduces the computational time of the attention layer, making it linear with the input size instead of quadratic. Additionally, enhances performance compared to the original Visual Transformers. The model is trained on ImageNet2IK(14M),
Noisy Student: Noisy Student EfficientNet-L.22 is pretrained using the Noisy Student paradigm { et al. 2020). The model is trained on ImageNet dataset and on the noisily labeled JFT-300M dataset. It has an input size of 475x475. The model is pretrained with the data augmentation method RandAugment (Cubuk et al. 2020), which among other distortions produces rotated images of the training set.
SWAG: SWAG-RegNetY (Singh al. 2022 is pretrained using semi-weakly supervised pretraining procedure. The pretraining dataset consisted of more than 3.6 billion Instagram images, which were labeled with hashtags of 27K classes.
C.3 Model Sources
The checkpoints for all the models we use are in PyTorch format. The models are from three different main Pytorch Image Model library (timm) (ViT-L16, ConvNext-XL, SWIN-L), Hugging Face Transformers library (BEiT-L16), and Torch Hub (EfN--2--S,, SWAG-RogNetY).
sources:
D
The 3D objects
Below are the 3D objects used in experiments (Figures 9
10
12
13
14
15
16
17
18
19
20
21
25
26
27
28
29
30
31
32
33
30
36
37
39
40
45
50
51
52
53
54
55
56
57
58
and
59
They are listed with the links to Sketchfab. Images each object
are also shown with the corresponding wrong labels (first upright, second rotated and correctly labelled by EfficientNee, and third rotated and incorrectly labelled by EfficientNet).
Acoustic guitar:
link
20

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 9: 1. Banjo, 2. Plectrum, 3. Whistle
Airliner: link
Figure 10: 1. Warplane, 2. Space shuttle, 3. Missile
Backpack:
link
Figure 11: 1. Sleeping bag, 2. Neck brace, 3. Neck brace
Binoculars: link
Figure 12: 1. Reflex camera, 2. Tripod
Bow tie: object deleted
Figure 13: 1. Windsor tie, 2. Band Aid, 3, Hammerhead shark
21

---

Published in Transactions on Machine Learning Research (01/2025)
Cannon:
link
Figure 14: 1. Rifle, 2. Hourglass, 3. Guillotine
Canoe:
link
LE7
Figure 15: 1. Speedboat, 2. Hair slide, 3. Toaster
Combination lock: link
RBE
Figure 16: 1. Safe, 2. Hook, 3. Airship
Corkscrew: link
Figure 17: 1. Can opener, 2. Syringe, 3. Unicycle
Crib: link
22

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 18: 1. Studio couch, 2. Plate rack, 3. Crate
Dial telephone:
link
Figure 19: 1. Smoothing iron, 2. Hook, 3. Joystick
Digital watch:
link
Figure 20: 1. Magnetic compass, 2. Buckle, 3. Chain
Dumbbell: link
Figure 21: 1. Honeycomb, 2. Nail, 3. Honeycomb
Electric fan link
Figure 22: 1. Spotlight, 2. Microphone, 3. Strainer
23

---

Published in Transactions on Machine Learning Research (01/2025)
Frying Pan:
link
Figure 23: 1. Spatula, 2. Gong, 3. Shield
Fire engine: object deleted
Figure 24: Tow truck, Trailer truck, 3. Chain
Folding chair: link
Figure 25: 1. Rocking chair, 2. Tray, 3. Shovel
Football helmet:
link
Figure 26: 1. Mask, 2. Balloon, 3. Bucket
Forklift: link
24

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 27: Crate, Crane, Lawn mower
Garbage truck: object disabled
Figure 28: 1. Trailer truck, 2. Stretcher, 3. Racer
Gasmask: link
Figure 29: 1. Ski mask, 2. Binoculars, 3. Spotlight
Hair dryer:
link
Figure 30: 1. Microphone, Computer mouse, Microphone
Hammer: object deleted
Figure 31: 1. Hatchet, 2. Can opener, 3. Bow
25

---

Published in Transactions on Machine Learning Research (01/2025)
Hatchet:
link
Figure 32: 1. Hammer, 2.
Cleaver, 3. Carpenter's plane
Jellyfish:
link
Figure 33: Parachute, Lampshade, Lampshade
Ladle: link
Figure 34: 1. Wooden spoon, Maraca, 3. Safety pin
Lawn mower:
link
Figure 35: 1. Harvester, 2. Mousctrap, 3. Projector
Mountain bike: link
26

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 36: Tricycle, Tricycle, 3. Unicycle
Mountain tent: link
Figure 37: 1. Dome, 2. Sleeping bag, 3. Crash helmet
Parachute:
link
Figure 38: 1. Balloon, 2. Lighter, 3. Swing
Park bench: link
Figure 39: 1. Studio couch, 2. Paper towel, 3. Swing
Piggy bank: link
Figure 40: 1. Hog, 2. Teapot
27

---

Published in Transactions on Machine Learning Research (01/2025)
Pitcher:
link
Figure 41: 1. Vase, 2. Conch, 3. Drum
Power drill:
link
Figure 42: 1. Hair dryer, 2. Screwdriver, 3. Mop
Racket:
link
m
Figure 43: 1. Strainer, 2. Ballpoint pen, 3. Mop
Rocking chair: link
Figure 44: 1. Folding chair, 2. Throne, 3. Cradle
Running shoe: object deleted
28

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 45: 1. Clog, Honeycomb
Scooter: link
Figure 46: Moped, 2. Moped, Tricycle
Shopping cart: link
T
Figure 47: 1. Shopping basket, 2. Shopping basket, 3. Shopping basket
Stretcher: link
Figure 48: Crutch, 2. Bobsled, 3. Matchstick
Table lamp: link
Figure 49: 1. Pedestal, 2. Plunger, 3. Toilet seat
29

---

Published in Transactions on Machine Learning Research (01/2025)
Tank:
link
Figure 50: 1. Canmon, 2. Stretcher, 3. Pedestal
Teapot:
link
Figure 51: 1. Water jug, 2. Plunger, 3. Speaker
Toddy bear:
link
Figure 52: 1. Piggy bank, 2. Knot, 3. Piggy bank
Tractor: link
Figure 53: 1. Forklift, 2. Mousetrap, 3. Forklift
Traffic light: link
30

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 54: 1. Pole, 2. Missile, 3. Nail
Trashcan: link
Figure 55: Bucket, 2. Lighter, 3. Revolver
Umbrella: link
Figure 56: 1. Mountain tent, 2. Kimono
Vacuum cleaner: link
Figure 57: 1. Lawn mower, 2. Chainsaw
Violin: link
Figure 58: 1. Banjo, 2. Harp, 3. Spindle
31

---

Published in Transactions on Machine Learning Research (01/2025)
Wheelbarrow: link
Figure 59: Shovel, 2. Shovel
32


Reviews:
[{"review_id": "lRvy41Vm2K", "reviewer": "Reviewer_PEXF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This study provides insights into human visual perception, the limitations of vision and vision-language networks, and the differences between the two (see insights/observations in the summary section). It contributes to understanding vision systems and suggests potential ways to enhance network performance, such as introducing recurrent processes.\n\nThe study also reveals a notable performance gap between vision-only networks and vision-language models like Gemini and GPT-4, with Gemini showing particularly strong accuracy on objects in unusual poses. However, the reasons for this difference remain unclear (partly due to limited documentation on these systems).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study investigates the performance of humans and neural networks in recognizing objects in challenging poses. The authors created a dataset of 147 images from 51 ImageNet categories, featuring objects in upright poses, \"Rotated-Correctly\" poses (rotated objects correctly classified by a pre-trained EfficientNet), and \"Rotated-Incorrectly\" poses (rotated object misclassified by the EfficientNet). Each image was paired with a two-choice question: the correct class and EfficientNet's top incorrect prediction.\n\nKey experiments included tests on vision-only networks (SWAG, ViT, SWIN, BEIT, ConvNext), vision-language networks (Gemini 1.5, GPT-4o, and Claude 3), and humans under three conditions: (1) time-unlimited exposure, (2) 40ms time-limited exposure, and (3) 150ms time-limited exposure. \n\nTheir main observations are: Humans outperformed networks in recognizing objects under unusual poses in the time-unlimited condition. Vision-only networks and most vision-language models performed poorly on rotated objects, except for the Gemini models. When humans were restricted to 40ms exposure, their accuracy dropped to the level of networks. However, error patterns differed between humans and networks: from visual inspection it seems that humans struggled when the object structure was unclear while networks struggled when the object details (texture) were unclear.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- While the study evaluates several vision-only networks, selected based on their performance in a previous related study, it would have been more complete to also include more recent and widely used models. For example, the state-of-the-art self-supervised model DINOv2 [A] and vision-contrastive models like CLIP [B] and its newer variations (EVA-CLIP [C] and SigLIP [D])are among the most popular image encoders today. Adding these models would give a broader view and better reflect the capabilities of current image encoders.  For instance, these models are often tested in recent research on robustness for tasks like semantic segmentation [F, G].\n\n- The analysis would have been richer if the work was also examining how the following factors affect the performance of the networks: a) the size of the pre-training dataset, b) the model size, c) the pre-training objective, and d) the resolution or size of the input images. Although the evaluated networks vary in these aspects, the study does not clearly address whether or how these factors affect performance. Such deeper investigation into these dimensions might have provided more insights on how to address the \"brittleness\" of vision-only networks on recognizing objects under challenging poses. For example, it would be informative to understand whether increasing both model size and pre-training dataset size increase the robustness of the models. \n\n- Gemini's behavior diverges so significantly from the other evaluated models. This raises the question of whether there is a possibility that the dataset used for evaluation in this study might have been included in the training data for such closed-source models. These models often rely on extensive internet-scraped datasets, which makes it hard to know exactly where the data came from. Do the authors consider this a possibility? What I mean is that, while it's nearly impossible to confirm such overlap, it would be helpful to clarify whether the images in this study were available online and could have been included in the training data of these models, which gather images from the internet.\n\n- Building on the previous point, it would have been valuable to include open-source and better-documented vision-language models, such as LLaVA 1.5 [E], in the evaluation. LLaVA 1.5 typically uses a CLIP-like model as its image encoder, and testing it in this context could provide insights into whether adding a language decoder (as LLaVA does) enhances performance on objects in challenging poses. \n\n\n[A] DINOv2: Learning robust visual features without supervision, In TMLR 2024    \n[B] Learning transferable visual models from natural language supervision, In ICML 2021   \n[C] EVA-CLIP: Improved training techniques for clip at scale.    \n[D] Sigmoid loss for language image pre-training, In ICCV 2023.    \n[E] Improved Baselines with Visual Instruction Tuning, In CVPR 2024    \n[F] How to Benchmark Vision Foundation Models for Semantic Segmentation?, In CVPR 2024 workshop.    \n[G] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation, In CVPR 2024", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zesLVqhNlg", "reviewer": "Reviewer_ZRm5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I believe this to be a nice and valuable study. It proposes a new experimental setting w.r.t. to other works that have probed robustness of DNNs to unusual pose changes, especially through the comparison with human subjects under various viewing conditions as well as a broad selection of state-of-the-art vision models. The experimental settings are explained and justified in detail. As described above, they mostly make sense (except for one key aspect discussed in the next section). The results are convincing and presented nicely together with a lengthy discussion that attempts to put them in a broader context with a focus on known differences between human and artificial vision systems.\n\nI don't find the results to be surprising, but that does not detract from its value at all since it is my view that artificial recognition systems (whether \"pure vision\" DNNs or VLMs) are highly overrated when it comes to robust object recognition, and that relatedly papers focusing on the flaws of such systems tend to be under-appreciated. However, in light of this view, I think that one aspect of the experimental design could use some strengthening, namely the choice to uniformly sample from the space of rotations, when in fact different kinds of rotations (e.g. rotations on the ground plane for upright objects vs. other more uncommon rotations) pose very different challenges. See next section for a lengthier discussion of this point as well as further suggestions for improvements.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a valuable comparative study of object recognition performance given challenging viewpoints. The study compares the performance of humans as well as state-of-the-art artificial recognition pipelines. This includes both \"pure vision\" feed-forward recognition networks, as well as large \"vision-language\" models.\n\nThe experimental setup largely makes sense. \n- Object selection: The authors select 51 object categories from the ImageNet classes which fulfill the following criteria: (a) Objects should have a canonical upright pose (this would e.g. include chairs, cars), (b) changes in pose should be reflected in significant appearance changes (this would exclude e.g. various types of balls), and (c) should be recognisable by a reference (\"pure vision\") model (in this case EfficientNet-L2 with large-scale training).\n- Stimulus generation: Given one 3D model per selected category, the authors then apply a random rotation relative to some canonical pose, and then render the model against a uniform background.\n\t\n- Task definition: To determine object recognition performance, a two-forced choice question is put together for each stimulus. The two choices are: (a) the correct label, (b) the highest-scoring incorrect label obtained from the outputs of the aforementioned reference model (EfficientNet-L2). If the reference model correctly identifies the stimulus, this is thus the label with the second-highest softmax probability.\n\nThe authors outline two potential issues with this setup, namely that (a) the incorrect labels might be biased against artificial systems given that they are derived from one such system, and (b) that the stimuli are not realistic enough. They convincingly address these two caveats in my view.\n\nFurthermore, they adapt this setup a little taking into account the different natures of the \"experimental subjects\" under consideration.\n\n- \"Pure vision\" models output one probability per object class from a closed set, and thus the answer to the forced choice question is derived by comparing the probabilities of the two classes in question, even if neither is the model's first choice.\n\t\n- VLMs require some kind of instruction to guide processing, so a fixed prompt template is devised in order to elicit a specific response. This prompt is as follows: \"What’s in this image? A. [label 1] B. [label 2] Choose either A or B and answer in one or two words.\" If the VLM provides an undesirable third answer as they are sometimes wont to do, the answer is considered incorrect -- which is similar to the approach above.\n\t\n- The human visual system is known to recruit recurrent processes over timescales that can vary depending on how challenging a visual task is. Artificial networks typically involve a bounded or even fixed amount of computation. Thus human subjects were presented with stimuli at one of two very short time scales: 40ms and 150ms. In the former case, this amount of time was picked as it likely precludes heavy use of recurrent processing, allowing for better comparisons with standard feed-forward vision networks. For reference, some human subjects were also given unlimited processing time as well.\n\nThe main findings:\n\n- Humans essentially demonstrate no degradation in performance when given unlimited viewing time for recognising objects in unusual poses.\n- The picture starts to change when limited time is available, especially in the 40ms case for which humans slightly underperform relative to the more advanced VLMs.\n- However, a qualitative analysis of the results shows that the mistakes made by humans are of a different nature even if the results are similar quantitatively: Humans tend to make mistakes when shape cues are misleading, and texture cues would be more helpful. This accords with the results of e.g. Geirhos et al. (2018) that show how DNNs and humans appear to rely on different cues for object recognition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concern is the following: I think a critical part of the study which I feel is underexplored is the comparison of mistakes among the different \"participants\" in the study. Is there perhaps a way to more systematically arrive at a characterisation of the different mistakes? Either through a lengthier/more rigorous qualitative analysis, or via a more controlled generation of objects in unusual poses? \n\nFor example with regards to the latter: The authors pick object classes with a canonical \"upright\" pose, but then randomly rotate the objects along either the x, y, and z axis. (For the sake of the discussion, I will assign directions to x, y, and z -- namely: up, side, and forward respectively) This equal treatment of axes I find to be a little problematic. Certain objects that have an upright position, such as chairs, can be rotated arbitrarily on the ground plane (w.r.t. to the up/x axis) without this resulting in a challenging image. In fact, such rotations will most likely occur in a natural image collection.  Similarly, a rotation around the z- or forward axis can perhaps be simulated via first rendering the 3D model to an image, and then applying an image rotation in 2D -- perhaps an augmentation operation that was used for training the more advanced image models. Significant rotations around the y- or side/lateral axis on the other hand (e.g. a chair that has been tipped over) are much less likely to occur in the training set, and thus pose a different challenge.\n\nAside from the choice of rotation axis, also the degree of rotation matters quite a bit, but as far as I can tell the rotation angle was sampled uniformly?\n\nMaking these kinds of distinctions is very challenging especially for many different types of object classes at once, but perhaps for a limited subset of the chosen classes it would have been worth doing some kind of quantitative analysis that attempts to put a number on how difficult a certain rotation is, or maybe to simply just break down the results by axis of rotation. This might shed more light on how humans and artificial networks differ.\n\nGiven that the stimuli are artificially generated, introducing some textural confounders (similar to the aforementioned Geirhos et al. 2018 study) might also give a more nuanced and well-grounded picture.\n\nFurther minor concerns:\n\n- I wonder what kind of experiments the authors performed to make sure results are robust against certain reasonable variations of the setup. For example, why just pick one 3D model per object type? What about the text prompt provided to VLMs? For example, what happens if you conduct the experiment twice with the same VLM, but by consistently switching the choices of A & B, or by varying the text prompt?\n\n- Deriving an answer to the two-forced choice question by simply looking at softmax activations and picking the bigger one largely makes sense, but should there perhaps be a cutoff in terms of how high the probability needs to be before accepting a correct answer? E.g. let's say that if for a certain stimulus, the DNN would assign to the correct label a probability of 0.01, and to the incorrect label a probability of 0.009, and to a third label entirely a probability of 0.9. Would this still count as a correct answer given that 0.01 is larger than 0.009, and the 0.9 probability answer is ignored?", "broader_impact_concerns": "I have no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HJp75YYNeX", "reviewer": "Reviewer_DiFG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The problem of recognizing objects in unusual poses is very interesting and differs from classical problems in out-of-distribution generalization in computer vision. This problem is rarely tackled although this paper highlights the fact that AI models have important limitations. This paper makes a step toward understanding this problem, with a clear and convincing experimental protocol and high quality analysis.\n\n- The list of models evaluated is very diverse and contains most of the popular and best models that the vision community is currently using. ViT is now the most popular vision backbone and there is still a strong interest in CNN with ConvNeXt . For VLMs, the authors evaluated all the best and most popular models such as Gemini, Claude and GPT-4o.\n\n- The paper incorporates strong knowledge in neurosciences. For example, the parameters of the human experiments, 40ms / 150ms display time, are not chosen randomly and are based on other neuroscience studies. Moreover, the paper presents interpretation and connections with various components of the brain involved in perception.\n\n- The paper is very well presented, clear and well written. In particular Figure 3 is very clear and summarizes well the conclusions of the study.\n\n\nWeaknesses:\n\n- One important conclusion is not much discussed in the paper: VLMs exhibit much better performance than vision models. What is your intuition for that ? Do you think language is a key difference ? Maybe it is just due to the details of how to probe these models ? Also, what makes Gemini a much better model than even the other VLMs ? You give hints on the fact that it could be the training data, what is your intuition ? The gap may also come from the prompting technique. Have you tried different prompts for all the VLMs ?\n\n- The biggest weakness of the paper is the lack of practical analysis of the results, and lack of recommendation for building better models. The paper identifies a strong limitation of AI models compared to humans, but does not elaborate on how to fix this limitation, even by giving intuitions or ideas. Now that you have these results, how would you improve existing models ?\n\n- The study is conducted with objects on white backgrounds. In realistic scenarios, some objects appear more frequently on some backgrounds, and this prior knowledge is used to help recognition, probably accelerating the process for humans. For example, if the object is a car, it will probably appear more frequently on roads, and seeing the road for 40ms for humans could be enough to help the recognition. This setting would be more realistic. However I understand the idea behind not being sensitive to these correlations, and I appreciate that the authors discuss this point in the caveat 2 section. Another study with background could be an extension of this work.\n\n\nQuestions:\n\n- For human experiments, you mention “normal vision or corrected-to-normal vision” ? Could you give more details on that ?\n\n- The protocol for comparing error patterns is to look at errors made by EfficientNet and checking whether humans make the same mistakes. Have you considered the reverse process of starting from human errors, and checking AI models ?\n\n\nConclusion:\n\nThis paper presents really cool findings, and highlights a strong limitation of vision models. It shows that VLMs are very good at unusual pose recognition but that there still exists a gap with humans. On the weaker side I would have liked to have more recommendations or ideas on how to close this gap, but overall I think that the study in itself is enough for acceptance at TMLR, and that closing the gap could be part of future work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a study on the robustness of various AI models, as well as humans, on the problem of recognizing objects in unusual poses in images. The authors collected a dataset of images with objects with various rotation angles on white backgrounds. They evaluated various popular purely vision based and vision-language models and performed the same evaluation with humans on various settings where humans can see the image a given amount of time. The paper finally compares performance across models and settings, patterns of errors, strengths and weaknesses and draws several conclusions such as the fact that humans and machines do not make the same mistakes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Incorporate a discussion on how to improve current AI systems to close the gap with humans.", "broader_impact_concerns": "This study involves human experiments and data. A broader impact statement could be incorporated to describe how privacy of the participants was preserved.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I think this paper clearly meets the standards of claims and audience. I'd ask the authors to make sure the limitations section of the final version addresses the points raised by reviewer ZRm5 as needed.
Decision: Accept (poster)