The Robustness Limits of SoTA Vision Models to Natural Variation
Authors: Mark Ibrahim
Quentin Garrido
Ari S. Morcos
Diane Bouchacourt
Venue: Accepted by TMLR ()

Abstract: Recent state-of-the-art vision models have introduced new architectures, learning
paradigms, and larger pretraining data, leading to impressive performance on tasks
such as classification. While previous generations of vision models were shown to
lack robustness to factors such as pose, the extent to which this next generation
of models are more robust remains unclear. To study this question, we develop a
dataset of more than 7 million images with controlled changes in pose, position
background, lighting color, and size. We study not only how robust recent state-of-
the-art models are, but also the extent to which models can generalize to variation in
each of these factors. We consider a catalog of recent vision models, including vision
transformers (ViT), self-supervised models such as masked autoencoders (MAE),
and models trained on larger datasets such as CLIP. We find that even today’s best
models are not robust to common changes in pose, size, and background. When
some samples varied during training, we found models required a significant portion
of instances seen varying to generalize—though eventually robustness did improve.
When variability is only witnessed for some classes however, we found that models
did not generalize to other classes unless the classes were very similar to those seen
varying during training. We hope our work will shed further light on the blind
spots of SoTA models and spur the development of more robust vision models.

Published in Transactions Machine Learning Research (04/2023)
The Robustness Limits of SoTA Vision Models to Nat- ural Variation
Mark Ibrahim
marksibrahin @meta. com
Quentin Garrido
Ari Morcos
Diane Bouchacourt
Fundamental Al Research (FAIR), Meta
Reviewed on OpenReview:
https: wptpe/vee.....eeeeeeopee net/ forum? id= QmHLum3DDY
Abstract
Recent state-of-the-art vision models have introduced new architectures, learning paradigms, and larger pretraining data, leading to impressive performance on tasks such classification. While previous generations of vision models were shown to lack robustness to factors such pose, the extent to which this next generation of models are more robust remains unclear. To study this question, we develop a dataset of more than 7 million images with controlled changes in pose, position background, lighting color, and size. We study not only how robust recent. slate-of- the-art models are, but also the extent to which models can generalize to variation in each of these factors. We consider a catalog of recent vision modcls, including vision transformers (ViT), self-supervised models such masked autoencoders (MAE), and models trained larger datasets such CL.IP. We find that even today's best models are not robust to common changes in pose, size, and background. When some samples varied during training, we found models required significant portion of instances seen varying to generalize-thonr
eventually robustness did improve. When variability is only witnessed for some classes however, we found that models did not generalize to other classes unless the classes were very similar those seen varying during training. We hope our work will shed further light on the blind spots of SoTA models and spur the development of more robust vision models.
1 Introduction
dalaset of natural images can be described by set of factors of variations which characterize the main axes along which samples sample vary; for example pose, position, illumination, size, [Bengio al. 2013 Bouchacourt et al. 2021 Importantly, test-time unseen data samples may

---

Published in Transactions Machine Learning Research (04/2023)
exhibit different variability across factors than those seen during training QQuinoner-Candeed al, 2009 It is thus desirable for stato-of-tho-art (SoTA) modcls to be robust to changes in these factors 2013 However, previous work has shown that vision models such Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs; Dosovitskiy et al. (2021) are quite brittle to changes in pose, illumination, or even slight rotations and translation transformations (Engstrom et al. 2019 Alcorn et al., 2019 Abbas & Deny 2022 p. Yet, much of the existing work focuses either on the effect of a single transformation or analyzes toy settings where variability can be controlled.
Beenio et al.
If we aim to deploy models in more realistic and challenging applications, however, we need to study their brittleness to more natural variations on more realistic data which can potentially appear together (e.g. multiple factors at the same time).
Here, we extend existing work to study models' susceptibility to changes in position, size, spot hue, background, and pose independently, as well as changes in all factors in conjunction. To do so, we develop a dataset allowing based on 3d warehouse objects (Trimble Inc) that we place in non-uniform backgrounds and for which we vary the aforementioned factors. Using the lypical evaluation procedures of self-supervised models (Caron al.] 2021 Dosovitskiy et al. 2021 Chen et al. 2020b) (finetuning and linear evaluation), examine robustness across a catalog of state-of-the-art vision architectures, such as CLIP (Radford et al. 2021 that have significantly outperformed earlier models on robustness benchmarks such as ObjectNet (Barbu et al. 2019), Masked AutoFncoders (MAF, (He et al. 2022 ) or ViTs Dosovitskiy et al. (221]) among hers.. This allows us to compare common inductive biases such as architectures, training paradigm or the amount of pre-training data. Furthermore, we examine the effect of variability for the factors,
that is (i) seeing some instances varying for a single factor offects the other factors and how (ii) sceing some some classes varying for factors affect other classes. To the best of our knowledge, generalization of robustness across classes has not previously been studied.
Our main findings and contributions, summarized in Figure are:
1. We study the robustness of wide range of SoTA models to variations in naturally occurring factors. We found SoTA pre-trained models fine-tuned with little or no variability are not robust to factor variations (Section 4
2. We compare the effect of different inductive biases realized through different architectures, training paradigms, quantity of pre-training data, and finetuming vs. linear evaluation. We found differences in architecture and training paradigm have minor impacts on robustness, but that more training data helps and that finetuning generally leads to worse robustness (Section 4
3. Increasing the amount of variability of all instances for each factor during training helps generalization (Section 5. However, increasing variability only for some instances can hurt if not enough variability is introduced (Section 5.2). Nonetheless, variability in single factors tends to improve robustness to other factors too (Section 5.2
4. By studying the effect of variability across classes, we find that if a class is seen varying for some factors during training, it helps to generalize to very similar classes that were not encountered varying, but generalizes worse to all classes that are even little dissimilar and much worse to those classes which are highly dissimilar (Section
5.4
2

---

Published in Transactions Machine Learning Research (04/2023)
Drop in Accuracy
Diversity across all samples
Generalization across
Cross-factor Effects
Pre-tmined Models
CUP
Varying
instances (duning training)
classes half (during traning)
factor accuracy
MAE
ML MM
pose
chance
training 50%
MLPWMbor
5%
RusNuddO
-49.%
-28.7%
-17.1%
-47.5%
60.3%
+ +1.2%
SiMCUR
position
1Bo1
-6.4%
-2.6%
-1.0%
-15.3%
-45.9%
+3.0%
background
-22.1%
-6,0%
-1.8%
-26.1%
-46.2%
+2.9%
canonical 82.7%
lighting color
33.2%
-1.5%
-0.9%
-15.1%
-49.7%
+1.2%
-20.2%
-8.7%
-5.3%
-24.2%
-47.9%
+2.3%
Figure 1: SoTA models are not robust to and struggle to generalize common variations in pose, background, size. We show the average drop in accuracy across models when we vary each factor. We find if we vary all samples during training, models require a significant portion variation (2 50%) close the robustness gaps. When variation is only seen for some classes or instances, models struggle to generalize variation across instances or classes. Finally, when factor varies during training the robustness of other factors is also affected.
2 Related work
Recently, there have been much interesting work studying models' brittleness. Our work falls in this body of literature, yet aim to provide more extensive analysis by (i) varying different factors alone and in combination (ii) studying the effect of different amounts of variability seen in the training data (iii) on extended list of standard vision models. Alcorn et al. (2019) focuses on models' robustness to pose changes, while EEgstrom et al. (2019) studies rotation and translation changes (both her and alone) and find that explicitly augmenting the data with such variations does not fix the problem, a conclusion shared by (Azulay Weiss 2019) who study small image changes e.g. translating scaling. Madan al. (2021) also find that ResNets and CLIP networks brittle pose and lighting changes. [Mdan et al. investigate the generalization of only CNNs to combinations of two factors (object category and 3D viewpoint), finding that increasing the number of combinations seen at training helps generalization and also that separate networks outperform shared oncs.
Some works explicitly study the invariance of models, e.g. Bordes et al. 2021 3 Bouchacourt et al. (201)) where the latter also found that data augmentation does not bring the expected invariance. In (yon Küggelgen et al. 2021 the preservation of natural transformation information was studied
3

---

Published in Transactions Machine Learning Research (04/2023)
in a self-supervised setting, where they found that the pretraining data augmentation policy plays an important role. [Eulig et al. (2021) develops benchmark for measuring gencralization to six factors (position, scale, hue, lighting, shape, and texture) using a synthetic dataset of digits focusing on several (mostly CNN-based) models. Similar to us, Djolonga et al. (2020) studies robustness of models with respect to factors of variation of objects on various backgrounds, and how inductive biases such as the amount of data or size of model impact performance. However, they only focus on Convolutional Neural Networks (CNNs), while we also study vision transformers based architectures, foodforward models and even zero-shot multi-modal models such as CLIP. Furthermore, they factors in the limited selting where only 2D in-plan orientation changes are possible, while we use realistic 3D renderings of objects. More importantly, we extend our study to the case where variability in cach factor is added during training, and not just at test-time. This allows us to gain an in-depth understanding of models ability to generalize changes seen during training to new objects, rather than out-of-the-box robustness. Perhaps closest to our work, (Abbas & Deny 2022) studied the sensitivity of a large body of models and a variety of effects (architectures, data augmentation, dataset modalities), albeit only to pose changes (orientation and scale). In this work, we extend the set of factors studied, including cross-factors effects and consider catalog of recent SoTA models spanning various learning paradigms, architectures, and sizes.
3 Methods
3.1
The dataset suited to perform robustness analysis
To study the brittleness of state-of-the-ar models with respect to data factors of variation, several data properties are desirable. First, while there exist variants of real image datasets that present naturally occurring variations (Hendrycks et al., (021). control over the data generation process allows detailed factor metadata. Second, we want the dataset to vary sufficiently for us to draw consistent conclusions. Finally, we want the images to remain as close to realistic images in terms of image quality as possible. Existing datasets developed to show robustness of models often vary in just one or a fow factors, and the images are not really realistic or span only a few classes (c.g. Shapes3D) (Kim & Mnih 2018), MPT3D (Gondal et al., 2019), dSprites (Higgins et. al.
2017] among
others). Therefore we develop our own dataset based on 3d Warehouse (Trimble Inc) objects that we place in non-uniform backgrounds.
We use 54 synsets from 3d Warehouse (Trimble Inch, and 50 objects for each synset. For the first 4 scalar factors (position, pose, size, lighting color), we use equally spaced scalar values. For the background, we use 5 background types (sky, water, city, home, gruss) and 5 different backgrounds per type, with natural images coming from Li al. (2022, 2021bla). We define for each factor a canonical value, that is, the most represented value for that factor, to mimic the fact that in natural images we often see objects in a set of given factors (e.g. their upright position). We then vary these factors in three different manners: (i) each factor independently (101 scalar values equally spaced for scalar factors + 25 backgrounds) (ii) factors varying in pairs (11 scalar values + 10 backgrounds) (iii) all factors varying together (drawing 1000 random combinations from the full grid of 11 equally spaced values and 10 backgrounds). This gives us roughly 7 million (M) images in total, divided as follows: single factor (1.1M), paired factors (3.1M), and all factors (2.7M). We provide more details on the generation process in Appendix Section B
4

---

Published in Transactions Machine Learning Research (04/2023)
3.2 Introduce pre-trained models
We select a of state-of-the-art (SoTA) vision models, with many achieving 80% top-1 accuracy on ImageNet, spanning learning paradigms, training dataset sizes, and architectures. We also include CLIP, a model trained with caption supervision impressive performance on several OoD benchmarks [Rafford et al. (2021). We also evaluate the zero-shot performance of CLIP trained on 2B images from the LAION dataset Iiharco et al. (2021).
over 400M text-pair images that has show
We select SoTA supervised models of varying architectures. For ResNet-50, a CNN-based model, we use ImageNet- 1k pre-trained model based on the an improved training recipe from Wightman al. 2021 achieving 80.4% top-1 accuracy on ImageNet and an ImageNee-221 weights from Ridnik al. 2021 achieving 82.0% top-1 accuracy ImageNet. For Vision Transformer (ViT), attention- based model, we use an ImageNet-21k pre-trained ViT-B/10 16 achieving 83.97% and ImageNet-1k pretrained weights from Ridnik et al. (2021). For MLPMixer, a multi-layer percetron-based model, we use ImageNet-21kk pretrained weights from Ridnik et al. (0211 and ImageNet- 1k weights from Wightman 2019 using Base- 16 architecture.
We also select several SoTA self-supervised learning models. For SimCLR (Chen al. 2020bb, contrastive learning method, we select ResNet-50 (CNN-based) backbone, trained ImageNet-1lk based on weights from Falcon & Cho (2020). For MAE He et al. (2022] method based on reconstruction objective, we select an attention-based ViT encoder. We use pre-trained weights from the official repo of He et al. (0022). For iBot Zhou et al. (2021), also a ViT-based model, we use ImageNet-1k 1k and ImageNet-21L. 21k pre-trained weights from the official repo of Zhou et al. 2021).
All of the models are pre-trained using the standard lata-augmentttiion pipeline proposed for each method. We load pretrained backbones for each model without modifying the pretraining data augmentations.
4 SoTA vision models are not robust to natural variations
We evaluate pretrained model's ability to generalize natural variation using two common protocols: linear evaluation and finetuning. We measure models' generalization by each model's classification accuracy for "canonical" settings and the same objects varying by one or more of the natural factors. Specifically, we define the generalization gap for each factor to be the (accuracy on held-out samples with randomly varying factor) (accuracy the canonical value of each factor is chosen arbitrarily, but fixed across all experiments such that the canonical value is simply the value which is dominant in the training data. We then evaluate these models on held-out objects which have factor values not seen in training, varying the values of one factor at a time. To control for differences in the performance of models on canonical data, we report the gap between the model's accuracy on the canonical and varying held-out, sets.
held-out sample in their canonical sctting). Note that
SoTA models are not robust to changes in pose, background, and size While models reached strong performance on canonical data, Table I demonstrates that even SoTA models suffer considerable drops in performance when objects vary across factors. Models were particularly sensitive to changes in pose, background, and size, while models were largely robust to changes in position and lighting color. We hypothesize that this difference robustness across factors may be related to how easily variation across a factor can be approximated by pixel-level augmentations. Both lighting color and position can be well approximated by color shift and translation, respectively.
5

---

Published in Transactions Machine Learning Research (04/2023)
(a) Linear evaluatiin gaps
Train accuracy
Held-out accuracy
Pose gap
Background gap
Siee
Position
Lighting color gxp
Average gap
CLIP
72.22 22
-12.40
-25.43
-1984
-5.70
-2.65
-19.20
MAE
30.12
21.11
-13.77
-10.77
-6.79
-2.30
-2.36
-7.20)
M
85.55
71.56
-44.19
-35.11
22.311
-10.39
-4.92
-24.22
MLPMieer2Lk
91.59
37
-3.25
-2.83
20.32
-5.45
ResNet50-1k 1k
86. 76
77.41
-42.00
-27.78
22.28
+5.10
-3.33
200.76
BesNe150-21k
76.22 22
-35.99
-23.20
-1488
-0.51
-1.04
-15.02
SinCLR
91.69
73.53
-51.05
-33,93
28.01
.8.85
1.11
-23.55
VTT-1K
90.47
79.63
-44.48
.24.43
-25.05
-6.53
-2.58
-20.81
ViI-21k
22.773
Bos It
93.75
81.11
-52.63
-28.88
-25.96
-7.54
-3.62
-33.6
Boss-21s
93.00 33.60
96
-52.90
-31.66
-30.46
-7.14
-1.06
-2164
Average
84.72
72.26
-42.05
26.17
-2.69
-5.79
-2.05
-19.75
(b)
Finetuning gaps
Train sceuracy
Held-out accuracy
Pose Kap
Background gap
Siee wap
Position gap
Lighting color gap
Averappe KAp
CLIP
94.36
81.85
-5084
-16.67
-16.63
-5,32
-1.96
-18.28
MAE
33
-50.50
-44.73
-24.74
-17.71
-14.62
-30-16
MLPMixrlls
90.73
80.37
-51.10
22.76
23.13
-7.08
-3.17
22.015
ML ML.Mere22
96 21
84. 44
-46.4
-15.67
16.53
-5.06
-2.48
-17.24
ResNe150-1k
-50.63
20.61
-6.66
ResNotb0-21k
05. 76
86. 07
-46.88
-2087
-9.54
-3.74
-2.35
-20.44
SunCLR
95.31
96
-56.38
-30.11
21.36
-8.30
-0.72
-23 -23.98
VTT-Ik
96.17
84.41
-46.61
-14.66
16.98
-3.34
-1666
-16.59
VIT-21k
96.00
81.41
-6.95
-9.83
-18.01
-2.78
-0.19
-15.50
91.56
-53.22
-25.57
23.98
22
iBos-21k
95. 70
86. 56
-50.55
-12.02
-17.61
-1.1
-1.18
-17.10
Average
91.85
82.71
-9.99
-22.09
-20.1
-6.36
-3.22
-20.37
Table SoTA models are not robust to common factors: we show the drop in accuracy (negative values) relative to each model's held-out accuracy when an object presented its canonical setting for linear eval (a) and finetuning (b). All models are trained (linear eval or finetuning) images of instances in their canonical sctting.
In contrast, posc, background and size (relative to fixed background) all require 3D manipulation of the object itself, and are therefore very difficult to replicate with pixel-level augmentations. We refer the interested reader to Tables A2
A3 A4 and A5 where we report the absolute as well as relative top-1 accuracy numbers associated with the gaps discussed here.
While finetuning consistently improved performance on canonical data (finetuned held-out, canonical accuracy of 82. 82.71% vs. 72.26% for linear), it actually hurt robustness relative linear evaluation. Performance gaps varying held-out instances increased after finctuning, demonstrating that while finetuning can improve in-distribution performance, it does at the cost of generalization (Table
4.1 architectural inductive biases matter?
Learning objective is more impactful than architecture for robustness In general, we found that robustness was similar across models with the notable exception of MAE. As shown in Table (b), the MAE model espocially susceptible to changes in background, with a-44.7% drop compared to an average 19.4% for other models. MAE also substantially more sensitive to position and lighting color. This sensitivity was not observed in other ViT based models, suggesting that stems from differences in the training objective rather than the architecture. While all other models
6

---

Published in Transactions Machine Learning Research (04/2023)
use either supervised InfoNCE based objectives, MAE uses reconstruction objective. This focus reconstruction may cause the model pay closer attention to factors like background, position, and lighting color, it is likely necessary to learn these correlations to effectively reconstruct.
Interestingly, the consistency across architectures also largely held for comparisons between CNNs and ViT based models, even for factors such as position (Iranslation) for which CNNs are widely believed to be robust, although several recent works have suggested otherwise (Kayhan & Gemert 2020
Liu et al., 2018; Bouchacourt et al. 2021 Biscione Bowers 2021 Ruderman et al. 2018
Zhang 2019). We also found comparable gaps when evaluating CLIP using zero-shot classification inchuding CLIP trained on 2B LAION images (see Appendix
4.2
Are self-supervised models more robust?
Self-supervised versus Supervised Generalization Gaps
self self-supervised (linear
supervised (liear evai)
self-supervised (finetuning)
supervised (finetuning)
Top-1 accuracc drop
-10
-40
gap
Background gap
gap
Lighting gap
gap
Gap
Figure 2: Supervised models benefit more from finetuning than self-supervised models: we compare generalization gaps for self-supervised and supervised models using box-plots. Models are trained only canonical instances.
Several recent works have suggested that pre-training with self-supervision may lead to increased robustness (Hendrycks et al. 2019 Geirhos al. 2020). To test this, we compared the robustness of self-supervised models to supervised models in Figure 2 For linear evaluation, supervised models slightly outperformed SSL models on average, though SST models were able to achieve higher ceiling. In the finetuning setting, however, this difference is far more striking, suggesting that the robustness of supervised models benefits far more from finetuning than SSL models. Previous works Fan et al., 2021 Chen et al. 2020a) noted that regular finetuning the full network weights does not preserve the robustness self-supervised might have learned during unsupervised pretraining (e.g. with adversarial pretraining). This finding in line with Kumar et al. (2022) which also studies in depth this phenomenon, both theoretically and experimentally, and find that finetuning leads to worse OOD performance compared to linear probing.
4.3
Can more training data improve robustness?
Recent works have shown that increasing the dataset size leads to substantial gains, especially for SSL models (Zhai et al. 2022 Goyal et al. 2021 Hoffmann et al., 2022 Kaplan al. 2020). However, the effect of additional data on robustness remains unclear. To this, in Figure 3 we focus on the comparison between ImageNet-21k (14 million training samples) and ImageNet-11 1k (1.2
7

---

Published in Transactions Machine Learning Research (04/2023)
million training samples). We found that for both finetuning and linear evaluation, models trained Imagc.et-221k were substantially more robust than those trained ImageNet-IK (Figure 3 Interestingly, this effect was pronounced in the context of finetuning than linear evaluation, with pose, size, and position benefitting most. Finetuning also led to less variance in accuracy drops across models, suggesting models robustness converges with finetuning compared to linear evaluation.
Effect of larger training data on generalization
ImageNet (Inear evail
Imagefitt 21k llinear eval)
ImageNet (fineturing)
ImageNett 21k (finetuning)
Top accuracy drop
-20
Pose
Background gap
Position gap
Lighting color
Size gap
Figure 3: Models trained on ImageNet-21k are more robust compared to those trained on ImageNet-1k. We compare the effect training size for linear evaluation and finetuning on canonical samples.
5 Can models generalize variation from seeing variability in the training data?
In the previous section, demonstrated that SoTA vision models struggle to generalize across several common factors such as pose size. We also observed that pre-training on larger datasets (ImageNet- 21k vs. ImageNet-1k; led to improved robustness, consistent with other results demonstrating the impact of additional data (Radford et al., 2021 Kaplan et al.] 2020 Hoffmann et al. 2022 Zhai et al. 2022). Here, we study the extent to which models can generalize variability from training to held-out samples across three scttings: 1) when all samples vary 2) when only some instances vary 3) when only some classes vary.
5.1 How much training variability is needed to close the generalization gaps?
We first. measure the extent to which seeing all instances varying during training can close the generalization gaps. In order to introduce variation, we ensure a particular fraction of samples per instance feature diverse values (i.e., departing from canonical). We increase the amount of variability from to 95% and evaluate how robustness variability on novel instances test time changes relative to the robustness of models trained only on data with canonical values for factors. During training, each sample has the given percent chance of the factor randomly varying, For example, 50% chance of varying pose means half of instances seen during training have a random pose while the other half are in the canonical pose. Figure rrpport the effect of increasing variability on the generalization gaps for each factor. While all factors benefit from introducing variability, some factors such as pose and size still incur quite a large gap even with 50% variability. This result demonstrates that while incorporating variability during training improves robustness, the magnitude of this effect varies substantially across factors.
8

---

Published in Transactions Machine Learning Research (04/2023)
CUI VIT-1k
MAE
ML MLPMixer1k
ML PMixer21k
ResNet5D-1k
ResNet50-21k
SimCLR
VIT-21K
IBot-1k
18ot-21k
best fit
Position
Lighting color
Size
Background
slopem0.04
slopem0.27 .27
slope=0.01
Slopem0.11
slope=0:11
40
50
100
50
100
50
100
50
100
50
100
5lope=0.02
slope-0.25
slope-0.0g 0.00
slope=0.06
stope-0.0r 0.06
-40
0
50
100
100
50
100
50
100
50
100
Percent of varying images across all instances
Figure 4: Modcls require significant variation across all samples to close generalization gaps. We show the generalization gaps as the variability across all samples increases using linear evaluation (top) and finetuming (bottom). The x-axis indicates how likely each training sample to randomly vary for each factor. For example, 50% indicates half of samples seen during training randomly vary for the factor while the other half are in the canonical setting.
5.2 Can models generalize variation across instances?
The previous experiment measured whether introducing variability across all training instances helped robustness, but it remains unclear whether models can generalize variability in one set of instances to different set of instances. This is analogous to the experiments of Alcorn et al. (2019) but extends their work to different levels of variability and additional factors. We thus introduce variability only for a subset of instances for each factor of variation. By contrast. to Section 5.1 variability in % now refers to the percentage of the instances that are seen undergoing variations, while the rest of the instances are seen only with their canonical factors values during training. For example, 5% variability in pose means 5% of instances (objects) are seen in random pose during training; the remaining 95% of instances (objects) are seen only in their canonical pose. This is a substantially more difficult generalization problem, as exemplified by the larger gaps observed in this setting. For example, as humans we don't need to see every dog rolling around to know dogs roll around. However, we found models struggle to generalize when the percent instances varying is low (<25%). Robustness improves with additional varying instances, with some factors reaching minimal gaps with as few as 50% of the training instances are seen varying (Figure A16). The pattern across factors was largely consistent, though both position and lighting color reached minimal gaps with comparatively less variability in training data, consistent with our previous observation that models are more robust to variance along these factors.
Interestingly, varying only a portion of instances led to substantial overfitting, especially when the proportion of varying instances is smaller than 50%. Compared to the original gap with no diversity Section 4 the gaps are higher when initially introducing variability, and only return to their baseline values once sufficient variability is reached. For example, while position and lighting have gaps of -5% and -2% respectively with variability (Table D their gaps when 5% of instances
9

---

Published in Transactions Machine Learning Research (04/2023)
Gonerall craniing
slope changes
finetuning slope changes
20.97
35,50
30.47
Dedkgrund
43.72
540
30.53
24.50
35.52
35.27
29.40
Lighting
99 64
24.36
38.50
33.05
28.10
ki
40.10
27.74
44.41
4.555
31.47
Poulier
20.35
40.08
37.21
Percent of varving training instances
Pas
Affected Pactors
Figure 5: Varying one factor can improve robustness to other factors. We illustrate the cross-factor changes when a factor varies by plotting the change gaps using the line of best fit as the number of varying instances increases.
vary are nearly -40% (Figure A16 This suggests that when the specific subset of training instances seen varying during learning is small, the model overfits to this small set of varying instances. We compare the effect. finetuning vs. linear evaluation Appendix Section
Does training with instances varying for single given factor improve robustness to variation in other factors? Does robustness to a single factor provide broader robustness to other factors as well? To lest this, we trained models with increasing amounts of variability for a single factor. In Figure ! (last column), we show the average change in gap for factors other than factor varying during training. We find average effects of 1-3%. We further isolate this effect for each factor in the heatmaps show in Figure 5 by plotting the slope of the line of best fit across models
increase the portion of varying instances seen during training. The diagonal of this heatmap represents cases where robustness evaluated for the same factor seen varying during training: off-diagonal entires measure changes for other factors. Inducing robustness to one factor consistently improved robustness for other factors by as much as 41% for linear evaluation and 48% for finetuning, though results varied across factor pairs. For example cross factor effects for pose are minor relative to those for position and lighting color. fact, we found position and lighting color most helped each other, suggesting that the impact of position and lighting color variability are somewhat entangled.
Does larger pretraining data improve generalization to varying held-out instances? To test the importance of pretraining data size, we compared models trained on ImageNet-lk to those trained on ImageNet-21k. can be seen in figure [AT]] ImageNet-21k pretraining consistently improves robustness compared ImageNet-Ik pretraining, whether for finetuning or linear evaluation.
5.3 Does training with instances varying in all factors improve robustness?
Training with variability for a single factor improves robustness both to the factor seen during training well as other factors, but how does training with variability for all factors impact
10

---

Published in Transactions Machine Learning Research (04/2023)
CLIP
Background gap
Background gep
MAE
MLMMxer11k
Size gao
Sire gap
ML MMMixer221
ResNet50-11 -kk
ResNet50-21K
Ligting color
Uigting color gap
SimCLR
ViT-1k
gep
Pose gap
VIT-21k
iBot-1k
iBot
Position gap
Position
10
20
30
40
Linear eval
Finetuning
Relative top-1 accuracy drop when (all factors vary no factors vary)
Figure 6: Varying all factors during training improves robustness We show show relative generalization gaps when all factors vary during training relative to no instances sceing varying (no variability). During finetuning or linear evaluation, half of training samples randomly vary across all factors; while the other half of samples are in the canoncial setting.
robustness? To test this, we selected random bascs in the five-dimensional factor space and sampled images with random values along these bases during training. During finetuning or linear evaluation, half of training samples randomly vary across all factors; while the other half of samples are in the canoncial setting. We report the change in the accuracy gap induced by incorporating factor variation during training (e.g., gap with varying training factors gap with all varying training factors) on held-out samples. Positive values indicate an improvement in robustness, while negative values indicate a decrease. We found that training with variability across all factors led to substantially improved robustness for most. factors, though lighting color received no benefit, perhaps because its baseline robustness was already quite high (Figure 6 Interestingly, pose benefited the most from training with variability across all factors, despite being helped the least from the individual cross-factor variability, suggesting that while variability in other factors can improve pose robustness, variability across multiple factors simultaneously is necessary to induce noticeable improvements.
5.4 Can models generalize variation across classes?
We have shown that introducing variability during training improves robustness for new instances, and in some cases, for entirely different factors of variation. However, in all prior experiments, the class distribution was held constant such that models were only asked to generalize to new instances from the same class. Can models generalize robustness across classes? To test this, we trained models with variability only present for single factor for half of the classes (randomly selected). For classes trained with variability, half of the instances within that class were seen varying for the given factor. Results are summarized in Tables A18 and Table 2
Models are significantly lcss robust when variation is only seen for some classes We found significant gaps in generalization when only half of classes were seen with variability for each of the factor, as shown in Table 2 The average gap across all factors is -50% more than double the gaps observed where no variability is seen during training at all. This implics that when variation
11

---

Published in Transactions Machine Learning Research (04/2023)
Position gap
Pose gap
Lighting color gap
Size gap
Background gap
Average gap
CLIP
44.34
-62.91
-33,25
-50.30
-53.87
-53.93
MAE
-37.17
-48.30
-51.14
-42.35
-9.21
-45.64
MLPMixerlk
-47.33
-60.10
-51.26
-46.41
-52.6
-51.49
MLPMixer21k
46.18
-62.77
-50.28
-47.92
-47.79
-50.99
ReaNet50-1k
-5.66
-53.22
-43.62
-49.03
-35.44
-45.39
RosNet50-21k
-3.85
-54.00
-47.54
-47.12
-43.85
-47.29
SimCLR
45.49
-59.69
-46.59
-44.24
-29.39
-45.08
VTT-Ik
-48.13
-66.16
-51.90
-49.01
-48.39
-52.72
ViT-21k
-47.17
-61.91
-9..93
-45.50
-47.66
-50.43
iBot-1k
46.53
-65,76
-49.82
-50.61
-51.47
-52.84
iRot-21k
-48.22
-67.85
-51.46
-54.14
-49.33
-54.20
Average
-45.91
-60.25
-49.71
-47.88
-46.24
-50.00
Table 2: Models have significant gaps generalization when only half of classes were seen varying. Table shows generalization gap differences between classes (27 randomly selected) seen with diversity and those not when finetuning.
is only observed for some classes, models generalize even more poorly and extends Alcorn et al. 22019's i results demonstrating lack of generalization across instances at the class level. Our finding suggests we should develop explicit mechanisms for improving model generalization across classes.
Models generalize equally poorly across classes, unless classes are very similar or very dissimilar to those seen varying during training It is possible that robustness can only be generalized across classes when the classes exceed some threshold similarity. To test this, we evaluated the cross-class robustness as a function of the distance between classes. Class distance was computed using a pre-trained word-embedding similarities (Honnibal & MMntann] (017). While the most dissimilar classes harmed more, the majority of classes exhibited similar ritrentaaal effect regardless of their similarity to the training classes that were varying (Appendix Figure A23
6 Discussion
In order to develop robust, trustworthy models which do not fail when presented with distribution shift, we much characterize the generalization capabilities of our current best approaches. In this work, we provided an extensive study of the robustness of SoTA models to naturally occurring variations, extending on previous work in a number of ways. Our experiments show that models fail to generalize to variations of set factors on held-out instances unless reasonable amount variability is seen during training. Surprisingly, we found that providing the model with training variability single factor help generalize to other factors which were not varied during training. However, models struggle to transfer their knowledge of variations across classes: when only some classes are seen undergoing variability in training, only very similar classes (not seen varying at training) benefited at evaluation. Finally, we found that inductive biases such as architecture and training paradigm had minimal impact on modcls' converged robustness, in contrast to the pre-training data size and the method of downstream training. We hope that our work, by shedding further light on the blind spots of state-of-the-art models, can help practitioners develop robust models that confidently and safcly be deployed at large.
12

---

Published in Transactions Machine Learning Research (04/2023)
References
Amro Abbas and Stéphane Deny. Progress and limitations of deep nerksrks to recognize objects in unusual poses. CoRR, abs/2207.08034 2022. URL ottpp:://bi..org110.44555/////// 2207
08034
Michael A. Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. 2019.
URL http://arxiv. org/abs 1111/11111111
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184):1 25, 2019. URL
htm.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://pproceedingg/ 77440hsskccca661aacc000770000--0
pdf
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pottern analysis and machine intelligence, 35(8):1798 1828, 2013. Publisher: IEEE.
Valerio Biscione and Jeffrey S. Bowers. Convolutional neural networks are not invariant to translation, but they can learn to be. Journal of Machine Learning Research, 22(229):1122) -2, 2021. URL http://jmmr.ce http:///jirr..ogpppeeeeeee
wwtp.///ppppee////210111113 .html
Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam. URL http::/www. httt:://ww.blendeer.. org
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised reprosentation knows about. arXiv preprint rrXiv:2112.09164 2021.
Diane Bouchacourt, Mark Ibrahim, and Ari S. Morcos. Grounding inductive biases in natural images:invariance stems from variations in data. 2021. URL http://aaxx.v.ogg//////0.......
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised lcarning of visual features by contrasting cluster assignments. 2021. URL http:
//arxiv.or//bbal www.:/ogg/bbb//eeeee hwwpg//bb//00..... 200s0/000020202222222
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savareso, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint ar arXiv:1512.03013 2015.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From superviee pre-training to fine-tuning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020a.
13

---

Published in Transactions Machine Learning Research (04/2023)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020b. URL httt:://arrv..orggaa///20222
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssel Zidan, Dmitry Olesir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint
arXic:1911.01111 2019.
Josip Djolonga, Jessica. Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' D'Amour, Dan Moldovan, Syl- vain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convolutional neural networks, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. 2021. URL attp://arxiv.. org/ abs/2a10. 11929
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseorch, pp. 1802 1811. PMLR, 09-15 Jun 2019. URL html
https://proceeeding: mlr ppress/v97eengstaae
Eliss Eulig, Piyapat Saranrittichai, Chaithanya Kumar Mummadi, Kilian Rambach, William Beluch, Xiahan Shi, and Volker Fischer. Diagvib-6: A diagnostic benchmark suite for vision models in the presence of shortcut and generalization opportunities. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 10655-10664, 2021.
William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing a new approach. arXiv preprint arXiv:200.001004 2020.
Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https::/opperrrr/// wtwprrrrwwwweeeeeooouuriiiii.... fntf:/foorrr//////00000j1aaa.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bothge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and self-supervised models. In NeurIPS 2020 Workshop SVRHM, 2020. URL https://openrevien/ wwt.rvwwwwweweo.eniettt.ti.
q2m14CJMHAX
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Mar- tin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Ste- fan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement datasct. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL @B720046661111114447011111111100 ppf
14

---

Published in Transactions Machine Learning Research (04/2023)
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Solf-supervised pretraining of visual features in the wild. CoRR, abs/2103.01988, 2021. URL https://arxiv atg/abs/2103.01998
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000 16009, 2022.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http://s//.ss/.o/...ooooo/o.oo20
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. 2021. URL http:///arxxv.orrr/aa//11111 07174
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL
Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Erie Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https://arxi::
15556
Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Gabriel Mharco, Mitchell Wortsman, Ross minh Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL http::////01.1110.05555555 If you use this soflware, please cile it as below.
51437 5143773
Jarod Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Sealing laws for neural language models. CoRR, abs/2001.08300 2020. URL
Osman Semih Kayhan and Jan C. van Gemert, On translation invariance in cmns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), June 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jonnifer Dy and Andreas Kranse (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649 2658. PMLR, 10- 15 Jul 2018 URL hntps //proceedings. mmr. wwt.:prressvxxineeeee/ http://immm..........e html
15

---

Published in Transactions Machine Learning Research (04/2023)
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference Learning Representations, 2022. URL http://ppnrriiee.neeef/trrr///// UYneF2XSJWh
Jizhizi Li, Sihan Ma, Jing Zhang, and Dacheng Tao. Privacy-preservinn portrait matting, In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3501-3509, 2021a.
Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. ar preprint arXiv:2107.07833 2021b.
Jizhizi Li, Jing Zhang, Stephen Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. International Journal of Computer Vision, 130(2):246 266, 2022.
Rosanme Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coord- conv solution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
and R. Garnett. (eds.), Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL 001068889977777111116666/6688388 Papeee pdf
wtw.:eceeddnnn..eur.p..... www.mes....apppeeeeeeee cc/pappr/2018/E/lle
NK Logothetis, Jon Pauls, HH Bülthoff, and Tomaso Poggio. View-dependent object recognition by monkeys. Current biology, 4(5):401 414, 1994.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how CNNs generalize to out-of- distribution category-viewrpinn combinations. URL
08032
Spandan Madan, Tomotake Sasaki, Tzu-Mao Li, Xavier Boix, and Hanspeter Pfister. Small in- distribution changes in 3d perspective and lighting fool both cnns and transformers, 2021.
Joaquin Quinoncro-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset shift in machine learning. 2009. ISBN 9780262170055. URL http:://ll. acm org/ wwgg/citation.o cfm?
id= id=1462129]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In International Conference on Machine Learning, pp. 1148-8763.. PMLR, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021.
Avraham Ruderman, Neil C. Rabinowitz, Ari S. Morcos, and Daniel Zoran. Pooling is neither necessary nor sufficient for appropriate deformation stability in cnns, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable elfectiveness of data in deep learning era. In ICCV, pp. 843 852. IEEE Computer Society, 2017. ISBN 978-1-5386-1032-9 URL http://ablp... wnw-triireri..... /ttp://eee..a/////oooo////////// html#
SunSSG17
16

---

Published in Transactions Machine Learning Research (04/2023)
Trimble Inc. 3d warehouse. httpp://3dwwarehousee. wtt.://ooo.sseettttee.....oooo
Accessed: 2022-03-07.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Solf-supervised learning with data augmentations provably
isolates content from style. 2021. URL
Ross Wightman.
Pytorch image
models.
https://github. wwwpp//...oowwwwgtaaaaaaaaaeaeee
pytorch- - iaaa--momeeeeeeee 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12104 12113, June 2022.
Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ar preprint arXiv:2111.07832 2021.
17

---

Published in Transactions Machine Learning Research (04/2023)
Parameter
Canonical value
Minimum value
Maximun value
Object rotation X,Y,Z
Translation X,Y,Z
0.4
0.4
Object scale
0.4
Lighting hue Background
0.1
0
25
Table A1: Values of the factors of variation used for the generation of our datasets, Each value is sampled uniformly from the given interval. Object rotationa are generated as Tayt-Bryan angles using extrinsic rotations. Background indicates the index the background image used.
A
Appendix
B
Dataset
B.1
Generation
we select 50 models in 54 synsets from 3d Warehouse (Trimble Inc). The synsets used the same ShapeNetCore (Chang et al. 2015), which gives diversity in the objects' classes and makes a link to popular datasets for 3d models.
As discussed in section 3.1
To generate the scenes and the renderings we rely on Blender (Blender Online Community) and use BlenderProc (Denninger et al. 2019) to simplify the automation of the generation process. We use background images coming from Li et al. (2022) 2021b order to generate more complex scenes. Particularly, we consider five types of background (sky, water, city, home and grass) and 5 backgrounds for each type, giving us 25 in total. We also change 4 scalar factors of variations that position, posc, object size and lighting color. We describe the ranges used for cach factor of variation in Table A1 We illustrate the ranges in Figures A1 to A10 by generaling renderings for various objects.
We study three different datasets that are generated as follows:
One Factor: We vary each factor independently using using 101 different scalar values equally spaced for scalar factors, or 25 for the backgrounds. This gives us 1.1M images.
Pairs of factors: We vary pairs of factors using 11 different scalar values that equally spaced for each factor, and 10 backgrounds. This gives 3.1M images.
We compute the full grid of possible combinations using 11 different scalar values that are equally spaced for each factor, and 10 backgrounds. We then sample 1000 combinations that we use on all objects. This gives us 2.7M images.
All factors:
For all scenarios, when a factor of variation is not changed, we use the canonical values. The generation of all images takes around 1500 hours on single NVIDIA V100 GPU, but can be easily parallelized.
18

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Bacligoondd
Figure A1: Examples from the datasct illustrating the different factors of variation.
19

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Bacligoonnd
Figure A2: Examples from the datasct illustrating the different factors of variation.
20

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting color
Bacligoonnd
Figure A3: Examples from the datasct illustrating the different factors of variation.
21

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting color
Bacligoondd
Figure A4: Examples from the datasct illustrating the different factors of variation.
22

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Bacligoonnd
Figure A5: Examples from the datasct illustrating the different factors of variation.
23

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Baclggoondd
Figure A6: Examples from the datasct illustrating the different factors of variation.
24

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Bacligoonnd
Figure A7: Examples from the datasct illustrating the different factors of variation.
25

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Bacligoondd
Figure A8: Examples from the datasct illustrating the different factors of variation.
26

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting color
Baclggoondd
Figure A9: Examples from the datasct illustrating the different factors of variation.
27

---

Published in Transactions Machine Learning Research (04/2023)
0 %
25 %
50 %
75 %
100 %
Position
Pose
Scale
Lighting coler
Figure A10: Examples from the dataset illustrating the different factors of variation.
28

---

Published in Transactions Machine Learning Research (04/2023)
Toiin canonical
Held-out canonical
Varying Pten
Varying Background
Varying Scale
Varying Poeition
Varying Lighting oolor
CLIP
$0.65
72:22
29.82
46.79
$2.38
06.53
69.57
MAE
30.12
21.11
7.34
10.31
14.32
18.81
18.75
MLPMiaer
86.55
71.56
27.37
36.45
45.24
01.97
66.85
MLPMinew21k
Ro
86.76
77.41
96.11
19.63
52.15
72.31
74.08
RenNec50:22 21k
92.89
76.22
00.73
58.05
61.37
75.71
75.18
SinCLR
91.00
73.33
22.28
39.
45.32
67.48
74.44
VTT IK
93.47
79.03
86.15
66.20
54.38
73.10
77.07
1Bot-1k
95.75
81.11
28.45
62.73
55.15
73.57
77.00
[Bot-214
95.60
82.90
30.07
51.30
32.50
75.82
$1.90
Table A2: Linear eval top-1 accuracy canonical images and as each factor varies for held-out
images.
Thii canonical
Held-out canouical
Varying Pose
Varying Background
Varying Scale
Varying Ponition
Varying Lighting color
94.36
76.53
MAE
92.91
73.33
22.84
28,
48.39
55,02
58.71
MLPMixerlk
77.20
MLPMisr211
96.21
84:44
38.00
68
67.92
70.30
81.96
ResNetta-1k
62.15
76.90
ReeNei500-11
85.76
00
36.79
67.13
82.08
84.32
SimCLR
95.34
82.96
26.18
62.85
08.00
74.67
82.24
VIT 21k
96.01
84.44
$7.50 50
74.61
86.45
81.07
$1.26
[Bot-1k
94.56
84.81
31.00 80
58:25
60.82
79.00
$1.90
1Bot-21k
95.70
85.56
35.01
73.54
07.92
81.41
$4.38
Table A3: Finetuning top-1 accuracy on canonical images and as each factor varies for held-out
images.
B.2 Dataset samples
C
Relative and absolute accuracy for each factor across models
addition the performance gaps, we show in Tables A2 and A3 the absolute top-1 accuracy numbers associated with the gaps we report. We show accuracy canonical instances (held-out and training) well as performance each factor varios (on held-out instances). These figures are for both lincar evaluation and uinetuned models.
We also show in Tables A4 and A5 the relative gap relative gap, that is, the gap value divided by canonical accuracy, for each model both in the lincar evaluation and finetuning protocols.
Teslo quansiol Hiliooa cusosical
Pue pobiiive upp (E)
Bacagpouid itaaive gap (5)
Stze rdatin zap (5)
Posilku shilveg (%)
Lightinn roloe riatiie pap (%)
-3.2
MLP5tise2kk
$1.09
-31.81
56.76
-31.8
4.00
211
$2.88
16222
46.96
09:11
4107
413
SmnLRi
91.00
15.33
40122
-91.1
38.20
-2.8
1.2
VIT-EE
58.47
70
15.96
00.05
J1.8
4.90
432
Vir-oik
91.82
19
1051
05.81
31.85
-.81
-1.1
58.76
81.11
-9.98
-31.7
-100
20330
-1.77
BBv. 211
58.00
32.96
-58
-3.15
-6.7
..61
-1.2
Table A4: Linear eval top-1 accuracy canonical images and the relative percent drop in accuracy each factor varies for held-out images.
29

---

Published in Transactions Machine Learning Research (04/2023)
Toato ouamiooi
Hokioaal o1MA)о1
volulie My (7)
Becigroud relaiv gi (7))
idatic pay
whhooggp (7))
Lighting cdlo nlatic py
CLPP
91.30
-00 11
-0130
-335
MAE
$2.91
15
00:35
33.21
-2115
ML PMtowe i
VEL73
01.37
-53
-2200
-33
ML ALPSM 23K
16.21
$4.65
85
18:22
18.27
-3.9
431
Redeetto ik
86.61
50.06
16 53
02.74
05.30
4.23
4.00
16.70
8000
15.80
-91.47
22:34
-131
27
SeaCLR
$6.84
82.0
65 PM
-91.3
22.31
1000
4/67
H6.17
-2.00
16.01
-1.40
-1407
Table A5: Finetuning top-1 accuracy on canonical images and the relative percent drop in accuracy as each factor varies for held-out images.
D Aggregated performances for linear vs finetuning and 21k 1k
-11
.01
Iopelee ikiner mageter
Fragetee ic ineer
Iiagetee
Irophhi 231 tratarin
Iragelid 20x Inelumeg
22
04
63 Divess swportie durii tabking
Divere proutton auring maining
-.1
0.0
-32
Inepoho-3kiing inee trupetto:21 Irupohic tratareg
Imagelid-2e tnser
imagetidd
-25
.05
Imagelid- 25x Enolumen
22
64
0.6
0.5
0.8
Civers propartion soan during baining
proporice unan duting trantig
Figure A11: Drops in performance averaged over all methods when varying the proportion of varying examples seen during training.
As can be seen in figure All finetuning usually leads to lower drops in performance with high variability rates during training. However linear evaluation is more robust when diversity was not encountered during training. Pretraining on ImageNet21k always improves robustness compared to ImageNet-1k pretraining, whether in finetuning or in linear evaluation. It is worth noting that for translation robustness, all settings exhibit similar performance, and finetuning only benefits ImageNet-Ik pretraining.
Model sensitivity to variation strength
To further understand the sensitivity of model performance the varying strengths of each variation, measure the accuracy vary the variation strength. We perform the analysis for the best
30

---

Published in Transactions Machine Learning Research (04/2023
Position
Accuracy Accuracy
Accuracy Accuracy
0.9
0.8
0.5
0.7
0.6
440
40
0.
0.4
0.6
Lighting color
0.5
M
M
parameter
parameter
Figure A12: Sensitivity increasing strength of variation for each factor for iBot pretrained on ImageNet-211 with linear evaluation.
Position
Scale
Accuracy Accuracy
Accuracy Accuracy
0.9
0.8
40
Pose
Lighting color
0.9
0.5
-100
100
20
40
60
80
100
parameter
parameter
Figure A13: Sensitivity to increasing strength of variation for each factor for iBot pretrained on ImageNet-21k with finetuning.
performing model, iBot trained on ImageNet-21kk, according to held-out accuracy. In Figure A12 we show the top-1 accuracy as the strength of each factor varies for iBot with linear evaluation. We find the canonical scale (1.0) and pose (0.0) have the highest accuracy with accuracy dropping as the variation strength increases. Position exhibits slightly higher accuracy with respect to the canonical (0.0), but we note the drop more muted compared that for pose and scale. Finally, in line with earlier findings, since Lighting color gaps quite small, do not observe significant changes in accuracy Lighting color strength varies. We show similar results in Figure A13 using the finetuning prolocol.
As expected, Pose, Position and Scale factors show strong correlations between their strengths and the drop in performance. For Pose particular, the accuracy almost halved when going beyond 45 degree rolations. These findings are consistent with previous studies involving primates (Logothetis jtt al. 1994) who were asked to identify an object after rotation, among distractor objects. As we in Figure 5 A from Logothetis et al. (1994) primates exhibit almost identical failure
31

---

Published in Transactions Machine Learning Research (04/2023)
Held-out canonical accuracy
Pose
Background
Scale
Position
Lighting color
BIT-IFT-300M
73.09
32.71
47.29
54.11
69.26
82.81
SerrdentNee-L2-JJ----
69.63
38.67
50.96
50.47
65.19
70.19
Table A6: Lincar eval top-1 accuracy on canonical images and as each factor varies for held-out images.
Held-out canonical accuracy
Pose
Background
Scale
Position
Lighting color
BIT-JFT-300M
79.01
32.54
58 94
62.55
74.40
78.09
80.49
40.93
69.87
71.82
78.13
71.33
Table A7: Finetuning top-1 accuracy on canonical images and as each factor varies for held-out images.
mode neural networks when varying the object's pose. Nonetheless the lasks not identical and identifiability is arguably harder than classification due to its more fine-grained nature.
Additional Models Trained on JFT-300M
In addition to the models discussed in the main paper, here we also evaluate two models trained on Google's properietary JFT datasets comprised of 300M images (unn al. (2017). We performed both linear evaluation and finetuning for BiT, Big Transfer model, as well Efficientett.2 (us 475) from retrained JFT 300M Sun al. (2017) We trained each model for 2k steps with batch size of 32 for BIT and 8 for Efficient Le accommodate GPU memory. In Table A6 show linear evaluation results and in Table A7 finctuning. Similar other pretrained models we examined, we observe comparable performance drops relative to each model's canonical performance. We note the higher held-out accuracy after finetuning as expected. This suggests even models trained on JFT exhibit similar performance drops as the set of factors we consider vary.
G CLIP Zero Shot Classification
We also evaluate CLIP's robustness using zero-shot classification. We assess both the standard Open Al CLIP model as well as CLIP trained on 2B LAION images. We prompt the model using "a photo of a I CLIP with LAION-2B accuracies are 31.9% for canonical, 15.9% when pose varies, 18.2% when scale varies, 31.9% when lighting color, and 26.8% background varies. CLIP with trained on 400M images has canonical 30.1%, pose 16.0%, scale 18.4%, lighting color 28.3%, and background 23.6% accuracy.
We examine other prompts ('I, inanimate object', 'a photo of a [], an inanimate object', "[], houschold item or vehicle") and observe similar classification performance (25-31% accuracy) using these variants.
H Cross Factor effects when varying all instances
The cross factor effects when all instances vary with increasing diversity levels are shown in Figures
A14 and A15
32

---

Published in Transactions Machine Learning Research (04/2023
linear eval slope changes
Background
1.45
2.80
0.08
2.77
11.22
25
Varying Factors
Size
1.01
1.81
-0.43
11.39
2,43
20
15
Lighting
0.08
0.25
1.34
0.01
0.15
Pose
1.68
27.41
-0.37
2.27
2.95
Position
3.51
2.33
0.11
4.60
0.78
Position
Pose
Lighting
Size
Background
Affected Factors
Figure A14: Cross factor changes when the given factor is varying for linear evaluation
finetuning slope changes
Background
1.26
2.69
0.41
5.28
5.86
20
Varying Factors
Size
0.10
0.31
0.04
5.55
1.34
15
Lighting
0.37
-0.45
1.02
-0.36
-0.57
10
Pose
1.44
24.80
-0.98
1.01
-0.03
5
Position
2.08
0.41
0.58
4.59
-1.18
Position
Pose
Lighting Affected Factors
Size
Background
Figure A15: Cross factor changes when the given factor is varying for finetuning
33

---

Published in Transactions Machine Learning Research (04/2023)
Varying subset of instances during training
We show the effect of increasing the number of instances seen varying during training. In Figure A16 we show the effect of each factor. We break down the effect by factor in figures A17 for linear
evaluation and finetuning A19 In addition, we show the overall accuracy in tables A8 A12 A9
A10 The val canonical column corresponds to held-out accuracy for canonical and the val diverse corresponds to the accuracy for a changing factor.
Finetuning vs. linear evaluation as variability increases during training To summarize these results across factors, for each subplot, compute linear fit to the average model curve and compute its slope. Models with higher slopes are more sensitive to the fraction of instances seen varying during training, while lower slopes indicate models which have the same generalization gap regardless how much instances was presented varying during training. The average slope across factors and models for finetuning was 0.359 = 0.035 vs. 0.441 + 0.020 for linear evaluation (mean std). This result demonstrates that, while both benefit from increasing the percentage of instances seeing varying during training, this effect is much more pronounced for finetuning, providing further evidence that the impact of supervision is larger for finetuned models, likely because of the increased expressivity introduced by allowing all the weights to change.
Position
Pose
Lighting
Size
Background
CUiFretrance
MAERetrrined
HI Peilee ettainnedi
MI PMiteed witaineee
ViThheraneeetk
0.5
diverse training
Position
Pose
Lighting
Size
Background
ELPPretrenee
MASPretrained
-70
-40
SimCLspctrained
-30
0.5
05
0.5
0.5
0.5
ViThroraine222k
percent diverse training samples
(a) Training with increasing percentage of variability across all instances using finetuning
Figure A16: Training with increasing percentage of instances seen varying during training using linear evaluation (top) and finetuning (bottom).
34

---

Published in Transactions Machine Learning Research (04/2023)
camonical top.. occuroey
rencccion top
diversc
Translation top
Iniin peop lo. vwyy
0.85
0.25
0.50
0.75
0.95
0.05
0.25
0.50
0.75
0.95
0.05
0.25
0.50
0.75
0.45
CLIPPeiniiul
92:2455
01.357%
312
2575
94.01%%
96.33%
77.78%
70.94%
68. 13.522
68.52%
58. 89.8955
30.8256
57.2096
64.5155
65.4555
06.58%
MAAPaerruioe
R2.0455
02.4135
600.45 4435
07.917%
83.77%%
20.37%
27.78%
08.331%
58. 837
27.7856
9.67%
12.9156
14.2555
15,00%
15.18%
ML. Pettateettttt
94.00%
83.09%
94.
96.70%
97.
77.78%
74.07%
72.22%
06.677
48.13%
36, 96.85%
53.70%
09.3356
63.00%
63.46%
ML. M.LPMiissPsetaaiine
94.55
97.13%
75.98%
76,07%
77.76%
70.57%
43.89%
71.2555
73.6995
76.18%
05 0000
94
07. 97.28%
88 50%
90.74%
67.94%
83.81%
70.57%
44.35%
63.26%%
68.9955
70.0455
70.03%
Res Ro=Ne600 Perrttid2i
05.5155
88 08-31P5
9075
077 07.399%
77-18%
-
12-2225
75,00%
70.07%
70-5775
06.6115
06.04%
13.0255
75 13.9095
72.13%
SinCLR Pariniied
96. 15%
95.74%
96 22%
97. 97.50%
61 48%
73468%
31.68%
72.22%
70.0055
43.7356
02.9655
09. 1055
70.6555
72.02%
05.7993
16.157
06.317
06 20780
97. 97.75%
58 801
80.335
11.785
77.415 411
70. 0056
19.34%%
07 07.02%
72.74%
13. 13.0555
17.22%
05.45%
05.01%
62%
30%
97.30%
65.33%
85.83%
63.53%
79. 76%
19. 2156
66. 59.9
67 2111
14.025
75.17%
iPe Peetaaettii
05.843
06.307%
97.61%
90.74%
55.19%
57.04%
81.08%
72.22%
70.55%
76.37%
5355
79.04%
Table A8: Position varying linear eval top-1 accuracy across multiple percentages of varying training instances.
Lexin conantiod tup. actsuney
vsl. causcciod lop
tol Aliveree
Rodaiion
Iup 0.50
Jermunscy 0.75
train prop to vary
0.85
0.25
0.30
0.75
0.95
0:05
0.25
0.50
0.75
0.05
0.05
0.25
0.95
CLJPPrettited
01.20%
01. 1155%
427%
05.988%
96.388%
81. 1858%
85. 195%
1975
70.1315
55. 5555
25 6155555
41.16%%
64.725%
06.11%
05.85%
MA RPreriidd
53-91%
67:249%
08.41%
633500%
95.905%
12.56%%
6.6855
7.935%
5.36%
94.24%
83.7695
95.72%
97.38%
76.07%
76.07%
72.22%
70.87%
46.9055
26.8436
30.4756
43.0455
66.6155
48.73%
ML.
01. 8935
81.811%
05.03%
95.2835
93.13%
79.53%
77.78%
71.788%
70.03 1336
72: 21.2255
36. 1K.0
55.865 86-8696
61.5755
63300755
65:3075
Ro So=Nne0PPeeraine
54.72%
91. 505%
91 297%
95.74%%
97.18%
85 10%
62.045
85. 19%
61.65 18%
68. 5955
$1.6456
46.2256
49.9095
51.7595
53.94%
Res Res 201
05.513%
01.907%
95. 007
96.12%
97.207%
77.78%
75.5036
75.305
72. 22%
06.6756
40,2995
57.6855
61,8356
63..915
63.0435
SmC Pretrnined
05.74%
95 05.62%%
16%
30%
97.31%
81 18%
63.33%
83.50%
S3.531
62, 0655
52 04.04%
47.300%
52.855
53.91%
56.91%
97.67%
87.011%
79.00%
13%
03.222
97.45%
85.19%
50.30%
63,331%
676
38.25 23.
58.345
00.923
62.28%
08.3095
96 22%
06 417
96.96%
97 263
80.20%
81 48%
485
72, 2256
54 82%
8856
895
82. 1995
61:1176
Bo SooPPe Petteauuuuu
06.5
96 42%
96.92%
97.61%
50.74%
88.80%
90.74%
57.04%
72 2256
41.05%
07. 85%
63.6255
65. 4955
07.34%
Table A9: Pose linear eval top-1 accuracy across multiple percentages of varying training instances.
Lexia conceniiid Lop
caamecion lop
diierree 0.05
Spect Burt top.
train__pm pcop_pttpppppppp tor__tt__oooe tary
0.05
movill
CLIPPretmined
02.40%
91 0000%
02: 46%
98.003
96.377
77.785%
75.90%
70.5v%
70.3:5
10. 3050
30.0 93.6/15
(3. 00.785%
05.7255
08: 06-2115
05/644%
MARPaerained
52.2696
55.115%
00:3655
07.78%
83.62%
22.22%
31.485%
37.01%
37.04%
20:3755
11.5156
15.05%5
16.1055
18.9195
15.38%
05.17% 17%
94.2105
94.98%
95.86%
97.36%
79.65%
77.78%
70.57%
66.07%
57.0455
40.09%
56.90%
64.4355
07. 07.3555
68.93%
MMPMiisePPettrannee Ik
01.80%
05:207%
4175
13%
97.20%
81 61.48%
70.635
72.22
74.00%
70.3055
$4.6855
09.30%%
13.3256
16.125%
TE.75%
Re-Ne Woerannnrd
63.30%
77.95%
NesNetto wetolned21k
95.9175
055.457%
97.41%
77.78%
75.50%
75.50%
72.22%
71.11%
44.22%
67.22%%
70.0376
T2.38%5
T4.99%
SmCC Pretrnined
06.307%
03.01%
06. 277
06.343%
97 09.200
70 13.%
75.50%
70.0%%
74.00%
06. 06.67%
13. 13.39%
63.22%
TL.325
T3.72%
72.26%
WTPretraieelllk
05 9999
96 36%
96.3 5-15
96 9676
97 30%
90 90.70%
67/0155
63. 53%
81 88%
74.07%
59. 53.5356
09.5395
71 8198
76.0195
77.963
95.07%
00.195%
95. 847
96.51%
97. 20%
86 10%
81.45%
89,33%
78%
75,0056
46.01%
77.666
11.97%
73, 75%
77.063
iBe.Petrainelkk
05.50%
86.58%
745
12%
97. 70%
79 19.00%
81,48%
77.78 78%
74.
40.32%
08.6
12. 7956
76 28%5
78.0475
07.01%
97.04%
68.80%
50.30%
76.935
45.83%
80.8995
81.6035
Table A10: Spot hue linear eval top-1 accuracy across multiple percentages of varying training instances.
train canontral top. aacurreyy
vol. cancntcaal
top 1 .aceurnccy
vol. diverse
Scale. 0.25
top
train prup to swry model
0.85
0.25
0.50
0.75
0.95
0.06
0.25
0.50
0.75
0.95
0.05
0.50
0.75
0.95
CIIPPootninnl
01.81%
91.511%
02:287%
95.50%%
96.26%
79.663%
72.223
71.077
70.577%
61.1155
36.8/95
51.20%%
57 (155
56. 93.9995
53.800%
MAAePeetratedd
51.98%
56.1575
60.55 8775
08.073
84.12%
25. 25.50%
35.195%
33.33%
37.04%
35.1996
7.015%
10.74%6
11,0556
11.299%
11.44%
81.27%
97.17%
79
33.59/%
30.3
ALLPMMaeePictta Pictrlloolk
94.003
95.00
85.50%
97.007%
79.50%
79.60%
75.90%
75.30%
74.077
40.53%
67.525
70003%
Res NetSoo sotrainedlk
04. 8453
94.57%
01. 5.%
95.063
97.16%
37.00%
90.74%
90.745
53.30%
72 .2.2
30. 22.22%
54.25 23.
39.1065
61,33 61.27%
60.80%
Ros Nww.ioaaaaaaaiiiiaao
95.6195
05.2895
85. 77%
96.08%
97. 97.30%
61.48%
77.78%
77.78%
74.00%
74.0755
41.139
60.8596
66.4155
09. 6965
70.15%
Sind (LR Peetnined
95.01%
00.535%
96. 05%
96.66%
97.48%
68.33%
77.41%
77.78%
72.22%
02 9659
20.79%
54.93%
61.9156
02 03.9
62.96%%
85.63%
96.017%
96.00%
07.607
87.00%
80.33%
79,63%
16.30%
44.10%
58.135
63.01%
67. 6378
68.82%
VIT
05.0
68.
63.50
73. 22%
41.40%
63.2555
05.14%
8630895
96. 0775
96.08%
97.6 6477
84.07%
77.78%
79.03%
77.78%
06, 67.67
41.84%
08.90%
07.245
68 6005
62008%
06.922
86.
18%
38%
2006
07 53%
85 19%
77.78%
81.18%
81.06%
75 .09%
44.59%
03.11%
08 9055
71.4595
70.83%
Table All: Scale linear eval top-1 accuracy across multiple percentages of varying training instances
35

---

Published in Transactions Machine Learning Research (04/2023)
Erain comniall top 1 pecuEaCy 0:05
camoniccl top
vol. Lidiinn Hadigrooud buth, top L.EEEESS
train prep. to
0.25
0.50
0.75
0.95
0.05
0.25
0.50
0.75
0.45
0.05
0.25
0.50
0.75
0.95
CLIPPostraine
90.5755
91.58%
92.75%
36555
65.199%
17.18%
77.78%
05.07%
54.70%
52.9695
01.7555
65.41%
MAEPretaaned
5U.25
02:317
67.05%
27.78%
45%
57.04%
37.047%
29.6356
9.275
12.21%
11.847%
12.07%
ML. Pa.Piiiaaattsiiaaaee
02.8455
90.52%
94.81%
95.4135
97.58%
76.98%
17.78%
72. 2212
74.07%
61.1356
57.9998
49.1975
52 72.4135
55.7318
56.74%
MLP MilPSeee Pe
95.4255
36.3315
95.6235
96.525
97,5875
83.3335
81. 455
75.533%
75:5835
77.7835
45.300%
64.37%
68.7995
70.13%
70.8425
RooNet Prttenallll
91.35 5555
96.12%
94.85%
95. 9555
77.46%
90.74%
92. 5959
88. 895
88 19.2955
65.3556
$4.890 11.8995
39.55%
04.15%
06. 7999
67: 07.29%%
155 20%
96, .111
03.897
16 63.655
97 07.67%
41.48%
130 6356
77.7855
7T.779%
75-9356
51.215
65.755%
69 4778
72.01%
73:90%
Sm CI PPeetaaneed
9056
95.985 95%
06 16%
96. 6556
50%
8535%
83.5556
77.765%
72 72-1296
5355
50.52%
05.3158
07.00 00.0555
08-95%
VIT Pretioinell
95.72%
96.42%
96.27%
96.67%
97.00%
94.44%
57.00%
77.78%
64.00%%
67.1425
72.33%
73.19%
94.00%
9600%
96. 4356
90. 71%
H3
50.00%
59-30%
17
49.
21%
67. 4226
70.17%
72 12.1
Be Pectminttlll
96.58%
06.005
07.
07. 2816
88 5975
83.85%
67.041
81,4895
79-6556
44.6818
5AYS
68.
71,209
73
iBot Pretminedd l.
06. 1655
96. 16%
06. 06.38%
07.3 3756
99. 71.
88 807
00 74%
14. 7456
63.35%
81. 4636
2096
68 08.58%
71. 11.577
74.6155
75.943
Table A12: Background path linear eval top-1 accuracy across multiple percentages of varying
training instances
teain consatie
1 senuueyc
canccieed Iup 1 .unuuncy
vol diivere 0.05
Tranalation top
peop
0.05
0.25
0.30
0:96
0.00
0.25
0.95
0.75
0.515
moddll
CLiPPettminn
07.4995
07. 07.01095
97.55%%
966.5175
87.04%
90.74%
77.78%
61.65%
76.
45.935
00.745%
74.0355
77.72%
75.199%
MARPrerrioed
03-0
0157
97.103
077.91%%
63.333%
71307%
00-67%
61.8155
79.2255
29.1756
51.3555
6: 01.0555
05:1095
7000755
96. 9595
96.755%
97. 1775
07.24%%
97.58%
68. 38.50%
77.78%
77.78%
74.07%
64.8115
41.6358
64.9656
70.0755
74.3495
75.95%%
BT. 7156
07.6395
97. 9175
07.98%%
98. 08.35%
65.191%
38.80%
85.19%
81.45%
77.7856
46,0356
70.0156
77.2755
79.7855
81.6835
Rea
07.00 97%
87 0222
91%
20.6
98 98.27%
67.01%
87.01%
72.225
51. 11.88%
81. 81.085
45.0555
04.8358
71 71.875
76. 56.5655
70665%
47.54%
201
53.33%
52.225
75.7455
88.1
Smo Pretrnined
07.4
M7.5
.1.1%
98.
90.74%
71.78%
67,045
63.20%
17. 75.55%
45.21%
74.599%
07.8075
07 8895
007%
87.50 9939
98 783
90 60.70%
90.745%
1958
81.065
81 1118888
09.3095
71.89%
26.0555
80.309
82.88%
07. 809
97. 50P5
97. 2915
07.91%
98 96.25%
67.01%
81.16%
61.98%
67.0 57.0155
15.8356
71.8066
25.6155
78 03.9695
81.8 86.8
07.7775
87. 537%
07. 647%
97.775 7737
98.007
88 605
85.1995
70.6335
75.00 9074
70. 635
45.09%
08 08.04%
74.7896
78, 639
79.833
07.07%
07. 83%
88%
903
98.38 13%
88 505
85.015
68.89%
79. 63%
09. 84.8
70.9%%
13%
82.53%
81.07%
Table A A13: Position finetuning top-1 accuracy across multiple percentages of varying training instances
texin eanonica top...
canenieal tep
val. diveree
Rotstion
tap 0.50
Ladin peup Lo sayy
0.05
0:25
0.50
0.76
0.96
0.06
0.25
0.50
0.75
0.95
0.05
0.25
0.75
0.515
model
CLIPPeetniun
07.6195
07.0155
07. 3135
97.58%
07.957%
68. 5015
88.59%
85.10%
63.331%
72,2256
54.27%
17.015
02.0755
05.0855
63.45/%
MA EPreeriieed
05.87%
96.7575
0576
07.59%
97. 97.96%
75.905%
08:5254
62.0015
68. 575
60. 9955
27.6455
45.24%
50.875
54.1895
55.82%
65.197%
57.76%
53.29%
51.455
53.6555
59.35%
07 07.603%
98.35%
87.04%
57.04%
77,7856
77.78%
75. 75.0056
67.07%
T3.47%
08 0396
97 811%
80%
07.983
98 08.24%%
84.81%
81 18.4
51.485
81. 6656
75. 0.50
53. 13.2
53.51%
62.50%
61.45%
67.47%
Ro-Ner Nei50 Ne500
07. 02.6996
07. 4595
97 65%
07.61%
98. 98.18%
85. 85.10%
70/439
63.333 339
87.04%
85. 1995
S7.4556
50.8955
65.3995
70.0095
15:133
SmnCI MnnCLRI Pretrained
07.37%
07. 47.43%%
07. 535%
97. 07.74%
18 98.07%
87.04%
ST.00%
81.33%
75, 75.0056
30, 35.30%
35,87%
64. 64.34%
86. 7996
69.34%
WrTPretaaeeiik
07.94%
87 81%5
98%
9676
98.11%
88 50%
S7.015
199
17.18 77.88%
16. 0055
40. 1356
62. 6696
68.50 5396
11.3225
75366%
07.68%
97.6
98.27%
74.00%
38.75% 70.74%
68.1996
71.51%
73.76%
iDoc Peteettoiookkkk
4974
5375
97.75%
80%
77 77.78%
5256
96.30%
08% 22%6
7000005
08.00%
97.
96%
98.01%
08.19%
88.3 58.30%
87.04%
83.53%
85.10%
72 2256
38. 866
02.35%
08.1555
71.9035
73.84%
Table A14: Pose finetuning lop-1 accuracy across multiple percentages of varying training instances
train cwnontcclll..cc..m....oo.......
cemontioal
vol. diverse Spot lue top.
prop vary
0.25
0.95
0.05
0.95
0.25
0:95
Cl.IPPatriind
07. 07.06%
87.13%%
40%
97.6
07. 07.58%%
50.23 50.7.1%
58.50%
67.01%
57.04%
81. S1.185
19. 19.9995
70.0655
87.30%5
82.48%
MARPastnined
07.04%5
96.6799
07.22195
97.113%
97.55%%
79.655%
77.78%
72.22%
74.00%
68. 5255
30.1095
$4.6356
00.7455
06. 06.7855
69.3495
ML P.PPMeeePeeraaanot
07.32%
96 06.9
07. 207%
91 31.3
97.20%
65.04%
53.33%
71.78%
70.02%
22. /2.2255
15.8.%
65. 65.0%%
70.92%
76. 83.8895
75.00%%
MI. Pnttaannrt IV
07.80%
87.85%
9979
07.599%
98.125
88 3055
87.015
81.68%
75.03%
79. 8358
49.8155
75.15%
75.2395
79.0195
82.35%
08.02%
07 07.9995
07.98%
98.28%
30%
57.04%
50.33%
$1.45%
77.78%
43. 45.48%
67.72%
74.63%
76.90%
75.837
08%
98.24%
90.74%
S7.045
77.78%
70. 13.00%6
54.55%
7L.89%
76.7096
03.03
86%
SimCT Postrnieed
07. 000
01.
725
97.81%
98 17%
90 007%
70.30%
85. 19%
77.18%
74.07%
45. 16.589
10.30 94.96%
79.072 01/0
78 0695
81.75%
07. 07.6885
07.9095
NuN
98.00%
98 98.37%
94.44%
58.099
NaN
$1,48%
$1.48%
54.38%
70.94%
NaN
52.5355
83.943
VrTPremamettk
07.77%
07 68%
07 94%
95 107%
98. 24%
12 50%
85. 105
71.78 71.78%
78 15%
$1.45%
50. 49%
72.5%%
76 76.38%
79 7996
82.10%
07.9195
07. 8895
97. 7676
97.58
98 08.08%
88 50%
81.48%
79.63%
70.305 30%
74.07%
48.07%
09.24%
75.00 0155
79.4495
80.813
Borr Pettainnoi211
98.20%
87.87%
07.96%
90.74%
80.33%
92.00%
79.63%
83.33%
49.39%
42.42%
80.67%
83.00%
85.02%
Table A15: Spot hue finetuning top-1 accuracy across multiple percentages of varying training
instances
36

---

Published in Transactions Machine Learning Research (04/2023)
train canontical top. DeCuEECY
canocitcal top
vol. diverse. Soale.
top aoccurcyyy
train prop to swy
0.85
0.25
0.50
0.75
0.95
0.35
0.25
0.50
0.75
0.45
0.05
0.25
0.50
0.75
0.95
1000
07. 07.05%
87. 07.095%
3975
07.63%
97.96%
50.715%
90.745%
67.0045
63.535
79. 0315
45.4755
06.5775
72-1855
75.055
75.05%
96.17%
95.647
97.407%
97.9195
31.49%
70.3756
70.3375
70.32%
53.70%
25. 25.7076
51.7356
00.8556
6224495
64.47%
ML PretttePeettttdiikkkkk
10.2455
07.8055
87.95%
98.377
55.19%
72.22%
77.7856
46.022
73,0556
76.0156
77.88%
Res Netail betralnedlk
07. 17.888
07.94%
07.95%
07.20%
98 2425
57.04%
S5.19%
$1.455
75. 75.53%
70, 72.63%
44.47%
62.96%
71.5356
73.81%
73.805
Res retrateedd
07. 07.4
07. 387%
07 8475
01.72%
98. 18%
90. 90.745
70.6335
S1 S1.885
00.0355
79. 8356
19.377%
88, T296
70.8955
76, 18.8596
19.18%
Sind1 LLt Paeinined
07. 57.07%
07.6455
60%
07.5 07.68%
08 107
88 50.3015
58.53%
68.53%
63.31%
74,445
42.25%
65.00%
71,8556
T4.8995
76.2536
07. 8076
71,855
17.40%
78.46%
ViTPest
07.805
08.04%
50.74%
83,83%
0355
45.14%
71,845
78.0055
07.889
07.7575
97.
97.97%
98. 98.12%
88. 88.80%
57.04%
57.04%
$1,65%
76, 90%
48. 75%7
65.52%
72,8556
78. 1055
800043
Table A16: Scale finetuning top-1 accuracy across multiple percentages of varying training instances
trxin. conmicel top.
1 seturary
vol. canmnica Iop
val. diverns
Eacleground path tep.
lenin
6.05
0.25
0.50
0.75
0.95
0.00
0.25
0.50
0.75
0.45
0.05
0.25
0.50
0.75
0.95
moodel
CLIPPretramed
96. 2996
77.00%
007.22%
87.8815
58.015%
90.44%
8155
15.999
-57.04%5
85%
56, 79.7955
67.24%
76.25%
79.79%
80-92%
MAEPestriiood
96.81%
97.38%%
81.80%
72.22%
77.78%
72.2275
02.965%
32.04%
68.5675
07.63%
57.04%
72.22%
ML.PMiisr Patrnisnn
98.10%
08.095
06.33 35.3355
96.715
88.8995
85.1995
79x6395
51.8458
72.0075
81.10%
RenNeetiPPetttnedd
97. 715
77.7
07.95%
Ds. 0176
92.30%
02 3655
92.309
83.3326
63.08%
703075
73.7856
71.72%
I7 3856
00.%
07.725
86. 05.5555
98.515 515
68.2975
K3.3556
87.0155
81 48%
85. 1996
50.8 6756
71.50%
76.91%
79.1856
85:263
StaCCLPP LRPeetrateed
08.35%
17.31%
07.77X
07.53%
58.00%
77. 77.78%
87.04%
$8.899 89%
90.747
83.335
-1.22%
61.47%
69.94%
76.9%%
79.07%
VIT Pretmoinedlss
1256
07.80%
7356
4.%
00.74%
4694
85.3596
11.3665
75.75%
8439236
87.04%
76.50%
07.0055
97.91%
07.77%
97%
97.00%
08.52%
6.%
90. 74%
65.1995
83.355
73.115
766625
83.6688
82.005
Table A17: Background path finctuning top-1 accuracy across multiple percentages of varying
training instances
J All Factor Gaps
We also study the setting where all factors vary during training. Figures A21 and A22 we show the generalization gaps when all factors vary for lincar evaluation and finetuning.
K Class generalization
In addition to the finetning results, we include here lincar evaluation results for class generalization gaps [A18] We show generalization gaps across classes by comparing the similarity of classes seen during training to held-out classes in Figure A23
L Cross factor effects
We study the effect of varying a factor on the generalization gaps of other factors. In Figures A24 and A25 we show the slopes of the generalization gaps as the number of varying training instances increases during training. We see how varying one factor can also close the robustenss gap of other factors. We also show normalized versions of these plots in A26 and A27
37

---

Published in Transactions Machine Learning Research (04/2023)
model
Position gap
Pose gap
Lighting color gap
Size gap
Background gap
Average gap
CLIP
-41.69
16.51
-43.73
-41.51
-48.09
-44.31
MAE
-3.32
-6.64
-7.69
-5.68
-14.61
-7.59
MLPMixerIk
-39.07
-43.70
-43.42
-36.32
-47.79
-42.06
MLPMixer21k
-43.07
-57.34
-42.90
-44.84
-52.09
-18.05
ResNet50-1k
.4.997
-51.06
-43.84
-42.18
-55.67
-47.54
ResNet50-216
46.34
-56.89
-44.06
-47.26
-57.81
-50.47
ViT-1k
-47.03
-58.69
-45.24
-48.67
-52.77
-50.48
ViT-21k
-50.41
-56.94
-49.17
-49.76
-55.23
-52.30
iBot-1k
-46.90
-61.71
-46.35
-51.56
-59.88
-53.28
iBot-21k
-53.00
-64.94
-50.99
-56.05
-64.83
-57.96
Average
-41.58
-50.44
-41.74
-42.38
-50.88
-45.40
Table A18: Linear eval class generalization top-1 accuracy gaps: shows validation top-1 accuracy difference between classes (27 randomly selected) seen with diversity and those not.
M Effect of class similarity on models' ability to generalize variation across classes
We study the effect of class similarity by measuring the generalization gaps per class for each factor relative to the class's similarity to the nearest class seen varying during training. Tf models' are able to generalize variation across classes, we might expect models generalize variation better when the class is similar to one seen varying during training. In Figures A28
A28
A32
and A30
N Experiments details
Tables and 16 show results for the best model after 10k steps of training with adam on log scale learning rates (Ie-2 1e-6) cross validated on canonical top-1 accuracy for validation images.
38

---

Published in Transactions Machine Learning Research (04/2023)
Generalization when Background varies training
Position
Lighting
eccurscyyy drop
-0.4
-0.8
percent diverse Background training samples
percent diverse Background training samples
percent diverse Background training samples
top- accurecy dhop
Size
Background
CUPPhetrained
NAEPatraimed
-0.2
Rechie Selerrietki
SMCLF Strceeottiien ained
ViTrerainedlk
0.
0.5 percent of diverse Background training samples
percent of diverse Background training samples
Botthetramdd21e
Generalization Pose varies training
Position
Lighting
top-1 dhop
-0.2
-0.4
percent of diverse Pose training samples
perzent civerse Pase training samples
percent of diverse Pose training samples
accuracy drop
Size
Background
CUFFrctroined
-0.4
Bestur Schwerinnedhic
-0.6
ViFrerraneddk
diverse training samples
percent Pase training
BotPhatramed21
Generalization when Size varies in training
Position
Pose
Lig Lighting
top-1 eccurecy
-0.
-0.4
8.
0.5
0.5
percent of diverse Size training samples
percent of diverse Size training samples
percent of diverse Size training samples ies
eccuracy drop
Background
CUPPrctrained
MöTReetrained
NI Wttieciieett
Reshir Sthveniieedd
Shatrthar
Vitheosneettk
ViTherainea2kk
percent diverse training samples
percent diverse Size training samples
ButPetrainsdk
Figure A17: Linear Evaluation Effect of Variability in Training (part 1)
39

---

Published in Transactions Machine Learning Research (04/2023)
Generalization when Lighting varies in tralining
Position
Pose
Lighting
Lighting training samples
of diverse Lighting training samples
Lighting training samples
accurstyy drop
Size
Background
CUfFretrainad
MARPhetrained
-07
MUPtisar
SmC SPretratned
ViTrerained221
percent diverse lighting training samples
percent of diverse i lighting training samples
BotPeetrahedkk
Generalization when Position varies trainiing
Lighting
top-1 accuracy drop
percent diverse Position training samples
percent of diverse Position training samples Background
percent of diverse Position training samples es
Size
CUIfhctrained
MAEPectroined
top-1 eccuracy
Reskec50hetrnned22k
Spretratted
-0.0
0.5
6.5
percent diverse Position training samples
percent of diverse Position training samplee
Figure A18: Linear Evaluation Effeect of Variability in Training (part 2)
40

---

Published in Transactions Machine Learning Research (04/2023)
Generalization when Background varies in training
Position
Lighting
eccurscyyyy drop
-2.0
percent diverse Background training samples
percent diverse Background training samples
percent diverse Background training samples
top- accuracy dhop
Size
Background
CUPPhetrained
NAEPatraimed
-022
Rechie Selerrietk
-066
SMCLF Sorrspooaiiie ained
ViTrerainedlk
0.
0.5 percent of diverse Background training samples
percent of diverse Background training samples
Botthetramed2ee
Generalization when Pose varies training
top-1 dhop
Position
Pose
Lighting
percent of diverse Pose training samples
perzent civerse Pase training samples
percent of diverse Pose training samples
accuracy drop
0
Size
Background
CUFFrctrained
-07
Bestur Schwerinnedhic
-0.6
ViTrerrannedk
diverse training samples
percent civerse Pase training
Botthatramed21e
Generalization when Size varies in training
Position
Pose
Lighting
top-1 eccurecy
-3.0
8.
0.5 percent of diverse Size training samples
0.5
percent of diverse Size training samples
percent of diverse Size training samplcs ies
top] eccuraty drop
Background
CUPPrctrained
MöTReetrained
NI Wttieciiteti
-0.2
Reskir S0hveniinedd
-0.4
Vitheosneettk
ViTherainea2kk
percent diverse Size training samples
percent diverse Size training samples
BotPeetrinndkk
Figure A19: Finetuning Effect of Variability in Training (part 1)
41

---

Published in Transactions Machine Learning Research (04/2023)
Generalization when Lighting varies in tralining
Position
Pose
Lighting
-0.2
-0.6
-0.8
0.5
Lighting training samples
of diverse Lighting training samples
Lighting training
accuraty drop
Size
Background
CUfFretrainad
MARPhetrained
-07
-0.6
SmC
SPretratned
ViTrerained221
percent diverse lighting training samples
percent of diverse i lighting training samples
BotPeetrahedkk
Generalization when Position varies trainingg
Lighting
top-1 accuracy drop
-0.5
percent diverse Position training samples
percent of diverse Position training samples Background
percent of diverse Position training samples es
top-1 eccuracy drap
Size
CUIPhctrrined
MAEPectroined
NUPtiwerrredrannedl..
Reskec50hetraned22k
SPretratted
0.5
6.5
percent diverse Position training samples
percent of diverse Position training samplee
Figure A20: Finetuning Effect of Variability in Training (part
42

---

Published in Transactions Machine Learning Research (04/2023
CLIPPretrained
Background gap
MAEPretrained
MLPMixxrPPeeraineedkk
MLPMiixerPretrrinee22k
Size gap
ResNett0PPrtraineedk
ResNeet5PPretriined22k
SimCLRPretrained
ViTPretrained1k
Lighting gap
VITPretrained21k
IBotPretrained1k
|BotPretriine221k
Pose gap
Position gap
-30
-20
-10
top-1 accuracy drop when all factors vary (linear eval)
Figure A21: Generalization gaps when all factors vary during training with linear evaluation
43

---

Published in Transactions Machine Learning Research (04/202)
CLIPPretrained
Background gap
MAEPretrained
MLPMixxrPPretrained11
MLPMiixerPrttraiee22k
Size gap
ResNet50PPetraineddk
ResNee550Prtrrinee222
SimCLRPretrained
VTTPretrained1k
Lighting gap
VITPretrained21k
iBotPretrained1k
iBotPretrained21k
Pose gap
Position gap
-30
-20
100
top-1 accuracy drop when all factors vary
(finetuning)
Figure A22: Generalization gaps when all factors vary during training with finetuning
Accuracy gapp
-50
622
0.8
Class similerity
Figure A23: Generalization gaps are smaller only for classes very similar to those seen during training and worse for classes that are very dissimilar. We plot the generalization gaps similarity the nearest class seen varying during training increases using the mean accuracy gap with error bars indicating the standard error.
44

---

Published in Transactions Machine Learning Research (04/2023)
varying factor
Position gap
Pose gap
Lighting gap
Size gap
Background gap
Position
0.385894
0.249188
0.411196
0.350276
0.287304
Pose
0.383425
0.369503
0.400835
0.308550
0.280720
Lighting
0.398418
0.243639
0.385000
0.330450
0.281797
Size
0.385311
0.245047
0.395171
0.352747
0.294040
Background
0.343635
0.209684
0.354958
0.304687
0.300778
Figure A24: Spill over effects: shows the average slope across models when a given factor varies during linear evaluation
varying factor
Position gap
Pose gap
Lighting gap
Size gap
Background gap
Position
0.448903
0.262276
0.475593
0.419783
0.370703
Pose
0.442200
0.454216
0.469521
0.372108
0.353449
Lighting
0.480997
0.277407
0.444130
0.415490
0.384728
Size
0.483358
0.274840
0.487531
0.451954
0.427255
Background
0.425475
0.242605
0.437238
0.406613
0.405921
Figure A25: Spill over effects: shows the average slope across models when given factor varies for finetuning
varying factor
Position gap
Pose gap
Lighting gap
Size gap
Background gap
Position
1.000000
0.674386
1.068042
0.992995
0.955206
Pose
0.993602
1.000000
1.041130
0.874707
0.933316
Lighting
1.032454
0.659368
1.000000
0.936791
0.936894
Size
0.998488
0.663178
1.026418
1.000000
0.977599
Background
0.890490
0.567474
0.921968
0.863755
1.000000
Figure A26:
Normalized Spill over effects: shows the average slope across models when a given factor varies during linear evaluation. Normalization is across rows by dividing the diagonal value isolate how much more given spill-over effect than the intended.
45

---

Published in Transactions Machine Learning Research (04/2023)
varying factor
Position gap
Pose gap
Lighting gap
Size gap
Background gap
Position
1.000000
0.577425
1.070841
0.928819
0.913239
Pose
0.985068
1.000000
1.057171
0.823333
0.870733
Lighting 1.071495
0.610738
1.000000
0.919320
0.947789
Size
1.076755
0.605087
1.097721
1.000000
1.052556
Background
0.947811
0.534118
0.984481
0.899679
1.000000
Figure A27: Normalized Spill over effects: shows the average slope across models when given factor varies for finetuning. Normalization is across rows by dividing the diagonal value to isolate how much more given spill-over effect than the intended.
Position accuracy as class similarity changes
0.6
0.4
0.2
accuracy gap
C30
0
-0.2
-0.4
-088
0.2
0.4
0.6
0.8
similarity
Figure A28: Position gap as class similarity to nearest neighbor increases to classes seen varying during training.
46

---

Published in Transactions Machine Learning Research (04/2023
Pose accuracy as class similarity changes
0.2
0
-0.2
accuracy
-0.4
-0.6
-0.8
0.2
0.6
similarity
Pose gap as class similarity to nearest neighbor increases to classes seen varying during training.
Figure A29:
47

---

Published in Transactions Machine Learning Research (04/2023
Size accuracy as class similarity changes
0.6
0.4
0.2
accuracy gap
0
-0.2
-0.4
-066
0.8
-1
0.2
similarity
Figure A30: Scale gap as class similarity nearest neighbor increases to classes seen varying during training.
48

---

Published in Transactions Machine Learning Research (04/2023
Background accuracy as class similarity changes
0.2
0
accuracy gap
-0.2
-0.4
-0.6
-088
0.2
0.6
similarity
Figure A31: Background gap as class similarity to nearest neighbor increases to classes seen varying during training.
49

---

Published in Transactions Machine Learning Research (04/2023
Lighting accuracy as class similarity changes
0.5
accuracy gap
0
-0.5
-1
0.2
similarity
Figure A32: Lighting color gap as class similarity to nearest neighbor increases to classes seen varying during training.
50


Reviews:
[{"review_id": "nEtMVNPYBL", "reviewer": "Reviewer_WbwZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses the important topic of robustness / generalization of current vision architectures to natural image variations. It provides strong empirical evidence that current architectures lack generalization wrt certain visual factors. The work is highly relevant for the CV community and raises concerns in using current vision architectures in safety critical use-cases. The paper is well written and structured.\n\n\nWeakness (minor):\n\n* In some figure's notations are rather small and should be enlarged to improve accessibility (Fig 2, 3, 4, 5, 6, 7)\n\n* The following reference addressing a similar question on largely the same visual factors should be added / discussed:\n\nDiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of\nShortcut and Generalization Opportunities, https://openaccess.thecvf.com/content/ICCV2021/papers/Eulig_DiagViB-6_A_Diagnostic_Benchmark_Suite_for_Vision_Models_in_the_ICCV_2021_paper.pdf\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work evaluates common vision models (image classification) including novel architectures (CLIP, SSL) wrt. their robustness under variation of natural visual factors (object pose, position, scale, color, background). The work highlights a category of significant generalization gaps for all current approaches, showing poor generalization to out-of-distribution cases. The work also evaluates other interesting generalization aspects, such as generalization across classes, under different amounts of factor variation, with / without fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper is already in a very good state and I would only have some minor change requests. Please see the minor weaknesses above.", "broader_impact_concerns": "The work sheds a light on the capabilities and in-capabilities of current vision models and emphasizes the need for more work on model generalization and data biases.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CbPDeJwiGN", "reviewer": "Reviewer_AUEY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A fairly thorough study, including a reasonable selection of modern vision models\n2. The conclusion about limited robustness of modern models is interesting, the study of transfer of invariances across classes and instances is interesting too\n3. It is interesting that MAE seems to behave substantially different from other models in terms of robustness\n\nWeaknesses:\n1. While I generally appreciate analysis papers, I wonder how exactly are the results on the proposed synthetic dataset relevant to real applications of image recognition models. It is no big surprise that the models do not work very well on out-of-distribution samples (like, say, upside-down objects on weird backgrounds), but do they even have to if such samples never appear in real applications? One argument could be that human visual recognition is invariant to all these variations, but then it’s necessary to explicitly point this out and support with experiments. Moreover, I would recommend to add evaluation on ObjectNet - a real-world dataset also concerned with measuring robustness of models - and see if conclusions there correlate with conclusions on the proposed synthetic dataset.\n2. I am confused by the fact that in many plots the gap is not zero when I would expect it to be zero. For instance, in Figure 4 how come many curves don’t reach 0 on the right, at 100%? Perhaps because the highest evaluated percentage is 95% not 100%, but that’s weird and I’d like to see 100% too just to be sure. Another example is that in Figure 5 some curves go above 0 on the right, how can that be? These results make me doubt the validity of all conclusions, perhaps there is a lot of noise in the numbers?\n3. From the numbers provided in the supplementary material, it seems the models overfit substantially during fine-tuning. It therefore seems that the specific fine-tuning protocol can have a large impact on the results and the conclusions of the experiments. However, I did not see a discussion of this in the paper. \n4. It would be great to add results on which specifically poses, positions, sizes, colors, backgrounds are easy or difficult for the models. The synthetic nature of the proposed dataset should allow to generate such results? Would be interesting to see.\n5. “For example, while position and lighting have gaps of -5% and -2% respectively with no variability (Table 1), their gaps when 5% of instances vary are nearly -40% (Figure A14).” - this is very strange, any explanation? Maybe training data imbalance during fine-tuning?\n6. In comparison to a related paper by Abbas & Deny, the authors state that they analyze “a larger catalog of more recent SoTA models and architectures” - does not seem quite accurate: while some of the models studied in this paper are not in Abbas & Deny, the opposite is also true. Afaict, Abbas & Deny study 38 models, while this submission - 11 models, is it right?  Then the statement about “larger catalog” does not seem quite correct.  \n    6a. In particular, Abbas & Deny show that large models and in particular EfficientNets trained with Noisy Student are quite a bit more robust than other models, it would be very valuable check this on the larger dataset proposed in this submission  \n7. Smaller issues  \n    7a. Missing reference “On Robustness and Transferability of Convolutional Neural Networks” (Djolonga et al.)  \n    7b. It seems confusing that for a generally poorly performing  model the gap is small and therefore it seems that the model is more invariant. Perhaps relative gap would be a better metric?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes the invariances of a range of modern image recognition models on a synthetic dataset showing rendered 3D objects against random 2D backgrounds. Factors of variation include object pose, position, size, color, as well as the background image. The paper shows that the models are not really robust to object transformations. Moreover, the paper shows that transfer of invariances across classes is difficult.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Fix the issues pointed out in \"weaknesses\". in particular:\n1. Add evaluation on ObjectNet and potentially human results (or otherwise explain the relevance of the proposed analysis)\n2. Clarify the strange seemingly noisy results in the paper\n3. Clarify the fine-tuning/linear protocols and evaluate their effect on the conclusions regarding robustness\n4. (nice-to-have) Add more detailed results on performance depending on specific values of different transformations\n5. Add missing reference(s) and polish writing, in particular related work w.r.t. Abbas & Deny\n6. Add more models, in particular larger models like in Abbas & Deny", "broader_impact_concerns": "No concerns, the paper studies robustness of models, which if anything could be a positive, contributing to our better understanding of the functioning of the models and potentially their biases.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ih3wDMwHw3", "reviewer": "Reviewer_AUUo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a thorough set of experiments, confirming the observed behavior across a wide range of architectures and pretraining methods, and evaluating multiple controlled transfer training conditions.  Some interesting observations are presented, in particular the measurements showing the rates at which generalization improves when increasing variation (how much diversity is needed to improve generalization?), and the effect varying one factor has on other factors.  The effects of pretraining dataset (larger is more robust) and architecture (most are similar behavior) are more widely recognized, but this offers further corroboration of these as well.\n\nAt the moment, however, I found many of the descriptions of data creation and evaluations vague, and details on the experimental setups are missing.  This makes the paper a somewhat hard to follow, as I was guessing at a lot.  More concrete figures and examples describing the data sampling and experimental setups would help.  \n\nIn particular:\n\n* I think it could be clearer including both absolute performance numbers as well as the \"gap\" measurement.  Currently, the size of the gaps is not put in context relative to the base point.  Plain accuracy numbers are reported here and there in the text, but it would help to be able to view and study these along with the relative drops.\n\n* More precisely defining measurements and definition of \"gap\" might help as well.  While a very formal description in set notation is probably not needed, an algorithmic or mathematical description that more precisely includes which objects and factors are used in which averages would be helpful.\n\n* The training procedure should be more concretely defined.  While the overall setting of linear classifier or end-to-end fine tuning is clear, exact setup and details are not.  Most importantly, the data sampling procedure is not very clear yet.  How are objects selected, and how many?  How is the canonical factor value assigned?  How long is each model trained (how many steps/samples) and how does this compare to the number of unique samples in the new training sets?  In particular, does adding or removing variation affect the number of training samples?  Algorithmic description and example images in figures describing what values/images are sampled in each experimental setting would help.\n\n* What image augmentations were applied at training time --- just color transforms and translation?  Or also scale, in-plane rotation, or others?  Some of these might actually approximate more of the 3d transforms and change the amount of robustness.  Is lack of robustness from object size changes even after similarly sized scale augmentations are applied to the images?  How much does this using vs not using this planar augmentation change robustness?\n\n\n\nThe paper is also very related to Kumar &al 2022 https://arxiv.org/pdf/2202.10054.pdf , which analyzes out-of-distribution robustness in the context of transfer learning with linear, end-to-end, and hybrid trained classifiers.  I think this paper should include a discussion on how these work relate, and relations between their conclusions.\n\n\nMore detailed questions:\n\nEntanglement of position and lighting:  In fact there is an entanglement created by 3D rendering.  When the object moves in world space and the light source remains fixed, this has the effect of (a) moving the object in the image plane, (b) moving the object relative to the light source, and (c) changing the result of perspective deformations by moving the object in the camera frustum.  I also noticed in the appendix images, that changing position seems to increase the size of the object between right and left.  Do these relations in the rendering pipeline account for the entanglement observed in performance robustness between factors?\n\nFig 1:  \"chance seen varying during training\" and \"half seen varying during training\":  I'm not sure what exactly these mean --- what is the selection procedure, and where does this chance fit in to this?  For example, is 50% the fraction of objects where all values are varied?  Or the fraction of times a random value is selected vs the canonical value, for all objects?  Or is it used in another way?  The use of \"varying\" e.g. in sec 5.1 is vague still in the text --- what is the sampling procedure?\n\nSec 5.2:  Fig 4 vs Fig A14:  The lines in A14 all trend upward, and don't clearly show the behavior described in the text where performance is worse than with zero variation.  To clearly demonstrate this, it would be best for the plot to show a downward dip followed by upward recovery.  In particular, if results at 0% variation are included, and a few more samples between 0% and 5% (e.g., 1%, 2%) that could show the down-up behavior.  In addition, the 0% values should coincide with the values for no variation in Table 1.\n\nFigures 2, 3:  Captions or text should explain what ranges are represented with box plots (mean, 50 and 90 percentiles?  or other ranges?)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes a controlled study of robustness in generalization to common variation factors for objects --- in particular, pose, position, background, lighting and size (relative to background) --- in the context of transfer learning.  3D models were rendered against pre-selected backgrounds in different configurations of these variables, with one configuration selected as the \"canonical\" representation used when not applying variations.  The key metric studied is the difference in average classification accuracy between images with varying factors, versus images with only the canonical values (\"generalization gap\").  Different training-time settings are compared, to study how training variation affects test-time generalization.  Training setups that use all variations, no variations, or variations of only some factors but not others are evaluated and compared in order to characterize generalization abilities, as are experiments on pretraining data and model architecture.  Several conclusions are drawn or confirmed, including (a) larger/diverse pretraining (imnet21k vs 1k) helps generalization, (b) more diverse training samples helps generalization not only in the factors varied, but sometimes others as well (however, see question below on this), (c) most current classification architectures show behavior in generalization gaps, (d) improving generalization on one class does not usually improve generalization on other classes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* discussion/comparison relative to Kumar &al https://arxiv.org/pdf/2202.10054.pdf\n\n* clearer descriptions of data sampling, transfer training and experimental procedures\n\nBoth of the above are necessary changes", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I believe this is important kind of work, since it furthers our understanding about the limitations of these models. I would encourage the authors to include some of the unresolved discussions mentioned by the reviewers (e.g. Kumar et al) in the camera ready manuscript, but also to shorten the main text to the page limit.
Decision: Accept (poster)