SlowGuess/NeurIPS_2023_OCR huggingface dataset:
respect to orientation of the camera yaw. We see that there is little to no equivariance to it, suggesting that image embeddings are more predictable when changing background than the camera yaw. Note that foundation models representations belong to the hypersphere, yet our measurement of equivariance as parallelism (measured with cosine similarity) relies on Euclidean geometry. Still, cosine similarity is a starting point to showcase how PUG: Animals can be used to study models'
Figure 11: Additional image equivariance results with respect to camera yaw.
representational spaces. This could also explain the higher equivariance values of text representations: since textual captions follow the same template, embeddings might be close to each other (relative to image embeddings distances). In this case the hypersphere locally behaves like an Euclidean space [Lee, 2000], for which the Euclidean geometry is better suited. We leave for future work the exploration more complex equivariance metrics potentially based on spherical geometry to study foundation models' representational spaces. Studying model's representations is key to better understanding and improving them Bouchacourt et al. [2021], Xie et al. [2022], Ushio et al. [2021], Lenc and Vedaldi [2018]. Our study showcases that PUG: Animals advantages (rich diversity of factors, knowledge of their values, control either one factor at a time but also all together) make it a great dataset to study state-of-the-art models representational properties.
C.2 Classification with held out sets
In section 3.2, we study how one can leverage PUG: Animsl to study OOD generalization in two settings: 1) Generalization on unseen factors 2) Generalization on unseen combination of factors.
For this experiment, we use PUG: Animals with held out sets. Typically, we random select a number of number of animals (0, 10 or 20) within our 70 assets. Then for the remaining animals we decided to remove from PUG: Animals a number of background, object size or object texture. This give us a training set. Then the images that were excluded from this training set are put as a test or held out set in which we measure the performance of a model. This model is typically a Resnet50 trained for 100 epochs with AdamW as optimizer with a batch size of 2048.
C.3 Robustness of SOTA models additional details
In addition to evaluating robustness for the models in the main paper, in Table 9 we provide an analysis of several additional models including the recent self-supervised DINOv2 model as well as BLIP a contrastive vision-language model. Parenthesis indicate the pretraining dataset: ImageNet 21k, LVD-142M, JFT 300M, LAION 400M, and LAION 2B. Models without parenthesis are pretrained on the standard ImageNet-1k dataset. For ResNet models we use the publicly available pretrained checkpoints from the Timm package based on the training recipe from Wightman et al. [2021]. For the vision transformer models and Swin we use the pretrained models from the Timm package with patch size 16 for ViT and Swin Base with a patch size of 4 and window size of 7. For the BiT model we use the pretrained checkpoint trained on Google's JFT 300M dataset from the Timm package with a ResNetv2 101 architecture. For DINOv2, we use the officially released repo to evaluate the base ViT architecture trained on 132 million samples [Oquab et al., 2023]. For BLIP we use the checkpoint available in HuggingFace and evaluate the model using zero shot classification via the prompt 'This is a photo of a [ ]'. We evaluate CLIP model variants in a similar zero shot fashion and rely on OpenCLIP's implementation. The parenthesis indicate the pretraining dataset size from the LAION dataset. We report the average accuracy as each factor (see columns of Table 9) varies.
We also measure the relationship between standard in-distribution accuracy and robustness based on the accuracy as each factor in PUG:ImageNet varies. We measure Pearson's correlation coefficient between ImageNet accuracy and accuracy for each factor in Table 10. We find no statistically significant relationship between standard classification accuracy and factor robustness.
C.3.1 Performances
To understand if the differences in performance between the real ImageNet and our PUG dataset is caused by a sim-to-real gap or by the factor of variations, we show below the zero-shot accuracy obtained with a pretrained resnet101 for each class in PUG: ImageNet. There is only 3 classes for
PUG: ImageNet
ImageNet	Camera_Yaw	Camera_Pitch	Camera_Roll	Object_Yaw	Object_Pitch	Object_Roll	Object_Scale	Object_Temperature	Scene_Light	Background
ResNet50	81.5	38.1	33.1	26.9	38.0	23.6	22.9	35.7	27.0	13.6	29.5
ResNet101	82.3	43.4	35.9	29.4	45.1	26.7	25.6	39.7	31.1	14.1	32.8
BiT (JFT300M)	80.3	40.5	32.3	26.0	42.1	23.6	22.8	37.3	23.4	6.3	20.5
DINOv2 (LVD-142M)	84.5	45.6	41.1	37.4	47.5	28.8	28.5	43.1	35.0	6.1	30.9
Flava (PMD 70M)	75.5	31.7	23.4	17.6	30.8	17.6	15.4	30.5	24.2	7.8	21.9
Swin	83.6	56.0	45.6	41.8	56.9	35.3	34.2	52.9	40.1	19.1	42.0
ViT-Base	84.3	37.5	34.3	31.7	38.0	21.8	20.5	33.0	28.5	4.1	26.6
ViT-Large	85.8	52.2	40.4	37.1	52.4	30.4	28.4	46.4	42.9	8.9	34.6
BLIP (100+M)	-	0.5	0.4	0.5	0.8	0.6	0.7	0.9	0.7	0.7	0.7
CLIPViTB32 (2B)	66.6	44.0	31.5	24.1	43.8	24.8	21.8	42.2	34.7	3.3	26.0
CLIPViTB32 (400M)	62.9	41.7	30.2	22.1	41.6	23.8	20.9	40.1	34.4	5.7	24.4
CLIPViTL14 (2B)	75.3	49.7	34.9	28.2	50.3	26.3	25.3	46.8	39.4	4.8	30.8
CLIPViTL14 (400M)	72.8	52.3	39.8	35.7	51.8	29.0	26.4	50.6	41.1	4.3	33.0
Table 9: Robustness measured by average accuracy across factors. We report zero shot classification accuracy for BLIP, Flava, all CLIP models. The pretraining dataset is indicated in parenthesis next to each model name with ImageNet-1k being the default unless otherwise indicated.
factor	correlation	pvalue
Object Pitch	0.29	0.45
Camera Roll	0.61	0.08
Camera Pitch	0.53	0.14
Camera Yaw	0.12	0.76
Background	0.43	0.25
Object Yaw	0.18	0.64
Object Texture	-0.10	0.81
Scene Light	0.53	0.14
Object Scale	-0.05	0.90
Object Roll	0.45	0.22
Table 10: We compute the correlation between standard ImageNet classification and robustness based on the accuracy for each factor. We find no statistically significant relationship for standard classification and factor robustness.
which there is not a single configurations of the factors that lead to a correct classification. For all the other classes, there is always at least one configuration for which the network is correctly predicting the class. In that instance, we assume that if the objects in a given class are correctly predicted at least one time, the failures in predicting the correct class for the same objects with different factors is probably due to the changes of factors.
Zero-shot top-1 accuracy per class Soccer_Ball: 100.00, Pineapple: 95.62, Barrel: 95.00, Cellular_telephone: 89.45, Pomegranate: 88.12, Jack_O_Lantern: 85.94, Vase: 85.62, BirdHouse: 81.88, BasketBall: 81.25, Sewing_Machine: 80.94, Umbrella: 80.00, Washer: 78.75, Pool_Table: 76.88, Baseball: 76.88, Safe: 76.25, Cabbage: 75.78, Cofee_Mug: 75.31, Mask: 74.06, Broccoli: 73.44, Starfish: 72.50, Rocking_Chair: 71.25, Punching_Bag: 70.94, Chicken_hen: 69.38, WineBottle: 66.88, Gasmask: 66.56, Joystick: 64.06, Television: 63.44, Chest: 63.44, Elephant: 62.50, Bell_Pepper: 61.46, Cheeseburger: 60.62, Pay_Phone: 60.00, Tennis_Ball: 58.44, Jean: 56.25, Binocular: 55.86, Racket: 55.62, Motor_Scooter: 55.00, Hay: 54.06, Park_Bench: 53.44, Bookcase: 53.44, Zucchini: 52.08, Banana: 50.00, Sliding_Door: 48.75, Military_uniform: 47.50, Ambulance: 47.50, Pizza: 47.19, Tractor: 46.88, Dishwasher: 46.88, Cowboy_Hat: 46.35, Drum: 45.31, Typewriter_Keyboard: 44.92, Toaster: 44.69, Obelisk: 44.38, Laptop: 44.38, Throne: 43.75, Backpack: 43.44, Shield: 41.88, Artichoke: 41.80, Penguin: 41.56, Bathtub: 40.31, WaterBottle: 40.00, SpaceShuttle: 37.19, Bagel: 36.88, Bear: 36.25, Vacuum: 35.94, Radiator: 35.62, Shovel: 35.55, Refrigerator: 35.00, Running_Shoe: 34.38, Goldfish: 34.38, Crate: 34.38, Polaroid_Camera: 33.98, Table_Lamb: 33.75, Bulletproof vest: 33.75, Microphone: 33.12, Traffic_Light: 32.50, Carton: 31.25, Volley_Ball: 30.62, Vending_MACHINE: 30.62, Lawn_Mower: 29.38, Car_Wheel: 29.38, Harmonica: 28.12, Lighter: 27.50, Carousel: 27.34, Mailbox: 27.19, Airliner: 27.19, Butternut_Squash: 26.95, Sweatshirt: 26.56, Sock: 25.62, French_Loaf: 25.00, Dial_telephone: 24.61, Rabbit: 24.06, Remote_controller: 22.81, Modem: 22.50, Chain-saw: 21.35, Screwdriver: 20.31, Power-drill: 19.69, Electric_Fan: 19.06, HairDryer: 18.75, Purse: 18.12, Wallet: 17.50, Sunglasses: 17.50, Minivan: 17.50, Cat: 15.94, Microwave: 15.62, Candle: 15.62, Mushroom: 15.31, Dining_Table: 14.06, Ice_Cream: 13.75, Perfume: 13.44, Komodo_dragon: 13.44, Bycicle: 13.12, Wooden_Spoon: 12.81, JellyFish: 12.81, Canoe: 12.81, Radio: 12.19, Desk: 12.19, African_crocodile: 11.88, Hatchet: 11.25, Sandal: 10.00, Stonewall: 9.69, Burrito: 9.38, Palace: 9.06, Mouse: 7.50, Convertible: 7.19, Espresso-maker: 6.88, Can_Opener: 6.56, Jeep: 6.25, Fox: 6.25, Tile_Roof: 5.86, Street_Sign: 5.62, WarPlane: 5.47, Frog: 5.47, Wolf: 5.31, Whale: 5.00, Torch: 5.00, Soup_Bowl: 4.69, Great_white_shark: 4.38, Kangaroo: 3.91 Digital_Watch: 2.81 Skirt: 2.50 Computer_Keyboard :2 .50 Piano :1 .88 Manhole_Cover :1 .88 Bridge :0 .94 , Aircraft_Carrier :0 .39 Screen :0 .31 ,Locomotive :0 .31 , Submarine :0 .00 , Shirt :0 .00 , Loupe :0 .00
C.4 Additional PUG:SPAR experiments
Instead of using all the background environments presented in the main part of the paper, we also introduce a much simple setup in which we have a single background (thus we do not need the background information in the caption anymore). The background that we choose is the simplest one named "salt flats". This is also the background for which the retrieval accuracy is the highest across all the backgrounds. In table 11, we present the performances of several VLMs when using this single environment. We can observe a significant boost in accuracy for the single object detection task for which the best model achieve  $94\%$  accuracy (this value is to contrast with the  $78\%$  accuracy obtained across all the background). This show that VLMs are definitively not robust to background changes. However as in the previous case, when probing for the positional information, the performance of the model is still decreasing significantly. We also illustrate in Figure 12 very simple failure cases on the best model.
Caption	Texture	ViTP-22 (OpenCLIP)	ViTP-22 (OpenCLip 2B)	ViTP-14 (OpenCLIP)	ViTP-14 (OpenCLip 2B)	ViTP-14 (OpenCLIP 2B)	BLIP	XVLM	NeeCLIP
Default	57.81	75.78	91.41	88.28	94.53	94.53	64.06	78.91	67.19	64.84
“A photo of a [character]”	Blue/Red	48.44	54.69	71.88	70.31	75.00	84.38	42.19	53.12	48.44	48.44
Grass/Stone	39.06	45.31	67.19	68.75	76.56	78.12	39.06	56.25	45.31	50.00
“A photo of a [character] on the (left/right) of the picture”	Default	29.69	37.50	39.06	40.62	50.00	51.56	32.81	42.19	34.38	31.25
“A photo of a [character] on the (bottom/top) of the picture”	Default	29.69	23.44	45.31	43.75	46.88	45.31	23.44	34.38	34.38	23.44
“A photo of a [character] and a [character]”	Default	24.56	32.47	68.85	59.67	72.66	80.96	18.55	40.58	24.71	21.04
Blue/Red	10.84	18.55	38.57	30.66	34.28	51.56	9.86	10.94	3.12	8.01
Grass/Stone	9.38	18.16	31.35	30.47	30.18	47.85	6.84	11.33	5.96	8.98
“A photo of a [character] on the left and a [character] on the right”	Default	14.11	14.92	38.10	30.85	40.83	42.14	13.71	23.99	14.01	14.21
“A photo of a [character] on the bottom and a [character] on the top”	Default	12.00	15.42	34.07	35.48	44.05	45.26	10.99	26.51	14.62	8.17
“A photo of a [character] textured with [texture1] and a [character] textured with [texture2]”	Blue/Red	4.44	6.65	19.15	15.52	20.56	29.84	6.15	10.28	5.44	4.13
Grass/Stone	3.43	5.44	15.73	17.24	19.05	27.32	5.54	7.76	6.35	4.03
Table 11: Setup and zero-shot evaluation of CLIP models on PUG: SPAR with caption retrieval in a single environment. In contrast with the figure presented in the main paper, we present the result only using the salt flats environment. The motivation for this experiment is to showcase the failures mode of VLMs in a very simple setup in which the model robustness to background does not impact the prediction.
C.5 CLIP fine-tuning details
We utilize the OpenCLIP framework [Ilharco et al., 2021] for all our CLIP experiments. The ViT-B/32 model is used as the image encoder for all our experiments. The CLIP model we fine-tune is the OpenAI 400M pre-trained model ('ViT-B-32', 'openai') $^{10}$ . Fine-tuning on PUG: AR4T is done for 10 epochs on the 200K dataset while training is done for 2 epochs on the 1M dataset.
<sup>10</sup>NOTE: We do not perform any training on the proprietary 400M dataset from OpenAI. We strictly only use the pre-trained models released, and fine-tune them on our PUG datasets.
A very simple failure mode of the OpenCLIP ViT-G-14 (2B)
The caption predicted by the model is presented below each image (correct/wrong)
A photo of a Goldfish
A photo of a Goldfish on the left of the picture
A photo of a Crow
A photo of a Goldfish
Figure 12: Failures mode of a OpenCLIP ViT-G-14. Our PUG: SPAR dataset provides very simple images and captions and yet even large models are failing on them.
A photo of a Goldfish
The model does not understand positional information
The model does not understand compositionality
A photo of a Orca
A photo of a Goldfish and a Orca