SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show in Figure 4 (right) results for equivariance with respect to the background. Plots for equivariance to texture and size are in Figure 10. Looking at Figure 4 results (right side, top row), we see that the foundation models' image embeddings present high equivariance to background  $(0.78 \pm 0.04$  on average over models). There is also (see Figure 10a) small image equivariance to texture  $(0.15 \pm 0.04)$ , but almost no equivariance to size  $(0.06 \pm 0.02)$ . Text equivariance is high with respect to background (average of  $0.87 \pm 0.03$ ), but is also strong for size and texture  $(0.71 \pm 0.11$  for size and  $0.81 \pm 0.03$  for texture, see Figure 10b) suggesting that foundation models' caption embeddings can be manipulated with vector arithmetic, similar to word vectors behaviours Ethayarajh et al. [2019]. This aligns with the recent work of [Trager et al., 2023] that show linear behavior of VLMs text embedding spaces. Across modalities, small equivariance is present with respect to background  $(0.22 \pm 0.03$  and Figure 4 right side, bottom row). However when size or texture change for a given object, its image and caption representations seem to move in non-aligned directions  $(0.07 \pm 0.01$  for texture and  $0.04 \pm 0.01$  for size, see Figure 10c). While more syntactically complex captions and other equivariance metrics could be designed, our aim here is to provide an initial study to showcase how PUG: Animals can be easily used to study state-of-the-art models representations.
Note that foundation model representations belong to the hypersphere, yet measuring equivariance as parallelism relies on Euclidean geometry, we discuss this in Appendix C.1. Still, cosine similarity is a starting point to showcase how PUG: Animals can be used to study models' representations.
3.3 PUG: ImageNet
Figure 5: We present PUG: ImageNet, a new photorealistic synthetic dataset with annotated factors of variations as an additional test set for ImageNet pretrained models.
As a second member of the PUG family, we introduce PUG: ImageNet (Figure 5), which contains 88,328 pre-rendered images using 724 assets representing 151 ImageNet classes with 64 backgrounds, 7 sizes, 9 textures, 18 different camera orientation, 18 different character orientation and 7 light intensity. In contrast to PUG: Animals, PUG: ImageNet was created by varying only a single factor at a time (which explain the lower number of images than PUG: Animals despite using more factors). The main purpose of this dataset is to provide a novel useful benchmark, paralleling ImageNet, but for fine-grained evaluation of the robustness of image classifiers, along several factors of variation.
An extensive evaluation of the robustness of SOTA models Our PUG: ImageNet dataset offers both photo-realism and precise control over how each object is depicted from pose and size to environment and camera-angle. We also provide a collection of objects with mappings to classes in the popular ImageNet dataset, enabling researchers to probe the robustness of SoTA vision models without retraining. We assess a variety of model architectures across several pretraining datasets including ImageNet-1/-21k, LAION (400M and 2B), and JFT300M [Kolesnikov et al., 2020, Liu et al., 2021, Dosovitskiy et al., 2021]. We observe in Table 1 that the models that perform the best on the ImageNet validation accuracy are not always the ones which offer the best robustness on PUG: ImageNet. For example, the pretrained ViT-B32 trained on ImageNet-21k is better on the ImageNet validation set compared to a Swin-B, but offers worse robustness across all factors. We confirm no statistically significant relationship exists between ImageNet accuracy and robustness by computing Pearson's correlation coefficients (Appendix C.3). This result showcases how PUG: ImageNet can be added as an additional benchmark to evaluate vision models.
PUG: ImageNet Top-1 Accuracy across Factors of Variation
ImageNet Val.	Camera (Yaw,Pitch,Roll)	Pose (Yaw,Pitch,Roll)	Size	Texture	Light	Background
ResNet50	81.5	(38.1, 33.1, 26.9)	(38.0, 23.6, 22.9)	35.7	27.0	13.6	29.5
ResNet101	82.3	(43.4, 35.9, 29.4)	(45.1, 26.7, 25.6)	39.7	31.1	14.1	32.8
ViTLarge	85.8	(52.2, 40.4, 37.1)	(52.4, 30.4, 28.4)	46.4	42.9	8.9	34.6
ViTBasePretrained21k	84.3	(37.5, 34.3, 31.7)	(38.0, 21.8, 20.5)	33.0	28.5	4.1	26.6
Swin	83.6	(56.0, 45.6, 41.8)	(56.9, 35.3, 34.2)	52.9	40.1	19.1	42.0
BiT (JFT300M)	80.3	(40.5, 32.3, 26.0)	(42.1, 23.6, 22.8)	37.3	23.4	6.3	20.5
DINOv2 (LVD-142M)	84.5	(45.6, 41.1, 37.4)	(47.5, 28.8, 28.5)	43.1	35.0	6.1	30.9
Flava (PMD 70M)	75.5	(31.7, 23.4, 17.6)	(30.8, 17.6, 15.4)	30.5	24.2	7.8	21.9
CLIPViTB32 (400M)	62.9	(41.7, 30.2, 22.1)	(41.6, 23.8, 20.9)	40.1	34.4	5.7	24.4
CLIPViTB32 (2B)	66.6	(44.0, 31.5, 24.1)	(43.8, 24.8, 21.8)	42.2	34.7	3.3	26.0
CLIPViTL14 (400M)	72.8	(52.3, 39.8, 35.7)	(51.8, 29.0, 26.4)	50.6	41.1	4.3	33.0
Table 1: Robustness measured by average top-1 accuracy across factors on PUG: ImageNet (We show on the second column the traditional ImageNet validation set accuracy for comparison). Pretraining dataset sizes are indicated in parenthesis with the default being ImageNet-1k. CLIP uses ViT-B32 or ViT-L14. Camera orientation and object pose indicate accuracy along (yaw, pitch, roll) axes.
3.4 PUG: SPAR for VLMs
As a third member of the PUG family, we introduce PUG: SPAR (Scene, Position, Attribute and Relation) for evaluating vision-language models (VLMs). In contrast to pure vision based models, VLMs should be able to predict the correct caption (from a given set of captions) that describe the content of a given image. Several benchmarks to evaluate VLM models already exist such as Winoground [Thrush et al., 2022] or ARO [Yuksekgonul et al., 2023]. However, recent works[Thrush et al., 2022, Diwan et al., 2022] have highlighted an important limitation in these benchmarks: some image_caption pairs in Winoground might be even too difficult to solve for a human whereas ARO
Scene recognition
A photo in a [background] environment
[background]  $= 10$  environment names [character]  $= 32$  animal names
Single Animal
Detection
A photo of a [character] in a [background] environment
Position
A photo of a [character] on the [pos] of the picture in a [background]
environment
Detection
A photo of a [character] and a [character] in a [background] environment
Multiple Animals
Relations
A photo of a [character] on the [pos1] and a [character] on the [pos2] of the picture in a [background] environment
Attributes
A photo of a [character] textured with [texture1] and a [character] textured with [texture2] in a [background] environment
Caption	Texture	ViTB-32	Open	ViTB-32	Open	ViTB-14	Open	ViTB-14	Open	Fiona	BLIP	XVLM	NegCLIP
“A photo in a [background] environment”	N/A	100.00	100.00	100.00	90.00	100.00	100.00	100.00	60.00	100.00	90.00	100.00
“A photo of a [character] in a [background] environment”	Default	39.61	41.80	78.44	60.00	70.23	78.36	31.41	52.73	44.84	32.42
Blue/Red	27.66	29.84	59.84	45.00	50.78	63.75	15.94	34.69	25.62	24.69
Grass/Stone	23.13	27.19	54.53	40.00	45.00	53.91	13.44	32.81	23.44	22.97
“A photo of a [character] on the (left/right) of the picture in a [background] environment”	Default	19.84	20.00	39.38	30.31	35.94	42.19	14.69	27.03	22.66	16.56
“A photo of a [character] and a [character] in a [background] environment”	Default	9.42	11.88	44.08	25.36	34.98	50.66	6.30	22.29	13.50	7.32
Blue/Red	3.54	6.03	23.00	14.16	14.59	28.54	1.93	7.02	2.49	2.94
Grass/Stone	2.89	4.94	16.98	11.21	12.21	21.89	1.11	5.67	2.48	3.28
“A photo of a [character] on the left and a [character] on the right in a [background] environment”	Default	4.82	6.75	22.20	13.00	20.94	29.83	3.43	11.51	7.42	4.92
“A photo of a [character] textured with [texture1] and a [character] textured with [texture2] in a [background] environment”	Blue/Red	1.69	3.03	9.18	6.37	8.25	15.89	1.73	4.20	2.50	1.44
Grass/Stone	1.20	2.24	8.46	6.14	7.08	13.50	1.50	3.50	1.89	1.52
Figure 6: Setup and zero-shot evaluation of CLIP models on PUG: SPAR with caption retrieval. By using synthetic data, we can increase progressively the difficulty of a scene. Our setup is presented in the image above the table in which we show 6 different types of image captioning. 1) caption for background scene recognition for which we have 10 different backgrounds which are easy to distinguish from each other. 2) caption for single animal class prediction, the model should predict the correct categories over the 32 possible animals and 10 backgrounds (for a total of 320 captions). 3) caption for single animal position prediction that increases the number of caption up to 640 and lead to a significant drop in accuracy for every models. 4) caption for two animals class prediction, the model should predict the correct categories of the two animals presented in the images (5120 captions). 5) caption for two animals positions prediction, the model should predict the position of the two animals in the picture (over a total of 10240 captions). 6) caption for two textured animals class prediction, the model should recognize a blue elephant from a red camel. The performances of several VLMs models are presented in the table for which each row corresponds to one of the scenario described previously.
has been shown by Lin et al. [2023] to be mostly solvable without even using the image information at all. Consider that for an image containing a horse eating grass, ARO will propose two captions: "the horse eating the grass" and "the grass eating the horse". The model should predict the correct caption between these two. However, the second caption is impossible, so even without looking at the image, any model can be confident that the first caption is the correct one.
Another shortcoming of current benchmarks is that most of them probe only if the model is correctly able to understand the relations or the attributes between objects. However it is not clear if the failures in finding the correct relations or attributes come from the model not understanding them or come
from not understanding which objects are present in the scene. For example, to understand complex relations like: A photo of an elephant on the left and a camel on the right in a desert background, the model should first be able to identify whether the background of the picture is an actual desert. Then, the model should identify whether there is an elephant on the picture. It should understand what is an elephant on the left. The model could be very effective at identifying individual elephants or camels, but it could unexpectedly fail when a camel and an elephant appear in the same picture. If the model does not fail in recognizing the animals, then we can probe the model to evaluate the position of each of them. We built PUG: SPAR with this goal of having a progressive evaluation scheme in which we can easily determine exactly what are the failure modes of a given VLM. From basic scene understanding to complex relations and attributes, our dataset offers a simple and yet effective way to get a better understanding of the capabilities and limitations of VLMs.
The dataset contains 43,560 images with the associated factors of variations: 10 backgrounds, 32 animals, 4 relations (left/right, bottom/top) and 4 animal texture attributes (blue/red, grass/stone). We have images containing either 1) only the background (for scene recognition) 2) only one animal at different left/right or bottom/top position 3) two animals at different left/right or bottom/top position. And for each of the scenes (either single or multiple animals), we vary the texture of the animals and the background to evaluate the robustness of the model. Our setup and experiments are presented in Figure 6 in which we display some images from the dataset with the corresponding captions used for evaluations. In our benchmark, we used 6 different types of captions to evaluate the following: 1) Scene Recognition (first row) 2) Single animal classification (second row) 3) Single animal position detection (third row) 4) Multiple animals classification (forth row) 5) Multiple animal position detection (fifth row) 6) Multiple animal and textures prediction (sixth row). For each of them, we evaluate the top-1 retrieval accuracy of the correct captions within the set of captions associated to each setup. We evaluate multiple models on these setups: OpenAI CLIP [Radford et al., 2021], OpenCLIP [Ilharco et al., 2021], Flava [Singh et al., 2022], BLIP [Li et al., 2022b], X-VLM [Zeng et al., 2021] and NegCLIP [Yuksekgonul et al., 2023]. Most of the models are correctly able to solve the scene recognition task (which is not surprising since we used only 10 environments which are very different from each other). Concerning the simple object recognition task when using a single animal, the performances across models is highly variable. Our experiments also highlight that the VLM performance in a multiple animals detection setting are much worse than the performance in a single animal detection setting. Those experiments show that despite their successes, VLMs are far from having a good understanding of the world and that improving the robustness of these models is a needed step for real-world robustness.
Inspired by Winoground [Thrush et al., 2022], we present an experimental setup in which we leverage hard-negative pair of images. Instead of performing caption retrieval within all captions associated to a given setup, we performed caption retrieval between the correct and the hard negative caption. For example, the hard negative caption of "An elephant on the left of the picture and a camel on the right of the picture" will be "A camel on the left of the picture and an elephant on the right of the picture". In addition of switching the relation (left/right and bottom/top), we also provide hard negative captioning for the attributes (blue/red and grass/stone). In Table 2, we present our results using the hard-negative pair. We clearly observe that none of the models are able to predict the correct captions, with many models being close to random performance (50%).
3.4.1 PUG: AR4T
Lastly, we introduce PUG: AR4T (Attributes and Relations for training). In contrast to PUG: SPAR which is only an evaluation benchmark, PUG: AR4T was created as an additional fine-tuning dataset for VLMs. As shown in the previous section, VLMs struggle to understand spatial relations or attributes and thus are good candidates for our fine-tuning scenario. PUG: AR4T contains 249,986 training images with captions and 23,216 test images. In Table 3, we present CLIP fine-tuning results on the ARO and PUG: SPAR benchmark. We also compare our results against Syn-CLIP, which is CLIP fine-tuned on the SyVIC synthetic dataset. Our results are very similar to Syn-CLIP, but Syn-CLIP training requires several additional tricks to arrive at this performance (Section 3.2 in
8The assets used to create PUG: SPAR and PUG: AR4T are different enough such that PUG: SPAR is still a good benchmark to evaluate models fine-tuned with PUG: AR4T.
<sup>9</sup>We also run experiments with a version of this dataset which contain 1M images but as shown in Table 3, adding more images does not increase performance on PUG: SPAR. We only release publicly the version of PUG:AR4T that contain 249,986 images.
Caption	Values	P2CLIP	P2CLIP 2B	L14CLIP	L14OpenCLIP 2B	H14OpenCLIP 2B	G14OpenCLIP 2B	Fliva	BLIP	XVLM	NegCLIP
Left/Right	Bottom/Top	49.53	47.66	50.00	46.72	49.84	50.31	49.22	51.88	52.03	53.28	54.06
“A photo of a [character] on the [position] of the picture in a [background] environment”	Left/Right	54.84	52.34	66.87	60.62	56.56	58.91	50.47	54.84	53.28	54.06
“A photo of a [character] on the [position] and a [character] on the [position2] in a [background] environment”	Left/Right	53.88	55.62	53.17	56.23	55.74	54.44	54.41	53.79	55.02	54.36
Bottom/Top	51.15	53.84	54.06	53.51	57.24	56.49	55.23	60.26	58.87	54.09
“A photo of a [character] textured with [texture1] and a [character] textured with [texture2] in a [background] environment”	Blue/Red	52.77	53.63	54.43	56.94	55.48	54.42	54.22	57.32	56.19	51.74
Grass/Stone	52.79	54.14	56.31	57.28	56.62	57.19	54.53	53.94	54.26	49.92
[Cascante-Bonilla et al., 2023]). On the other hand, the photo-realistic nature of PUG: AR4T enables us to match Syn-CLIP without any of these additional bells and whistles. However, even if we note some improvements on ARO, we are still far from having a model able to understand spatial relations. This is highlighted by the results given on our PUG: SPAR benchmark for which the improvement on single animal position prediction is still only above random chance while there is no improvement on the double animal location prediction task. This confirm the unreliability of the ARO benchmark highlighted by Lin et al. [2023].
Table 2: We present the performances of several VLMs with hard negatives captioning on PUG: SPAR in which we perform retrieval between two captions: the correct caption and the hard-negative corresponding caption. In that instance, the model should choose the correct caption between both of them (the probability to get the correct one with a random model would be  $50\%$ ). Interestingly, none of the model presented in this table seem to be able to get a real understanding of simple position (left, right, bottom, top) or colors.
VG-Relation (Macro-Accuracy%)	VG-Attribution (Macro-Accuracy%)	ARO COCO-Order (Precision@1)	Flickr30k-Order (Precision@1)	Average	PUG: SPAR (left/right)
Single (Precision@1)	Double (Precision@1)
CLIP-VIT-B/32 (400M)	59.16 | 55.50	62.18 | 61.52	47.96	59.98	57.32	49.84	54.42
+FT w/Syn-CLIP	71.40	66.94	59.06	70.96	67.09 (+9.77)	N/A	N/A
+FT w/PUG:AR4T (200K)	68.36 | 75.18	65.54 | 64.44	57.80	69.74	65.36 (+8.04)	50.78 (+0.94)	54.23 (-0.19)
+FT w/PUG:AR4T (1M)	71.03 | 76.57	65.15 | 64.32	61.07	72.84	67.52 (+10.3)	50.16 (+0.32)	54.19 (-0.23)
Table 3: Fine-tuning CLIP on PUG: AR4T. For VG-Relation and Attribution, the results (Acc1 | Acc2) indicate macro-accuracy across all relations and attributes (Acc1), and macro-accuracy on the subset of relations and attributes present in both ARO and PUG (Acc2). For PUG: SPAR, we evaluate on images in which there is only one animal (Single) or two animals (Double) with the relation being left or right. We were not able to run SynCLIP on PUG: SPAR because the model was not public at the time of the publication.
The fine-grained controllability of synthetic rendered image data makes it ideal for designing challenging evaluation benchmarks to better understand the properties and limitations of vision models, as well as for controlled training scenarios – if only it was closer to real data. To this effect, we introduced PUG datasets for representation learning. By leveraging the photorealism of the Unreal Engine[EpicGames], we created 4 new datasets and showcased their utility for robust evaluation. We showed how PUG: Animals could be leveraged for OOD generalization and to study properties of the representation spaces. We developed PUG: ImageNet as a new challenging benchmark that researchers can easily use to assess and compare the robustness of image classifiers. With PUG: SPAR we provide a reliable benchmark for vision-language models while PUG:AR4T offer additional data that could be leveraged to fine-tune VLMs. Together, the PUG family of datasets represents a new standard of photorealism and control for synthetic image data.
S. Xie, A. S. Morcos, S.-C. Zhu, and R. Vedantam. COAT: Measuring object compositionality in emergent representations. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 24388-24413. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/xie22b.html.
R. X. H. Y. Z. S. P. C. Xingxuan Zhang, Yue He. Nico++: Towards better benchmarking for domain generalization, 2022.
M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX.
Y. Zeng, X. Zhang, and H. Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276, 2021.
Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
In this work, we introduced 4 new datasets that were created using the Unreal Engine. We presented a set of case studies, demonstrating how these datasets can be leveraged to improve both evaluation and training for representation learning. Deeper work would be needed to fully unlock the potential these datasets can offer. In addition, we merely scratch the surface of what a powerful engine such as the Unreal Engine can offer. With advanced techniques such as Lumen, Nanite and Megascans, it is now possible to create even more realistic environments. In addition, the datasets we provide have a single simple label, whereas future uses could easily provide detailed rich labels for the entire scene underlying each generate image.
A.2 Future Work
A long vision for the PUG family would be to yield a series of benchmarks that can probe the robustness of vision models. PUG: ImageNet is a first step in this direction, however we might want to get more factors of variation such as weather and occlusion. A second take would be to increase the richness of the labelling by making available detailed segmentation masks and labels. Making a short video dataset in which we have complete control over the factors is also a promising future direction since AI research on video is still far behind what can be done with images. The active learning pipeline direction is also worth to explore since the model could bias the PUG environment to produce samples that are the best for a specific downstream task[Mishra et al., 2022].
B PUG Datasets
The datasets PUG: Animals, PUG: ImageNet, PUG: SPAR and PUG: AR4T are available under the cc-by-nc license with the restrictions that they should not be used to train generative AI models. They are available to download on the following website: https://pug.metademolab.com/. The datasets can be read by a torchvision ImageFolder. We have one class by folder and all the images associated to one class are saved in this folder as png. There is a csv file associated to each dataset that map a filename (with unique ID) to its associated factors of variations. Examples of dataloaders are available at https://github.com/facebookresearch/PUG.
B.1 Datasheet
Table 4: Datasheet for PUG, following the framework introduced by Gebru et al. [2021].
B.2 PUG: Animals
PUG Animals contains 215 040 pre-rendered images using 70 animals assets, 64 backgrounds, 3 sizes, 4 textures, under 4 different camera orientations. To create PUG: Animals, we use Animals assets from the following bundle in the Epic Game Marketplace ( https://www.unrealengine.com/ marketplace/en-US/product/complete-animals/reviews). The list of environments used can be found in the dataset folder or at https://github.com/facebookresearch/PUG. Below, we list all the values for the factors of variation, we have used:
- World_Name: ["Egypt", "Desert", "AmusementPark", "ArcadeClub", "Arena", "Battleground", "Catacombs", "Tableland", "EuropeanStreet", "JunkYard", "OceanFloor", "Racetrack", "Ruins", "SciFiCity", "SciFiGarage", "SpaceIsland", "SpaceHangar", "SpatialStation", "TokyoDay", "TokyoNight", "TrainStation", "Bridge", "Beach", "BusStationInterior", "BusStationExterior", "Subway", "IndoorStairs", "Bar", "ScreeningCheckpoint", "Circus", "Apartment", "Hallway", "TrashRoom", "FuturisticSubway", "Footbridge", "BoxingRing", "Hangar", "Mansion", "ShoppingMall", "ConferenceRoom", "SpacePort", "VillageOutskirt", "VillageSquare", "Courtyard", "ElvenRuins", "Forge", "Library", "Museum", "Gallery", "ModernGallery", "Opera", "AncientAgora", "Restaurant", "RuralAustralia", "AustralianRoad", "ShadyRoad", "SaltFlats", "Castle", "StylizedEgypt", "Temple", "Snow", "Grass", "DryGrass", "Forest"],