PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
Authors: Florian Bordes
Shashank Shekhar
Mark Ibrahim
Diane Bouchacourt
Pascal Vincent
Ari S. Morcos
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Synthetic images
Unreal Engine
Representation learning
TL;DR: We leverage the Unreal Engine to produce the next generation of dataset for representation learning

Abstract: Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation.
Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear.
In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. 
We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. Using PUG for evaluation and fine-tuning, we demonstrate the potential of PUG to both enable more rigorous evaluations and to improve model training.

PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
Florian Bordes 1.rdes!.2.3 2,3
Shashank Shekhar!
Mark Ibrahim'
Diane Bouchacourt
Pascal Vincent 1.233
Ari S. Morcos
FAIR Meta
Mila Quebec AI Institute
Universitéé de Montréal, DIRO
Abstract
Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use synthetic image data still limited - and often played down mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present path to democratize the of photorealistic synthetic data: develop new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphies) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models. The datasets can be downloaded at https:: //pug. metademolab com/
Introduction
A grand goal of machine learning is to learn representations of data that are useful across many tasks. Essential to measuring and making progress towards this goal is the availability of ample controllable, realistic data for evaluation and training. This is especially true when considering deep neural network models not only in terms of their raw accuracy, but also their robustness and fairness-crucial properties for models deployed in real-world applications. However, collecting such data is challenging, presenting issues with privacy, bias, and copyright. Furthermore, the majority of available image datasets lack fine-grained labels and are challenging to manipulate beyond coarse image augmentations (e.g. with a photograph, it is hard to change the viewpoint or the time of day).
Using synthetic image data where precisely control all the factors affecting the rendered scene gives easy access to the corresponding rich set of factor labels. This enables evaluating the extent of trained deep neural network's abilities, most importantly its robustness. Is the network robust to change in pose? Are the predictions similar for different textures? All these questions may be answered systematically by using synthetic data, enabling highly rigorous evaluations of deep neural network models. In addition, training could also benefit from controllable factore"] by increasing the robustness of models with respect to these factors. They may also be used monitor training, e.g. tracking which factors a model focuses on or becomes most invariant to, and in which order, as training progresses. This potentially enables better understanding of the training and generalization
We define factors here as distinctive attributes that describe the data, such as color pose of an object.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

PUG Environment Generation
Dataset Generation
The PUG Family PUG: ImageNet
PUG: Animals
PUG: SPAR
Unieal Enginn 5.0
Complation
d'
PUG
Joon Datawet Contrg
TorchMultiverse
reedor De hoolgrond emmionmen""
Figure 1: The PUG Dataset Family (left) Cartoon illustration of our dataset creation setup, which consists of two steps: environment creation and then data creation. (right) Example images from PUG: Animals, PUG: Image-Net and PUG: SPAR.
dynamics in deep neural networks. However the lack of realism typical in many of the currently available synthetic image datasets, and their usually very limited scope greatly limits their usefulness for general image representation learning research.
To address this, we introduce new family of synthetic Photorealistic Unreal Graphics (PUG) datasets, designed for ease of use by the representation learning research community, where image realism is significantly improved compared to current public synthetic image datasets. The environ- ments were built using the Unreal Engine [EpicGames], which is widely used in the video game and entertainment industries and praised for its realism. In addition to pre-rendered static image datasets, also introduce the TorchMultiverse python library, which offers simple python interface to enable easily controlled dataset creation from any given PUG environment. Using these tools, we contribute 4 new datasets and show their usefulness across several different research domains. To summarize:
We introduce new family of environments and image datasets (coined as PUG) for representation learning, based on the Unreal Engine (EpieGames].
We present PUG: Animals for research out-of-distributtin (OOD) generalization and study the representational space of foundation models.
We introduce PUG: ImageNet as an additional robustness test set to ImageNet, containing a rich set of factor changes such pose, background, size. texture, and lighting.
We introduce PUG: SPAR for evaluating vision-language models. We use it to demonstrate how synthetic data can be utilized to address known benchmark limitations. In addition, we introduce PUG: AR4T for fine-tuning vision- language models and use it to demonstrate the reliability of PUG: SPAR in contrast to other benchmarks.
2
Related work
Synthetic data for representation learning To address robustness shortcomings, researchers today commonly study representations using lower-fidelity controlled datasets such as CLEVR, Biased Cars, and ShapeNet [Johnson et al., 2017 1017,Maddn et al. Chang et al., 2015). Other datasets also contain precise factor labels useful for probing how well a representation encodes each factor in a structured form [Gondal et al.) 2019 Struckmeier et al. 2023 Weng 2018). While these datasets offer control in terms the factors that change as well as the train evaluation splits enabling controlled scientific experimentation, they lack realism. This gap between the pwwerfiielity controlled data and the real world poses a challenge for the broader application of these studies. On the other hand, photorealistic datasets have been explored in various application-speccfic domains in machine learning (outside
2 As reminder, any use of content or technologies made available by Unreal and/or Epic Games, or any other provider, should comply with their applicable terms (such as the Content License Agreement available at https://wwe :/wps.:/wwww... unreal eng ine, com/ en US/ hae//ella/cco/ www./cootleettteeeeeeeeeee or any other direct agreement you may have with Epic / Unreal)

---

of representation learning.) This is especially relevant when trying to evaluate and train models on rare events in which getting real data might be really difficult, such as for autonomous driving. CARLA (Dosovitskiy et al. 2017] is popular self-driving car simulator which offer highly realistic environment with a significant amount of controllable factors such as environmental conditions, full control of all static and dynamic actors and maps rendering. Another domain where simulated environments are commonly used is reinforcement learning (RL), as RL algorithms often requires the ability to run millions of simulations to learn to master non-trivial tasks, and this cannot be done in a real environment. Data environments based on video games like Atari have been very popular to design and evaluate RL algorithms. Alternatively, platforms like Habitat (Szot al.. 2021 offers indoor scene for training home assistant agents. While these simulators, games or datasets can offer some photo- reato-realism and mimic real world interactions for agents, they are relegated to domain-specific applications making them challenging to use for evaluating the representations of deep neural networks more broadly. Since focus is not RI., we do not need to embed a fast simulator capable of rendering several thousands frames per second for effective online-training, Instead we can pre-render custom high-quality datasets offline. Photorealistic environments and datasets have also been explored for more general domains with the ThrleDWorld platform [Gan et al., 2021]. Based on the Unity game engine, it offers an interactive environment that can be leveraged to create datasets. The environment is presented as a simulator that is generic enough to handle multiple uses cases, and users can customize the setup of a scene and the data smapling through a low level API. One such dataset that utilizes ThreeDWorld is the Synthetic Visual Concepts (SyVIC) dataset Gaasantee-Bonlle et al. 2023], which uses the API to create scene images and descriptive captions for training vision- vision-language language models. One of the downsides of TWrle is that the back-end, the simulator itself, is closed source which limits external contributions. In contrast with DWrld, we do not provide a platform or generic simulator for people to use. In fact, we believe that tools like the Unreal Engine are simple enough to be used directly by researchers to create the environments they want without the need to use intermediate platform. In addition, being free of such intermediate platform allows us to leverage most of the content created for video gaming directly into our simulator by using the existing Epic Games marketplace.
Evaluating model robustness To study model robustness, there is an inherent trade-ofi off between photo-realism and control. Photo-realism depicts objects as they appear in the real world, but often lacks control to precisely define the factors to describe the object such as pose or background. Prior
works either collect natural images with specific factor changes (Xiao al. 2020 Barbu al. 2019
or label distinctive factors in existing datasets [ldrissi et al.] 2022]. Such datasets allow researchers to measure average accuracy on photo-realistic images, but lack granular control necessary for precisely controlled robustness experiments. On the other hand, prior studies [lbrahim et al. [022 Abbas and Deny 2022. Alcomm et al. 2019] examine model robustness with respect to factors such as pose and size by rendering Dboejetts such as buses. These studies precisely control how each object is depicted, but lack in realism. In this work, we advance the photo-realism of these prior works by using the Unreal engine 5.0 (EpicGames), rendering engine commonly used in high-end cinematic CGI and high-resolution video games which allow us to measure robustness with respect to factors of variation such as lighting.
Benefits and limitations of using generative models as data generator Another way to generate realistic datasets is use generative models(Ho et al. 2020 Goodfellow al., 2020). However, one limitation of such models, despite impressive improvements in the last few years [Dhariwal and Nichol 202T], is the lack of quality control on what the model can produce [Gandikota et 2023) It's not uncommon to find cases in which the model will ignore parts of the conditioning prompt.
2023)
Despite such limitation many works have tried to leverage generative model as an additional source of
data to train deep neural networks with some success [Astolfi et al. 2023 Bansal and Grover 2023
Trabucco et al. [2023] Azizi et al. 2003 /Zhang et al. 202 et al. 2022a Jahanian et al. [022 Jain
et al. 2023] Sariyildiz et al.) 2023 He et al. 2023]. Another limitation of using generative models s are
privacy concerns that arise from such models replicating data from their training datasets [Somepalli]
et al. (022). Finally, (Shumailov et al. [2023) recently demonstrated that training on data recursively generated from such models results increasing underestimates of the tails and overestimates of the mode, amplifying bias in datasets. In contrast to generative models that might produce unreliable results, we use an entirely controllable environment for which we can have known and accurate generation with respect to set of factors.
3

---

3 Photorealistic Unreal Graphies (PUG) environments and datasets
3.1 Leveraging Unreal Engine to create environments and datasets for representation learning
We introduce the Photorealistic Unreal Graphics (PUG) environments, a family of 3D graphics environments that leverage Unreal Engine for rendering image data for representation learning research. To create PUG environment, we first obtain number of assets 2 which can be 3D objects or 3D backgrounds. Then, we import them in the Unreal Engine editor and create blueprints that yield a simple generic 3D environment. Once this generic and controllable environment is created, it is compiled into Linux binary file, which can be run standard GPU clusters. This environment is programmed such way that when running, through WebRTC which can specify instructions about how change scene. Since most machine learning practitioners are used to python scripting, we wanted to have a very simple approach by which a user request image data rendered from packaged PUG environment, through very simple python code and JSON config files. To do so, we developed a python API, TorchMultiverse, that allows user to easily specify scene configuration in JSON and request rendered images from the PUG by using WebRTC. Once the factors have been set as requested by the user, for a specific environment configuration, the user can send command receive back an image. It takes around second to render an image at a resolution of 512x512 on a We illustrate this setup in Figure 1 It shows how, starting from 3D assets, we design
listening for incoming packets
freeze the current environment and
V100 GP f interactive environments that enable us to create different datasets. In the present work, we focus on pre-rendered static image datasets, however our setup also allows dynamic communication between PUG environment and pytorch program, meaning that new data could be requested and rendered on the fly while training deep neural network. We leave the exploration of such active learning setups, as well as the rendering of videos, as future work 5
3.2 PUG: Animals
Figure 2:
We present PUG: Animals, new photorealistic synthetic dataset with annotated factors of variations to evaluate the out-of-distribttion (OOD) robustness of models.
As the first member of the PUG family we introduce PUG: Animals (Figure 2 which contains 215 040 pre-rendered images using 70 animals assets, 64 backgrounds, object sizes, 4 textures, under 4 different camera orientations. PUG: Animals is designed with the intent to create a dataset with every combination of the factors of variation available. PUG: Animals allows one to precisely control distribution shifts between training and testing which can give researchers better insight on how a deep neural network generalizes on held out factors of variations. Surprisingly, the usage of 3D realistic synthetic data is limited in OOD generalization research - with the exception of Biased- cars Madaan et al. 2022 that has been used to study generalization
new nategory-viewpond Commons OOD datasets are Colored MNIST [Arjovsky et al., 2020 to study how well network can generalize to unseen combinations of digits and colors and MNIST-R [Ghifary al. 2015 to study generalization on new combination of digits and rotations. However, MNIST based dataset
'We purchased assets from the Epic Game Store and used assets from Sketchfab Deitke et al. 2022 The complete list of assets have used is available hhttpa://githui ccmm/aacebokkeeeercc... PUG
"In our setup, we paralyze the rendering across 64 GPUs. dataset like PUG: Animals which contains 200K images has taken around lh to be entirely rendered.
used a photorealistic interactive environment for reinforcement learning (RL), but the high quality image rendering achieved in this system currently appears too compute-intensive intensive and slow to be practically useful in the context of current RL research. Our initial targeted research community and case is that of supervised and self/unsupervieed representation learning from image data, rather than RL.
'It might also conceivably

---

might be too toyish to evaluate modern architectures. A more realistic dataset based on real images Nico++ (Xingxuan Zhang/ 2022] - to study generalization with respect to different domains or environments. However, in Nico++ the objects and backgrounds are never entirely disentangle (the context background is different for each image). Thus, it is never clear the model failing because of the context or because of specific object (since the contexts and the objects are never disentangle). In contrast, in PUG: Animals the animal asset is always the same, in that case the environment factor and the objects are perfectly disen- tangle such that if the model is able to classify correctly an elephant
Holdd out Accuracy
100
on road and is not able to classify the elephant in museum, we can rigorously say that the failure is caused by the change in context background. In addition of analysis the robustness with respect to the
20
40
30
20
background scene, it is also possible to analyse with PUG: Animal the robustness with respect to the camera position, asset size and texture (as we demonstrated in Appendix 3.2.
Nb.
training
Held out Accuracy is & 2
Classification with held out sets In the first experiment, we held out some factors of variation during training (backgrounds, sizes or textures) except for a specific number of animals C' and use the held out data as validation set. Thus, C = 0 means that the network never saw
"mmal
Modium
Asset Siiza
Iteld out Accurncy
the factor during training (this is an OOD scenario with unseen factors)
while C 10 imply that the network saw this factor for least 10 different animals (OOD scenario with unseen combinations of factors). In Figure 3 we present our results training a ResNet50 with different held out factors. Every model reached more than 99% accuracy on the training set. First, we trained on 50 backgrounds and used the remaining 14 backgrounds for validation: here, the network reached an accuracy of 80%. However, when using only 30 backgrounds for training and using the remaining 34 as validation, the accuracy drop significantly. Interestingly, showing every background for some of the animals (having unseen combination of factors instead of just unseen factors) decrease performance. In contrast, for texture, we found that having at least 10 animals for which every possible textures are seen during training improves generalization. Interestingly, the network overfits much more to the grass texture relative the default network. Lastly, when looking at the size factor, it seems that training on medium size assets leads to good generalization assets while training only on small asset leads to worse performance on medium and large assets.
20
Grass
Aspbalt
Training Texturn
Figure 3: Accuracy on held out factors with PUG: Animals. Each line and value C corre- spond to the number of animals for which
the factors are seen. The test space is built by taking all the factors minus the training factors. If we train on the Default texture, then the network is evaluate on Grass, Sky and Asphalt. If we train on 50 backgrounds, then we evaluate on 64 (total number of background) 50 (training background) = 14 backgrounds.
small and large
Studying foundation model representational space PUG:
Animal can also be to study the equivariance of foundation models'
representations. For this, we augment each image in PUG: Animals with caption that describes it according to its factor values (sizes are converted to three adjectives: "small", "medium" or "big", see Appendix C.1 for details), using the following template "A photo of a [size] sized [character] textured with [texture] on a [background] background" " Informally, equivariance of model's representation with respect to factor means that when the factor changes from one value to another, the embedding of the corresponding image (or caption) changes in a predictable manner. Equivariance is a sought-after property to improve sample efficiency and robustness to transformations [Keee et al.] 2023 Tai et al. 2019 Wang et al. [2023] Similar to previous works on equivariance and compositionality [Bouchacourt et al. [202]], Xie et al. [222]. we measure equivariance as the alignment (i.e. parallelism) between embedding differences. First, we feed images and their corresponding captions to pretrained vasion-language models including multiple CLIP models Radford et al.) [2021], NegCLIP Yuksekgonul et al. (2023) Flava (Sighh et al. [2022), BLIP Li et al.] [2022b] and X-VLM Zeng et al.] [2021] and collect their embeddings of PUG: Animals images and created captions. For each model, we compute difference vectors between the embeddings of two images (or captions) of an object undergoing factor change: e.g. a big penguin textured with grass on background "village square" modified to the same penguin but
'Not that the camera and character orientations are not described, well the texture when it is default.
5

---

with background "restaurant", see arrows in Figure Specifically, for sample i, undergoing change from background by for background bt we denote the difference vector between the embedding of the image of the sample with backgorund by and the image of the same sample but with background by by -b, Similarly, we denote by -b the difference vector between the embedding of each of the two captions accompanying the images.
Then, we measure the alignment of difference vectors across pairs undergoing the same factor change (here, the penguin and the cat) as their cosine similarity We estimate three types of equivariance: (i) Image equivariance: how parallel (measured with cosine similarity) are difference vectors across image pairs? (lined and dashed red arrows) (ii) Text equivariance: same but for caption pairs (parallelism of lined and dashed green arrows) (iii) Across modalities equivariance: for the same object, alignment of difference vectors between pairs of image-caption (i.e. alignment of the two arrows for the penguin). Specifically, for image equivariance between sample and for background change by to b,, we compute:
T 2/
1/ +b
sim (on b (bp
+b
11 bb
by bb
For text equivariance, we compute be sim sim (uh,, - , "y - while for across equivariance, we com- pute sim +b, "8 )b,) We report cosine similarity averaged over pairs and possible changes for each factor (higher value means higher equivariance, 1 is the maximum). Specifically, the image equivariance for background writes as
B(B EE bg
ΣΣ sim -be P/n
N(N 1)
where B is the number of possible backgrounds and N is the number of samples.
Image
Text simtt bf
Across
fentured with gioss on resdaurent beckgrounnd
CST lostrrd with sky restaurend bnsigraunet
sinte Spree
photo big sloud parpuiy wetturedwuth with grass or village square backgooud
Apllooo a moniiom sloed testured with sky on village square coolground
Equavariance tpes
Figure 4: Measuring foundation models equivariance thanks to PUG: Animals. Left: Illustration of how to use PUG: Animals to compute equivariance. Right: Image and text equivariance is present with respect to background, while across modalities equivariance to background doesn't 't hold as much. See main text for detailed results.
We show in Figure 4 (right) results for equivariance with respect to the background. Plots for equivariance to texture and size are in Figure T0 Looking at Figure/Jreuulle (right side, top row), see that the foundation models' image embeddings present high equivariance background (0.78 + 0.04 on average over models). There is also (see Figure 10a) small image equivariance to texture (0.15 + 0.04), but almost no equivariance to size (0.06 + 0.02). Text equivariance high with respect to background (average of 0.87 + 0.03), but is also strong for size and texture (0.71 = 0.11 for size and 0.81 + 0.03 for texture, embeddings can be manipulated with vector arithmetic, similar to word vectors behaviours Ethayarajh al. [2019]. This aligns with the recent work of [Trager et al., [023) that show linear behavior of VLMs text embedding spaces. Across modalities, small equivariance is present with respect to background (0.22 + 0.03 and AggureFrigtt side, bottom row). However when size or texture change for a given object, its image and caption representations seem to move in non-aligned directions (0.07 = 0.01 for texture and 0.04 0.01 for size, see TOc). While more syntactically complex captions and other equivariance metrics could be designed, our aim here is to provide an initial study to showcase how PUG: Animals can be easily used to study state-of-the-ard models representations.
Figure Tob) suggesting that foundation models' caption
'Note that foundation model representations belong to the hypersphere, yet measuring equivariance as parallelism relies on Euclidean geometry, we discuss this in Appendix C.] Still, cosine similarity is starting point showcase how PUG: Animals can be used to study models' representations.

---

3.3 PUG: ImageNet
Figure 5: We present PUG: ImageNet, a new photorealistic synthetic dataset with annotated factors of variations as an additional test set for ImageNet pretrained models.
As a second member of the PUG family, we introduce PUG: ImageNer (Figure 5), which contains 88,328 pre-rendered images using 724 assets representing 151 ImageNet classes with 64 backgrounds, 7 sizes, 9 textures, 18 different camera orientation, 18 different character orientation and light intensity. In contrast to PUG: Animals, PUG: ImageNet at a time (which explain the lower number of images than PUG: Animals despite using more factors). The main purpose of this dataset is provide novel useful benchmark, paralleling ImageNet, but for fine-gree grained evaluation of the robustness of image classifiers, along several factors of variation.
created by varying only single factor
An extensive evaluation of the robustness of SOTA models Our PUG: ImageNet dataset offers both photo-realism and precise control over how each object depicted from pose and size to environment and camera-angle. We also provide collection of objects with mappings to classes in the popular ImageNet dataset, enabling researchers to probe the robustness of SoTA vision models without retraining. We assess variety of model architectures across several pretraining datasets including ImageNet-1/-21k. LAION (400M and 2B), and JFT300M (Kolesnikov et al., 2020 Liu al. 2021 Dosovitskiy al. 202T]. We observe in Table I that the models that perform the best on the ImageNet validation accuracy are not always the ones which offer the best robustness on PUG: ImageNet. For example, the pretrained ViT-B32 trained on ImageNet- 1k is better on the ImageNet validation set compared to Swin-B, B, but offers worse robustness across all factors. We confirm no statistically significant relationship exists between ImageNet accuracy and robustness by computing Pearson's is correlation coefficients (Appendix CK3 This result showcases how PUG: ImageNet can be added as an additional benchmark evaluate vision models.
PUG: ImogeNer Top- Accuracy screes Factors of Variation
ImageNet Val.
Camera (Ysw.Pach.Rol Roll
Peee (Yow Www.PMccR.oo Roll)
Size
Teature
Light
Backgrund
ResNe150
81.5
(38.1. 33.1, 26.9)
(38,0) 23.6, 22.9)
35.7
27.0
13.6
29.5
ResNerl01
82.3
(45.4, 35.9, 29.4)
(45 I. 7, 25.60
39.1
31.1
16.1
32.8
ViTLarge
85.8
(52.5. 40.4, 37.0)
(53.4, 50.4, 28
46.4
42.9
3.9
34.6
84.3
(37.5, 14.1, 31.5)
(3 21.8. 20.5
$3.0
18.5
4.1
26.6
Swin
83.6
056.0 45.6, 41.5)
(SK 35.3 34.21
53.9
46
19.1
42.0
BIT (FT
80.3
(40.5, 32.3, 36.0)
(43.1 23.6. 22.8)
37.3
234
6.3
20.5
DENO (LMD-14IM)
84.5
(45.6 41.1, 37.4)
(47.5, 28.8. 25.5)
43.1
35.0
6.1
30.9
Flaa
(PMD (PMD 70M)
75.5
(31.7. 234, 17.6)
(30.8. 17.6. 15.4)
30.5
26.2
7.8
21.9
CLIPATB3 (400M)
62.9
(41.3. 30.3: 22.1)
(41 23.8. 20.90
40.1
364
5.7
24.4
CLPPATB32 (28)
66
(480) 31.5, 24.1)
(41.8. 24.8. 21.8)
43.2
34.7
3.3
76.0
CLIPATI.14 14 (400M)
72.8
(52.3. 39.8, 35.7)
(51.8,2.00) 20.0)
29.0) 26.45
50.6
41.1
4.3
35.0
Table Robustness measured by average top-1 accuracy across factors on PUG: ImageNet (We show
on the second column the traditional ImageNet validation set accuracy for comparison). Pretraining dataset sizes are indicated parenthesis with the default being ImageNet- kk. CLIP uses ViT-B32 or VIT-L14. Camera orientation and object pose indicate accuracy along (yaw, pitch, roll) axes.
3.4 PUG: SPAR for VLMs
As third member of the PUG family, we introduce PUG: SPAR (Scene, Position, Attribute and Relation) for evaluating vision- -aiiuala models (VLMs). In contrast to pure vision based models, VLMs should be able to predict the correct caption (from a given set of captions) that describe content of given image. Several benchmarks to evaluate VLM models already exist such as Winoground Thrush et al. 2022 or ARO YYuksekgonul et al. 2023 However, recent Works(Thrushi
al. 2022 Diwan al. 2022 have highlighted an important limitation in these benchmarks: some
image-caption pairs in Winoground might be even too difficult to solve for a human whereas ARO
7

---

Scene recognition
Single Animal
Multiple Animals Relations
Dataction
Position
Datection
Attributes
Aplcto beokgrandy onsroimentt
baclgronnd divenmer
Aiphoto paoneced the bonj try pickor Jackgrunn
enda a Stenans) It terelgaunal
jntonater the and rchenclee on lo foods of plouuo
Aptoso jolanccee with and e joheecced sedurdd pootesst baokgroand envermment
eackgranng inderment
booj Sobrigh
VTTBB 32 (OyonA) CLIP)
VT----2 22 (Opent 2B)
WTFL (OpenA CLIP)
VIT (Opentiip 38)
VI-R-14 Open LP 38)
WI-G-14 (OyenCLin
Flanaa
BLIP
XVLM
NeyC
Caption
Texture N/A
"A photo [hockgrownd] eninnon
100.00
100.00
100000
90,00
100.000
100,00
60000
100,00
90,00
100000
"A photo of [charncter] in (background) onviroment"
Defult
39.61
41.80
78.44
60.00
70.23
78.36
31.41
53.73
44.84
12.42
Bluc/ed
27.66
29.84
59.84
45.00
50.78
63.75
15.94
34.69
25.62
24.69
GmesdStone
53.91
"A photo [character] on the
Desault
19.84
20000
39.38
30.31
35.94
42.19
14.09
27.03
22.66
16.56
(loferight) the picnure in (back-
grand] ewiroonenn"
"A ploto of [charunter]
Defsuit
9:42
25.36
34.98
[characser] in (bankgmund)
Blue/Ked
3.54
6,013
23.00
14.16
14.59
28.54
1.93
7,02
2.49
2.94
muboommenn
GiessStonee
2,89
4.84
16.90
11.21
12.21
21.89
1.31
5.67
2,48
3.28
photo [character]
and tcharacter on the right in
(backroound) otritcommntt
photo of
Blue/Red
3,05
9.18
6.37
15,89
4.20
2.50
with [tesmorl]] I and a [choracter] wetnedd with [vesture in
Gieea/tinee
1,20
2.24
8.46
7.08
1.50
3.50
1.89
[oackround] envieoment"
Figure 6: Setup and zero-shot evaluation of CLIP models on PUG: SPAR with caption retrieval. By using synthetic data, we can increase progressively the difficulty of scene. Our setup presented in the image above the table in which we show different types of image captioning. 1) caption for background scene recognition for which we have 10 different backgrounds which are easy to distinguish from each other. 2) caption for single animal class prediction, the model should predict the correct categories over the 32 possible animals and 10 backgrounds (for total of 320 captions). 3) caption for single animal position prediction that increases the number caption up to 640 and lead to a significant drop accuracy for every models. 4) caption two animals class prediction, the model should predict the correct categories of the two animals presented in the images (5120 captions). 5) caption for two animals positions prediction, the model should predict the position of the two animals in the picture (over total of 10240 captions). 6) caption for two textured animals class prediction, the model should recognize blue elephant from red camel. The performances of several VL.Ms models are presented table for which each row corresponds to one of the scenario described previously.
has been shown by Lin et al. 2023] to be mostly solvable without even using the image information at all. Consider that for an image containing a horse eating grass, ARO will propose two captions: "the horse eating the grass" and "the grass eating the horse". The model should predict the correct caption between these two. However, the second caption is impossible, the image, any model can be confident that the first caption is the correct one.
without looking at
Another shortcoming of current benchmarks is that most of them probe only if the model is correctly able to understand the relations or the attributes between objects. However is not clear if the failures in finding the correct relations or attributes come from the model not understanding them or come
8

---

from not understanding which objects are present in the scene. For example, to understand complex relations like: photo of an elephant on the left and a camel on the right in a desert background. the model should first be able to identify whether the background of the picture is an actual desert. Then, the model should identify whether there an elephant on the picture. It should understand what is an elephant on the left. The model could be very effective at identifying individual elephants or camels, but it could unexpectedly fail when a camel and an elephant appear in the same picture. If the model does not fail in recognizing the animals, then we can probe the model to evaluate the position of each of them. We built PUG: SPAR with this goal of having a progressive evaluation scheme in which we can easily determine exactly what are the failure modes of given VLM. From basic scene understanding to complex relations and attributes, our dataset offer a simple and yet effective way to get a better understanding of the capabilities and limitations of VLMs.
The dataset contains 43,560 images with the associated factors of variations: 10 backgrounds, 32 animals, 4 relations (left/right, botton/top) and 4 animal texture attributes (blue/red, grass/stone). We have images containing either 1) only the background (for scene recognition) 2) only one animal at different left/right bottom/top position 3) two animals at different left/right position. And for each of the scenes (either single or multiple animals), we vary the texture of the animals and the background to evaluate the robustness of the model. Our setup and experiments are presented in Figure 6 in which we display some images from the dataset with the corresponding captions used for evaluations. In our benchmark, we used 6 different types of captions to evaluate the following: 1) Scene Recognition (first row) 2) Single animal classification (second row) 3) Single animal position detection (third row) 4) Multiple animals classification (forth row) 5) Multiple animal position detection (fifth row) 6) Multiple animal and textures prediction (sixth row). For each of them, evaluate the top-1 retrieval accuracy of the correct captions within the set of captions associated to each setup. We evaluate multiple models on these setups: OpenAl CLIP RRaford et al.) [021]]
bottom/top
OpenCLIP [lhaacoo et al. [021], Flava (Singh et al. 2022]. BLIP (Li al. 20226), X-VLM [Zeng] iet al. 1021 and NegCLIP [Yuksekgonul et al. 2023). Most of the models are correctly able to solve the scene recognition task (which is not surprising since we used only 10 environments which are very different from each other). Concerning the simple object recognition task when using a single animal, the performances across models is highly variable. Our experiments also highlight that the VLM performance in a multiple animals detection setting are much worse than the performance in a single animal detection setting. Those experiments show that despite their successes, VLMs are far from having good understanding of the world and that improving the robustness of these models is a needed step for world robustness.
Inspired by Winoground (Thrush et al. 2022), we present an experimental setup in which we leverage hard-negative pair of images. Instead of performing caption retrieval within all captions associated to given setup, we performed caption retrieval between the correct and the hard negative caption. For example. the hard negative caption of "An elephant on the left of the picture and camel on the right of the picture" will be "A camel on the left of the picture and an elephant on the right of the picture". In addition of switching the relation (left/right and bottom/top), we also provide hard negative captioning for the attributes (blue/red and grass/stone). In Table 2 we present our results using the hard-negand negative pair. We clearly observe that none of the models are able to predict the correct captions, with many models being close to random performance (50%).
3.4.1 PUG: AR4T
Lastly, we introduce PUG: AR4T (Attributes and Relations for training). In contrast to PUG: SPAR which is only an evaluation benchmark, PUG: AR4T was created as an additional fine-tuning dataset for \ 8 As shown in the previous section, VLMs struggle to understand spatial relations or attributes and thus are good candidates for our fine-tuning scenario. PUG: AR4T contains 249,986 training images with captions and 23,216 test images"] [ In Table 3 we present CLIP fine-tuning results on the ARO and PUG: SPAR benchmark. We also compare our results against Syn-CLIP, which is CLIP fine-tuned on the SYCIC synthetic dataset. Our results are very similar to Syn-CLIP, but Syn-CLIP training requires several additional tricks to arrive at this performance (Section 3.2 in
"The assets used to create PUG: SPAR and PUG: AR4T are different enough such that PUG: SPAR is still a good benchmark to evaluate models fine-tuned with PUG: AR4T.
"We also run experiments with a version of this dataset which contain IM images but as shown in Table 3 adding more images does not increase performance on PUG: SPAR. We only release publicly the version of PUG:AR4T that contain 249,986 images.
9

---

0-320paCCpp 2B
G-14 OpenE'LIP LI 3B
1-14 OpeaClip
OpencLIP LI
B-32
1.14
M-14
Fiara
BL.1P
XVLM
NegC7 LI
Caption
Values
photo of [choracter]
LeitRight
49.53
72
49.84
19.22
48.91
[pocttion] of the picture in (hor groundd pevirommnnt
Botiom/Top
54.84
$2.34
66.87
60.62
56. 56
38.91
50.47
34.84
53.28
54 06.06
platto of [chaeurter] [postion]] and [choracter] on the [pesition]] in a [bockgrood] envirounent
LettRight Betcom/Lop 31.15
53.88
56.23
55.74
54.44
54.36
33.84
33.51
57.24
56.49
35.25
58.87
54.09
"A photo of Character) retured with [esturll] I and a [churacter] resuoed with [resture] in (backrouud)) onvrrommnnt
BlueRRed GreseStone $2.79
32.77
33,63
56,43
56.59
55.48
54.32
54.22
37.33
56.19
51.74
54,14
56.31
$7.28
56.62
57.19
54.53
53.94
$4.26
49.92
We present the performances of several VLMs with hard negatives captioning on PUG: SPAR in which we perform retrieval between two captions: the correct caption and the hard-negative corresponding caption. In that instance, the model should choose the correct caption between both of them (the probability get the correct one with a random model would 50%). Interestingly, none the model presented in this table seem to be able to get a real understanding of simple position (left, right, bottom, top) or colors.
Table 2:
Cascante-Bonile et al.) 2023)). On the other hand, the photo-realistic nature of PUG: AR4T enables us to match Syn-CLIP without any of these additional bells and whistles. However, even if we note some improvements on ARO, we are still far from having a model able to understand spatial relations. our PUG: SPAR benchmark for which the improvement on
This is highlighted by the results given single animal position prediction is still only above random chance while there is no improvement on the double animal location prediction task. This confirm the unreliability of the ARO benchmark highlighted by Lin et al. [2023].
ARO
PUG: SPAR destriggt
VCI-Relation (Macro Accuracy
VG-Amtihniin (Macno Accuracy
COCO-Onter (Precision
Fickr10l-Order (Precision@
Average
Single
Druble
(Precisson 1)
(Precision
CLIP-VIT- B/3- (4005))
59.16 135.50
62.18
61.52
47.96
$9.98
$1.32
49.84
$4.42
FT W/ Syn-CLPPP
71.40
66.94
59.06
70.96
67,09 (49.77)
N/A
NIA
PUGAARR 2200)
68:36 173.6115.18
65:54
64.44
$7.80
69.74
65.36 (+8.04)
50.78:+0.94)
S4.231-0 (9)
PT PUGAARTT
71.03 176.7
65.15
64.32
61.07
72.84
67.52 (+103))
50,
30,18+00.32)
S4. $4.19(-0.33)
Table Fine-tuning CLIP on PUG: AR4T. For VG-Relation and Attribution, the results (Acel Ace2) indicate
macro-accuracy across relations and attributes (Ace1), and macro-accuracy on the subset of relations and
attributes present in both ARO and PUG (Acc2). For PUG: SPAR, we evaluate on images in which there is only one animal (Single) or two animals (Double) with the relation being left right. We were not able to run SynCLIP PUG: SPAR because the model was not public the time of the publication.
4 Conclusion
The fine- ane-grained controllability of synthetic rendered image data makes ideal for designing chal- lenging evaluation benchmarks to better understand the properties and limitations of vision models, as well as for controlled training scenarios - if only was closer to real data. To this effect, we introduced PUG datasets for representation learning. By leveraging the photorealism of the Unreal we created 4 new datasets and showcased their utility for robust evaluation. We showed how PUG: Animals could be leveraged for OOD generalization and to study properties of the representation spaces. We developed PUG: ImageNet as a new challenging benchmark that researchers can easily use to assess and compare the robustness of image classifiers. With PUG: SPAR we provide a reliable benchmark for vision- Language models while PUG:AR4T offer additional data that could be leveraged to fine-tune VLMs. Together, the PUG family of datasets represents new standard of photorealism and control for synthetic image data.
10

---

References
A. Abbas and S. Deny. Progress and limitations of deep networks to recognize objects in unusual poses, arXiv preprint arXiv:2207.0800 08034, 2022.
M. A. Alcorn, Q. Li, Z. Gong, C. Wang, L. Mai, W.-S. Ku, and A. Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4845 4854, 2019.
M. Arjovsky, Bottou, I. Gulrajani, and Lopez-Paz. Invariant risk minimization, 2020.
P. Astolfi, A. Casanova, J. Verbeek, P. Vincent, A. Romero-Soriano, and M. Drozdzal. Instance- conditioned gan data augmentation for representation learning, 2023.
S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet. Synthetic data from diffusion models improves imagenet classification, 2023.
H. Bansal and A. Grover. Leaving reality imagination: Robust classification via generated datasets, 2023.
A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
D. Bouchacourt, M. Ibrahim, and A. Morcos. Grounding inductive biases in natural images: in- variance stems from variations in data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19566-19577 Curran Associates, Inc., 2021. URL http::ttttttttteeeeeeeeeee http:///wwopppeeoooooooooooooora
neurips cc/ pdf
Eeccante-Bonilla K. Shehada, J. Smith, S. Doveh, D. Kim, Panda, G. Varol, A. Oliva, V. Ordonez, R. Feris, et al. Going beyond nouns with vision & language models using synthetic data. arXiv preprint 2rXiv:2303.17590 2023.
A. X. Chang, T. Funkhouser, I.. Guibas, P. Hanrahan, Q. Huang, 7. Li, S. Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
M. Deitke, D. Schwenk, Salvador, I.. Weihs, O. Michel, E. VanderBilt, I.. Schmidt, Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022.
P. Dhariwal and A. Q Nichol. Diffusion models beat GANs image synthesis. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021. URL forum?id AAWuCvzaVE
net/
A. Diwan, L. Berry, E. Choi, D. Harwath, and K. Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionaliit. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2236-2250, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguisties. URL https://aclantholoogy org/
2022. emn www.eennmmmme w-mppmmmnnnmmmmm 1ammiin.113... 143
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale, 2021.
C. Eastwood and C. K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. URL httpss //openreview.o net/

---

EpicGames. https:///www unrealengine. com. Unreal Engine is a copyright of Epic Games, Inc. and its affiliates (collectively, "Epic"). Any use of images, datasets, or other content made available by Epic, including without limitation through the Unreal Engine Marketplace or the Epic Games Launcher, in connection with your use of the PUG environments and datasets we' ve outlined in this paper and released publicly in connection hereto (the "PUG environments and datasets") or otherwise, is subject to the Epic Content License Agreement available at https //www. unreal engine. com/ en /S//ela//e///// cwwttetoottt//////eeeeeeeeeaactt or other agreement between you and Epic.
K. Ethayarajh, D. Duvenaud, and G. Hirst. Towards understanding linear word analogies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3253-3262, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/ 11/P19-1315. URLhttpss httpp/////laaahhoooogo.o
otg/P19- 1315
C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al. Threedworld: platform for interactive multi-modal physical simulation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
R. Gandikota, J. Materzynska, J. Prootto-Kaffmnn and D. Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2333.073345, 2023.
T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 92, 2021.
M. Ghifary, W. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. 2015 IEEE International Conference Computer Vision (ICCV), org//orrpuu 12825123
pages 2551-2559, 2015. URL /Rttpsts://ppp http:/app.ssmaaaan www.eetiiccnoooo.eeee.
M. W. Gondal, M. Wuthrich, D. Miladinovic, F. Locatello, M. Breidt, V. Volchkov, J. Akpo, O. Bachem, B. Schölkopf, and S. Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus Buché-Buc,,a E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:tts:: http::::::po.ccccpiiiggeoooooeee neurips cc/ Paper....
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. QL IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? In The Eleventh International Conference on Learning Reprexentations, 2023. URL htttps://openreviewt nUmCCZ5RKF]
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
M. Ibrahim, Q. Garrido, A. Morcos, and D. Bouchacourt. The robustness limits of sota vision models to natural variation. arXiv preprint vrXiv:2210.13600 2022.
B. Y. Idrissi, D. Bouchacourt, R. Balestricro, I. Evtimov, C. Hazirbas, N. Ballas, P. Vincent, M. Drozdzal, D. Lopez-Paz and M. Ibrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. arXiv preprint arXiv:22/1.018866 2022.
G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https://doi. http:///ooooorgg111/ 10. 5281/zenodo. 5143773 If you use this software, please cite it as below.
A. Jahanian, X. Puig, Y. Tian, and P. Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations, 2022. URL
https://opeerevierr
S. Jain, H. Lawrence, A. Moitra, and A. Madry. Distilling model failures as directions in latent space. In The Eleventh International Conference on Learning Representations, 2023. URL htttss//opeereviiee wtt.:eeeeeeeeooo/d/dddg//rrr/aaa
12

---

J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
D. Klee, O. Biza, R. Platt, and R. Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In The Eleventh International Conference on Learning Representations, 2023. URL
A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, Big transfer (bit): General visual representation learning, 2020.
J. Lee. Introduction to Topological Manifolds. Graduate texts in mathematics. Springer, 2000. ISBN 9780387950266. URL http:://oookk...e google. fr wttpbooksidddloeoooo.oo
MC
K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. International Journal of Computer Vision, 2018. doi: 10.1007/s1
D. Li, H. Ling, S. W. Kim, K. Kreis, S. Fidler, and A. Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 21298 21088821118.... IEEE, 2022a. doi: 10999CVVR22688.2222222222222 URL htt.t:htp//doo org/ 10V9CC/P5/66 2022. 02064
J. Li, D. Li, C. Xiong, and Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022b.
T. Lin, M. Maire, J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll'a r, and C. L. Zitick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312. 2014.
org/ abs/ 1405. 0312
Z. Lin, X. Chen, D. Pathak, P. Zhang, and D. Ramanan. Visualgptscore: Visio-linguistic reasoning with multimodal generative pre-training scores, 2023.
J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui. Towards out-of-distribution generalization: A survey, 2023.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
S. Madan, T. Sasaki. T.-M. Li, X. Boix, and H. Pfister. Small in-distribution changes in 3d perspective and lighting fool both CNNs and transformers. URL URL http::/arxiv org/ abs/ 2206 16198
S. Madan, T. Henry, J. Dozier, H. Ho, N. Bhandari, T. Sasaki, F. Durand, H. Pfister, and X. Boix. When and how convolutional neural networks generalize to -utttfidiitriiuttio sategory-viewpoing combinations. Nature Machine Intelligence, 162)):16-113, Feb 2022. ISSN 2522-5839. doi: 000888442256-001-0047777 URL, httttp:://aal.orr/110113333 11w.:1/////2225002-000---80
Mishra, R. Panda, C. P. Phoo, C.-F. R. Chen, L. Karlinsky, K. Saenko, Saligrama, and R. Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition, pages 9194-9208 9204, 2022.
M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fermandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
S. Sagawa*, P. W. Koh*, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https: ftttpp://operreviie net/ forum? Darumfid=yySGuJFFvv
13

---

M. B. Sariyildiz, K. Alahari, D. Larlus, and Y. Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
I. Shumailov, Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and Anderson. The curse of recursion: Training on generated data makes models forget, 2023.
A. Singh, R. Hu, V. Goswami. G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.038860 2022.
O. Struckmeier, Tiwari, and V. Kyrki. Autoencoding slow representations for semi-supervised data-efficien regression. Machine Learning, pages 19 2023.
A. Szot, A. Clegg, E. Undersander, Wijmans, Y. Zhao, J. Tumer, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, Malik, M. Savva, and Batra. Habitat 2.0: Training home assistants to rearrange their habitat, 2021.
K. S. Tai, P. Bailis, and G. Valiant. Equivariant transformer networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages v97/tai19a. html
PMLR, 2019. hotLitttp//prrceeddine mlr press/
T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross, Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
B. Trabucco, K. Doherty, M. Gurinas, and R. Salakhutdinov. Effective data augmentation with diffusion models, 2023.
M. Trager, Perera, L. Zancato, A. Achille, P. Bhatia, and S. Soatto, Linear spaces of meanings: Compositional structures in vision-language models, 2023.
A. Ushio, L. Espinosa Anke, S. Schockaert, and J. Camacho-Collann AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, Aug. 2021. Association for Computational Linguistics.
BERT is to NLP what
T. Wang, K. Lin, L. Li, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang. Equivariant similarity for vision-language foundation models, 2023.
L. Weng. From autoencoder to vae. fllianweng.github... 2018. URL httpp:://iiliaaaoon 1aa---a-ae-- vae/
R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm, 2021.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, Xu, L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: ofate-of-the--e language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Oct. 2020. Association for Computational Linguistics. URL. https:://////t aclweb. orgoooggnaoo anthhooothhy/o/oe 2020 2020. emnlp-
natural
demos.6 6
K. Xiao, I.. Engstrom, A. llyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.0999 2020.
14

---

S. Xie, A. S. Morcos, S.-C. Zhu, and R. Vedantam. COAT: Measuring object compositionality in emergent representations. In Chaudhuri. S. Jegelka, L. Song, Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 14388-24411 PMLR, 17-23 Jul 2022. URL.httpe htattps:/prroceedingg. mlr paress/v662/xee
html
R. x. H. Z. S. P. C. Xingxuan Zhang, Yue He. Nico+:: Towards better benchmarking for domain generalization, 2022.
M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. URL
Y. Zeng, X. Zhang, and H. Li. Multi Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276 2021.
Y. Zhang. H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
15

---

A Limitations and Future Work
A.1 Limitations
In this work, we introduced 4 new datasets that were created using the Unreal Engine. We presented of case studies, demonstrating how these datasets can be leveraged to improve both evaluation and training for representation learning. Deeper work would be needed to fully unlock the potential these datasets can offer. In addition, we merely scratch the surface of what a powerful engine such as the Unreal Engine can offer. With advanced techniques such as Lumen, Nanite and Megascans, it is now possible to create even more realistic environments. In addition, the datasets we provide have single simple label, whereas future uses could easily provide detailed rich labels for the entire scene underlying each generate image.
A.2 Future Work
A long vision for the PUG family would be to yield a series of benchmarks that can probe the robustness of vision models. PUG: ImageNet is a first step this direction, however we might want to get more factors of variation such weather and occlusion. A second take would be to increase the richness of the labelling by making available detailed segmentation masks and labels. Making short video dataset in which we have complete control over the factors is also a promising future direction since research on video still far behind what can be done with images. The active learning pipeline direction is also worth to explore since the model could bias the PUG environment to produces samples that are the best for a specific downstream task (sishrr et al. 2022
B PUG Datasets
The datasets PUG: Animals, PUG: ImageNet, PUG: SPAR and PUG:AR4T are available under the ce-by-ne license with the restrictions that they should not be use to train generative AI models. They available to download the following website: https://puu datasets can be read by a torchvision ImageFolder. We have one class by folder and all the images associated to one class are saved in this folder as png. There is csv file associated each dataset that map a filename (with unique ID) to its associated factors of variations. Examples of dataloaders http:/ttw..oooeaccccooeeeeeaaaaa
me metademolab ab. com/ The
available
B.1 Datasheet
MOTIVATION
For what purpose was the dataset cre- ated?
The 4 datasets presented in this paper were created for representation learning research. PÚG: Animals is a strong dataset for OOI) research as well as for being able to better probe the representation of vision models. PUG: ImageNet was designed an additional benchmark for ImageNet pretrained model to offer a better comprehen- sion of vision models capabilities in term of robustness to specific factor of variations. PUG: SPAR showcase how synthetic data can be used to evaluate VLMs under- standing while PUG: AR4T can be leveraged to fine-tune them.
Who created the dataset and on behalf of which entity?
This dataset was created by the FAIR team at Meta AL.
Who funded the creation of the dataset?
Meta.
COMPOSITION
16

---

What do the instances that comprise the dataset represent?
The instances represent images of animals in various envi- ronment for PUG: Animals. In contrast PUG: ImageNet contains 151 object classes (the full list is available in Ap-
pendix [B.3p. PUG: SPAR uses the sames assets PUG:
Animal while PUG: AR4T use the same objects as PUG: ImageNet.
How many instances are there in total?
PUG: Animals: 215 040 images; see Appendix B.2
PUG: ImageNet: 88,328 images;
see
Appendix
B.3
PUG: SPAR: 43,560 images; see Appendix B.4
PUG: AR4T: 249,986 images for training and 23,216 test
images; see Appendix B.5
Does the dataset contain all possible in- stances or is it sample (not necessarily random) of instances from larger set?
PUG: Animals contains all possible combination of factors of variations. In contrast PUG: ImageNet was sampled by changing only 1 factor at time and is therefore random sample of the distribution. Images in PUG: SPAR sampled using all possible combination of factors of variations (with the exception that for the attributes the blue grass animal is always the left). Image-text pairs in PUG: AR4T were randomly sampled.
What data does each instance consist of?
Animals, PUG: ImageNet, PUG: SPAR release images along the factor variation. For PUG: SPAR, release the script to generate the captions from the factors of variations. For PUG: AR4T, we release images along with corresponding captions.
For PUG:
17

---

Is there label or target associated with each instance?
Yes, csv file. Each instance have row in this esv files with all the factors of variation used to generate this image. For PUG: Animals, the csv files contains the following columns:
filename, world_name, character_nang character_scale camera_yaw, character_ sexacteer_exture
while for PUG: ImageNet, it contains:
filename, world_nama character_nee _nam,, character_
character pearacter_rotation_roo character perratterrootatioonppttee camera_roll, camera_pitch, camera_yaw, character_texture scene_light.
charactersscale
For PUG: SPAR, the csv contains:
filename, world_name, character_name character2_ _eaate,e___m_e,,,, characteri_pon character2.pos. Character_texturee character2_textun
For PUG: AR4T, the csv contains:
Relation, Actor Actor2Category, ActorlName, Actor2Name, ActoriLocation, Actor2Location, ActorlBotation, Actor2Rotation, Actor1Scale, Actor2Scale, Actor[Texture, Actor2Texture,
ActorlAttribute, Actor2Attribute, Camera_roll. Camera_pitch, Camera_yaw, caption, alt_ alt_caption, Level, World. Name, filename, filename_neg, filepath
any information missing from individ- ual instances?
No, relevant information is included.
Are relationships between individual in- stances made explicit?
N/A.
Are there recommended data splits?
There is no specific split concerning PUG: Animals be- cause this dataset should be used for OOD research. We primarily let the researchers choose their own held out vriining/validaationttssng split to train their models. In contrast, Ima:leeeeNet and PUG:SPAR should only be used as an additional test set. For PUG: AR4T, splits are described in Appendix B.5
Are there any errors, sources of noise, redundancies in the dataset?
We did explicitly filter for occlusion, so some im- ages may contain occluded views. PUG: Animals and PUG:SPAR are very clean and each animal easily iden- tifiable. In contrast, PUG: ImageNet and PUG: AR4T leverage assets from Sketchfab and the asset quality vary significantly.
18

---

Is the dataset self-contained, or does it link or otherwise rely on external re- sources?
The dataset is self-contained however the assets that were used build the dataset belongs to external sources which are listed in the github at htttps://github.... /ttp//o.othbocco//
facebookresserchh/PUO
Does the dataset contain data that might be considered confidential?
No.
Does the dataset contain data that, if viewed directly, might be offensive, in- sulting, threatening, or might otherwise cause anxiety?
No.
COLLECTION
How was the data associated with each instance acquired?
The data (3D assets) were acquired through the Unreal En- gine Marketplace https: //s:://wwww www. unrealengine com/ marketplace/ en USe store and Sketchfab https:// sketchfab... cote/f/ba.omm///. Assets were then incorporated into the Unreal Engine to generate realistic 3D scenes and cor- responding images. The 3D assets were manually selected to ensure high quality.
What mechanisms or procedures were used to collect the data?
Manual human curation. Assets were manually collected.
If the dataset is a sample from a larger set, what was the sampling strategy?
For PUG: Animals PUG: SPAR, combinations are included. For PUG: ImageNet and PUG: AR4T, random sample of possible combinations provided.
Who was involved in the data collec- tion process and how were they compen- sated?
Only the authors of this work were involved.
Over what timeframe was the data col- lected?
The data were collected between June 2022 and June 2023
Were any ethical review processes con- ducted?
No.
Did you collect the data from the indi-
Third parties: Unreal Engine Marketplace http:l/cccceetttt///::::: umrealengine wwt.///www///aece///////e/eaaaee US/ store and Sketchfab htttps://kkttchfabbb com/
viduals in question directly, or obtain it via third parties or other sources (e.g., websites)?
Were the individuals in question notified about the data collection? If so, please de- seribe (or show with sereenshots other information) how notice was provided, and provide link or other access point to, or otherwise reproduce, the exact lan- guage of the notification itself.
personally identifiable information in datasets as they are purely synthetic and contain no images of people. We purchased 3D assets from different mar- ketplaces where required, however we did not explicitly contact the individual creators.
There
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots other information) how consent was requested and provided, and provide a link or other access point to, otherwise reproduce, the exact language to which the individuals consented.
N/A. See above.
19

---

If consent was obtained, were the con- senting individuals provided with mech- anism to revoke their consent in the fu- ture or for certain uses? If so, please provide description, as well as a link or other access point to the mechanism (if' appropriate).
N/A. See above.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g,, data protection impact analysis) been conducted? If so, please provide description of this analysis, including the outcomes, as well as link or other access point to any supporting documentation.
No data about specific individuals included in these data. See above.
PREPROCESSING
Was any pregroessingcctaninn/aaiiiig of the data done?
N/A.
Was the "raw" data saved addition to the preprocssa//clemccllabeeedd data?
N/A.
Is the software that used to prepro- cess/clean/label the data available?
N/A.
USES
Has the dataset been used for any tasks already?
Yes, these data used for the experiments that presented in this paper.
Is there repository that links to any or all papers or systems that use the dataset?
No.
What (other) tasks could the dataset be used for?
These Datasets could be used widely for evaluating and training neural networks. For example, assessing disentan- glement of models with respect to PUG: Animals factors of variation (e.g. with DCI metric Eastwood and Williams (2018))
Is there anything about the composition of the dataset or the way it was col- lected and preprocesedddlennd//hhbee that might impact future uses?
No.
Are there tasks for which the dataset should not be used?
These datasets should not be used for generative mod- elling purposes.
DISTRIBUTION
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?
Yes, the dataset will be publicly distributed.
How will the dataset will distributed?
Tarball on a website.
Will the dataset be distributed under copyright other intellectual prop- erty (IP) license, and/or under applicable terms of use (ToU)?
The license of the dataset ce-by-nc with the mention that these data should not be used for generative AI purposes.
Have any third parties imposed IP-based or other restrictions on the data associ- ated with the instances?
See EpicGames
20

---

Do any export controls or other regula- N/A tory restrictions apply to the dataset or to individual instances?
MAINTENANCE
Who will be support-
Meta AI.
ing/hosting/maiitaiinng the dataset?
How can the owner/curatorr/manger of the dataset be contacted?
Please contact the corresponding author of this paper.
Is there an erratum?
No.
Will the dataset be updated?
Yes the dataset will be updated with versioning.
If the dataset relates people, there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
N/A.
Will older versions of the dataset continue to be sup- ported//husteummintaineee
It depends. If the dataset is updated because one of the asset creators has requested to remove their assets, we will not continue to host the dataset containing these sets. Only the newer version of the dataset which will not contain these assets will be available.
If so, please describe how. not, please describe how its obsolescence will be communicated to dataset consumers.
If others want to extend/augment/build on/contribute to the dataset, is there mechanism for them to do so?
No mechanisms are in place yet, but they can contact the authors of this paper if they would like contribute.
Table 4: Datasheet for PUG, following the framework introduced by Gebru et al. 2021
B.2
PUG: Animals
PUG Animals contains 215 040 pre-rendered images using 70 animals assets, 64 backgrounds, sizes, 4 textures, under 4 different camera orientations. To create PUG: Animals, from the following bundle in the Epic Game Marketplace (https://wwy unrealengine com/ marketplace/ Aaa-US/producct complete wwwwaannmmeaiiiaiaaa hotpa//ooocoooaaaaaaaaaaaeerrrre can be found in the dataset folder or at htthttpp///githhh ccmm/facceboooeeeeercc//UUu Below, list all the values for the factors of variation, have used:
Animals assets
The list of environments used
World Name ["Egypt", "Desert". "AmusementPark" "ArcadeClub" "Arrenaaa"."""na" "Battlegroudd". "Catacombs". "Tableland", "EuropeanStreel", "Junk""rd", "OveanFloor", "Racetrack", "Ruins". "SciFiCity". "SciFiGarage", "Spacelsland", "Spacellangar", "SpatialStation". "TokyoDay", "Toky- oNight", "TrainStation", "Bridge", "Beach", "BusStatioonnnerrio" "BunStaaionFxteerio""" "Suhway". "IndoorStairs", "Bar", "ScreeningCheekpoinn"" "Cireus", "Appartment", "Hallway", "TrashRoom", "FuturisticSubway" "Footbridge", "BoxingRing". "Hangar", "Mansion", "ShoppingMall", "Confer- enceRoom". "SpacePort". "VillageOutskin"" "VillageSquare, "aaqquarr"""Couttteei
"Courtyard", "ElvenRuins". "Forge", "Library", "Museum". "Gallery", "ModernGalleryy. "Opera". "AncientAgora", "Restaurant". "Ru- ralAustralia", "A "AustralianRoad". "ShadyRoad", "SaltFlats", "Castle", "StylizedEgypt", "Temple", "Snow", "Grass,, "DryGrass", "Forest"],
Character Name ["Goldfish', "Caribou", "Elephant", "Camel", "Penguin",
"Cassowary". " "Z"ebera""""""ebra...
"Turte": "Bearr"."Beer", "Bear". "Beaver", "Capyhara.,"'rooodiee "Crocodile", "Armadillo"; "aaillo"""""""" "Cat". "Gecko","Car "Crow", "oow""""o GiantAntttet,"..""""""
"GiantTortoiis"" ; "KomodoDragon" "Rhinoceros" "Dolphin", "EarlessSell", "FruitBat", "Goat", "Hippopotamus", "Horse"; ""Horse","Impatei" "Impala", "Lion", "Orea", "Pig". "Rag"."Rabit",, """abii",""quirre"""" "Tapir"; "Wildbeest". "Wolf" "Wolf","Aly""" "Anlylosaurus", "BlackRockFish "PParasurolophus" "PoisonDDarFrro" "Spinosaurus", "Triceraptos", "Chicken", "Roooi
"HarpyEagle" "Ostrich"" "Raven". "RedCrownedCrane" "Robin" "Seagull", "Secretarybird". "Shoebill", "Swan".
21

---

"Toucan", "Vulture" ," "Ammonite", "Ant", "Scoorpion"" "GoldBeetle", "Hornet", "SnnoCrab", "Tarantula""""""""" "WhiteShark", "Tuna", "Arowana", "Ayu", "Betta", "Koi", "Pirarueu". "Salmon", "Cattle", "Jerboa"),
Character_Seaie [0.7, 1.0, 1.3],
Camera Yaw [0, 45, 225, 315],
Character_Texturee ["Default", "Sky", "Grass", "Asphalt"]
PUG: Animals is built by using all combinations of the factor of variation above. Figure 7 we show random images from the PUG: animal dataset that highlight the diversity of this dataset.
Following Liu al. 20233. we present comparison Table with other datasets that are often used in OOD research:
Colored MINST
MNIST-K
Waterbirds Sagawa" al. 2020
Rased-Carr
Nieo++ Xingxuan 2hang
PUG: Animals
Image Data
Anjosky
2020
Ghifary
2015
Maden
2023
X02
Doomains
10
54
CCuegonnes
30
70
Ecamples
4.8L
450k
250k
21%
Shift Type
Coke
Anglec
Background
Views
Backgroug Objects
Back/Text View
Image Type
Digits
Digis
Birds
Synthetic Curs
Synthetic Animals
Table 5: Comparing PUG: Animals with other datasets traditionally used for OOD research. In
contrast to other datasets that have variations across only single domain, that have noisy annotations that are unrealistic, PUG: Animal over high quality images with reliable annotations across
different domains such as the background, and view.
B.3 PUG: ImageNet
PUG: ImageNet contains 88,328 pre-rendered images using 724 assets representing 151 ImageNet classes with 64 backgrounds, 7 sizes, 10 textures, 18 different camera orientation, 18 different character orientation and 7 light intensity. Below is the values we used for each of these factors:
World_Name l'Egypt" "Desert", "AmusementPark"" "ArcadeClub", "Arena", "Battleground", "Catacombs", "Tableland", "EuropeanStreet", "JunkYard" "OceanFloor", "Racetrack", "Ruins". "SciFiCity". "SriFiGarage". "Spacelsland", "SpaceHaagarr, "SpatialStation"" "TokyoDay". "Toky- oNight", "TrainStation", "Bridge", "Beach", "BusStationInterior"" "BusStationExterior." "Subway". "IndoorStairs". "Bar", "ScrreningChheckpoint, "Circus". "Appartment", "Hallway". "TrashRoom". "FuturisticSubwaa"" "Footbridge", "BoxingRing". "Hangar". "Mansion", "ShoppingMall", "Confer- enceRoom", "SpacePort", "VillageOutskin" "VillageSquare", "Courtyard", "ElvenRuins", "Forge", "Library", "Museum", "Gallery", "ModernGallery" "Opera", "AncientAgora", "Restaurant", "Ru- ralAustraliia, "AustralianRoad", "ShadyRoad", "SaltFlats", "Castle", "StylizedEgypt", "Temple", "Snow", "Grass", "DryGrass", "Forest"],
Character _ame 724 Sketchfab assets (See github for the list)
Character Charrcter_Rotattonn Laaation_aa>">a [0, 45, 135, 180, 225, 270], Character_Rotation Roll [45, 90, 135, 180, 225, 270].
Character_Rotationn Pitch [45, 90, 135, 180, 225, 270].
Character_Scake [0.5, 0.6, 0.7, 0.8, 1.3, 1.6). Camera Roll [45, 90, 135, 180,
Camera_Pitch [240, 260, 280, 300, 320, 340),
Camera Yaw: : [0, 45, 135, 180, 225, 270].
Character Texture Textuue ["Default", "Sky", "Green", "Gray", "Red", "Grass", "Color", "Black",
"Curtain").. I.,
Scene_ Light: ["255,255 255,255,0". '0,0,255,0", "0,255,0,0", "255,0,0,0", "0,255,255,0", "255,0,255,0", "255,255,0,0") (The value for the lights are in RGBA format)
To generate PUG:ImageNet, we change only one factor at time for each assets. When changing the background (World _ame), all the other factors (Camera//Oject Orientation, Size, Texture, Light)
22

---

Figure 7: Random images taken from the PUG: Animals dataset.
at 0 or at their default value. When changing the other factors (Camera/Object Orientation, Size, Texture), the background is set to "SaltFlats 0" (Which is the most basic background). When changing the light of the scene, have used the environment "Opera".
The assets PuG:ImageNet were selected base on 151 ImageNet classes which are listed below: ['BirdHouse', 'Chest', 'Bagel', 'WarPlane', 'Rocking_ Chairr, 'Bridge', 'Street_Signi 'Cabbage', 'Pay Pay_e', "ButternttSSquuu 'JellyFish', 'Jack O O_Lateemn, 'Bookcase'' 'Stonewall', 'Punching_ Bag', "Toaster', 'Mush- room'. FFrog.. 'Jeep'. "Television', 'Pineapple', 'Vacuum', 'Torch', "Carousel', 'Desk", 'WineBottle, "Wallet',
23

---

'Dining Table'. 'Military uniform', 'Car Wheel', "Table Lamb', 'Digital Watch', 'Electric Fan', 'Sweat- shirt', "Komodo dragon', 'Racket'. 'Cheeseburger', 'Can Opener', "Pomegranate", 'Convertible', 'Laptop'. "Chicken_hen hene,, "Wolf', "Bulletproof_ _eett, 'Shield', 'Bathtub', "Throne', 'Lighter', 'Byciele', 'Cofee_Miu M_og_M Sionor_Soottrr,, 'Jaan', 'Soccer 'Socer_Balll, "Vending_macee machine', 'Hatchet', 'Umbrella', 'Bear', 'Artichoke', 'Vase', 'Radiator', 'SpaceShuule', "Manhole_ Manhole_Cover', "Polaroid_ "Cannrdd_'_a,,, "Traffic 'Traffic_Light', 'Radio', "Sou__owwl 'Zac- chini', 'Barel', Tennns Ball', "Sunglasses', 'Microwave', 'Joystick', "Aireraft_ Carrier', 'Fox'. 'Subbmrine", 'BasketBall', 'Running Shoc', 'Chain-saw', 'Piano', "Crate', 'Loupe'. 'Minivan', 'Shirt', 'Airliner', 'Sock', 'Shovel', 'Mask', 'Tractor'. 'Sandal', 'Wooden_ Spooo', 'Drum', 'Goldfish', 'Gasmask', 'Mailbox', "Volley Ball', 'Banana', 'Penguin', 'Sliding Door'. 'Pool Table'. 'uurito', 'Candle', 'Purse'. 'Canoe', "Typewriter Keyboard', 'Espresso maker', 'Carton', 'Park Bench', 'Screen', 'African crocodile', 'Cat', 'Hay'. 'Elephant', "WaterBottle, 'Modem', 'Palace'. 'Ie Cream', 'Washer', 'Sewing Machine'. 'HairDryer', 'Rabbit', 'Dishwasher', "Bell_Peppes Ambullacce,, Frrech__aaat 'Refrigerator', "Mouse", 'Ohelisk', 'Starfish', 'Brocolli,, 'Microphone', "Great white shark', 'Power-drill', 'Locomotive', 'Perfume', 'Whale", 'Serewdriver', Dial_telephon 'Backpack', 'Harmonica', 'Binocular', 'Skirt', 'Pizza', 'Cowbny_Har 'Computer Keyboard', 'Kangarou', 'Baschall', "Tile_ Roof", 'Lawn nownn_Mowr', 'Safe', Gellular_telephone
In Figure g
we show random images taken from the PUG: ImageNet dataset.
B.4
PUG: SPAR
SPAR contains 43,560 pre-rendered images using 32 animals assets, 10 backgrounds, positions and 4 textures. In contrast with the other PUG datsets, PUG: SPAR contain up to two animal in a single scene. For generating the PUG: SPAR dataset, we utilize the same subset of assets as PUG: Animals.
PUG:
World_Nan Laadd_Name:: 'Desert', 'Arena', 'OccanFloor', 'Racetrack', "TokyoDay', 'Circus', 'BoxingRing',
AAusrraianRoad'' 'SaltFlats', 'Museum'J.
Character _ame : ['Goldfish', 'Caribou', 'Elephant', 'Camel', 'Penguin', 'Zebra', 'Bear', 'Beaver', 'Cattle', 'Armadillo', 'Gecko', 'Crow', 'Scorpion', 'GianfTortoise', 'Tarantula', 'Rhinoceros', 'Dolphin', 'EarlesSSeal' 'Goat', 'Hippopotamus', 'Horse", 'Impala', 'Lion', 'Orca', 'Pig', 'Rabbit', 'Squirrel', 'Chicken', 'WhiteShark,, Anlylosaurus', 'BlackRockFish', "PoisonDart- Frog").
Character_pes ["Left/Right, "Botton/Top"]
Character_ Texture ["Default", "Blue/Red", "Grass/Stone"]
In Figure S
we show random images taken from the PUG: SPAR.
B.5
PUG: AR4T
Assets and Environments For generating the PUG: AR4T dataset, we utilize the same subset of Sketchfab assets as used in PUG: ImageNet. This leaves us with a set of 680 unique assets chosen from across 151 ImageNet categories, each manually inspected to quality control for photo-realism as much as possible. For this PUG dataset we are primarily concerned with object-object and object- attribute information, hence we utilize single camera and character orientation. Since Sketchfab assets differs widely in their scales, we first scaled down the longest edge of the asset bounding box to 150 pixels to normalize the order of magnitude of the asset dimensions, before any further scaling based on attributes. Next, we select total of 26 unique environments as our background environments. The richness of some environments enables us to manually select different camera views each environment as a novel environment view, and we generate a total of 28 visually unique backgrounds for our PUG: AR4T dataset. We provide each background with human-intelligiuo descriptive names for use in the dataset captions (Table The PUG: AR4T dataset is composed with two subset described in the following sections.
PUG: ARaaiinsations For generating the PUG: AR4T-Relations subset, we utilize the spa- tial relationships from Visual Genome that are not symmetric, as noted in the ARO bench- mark Yuksekkonul et al. (2023). The set of relationships used in PUG: AR4T-Relations given in Table [ consists of three unary relations (at, in, inside) and ten binary relations (above, on, on top of, behind, in front of, below, beneath, under, to
24

---

ONEW W
Figure 8: Random images taken from the PUG: ImageNet dataset.
the left of, to the right of.) For each relation, the corresponding objects are picked ran- domly from the set of assets, and placed in the scene based on the co-ord co-ordinates given in Table 7 add a random offset in range [0-25) along each dimension every scene, so that the dataset does not contain shortcuts where the object locations can directly inform the underlying relation. The size of each asset is chosen randomly on a scale of 1-10, where corresponds to 1 110 pixels and 10 to 200 pixels for the longest edge of the scaled asset.
every individual object
25

---

Figure 9: Random images taken from the PUG: SPAR dataset.
The caption for a scene is generated using one of two templates based on whether the spatial relationship in the scene unary or binary:
Unary Relation: "/ImageNet class label of asset 1] [relation] [humann-nttlligible background description]" e.g. Banana inside museum
26

---

Knvironment Name
Descriplive Cuption for PUG
Environment Name
Descriptive Caption for PUG
(with camera view variant)
(with camera view variant)
Arena
arena
IeeRoad
Icy
OulskgeOOtskin
village
Jungle
jungle
VillageSquare
village
Library
librury
Batrlegrund
buttlegrand
Museum
museuum
Beach
beach
OceanFloovr
floor
Bridge
hridge
AncientApon
castle outskirts
Circus
circus
Rucetrack
race track
Cliff
cliff
RuralAustalia
rural wilderness
Courtyard
courtyard
AustralianRoud
desert road
Egypt
SaltFlats
salt flats
RlvenRuins
Spacelsland
space cgypt temple
EuropeanStreet
curopean street
StylizedEgypt
FighringArena
fighting lorge
Temple
l'orge
TrainStation
train station
Table 6: Environments and descriptive captions used in PUG: Attributes & Relations
Object 1 coordinates
Object coordinates
Relation
Negative Relation
(wrt origin)
(wri origin)
At
(0, 0, 0)
N/A
N/A
In
(0, 0)
N/A
N/A
Inside
0)
N/A
N/A
Above
300)
(0, O,
Below, Beneath, Under
top
300)
(0,
Below, Beneath, Under
(0. 300)
(0, 0, 0)
Below, Beneath, Under
Below
(0, 0)
(0, 0, 300)
Above, Top Of,
Beneath
(0, 0, 0)
(0, 0, 300)
Above, On Top Of, On
Under
(0, 0, 0)
(0, 0, 300)
Above, On Top Of, On In front of Behind
Behind
(150, 0, 0)
(-150, 0, 0)
In front of
(-150, 0, O)
(1500, 0, 0)
To the left of
(0, -150, 0)
(0. 150, 0)
To the right of
To the right of
(0, 150. O)
(0. -150, 0)
To the left of
Table 7: Relations, corresponding asset locations wrt to camera origin, and corresponding hard negative relation used PUG: AR4T
Binary Relation: "[ImageNet class label of asset 1] [relation] [ImageNet class label of asset 2| [(random) unary relation] |human--ntelligglle background description]" J" e.g. Banana to the left of chair inside museum
For each binary relation, we also sample the corresponding hard negative scene and caption, by replacing the relation with its negative from Table 7 such that the semantic meaning of the scene and caption represents switch from the original relation between objects. For example, the negative of "Banana to the left of chair inside museum" is given by "Banana to the right of chair inside museum".
We sample a total of 310 binary relation scenes (155 random 155 negatives), and 310 unary relation scenes for each background, thus leading dataset of 112,840 image-caption samples (28 backgrounds x (310 pairs x 10 binary relations 310 pairs 3 unary relations)). The Rel:tilasions dataset generation process described above is summarized psueduocode in Algorithm [
Lastly, we split the dataset into 101,920 train and 10,920 test image-caption samples, such that the test set is balanced by background and relations (28 backgrounds 13 relations x 30 samples). We also release the subset of training and test samples that only contain pairs of objects in scenes along with their hard negatives, which contains 78,400 training and 8,400 test samples, 39,200 training pairs and 4,200 test pairs. This subset enables Winoground Thrush et al. [2022] style evaluation well as training with hard visual negative mining for VLMs in future work. The PUG:Relations dataset generation process described above is summarized as psueduocode Algorithm
For PUG: AR4T-Atrribues the selection of assets, relations between assets, and spatial locations of assets is done exactly as for PUG: AR4T-Relation However, since the focus this dataset is on the object-atrribute pairs, the relations between objects are not represented in the corresponding scene caption in any form. The attribute for each object is chosen randomly from the set of 53 attributes described in Table g For size based attributes, the asset's material instance remains the same but its size is varied between 50 pixels (short, small, little, tiny) and
PUG: AR4T-Attribute
27

---

Algorithm PUG:
AR4T-Relations subset generation
Unary Att In, Inside)
Categories (Set of ImageNet Class Labels)
Environments = (Set of Unreal Environments)
Relations (Set of Relations)
NegRelations = (Dictionary of relations as key, semantically negative relations as values)
Assets (Dictionary of Sketchfab assets, keys being ImageNet Class Labels] }
AssetLocations = (Function that returns locations of assets based on relation with random offset)
Dataset =
for env in Env do
for rel in Rel
& For binary relations we also add the negative scene+caption to the dataset, thus the effective number of samples per background is 2 if rel Unary then
num samples 15
else
num hum_samples 30 end if
for i
to nwm_nammmnnaaaaaaa
caann_ssmlessssee do
catl random. caandom.choiceCCatggrieee
assetl = random. (hsoonchooocccAsssssscattt))
if rel & Unary then
car2 = rww.daaacooiiccccc (aa.ddow..cocceCCaaageiiii.
asset2 www.daaaaooiiicccco wawndoomccooceeesssttssseeeoo
end if
location location2 AssseLoocaionssree)
scenel TorchhMulivverseesssell asset2, locationl, location2, env)
if rel & Unary then
rel2 wwwndom.choiceeUnne
caption| cat +
cat2 + ' rel2 '' env
Dataset U [scenel, caption
Generate hard negative scene and caption
scenel Torchhuuliverss(asseee asset2, location2, location env)
caption2 cat + NegRelations(rel]
cat2 + rel2 env
Dataset |scene2, caption2]
else
captionl catl + rel + env Dataset {scenel, caption!)
end if end for
end for
end for return Dataset
200 pixels (big, long, tall, large). For all other attributes, the material instance of the object changed in Unreal using Pytorch Multiverse.
The caption of the scene is generated using one of templates based on whether the spatial relationship in the scene is unary or binary:
Unary Relation: "[Attribute of asset 1] [ImageNet class label of asset 1] [relation] [human- intelligible background description]" e.g. Green banana inside museum
Binary Relation: "[Attribute of asset [[mmageNet class label of asset 1] and [Autribuee of asset 2][ImageNet class label of asset 2] [(random) unary relation] [human-intellige background description]" e.g. Green banana and large chair inside museum
For each binary relation, we also sample the corresponding hard negative scene and caption, by swapping the attributes between objects such that the semantic meaning of the scene and caption
28

---

Attribute Category
Attribute
Variants
Attribute Category
Attribute
Variants
Brick
Black
2
Metal
Blue
Wood
Yellow
2
Material
Glass
White
Cloth
Green
Plastic
Gray
Color
Rock
2
Brown
Big
Red
Long
Silver
Tall
Orange
Large
Pink
Size
Short
Gold
Small
Striped
Little
Dark
Texture
Tiny
Cloudy
Total= 53
Table 8: Attributes and corresponding number of variants used in PUG: AR4T-Attribuie
represents switch from the original object-attribute associations. For example, the negative of "Green banana and large chair inside museum" given by "Large banana and green chair inside museum". For each of the 53 object attributes, we sample 2 scenes of it in conjugation with another object and attribute, and scenes of the attribute isolation in a scene. We repeat this for each possible background, thus leading to a dataset of 160,272 image-caption samples (28 backgrounds (53 attributes x (53 attributes x 2 samples) + samples)). The PUG:Attributes dataset generation process described above is summarized psueduocode in Algorithm 2
Lastly, we split the dataset into 147,976 train and 12,296 test images. For each attribute pair in a scene, we select 4 image-captin caption samples in the test set (4 samples x 53 attributes x 53 attributes 1 11,236 sample). And we sample the remaining test samples by selecting 20 image- for each attribute in isolation in a scene (20 samples x 53 attributes = 1,060 samples). Similar to PUG-attributes, we also separately release the subset training and test set with scenes and captions containing only pairs of scenes with their corresponding hard negatives, leading to I 146,155 training samples and 1 11,236 test samples, or 73,0739 training and 5,618 test sample pairs.
samples
Caption variants in PUG: AR4T In order to emphasize the idea that each scene can have multiple descriptive captions associated with it, we utilize a simple template to generate alternate captions for scenes with binary relations that are semantically consistent but linguistically different. During fine-tuning, the model randomly sees either the original caption or the alternate caption. The presence of alternate captions also prevents the VLM to learn shortcuts between the position of the object descriptions in captions and the underlying spatial relationship or object-atrribute association.
PUG: Relations "[ImageNet class label of asset 2] [negative relation] J [ImageNet class label of asset 1] [(random) unary relation] Thaman-intelligiie intelligible background description]" e.g. The alternate caption for 'Banana to the left of chair inside museum" is 'Chair to the right of banana inside museum'
PUG: Attributes "[Attribute of asset 2][ImageNet class label of asset 2] and [Attribute of asset I [[lmageNNer class label of asset 1] [(random) unary relation] [human- inttaan-ittllliibble background description]" e.g. The alternate caption for 'Green banana and large chair inside museum' is LLarge chair and green banana inside museum'
29

---

Algorithm 2 PUG: AR4T-Attributes subset generation
Unary At, In, Inside)
Categories (Set of ImageNet Class Labels]
Environments (Set of Unreal Environments
Relations SSt of Relations}
Attributes (Set of Attributes)
Assets = [Dictionary of Sketchfab assets, keys being ImageNet Class Labels]
AssetLocations = (Function that returns locations of assets based relation with random offset)
Dataset = Ø
for att in Attributes do
for att2 Attributes [None] do if att2 = None then
num samples 20
else
& For binary relations we also add the negative scene+caption to the dataset, thus the effective number of samples per background and attribute is 2 = num ham_ssmples= =
end if for
= to num_samples do env = random..hhoice((Eiiinnnne if att2 None then
rel = rwandom.chiiceUnni
else
rel = random..hhocceRRllatinn Unary)
end if
catl = www.daa.cooiieeeco random.choice(Categorres
assetl random. www.ra..cooiceeee... chsiccc(((ssstss/catll)
if att2 != None then
car2 = aandom.chooceeCattgoiiee.
asset2 randomm.choiceeAsssttt...c
end if
locationl, lww.loooccnnccc................. Asseooocciccs))
scenel Torchhullivvaass((sseel, asset2, locationl, location2, attl, att2, env)
if rel & Unary then
rel2 = rrandom..coice(Unaryy
caption attl + catl + and att2 +' cat2 *+rel2 +
Dataset |scenel, caption
Generate hard negative scene and caption by switching abject-atriibte associations
asset2, location), location2, att2, att], env) caption2 = att2 + catl + and + attl cat2 rel2 + env
scene2 Tortih
Dataset [scene2, caption2)
else
captionl attl + catl + rel+* env Dataset U |scenel, caption
end if end for
end for
end for return Dataset
30

---

Additional experimental details
C.1 Equivariance study details
In section 3.2 we used PUG: Animals to study how foundation vision-language models behave with respect changes in factors of variations. We showed high image and text equivariance with respect to background, and text equivariance with respect to size and texture too. Here, we provide more details and results.
our study, we use the following pretrained models:
BLIP with ViT base backbone from the Huggingface transformers library
(Wolf et al. 20200]
trained on COCO dataset (Lin et al.
2014].
NegCLIP from Yuuksekgonul et al.
2023
As in Yukkekkonull et al. 2023 we use X-VLM pretrained on COCO dataset from https
//github. wwt. www.o////w/.nn.caooo
17/X-VLM
Flava with ViT-B/32 backbone from Huggingface transformers (https://huggingfee
co/facebookkfllav--tu
CLIP models all come from OpenAl CLIP http://pp////gtth....ooo/oooo//a We use the versions with ResNet50, ResNet101, ViT-L/14, ViT-B/16, ViT-B/32.
We compute equivariance each of the factors of variations. Since the captions do not take into account camera and asset orientations, when we compute the equivariance with respect to the other factors take only samples for given orientation of the character and camera (0 for roll, pitch and yaw in both cases). Furthermore, in the text caption we replace sizes by three adjectives as follow: 0.7 is mapped to small, 1.0 medium and 1.3 to big. Inspired by [Bouchacour] et al., [021], we compute equivariance as the alignment between embedding difference vectors. That is, we compute embedding difference vectors 2; 2, where 2; and 2j are the (normalized) embeddings of two images (or texts) of an object undergoing a given factor change. We then measure pairwise alignment cosine similarity of embedding difference vectors (either between images pairs, text pairs or image-text pairs) corresponding to the same factor change. We report averaged cosine similarity randomly paired vectors, and a higher value implies higher equivariance.
Note that model can present image equivariance age not guaranteed to be encoded in the same vector), or have high equivariance across modalities but no image or text equivariance and vice-versa. In Figure we report image equivariance with
no text equivariance (caption and im-
badigroued
1.5
Toture
(a) Image equivariance
(b) Text equivariance
(c) Across modalities equivariance
Figure 10: Mcasuring foundation models equivariance thanks to PUG: Animals: all three factors.
respect to orientation of the camera yaw. We see that there is little to no equivariance to it, suggesting that image embeddings are more predictable when changing background than the camera yaw. Note that foundation models representations belong to the hypersphere, yet our measurement of equivariance as parallelism (measured with cosine similarity) relies on Fuclidean geometry. Still, cosine similarity is a starting point to showcase how PUG: Animals can be used to study models'
31

---

camera yaw
GUPRN101
CUPHiSo
cUPViTate
CUPVITa22
CUPPWILL
Flava
HegrLUP
XVIM
Figure 1I: Additional image equivariance results with respect to camera yaw.
representational spaces. This could also explain the higher equivariance values of text representations: since textual captions follow the same template, embeddings might be close to each other (relative to image embeddings distances). In this case the hypersphere locally behaves like an Euclidean space [Lee] 2000). for which the Euclidean geometry is better suited. We leave for future work the exploration more complex equivariance metrics potentially based on spherical geometry to study foundation models' representational spaces. Studying model's representations is key to better understanding and improving them [Bouchacourt et al. [2021). Xie et al. (2022). Ushio al. (202T). Lene and Vedaldi [2018]. Our study showcases that PUG: Animals advantages (rich diversity of factors, knowledge of their values, control either one factor at a time but also all together) make it a
great dataset to study state-of-the-ar models representational properties.
C.2 Classification with held out sets
In section 3.2 we study how one can leverage PUG: Animsl to study OOD generalization in two settings: 1) Generalization on unseen factors 2) Generalization unseen combination of factors.
For this experiment, we use PUG: Animals with held out sets. Typically, random select number of number of animals (0, 10 or 20) within our 70 assets. Then for the remaining animals we decided remove from PUG: Animals number of background, object size object texture. This give training set. Then the images that were excluded from this training set are put as test or held out set in which we measure the performance of model. This model is typically Resnet50 trained for 100 epochs with Adam W as optimizer with a batch size of 2048.
C.3 Robustness of SOTA models additional details
In addition evaluating robustness for the models in the main paper, in Table provide an analysis of several additional models including the recent self-supervised DINOv2 model as well as BLIP a contrastive vision-langee language model. Parenthesis indicate the pretraining dataset: ImageNet 21k. LVD-142M, JFT 300M, LAION 400M, and LAION 2B. Models without parethesis are pretrained on the standard ImageNet-II 1k dataset. For ResNet models we use the publicly available pretrained checkpoints from the Timm package based on the training recipe from Wightman et al. [2021]. For the vision transformer models and Swin we use the pretrained models from the Timm package with patch size 16 for ViT and Swin Base with a patch size of 4 and window size of 7. For the BiT model we use the pretrained checkpoint trained on Google's 's JFT 300M dataset from the Timm package with ResNetv2 101 architecture. For DINOv2, we use the officially released repo to evaluate the base ViT architecture trained on 132 million samples (Oquab et al., 2023). For BLIP we use the checkpoint available in HuggingFace and evaluate the model using zero shot classification via the prompt 'This is a photo []. We evaluate CLIP model variants in a similar zero shot fashion and rely on OpenCLIP's implementation. The parenthesis indicate the pretraining dataset size from the LAION dataset. We report the average accuracy as each factor (see columns of Table 9 varies.
We also measure the relationship between standard in-distribution accuracy and robustness based on the accuracy as each factor in PUG:ImageNet varies. We measure Pearson's 's correlation coefficient between ImageNet accuracy and accuracy for each factor in Table T0] We find no statistically significant relationship between standard classification accuracy and factor robustness.
C.3.1 Performances
To understand if the differences in performance between the real ImageNet and our PUG dataset is caused by a sim-to-real gap or by the factor of variations, we show below the zero-shot accuracy obtained with a pretrained resnet101 for each class in PUG: ImageNet. There is only 3 classes for
32

---

Pil InvgiNn
IngeeNe
Yoe
OleeaL
Oleee
Scae
ReNNeco
36.
38.0
15.6
37.
136
43.5
360
37.3
13.4
DeST (VV (13M)
84.5
-5.6
41.
114
87.5
753
18.5
811
15.0
6.1
30.
Fasa
N00
31.1
128
ViT-Brac
54.
315
343
3L7
560
213
263
33.0
28.5
4.1
65.8
922
464
31
524
304
24
664
423
83
566
ET IPVITE
440
618
16.
33
(T pPVoTE (000))
67.
48.7
30.3
25.1
61.6
713
100
001
764
6.1
28.
19.
903
688
Table 9: Robustness measured by average accuracy across factors. We report zero shot classification accuracy for BLIP, Flava, all CLIP models. The pretraining dataset is indicated parenthesis next to each model name with ImageNet- Ik being the default unless otherwise indicated.
factor
correlation
pvalue
Object Pitch
0.29
0.45
Camera Roll
0.61
0.08
Camera Pitch
0.53
0.14
Camera Yaw
0.12
0.76
Background
0,43
0.25
Object Yaw
0.18
0.64
Object Texture
-0.10
0.81
Scene Light
0.53
0.14
Object Scale
-0.05
0.90
Object Roll
0.45
0.22
Table 10:
We compute the correlation between standard ImageNet classification and robustness based on the accuracy for each factor. We find no statistically significant relationship for standard classification and factor robustness.
which there is not single configurations of the factors that lead correct classification. For all the other classes, there always at least one configuration for which the network is correctly predicting the class. In that instance, least one time, the failures in predicting the correct class for the same objects with different factors is probably due the changes of factors.
assume that if the objects in given class correctly predicted at
Zero-shot top-1 accuracy per class Soccer_Bai Ball: 100.00, Pincapple: 95.62, Barrel: 95.00, Cellular hetprtttllehoeeeeeeee 89.45,
Pomegranate: Jack Lanter::
BirdHouse: BasketBall: 81.25 Sewing Machine:
Sewing_Machinee
Umbrella:
80.00. Washer: 78.75
Table: 76888 Basoball: 76.88. Safe: 76.25
Cabbage: 75.78. Coffe Cofoe_Mug: 75,31 Mask: 74.06. Brocolli:
73.44. Starfish: 72.50. Rocking Chair: 71.25. Punching Bag: 70.94, Chicken 69.38. WineBettle: 66.88 Gasmask: Joystick:
64.06. Television: 63.44, Chest: 63.44, Elephant: 62.50, Bell Pepper: 61.46, Cheeseburger: 60.62, Phone: 60.00, Tennis Ball: 58.44
Jean: 56.25,
55.62. Scooter:
Hay: 54.06. Bench:
Zucchini:
$2.08 Bananu: 50.00, Sliding Door: 48.75 Military_unifomm: unifomm: 47.50 Ambulance: 47.50. 17,50, Pizza: 47.19 29199, Tractor: 46.88 Dishwasher:
46.88, Cowboy Hat: 46.35, Drum: 45.31, Kyyyoriter_Keyyoordd:
Keyboard: 44.92, Toaster: 44.69 Obelisk:
Laptop: 44.38, Throne: 43.75,
Backpack: 43.44, Shield: 41.88, Artichoke: 41.80, Penguin: 41.56, Bathtub: 40.31, WaterBortle: 40.00, SpaceShuttl:: 37.19 Bagel: 36.88.
Bear: 36.25, Vacuum: 94, Radiator: 35.62, Shovel: 35.55, Refrigerator: 35.00, Running Shoe: 34.38 Goldfish: 34.38 Crate: 34.38
Polaroid Camer: 33.98, Table J Lamh: 33.75, Rulletpmof_ yest: 33.75, Microphane: 33. Traffic J Light: 32,50,
Polaroid_Camem::
Carton:
Carton: 31.25, Volley Ball:
30.62, Vending_maat machine: 30.62. Lawa_Mower: 29.38. Car, Wheel: 29.38 Harmonica: 28.12. Lighter: 27.50. Carousel: 27.34. Mailbox:
27.19 Aitliner: 27.19. Butternut_ Squash: 26.95 Sweatshirt: 26.56 Sock: 25.62. French_ 25.00, Dial tel_phecp::::: 24.61. Rabbit: 24.06
Remote_controler: 22.81, Modem: 22.50. Chain-saw: 21.35, Screwdriver: 20.31. Power-drill: 19.69. Electric_Fas Fan: 19.06. HairDryor: 18.75, Purse: 18.12, Wallet: 17.50, Sunglasses: 17.50. Minivam: 17.50. Cat: 15.94,
Microwave: 15.62. Candle: 15.62. Mushroam: 15.31,
Table:
Table: 14.06. Ice. Cream: 13.75, Perfume: 13.44, Komodo dragon: 13,44, Bycicle: 13.12, Wooden_ Spoen: 12.81. JellyFish: 12.81,
Canoe: 12.81 Radio: 12.19, Desk: 12.19. Affican crocndile: 11.88. Hotchet: 11.25
Sandal: 10.00, Stonewall: 9.69 Burriko: 9.38 Palace:
9.06. Mouse: 7,50, Convertble: 7.19, Espresso_makee maker: 6.88, Can Openor: 6.56 Jeep: 6.25 Fox: 6.25 Tile Roof: 5.86, Street Sign:
5.62, WarPlane: 5.47, Frog: 5.47, Wolf: 5.31, Whale: 5.00, Torch: 5.00 Sowp_Bowl: Bowl: 4.69, Great, white shark: 4.38, Kangarou: 3.91,
Digital Watch: 2.81 Skkrt:: 2.50, Computer Keyhoard: 2.50, Piano: 1.88, Manhole Cover: 1.88, Bridge: 0.94, Aircraft Carrier: 0.39.
Screen: 0.31, Locomotive: Submarine:
Loupe:
33

---

C.4 Additional PUG:SPAR experiments
Instead of using all the background environments presented in the main part of the paper, we also introduce much simple setup in which we have a single background (thus we do not need the background information in the caption anymore). The background that we choose is the simplest one named "salt flats". This is also the background for which the retrieval accuracy is the highest across all the backgrounds. In table 1T we present the performances of several VLMS when using this single environment. We can observe significant boost in accuracy for the single object detection task for which the best model achieve 94% accuracy (this value is to contrast with the 78% accuracy obtained across all the background). This show that VLMs are definitively not robust to background changes. However as in the previous case, when probing for the positional information, the performance the model is still decreasing significantly. We also illustrate in Figure best model.
12 very simple failure cases on the
VIT-B-32 (0T-8-22 (OpenA) CLPP)
VIT-B-32 (OponClip 38)
VIT OppAA CLIP)
VI-L-14 (Opent 2011
WT rOpen 28)
NIT 14 (0pentLIP 28)
XVLM
NocCLIP
Flaa
BLJPP
Caption
Texture
Desault
57.81
94.53
78,91
67.19
"A ploto a [character]"
BluetRed
48.44
54,09
71,88
70.31
75.00
84.38
42.19
53.12
48.44
48.44
GinedStanee
39.06
45.51
67.19
68.75
76.56
78.12
39.06
56.25
45.31
50.00
[character] (leptright) the picture"
29.69
37.50
3000
32.81
31.35
"A phuto [churacter] the (banonrrap) picture"
Detauit
29.69
23.44
45.31
43.75
46.88
45.31
23.45
34.38
34,38
25.44
"A photo of [choroster] and [choraises
Blue/Red
Desault
24.36
10.84
32,87
18.55
68.85
38.57
59.67
30.66
72.00
34.28
$0.96
51.56
9.86
13.35
40.55
10.94
3.12
26.71
8.01
21.01
GraseStone
31.35
30.47
30.18
47.85
6.84
11.33
pluto of [character] the left and a [choracterj on the right"
14.11
40.33
14,01
photo [character] on
Detault
12.00
15.42
36,07
35.48
44.05
45.36
10.99
26.51
14.62
tom (eharacter) on tap"
"A photo of (charnster) tenured with (iemmrel] and [churacteri
GiessSttne
Blue/Red
3.43
4.44
6.65
5.44
19.15
15.73
15.52
17.24
20.56
19.05
29.84
27.32
6.15
5.54
7.76
10.28
5.44
6.35
4.03
4.13
(resure2)"
Table 1I: Setup and zero-shot evaluation of CLIP models on PUG: SPAR with caption retrieval in single environment. In contrast with the figure presented in the main paper, we present the result only using the salt flats environment. The motivation for this experiment showcase the failures mode VLMs in very simple setup in which the model robustness to background does not impact the prediction.
C.5 CLIP fine-tuning details
We utilize the OpenCLIP framework [Ilharco et al. 2021 for all our CLIP experiments. The ViT-B/32 model is used as the image encoder for all our experiments. The CLIP model we fine-tune is the OpenAl 400M pre-trained model ('ViT-B-32', 'openai' 10 Fine-tuning on PUG: AR4T is done for 10 epochs on the 200K dataset while training is done for epochs on the IM dataset.
NOTTE We do not perform any training on the proprietary 400M dataset from OpenAl. We strictly only use the pre-trained models released, and fine-tune them our PUG datasets.
34

---

A very simple failure mode of the OpenCLIP ViT-G-14 (2B)
The caption predicted by the model is presented below each image (correct/wrong)
photo o Goldlish
photo Goldfish falt the picture
A pholo a Crow
photo of Oroa
photo Golditsh on the
photo of Golditsh a Oroa
model does not understand positional information
The model does not understand compositionality
Figure 12: Failures mode a OpenCLIP ViT-G-14, Our PUG: SPAR dataset provides very simple images and captions and yet even large models failing on them.
35


Reviews:
[{"review_id": "3DyrIu5aAT", "reviewer": "Reviewer_PGeG", "summary": "", "questions": "", "limitations": "The limitations section (A.1) is short and barely talks about concrete limitations, instead just saying that more could be done. This section would add a lot more if it were more precise. What about limitations in terms of 1) difficulty of creating assets, 2) runtime (mentioned a bit w.r.t. RL), 3) lack of diversity in the data / biases, 4) specific sim2real gaps that remain, and their effects on the paper's experiments, and so on. A richer discussion would be appreciated.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The data appears to be more photorealistic than prior large-scale 3D datasets\n* The datasets are large-scale; significant effort went into making this\n* Evaluation covers a good range of recent and popular vision foundation models\n* Unlike real datasets, the synthetic data can be manipulated more systematically to ask more pinpointed questions about model robustness and generality\n* Good to see that training on the data can substantially increase CLIP's performance, without many tricks", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BNAJn05sy3", "reviewer": "Reviewer_hTqw", "summary": "", "questions": "", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThe process of how to construct the dataset is detailly described.\n2.\tThe paper is easy to read and follow.\n3.\tThe targeted problem is worth exploring.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "oumAzdAH5U", "reviewer": "Reviewer_7VBW", "summary": "", "questions": "", "limitations": "Code/web-site are not published yet, only the data examples.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Idea of PUG is clear and well-motivated.\n2. Multiple use cases have been discussed and evaluated\n3. Paper is well-written and can be widely-used by the community", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Go21BU0dXf", "reviewer": "Reviewer_b1EE", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The controllability of the dataset is a major strength of PUG. The idea of using game engines for representation learning research is interesting and promising.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "gErLs4PChk", "reviewer": "Reviewer_bwEW", "summary": "", "questions": "", "limitations": "* Limitations are discussed in the appendix and are sufficient.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The paper proposes a semantically controllable framework for rendering 2D images from a 3D scene using the Unreal engine. Such research direction is valuable as it enables to evaluate computer vision models in controlled scenarios, which would not be feasible with with real natural images. As such it is valuable that the paper explores factors of variation related to the camera pose, the object pose, the object size, the object texture, the lighting of the scene, or the scene itself.\n* As mentioned in the paper, synthetic data also have the advantage of presenting less issues related to privacy, bias and copyrights than real natural images.\n* The paper proposes relevant benchmarks that rely on the rendered images. It showcases the potential of such images in the computer vision context as it helps to highlight issues related to the model robustness or how such models can be fine-tuned for improved performance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The ratings of this paper are 7-6-8-5-7. Reviewers are generally positive. This paper introduces a platform PUG for generating 3D synthetic data using the Unreal Engine. This platform is accompanied by three datasets designed for various purposes, including model evaluation and fine-tuning. Reviewers acknowedge that it is impressive to achieve a high level of photorealism in the synthetic data, surpassing prior large-scale 3D datasets. Besides, the paper effectively covers a range of recent and popular vision foundation models, providing valuable insights into their performance with synthetic data. This paper also demonstrates that training on the synthetic data can significantly enhance CLIP's performance without resorting to complex tricks. Authors addressed reviewers' concerns well in the response. Given these considerations, AC recommends to accept this paper.
Decision: Accept (Poster)