SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Florian Bordes $^{1,2,3}$   
Shashank Shekhar $^{1}$   
Pascal Vincent $^{1,2,3}$   
Mark Ibrahim $^{1}$   
Diane Bouchacourt $^{1}$   
1FAIR, Meta  
2Mila - Quebec AI Institute  
3Université de Montréal, DIRO
Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited – and often played down – mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models. The datasets can be downloaded at https://pug.metademolab.com/.
A grand goal of machine learning is to learn representations of data that are useful across many tasks. Essential to measuring and making progress towards this goal is the availability of ample controllable, realistic data for evaluation and training. This is especially true when considering deep neural network models not only in terms of their raw accuracy, but also their robustness and fairness—crucial properties for models deployed in real-world applications. However, collecting such data is challenging, presenting issues with privacy, bias, and copyright. Furthermore, the majority of available image datasets lack fine-grained labels and are challenging to manipulate beyond coarse image augmentations (e.g. with a photograph, it is hard to change the viewpoint or the time of day).
Using synthetic image data where we precisely control all the factors affecting the rendered scene gives easy access to the corresponding rich set of factor labels. This enables evaluating the extent of a trained deep neural network's abilities, most importantly its robustness. Is the network robust to change in pose? Are the predictions similar for different textures? All these questions may be answered systematically by using synthetic data, enabling highly rigorous evaluations of deep neural network models. In addition, training could also benefit from controllable factors<sup>1</sup>, by increasing the robustness of models with respect to these factors. They may also be used to monitor training, e.g. tracking which factors a model focuses on or becomes most invariant to, and in which order, as training progresses. This potentially enables better understanding of the training and generalization
<sup>1</sup>We define factors here as distinctive attributes that describe the data, such as color or pose of an object.
Figure 1: The PUG Dataset Family (left) Cartoon illustration of our dataset creation setup, which consists of two steps: environment creation and then data creation. (right) Example images from PUG: Animals, PUG: Image-Net and PUG: SPAR.
dynamics in deep neural networks. However the lack of realism typical in many of the currently available synthetic image datasets, and their usually very limited scope greatly limits their usefulness for general image representation learning research.
To address this, we introduce a new family of synthetic Photorealistic Unreal Graphics (PUG) datasets, designed for ease of use by the representation learning research community, where image realism is significantly improved compared to current public synthetic image datasets. The environments were built using the Unreal Engine [EpicGames], which is widely used in the video game and entertainment industries and praised for its realism. In addition to pre-rendered static image datasets, we also introduce the TorchMultiverse python library, which offers a simple python interface to enable easily controlled dataset creation from any given PUG environment. Using these tools, we contribute 4 new datasets and show their usefulness across several different research domains. To summarize:
- We introduce a new family of environments and image datasets (coined as PUG) for representation learning, based on the Unreal Engine [EpicGames].
- We present PUG: Animals for research on out-of-distribution (OOD) generalization and to study the representational space of foundation models.
- We introduce PUG: ImageNet as an additional robustness test set to ImageNet, containing a rich set of factor changes such as pose, background, size, texture, and lighting.
- We introduce PUG: SPAR for evaluating vision-language models. We use it to demonstrate how synthetic data can be utilized to address known benchmark limitations. In addition, we introduce PUG: AR4T for fine-tuning vision-language models and use it to demonstrate the reliability of PUG: SPAR in contrast to other benchmarks.
Synthetic data for representation learning To address robustness shortcomings, researchers today commonly study representations using lower-fidelity controlled datasets such as CLEVR, Biased Cars, and ShapeNet [Johnson et al., 2017, Madan et al., Chang et al., 2015]. Other datasets also contain precise factor labels useful for probing how well a representation encodes each factor in a structured form [Gondal et al., 2019, Struckmeier et al., 2023, Weng, 2018]. While these datasets offer control in terms of the factors that change as well as the train and evaluation splits enabling controlled scientific experimentation, they lack realism. This gap between the lower-fidelity controlled data and the real world poses a challenge for the broader application of these studies. On the other hand, photorealistic datasets have been explored in various application-specific domains in machine learning (outside
<sup>2</sup>As a reminder, any use of content or technologies made available by Unreal and/or Epic Games, or any other provider, should comply with their applicable terms (such as the Content License Agreement available at https://www.unrealengine.com/en-US/eula/content or any other direct agreement you may have with Epic / Unreal)
of representation learning.) This is especially relevant when trying to evaluate and train models on rare events in which getting real data might be really difficult, such as for autonomous driving. CARLA [Dosovitskiy et al., 2017] is a popular self-driving car simulator which offers highly realistic environment with a significant amount of controllable factors such as environmental conditions, full control of all static and dynamic actors and maps rendering. Another domain where simulated environments are commonly used is reinforcement learning (RL), as RL algorithms often require the ability to run millions of simulations to learn to master non-trivial tasks, and this cannot be done in a real environment. Data environments based on video games like Atari have been very popular to design and evaluate RL algorithms. Alternatively, platforms like Habitat [Szot et al., 2021] offers indoor scene for training home assistant agents. While these simulators, games or datasets can offer some photo-realism and mimic real world interactions for agents, they are relegated to domain-specific applications making them challenging to use for evaluating the representations of deep neural networks more broadly. Since our focus is not RL, we do not need to embed a fast simulator capable of rendering several thousands frames per second for effective online-training. Instead we can pre-render custom high-quality datasets offline. Photorealistic environments and datasets have also been explored for more general domains with the ThreeDWorld platform [Gan et al., 2021]. Based on the Unity game engine, it offers an interactive environment that can be leveraged to create datasets. The environment is presented as a simulator that is generic enough to handle multiple uses cases, and users can customize the setup of a scene and the data smapling through a low level API. One such dataset that utilizes ThreeDWorld is the Synthetic Visual Concepts (SyVIC) dataset [Cascante-Bonilla et al., 2023], which uses the API to create scene images and descriptive captions for training vision-language models. One of the downsides of ThreeDWorld is that the back-end, the simulator itself, is closed source which limits external contributions. In contrast with ThreeDWorld, we do not provide a platform or a generic simulator for people to use. In fact, we believe that tools like the Unreal Engine are simple enough to be used directly by researchers to create the environments they want without the need to use an intermediate platform. In addition, being free of such intermediate platform allows us to leverage most of the content created for video gaming directly into our simulator by using the existing Epic Games marketplace.
**Evaluating model robustness** To study model robustness, there is an inherent trade-off between photo-realism and control. Photo-realism depicts objects as they appear in the real world, but often lacks control to precisely define the factors to describe the object such as pose or background. Prior works either collect natural images with specific factor changes [Xiao et al., 2020, Barbu et al., 2019] or label distinctive factors in existing datasets [Idrissi et al., 2022]. Such datasets allow researchers to measure average accuracy on photo-realistic images, but lack granular control necessary for precisely controlled robustness experiments. On the other hand, prior studies [Ibrahim et al., 2022, Abbas and Deny, 2022, Alcorn et al., 2019] examine model robustness with respect to factors such as pose and size by rendering 3D-objects such as buses. These studies precisely control how each object is depicted, but lack in realism. In this work, we advance the photo-realism of these prior works by using the Unreal engine 5.0 [EpicGames], a rendering engine commonly used in high-end cinematic CGI and high-resolution video games which allow us to measure robustness with respect to factors of variation such as lighting.
Benefits and limitations of using generative models as data generator Another way to generate realistic datasets is to use generative models[Ho et al., 2020, Goodfellow et al., 2020]. However, one limitation of such models, despite impressive improvements in the last few years [Dhariwal and Nichol, 2021], is the lack of quality control on what the model can produce [Gandikota et al., 2023]. It's not uncommon to find cases in which the model will ignore parts of the conditioning prompt. Despite such limitation many works have tried to leverage generative model as an additional source of data to train deep neural networks with some success [Astolfi et al., 2023, Bansal and Grover, 2023, Trabucco et al., 2023, Azizi et al., 2023, Zhang et al., 2021, Li et al., 2022a, Jahanian et al., 2022, Jain et al., 2023, Sariyildiz et al., 2023, He et al., 2023]. Another limitation of using generative models are privacy concerns that arise from such models replicating data from their training datasets Somepalli et al. [2022]. Finally, Shumailov et al. [2023] recently demonstrated that training on data recursively generated from such models results in increasing underestimates of the tails and overestimates of the mode, amplifying bias in datasets. In contrast to generative models that might produce unreliable results, we use an entirely controllable environment for which we can have a known and accurate generation with respect to a set of factors.
3 Photorealistic Unreal Graphics (PUG) environments and datasets
3.1 Leveraging Unreal Engine to create environments and datasets for representation learning
We introduce the Photorealistic Unreal Graphics (PUG) environments, a family of 3D graphics environments that leverage Unreal Engine for rendering image data for representation learning research. To create a PUG environment, we first obtain a number of assets<sup>3</sup> which can be 3D objects or 3D backgrounds. Then, we import them in the Unreal Engine editor and create blueprints that yield a simple generic 3D environment. Once this generic and controllable environment is created, it is compiled into a Linux binary file, which can be run on standard GPU clusters. This environment is programmed in such a way that when running, it is listening for incoming packets through WebRTC which can specify instructions about how to change a scene. Since most machine learning practitioners are used to python scripting, we wanted to have a very simple approach by which a user can request image data rendered from a packaged PUG environment, through very simple python code and JSON config files. To do so, we developed a python API, TorchMultiverse, that allows a user to easily specify a scene configuration in JSON and request rendered images from the PUG by using WebRTC. Once the factors have been set as requested by the user, for a specific environment configuration, the user can send a command to freeze the current environment and receive back an image. It takes around 1 second to render an image at a resolution of  $512 \times 512$  on a V100 GPU<sup>4</sup>. We illustrate this setup in Figure 1. It shows how, starting from 3D assets, we design interactive environments that enable us to create different datasets. In the present work, we focus on pre-rendered static image datasets, however our setup also allows dynamic communication between a PUG environment and a pytorch program, meaning that new data could be requested and rendered on the fly while training a deep neural network. We leave the exploration of such active learning setups, as well as the rendering of videos, as future work.<sup>5</sup>
3.2 PUG: Animals
Figure 2: We present PUG: Animals, a new photorealistic synthetic dataset with annotated factors of variations to evaluate the out-of-distribution (OOD) robustness of models.
As the first member of the PUG family we introduce PUG: Animals (Figure 2), which contains 215 040 pre-rendered images using 70 animals assets, 64 backgrounds, 3 object sizes, 4 textures, under 4 different camera orientations. PUG: Animals is designed with the intent to create a dataset with every combination of the factors of variation available. PUG: Animals allows one to precisely control distribution shifts between training and testing which can give researchers better insight on how a deep neural network generalizes on held out factors of variations. Surprisingly, the usage of 3D realistic synthetic data is limited in OOD generalization research – with the exception of Biased-cars[Madan et al., 2022] that has been used to study generalization on new category-viewpoints. Commons OOD datasets are Colored MNIST [Arjovsky et al., 2020] – to study how well a network can generalize to unseen combinations of digits and colors and MNIST-R [Ghifary et al., 2015] – to study generalization on new combination of digits and rotations. However, MNIST-based dataset
<sup>3</sup>We purchased assets from the Epic Game Store and used assets from Sketchfab [Deitke et al., 2022]. The complete list of assets we have used is available at https://github.com/facebookresearch/PUG
4In our setup, we paralyze the rendering across 64 GPUs. A dataset like PUG: Animals which contains 200K images has taken around 1h to be entirely rendered.
It might also conceivably be used as a photorealistic interactive environment for reinforcement learning (RL), but the high quality image rendering achieved in this system currently appears too comput-intensive and slow to be practically useful in the context of current RL research. Our initial targeted research community and use case is that of supervised and self/unsupervised representation learning from image data, rather than RL.
might be too toyish to evaluate modern architectures. A more realistic dataset based on real images is  $\mathrm{Nico + + [Xingxuan Zhang,2022]}$  - to study generalization with respect to different domains or environments. However, in  $\mathrm{Nico + + }$  the objects and backgrounds are never entirely disentangle (the context background is different for each image). Thus, it is never clear if the model is failing because of the context or because of a specific object (since the contexts and the objects are never disentangle).
In contrast, in PUG: Animals the animal asset is always the same, in that case the environment factor and the objects are perfectly disentangle such that if the model is able to classify correctly an elephant on a road and is not able to classify the elephant in a museum, we can rigorously say that the failure is caused by the change in context background. In addition of analysis the robustness with respect to the background scene, it is also possible to analyse with PUG: Animal the robustness with respect to the camera position, asset size and texture (as we demonstrated in Appendix 3.2).
Classification with held out sets In the first experiment, we held out some factors of variation during training (backgrounds, sizes or textures) except for a specific number of animals  $C$  and use the held out data as validation set. Thus,  $C = 0$  means that the network never saw the factor during training (this is an OOD scenario with unseen factors) while  $C = 10$  imply that the network saw this factor for least 10 different animals (OOD scenario with unseen combinations of factors). In Figure 3, we present our results training a ResNet50 with different held out factors. Every model reached more than  $99\%$  accuracy on the training set. First, we trained on 50 backgrounds and used the remaining 14 backgrounds for validation: here, the network reached an accuracy of  $80\%$ . However, when using only 30 backgrounds for training and using the remaining 34 as validation, the accuracy drop significantly. Interestingly, showing every background for some of the animals (having unseen combination of factors instead of just unseen factors) decrease performance. In contrast, for texture, we found that having at least 10 animals for which every possible textures are seen during training improves generalization. Interestingly, the network overfits much more to the grass texture relative the default network. Lastly, when looking at the size factor, it seems that training on medium size assets leads to good generalization on small and large assets while training only on small asset leads to worse performance on medium and large assets.
Figure 3: Accuracy on held out factors with PUG: Animals. Each line and value C correspond to the number of animals for which all the factors are seen. The test space is built by taking all the factors minus the training factors. If we train on the Default texture, then the network is evaluate on Grass, Sky and Asphalt. If we train on 50 backgrounds, then we evaluate on 64 (total number of background) - 50 (training background) = 14 backgrounds.
Studying foundation model representational space PUG: Animal can also be to study the equivariance of foundation models'
representations. For this, we augment each image in PUG: Animals with a caption that describes it according to its factor values (sizes are converted to three adjectives: "small", "medium" or "big", see Appendix C.1 for details), using the following template<sup>6</sup>: "A photo of a [size] sized [character] textured with [texture] on a [background] background". Informally, equivariance of a model's representation with respect to a factor means that when the factor changes from one value to another, the embedding of the corresponding image (or caption) changes in a predictable manner. Equivariance is a sought-after property to improve sample efficiency and robustness to transformations [Klee et al., 2023, Tai et al., 2019, Wang et al., 2023]. Similar to previous works on equivariance and compositionality Bouchacourt et al. [2021], Xie et al. [2022], we measure equivariance as the alignment (i.e. parallelism) between embedding differences. First, we feed images and their corresponding captions to 9 pretrained vision-language models including multiple CLIP models Radford et al. [2021], NegCLIP Yuksekgonul et al. [2023] Flava Singh et al. [2022], BLIP Li et al. [2022b] and X-VLM Zeng et al. [2021] and collect their embeddings of PUG: Animals images and created captions. For each model, we compute difference vectors between the embeddings of two images (or captions) of an object undergoing a factor change: e.g. a big penguin textured with grass on a background "village square" modified to the same penguin but
Note that the camera and character orientations are not described, as well as the texture when it is default.
with background "restaurant", see arrows in Figure 4 (left). Specifically, for a sample  $i$ , undergoing a change from background  $b_{k}$  for background  $b_{l}$ , we denote the difference vector between the embedding of the image of the sample with background  $b_{k}$  and the image of the same sample but with background  $b_{l}$  by  $v_{b_{k} \to b_{l}}^{i}$ . Similarly, we denote by  $u_{b_{k} \to b_{l}}^{i}$  the difference vector between the embedding of each of the two captions accompanying the images.
Then, we measure the alignment of difference vectors across pairs undergoing the same factor change (here, the penguin and the cat) as their cosine similarity<sup>7</sup>. We estimate three types of equivariance: (i) Image equivariance: how parallel (measured with cosine similarity) are difference vectors across image pairs? (lined and dashed red arrows) (ii) Text equivariance: same but for caption pairs (parallelism of lined and dashed green arrows) (iii) Across modalities equivariance: for the same object, alignment of difference vectors between pairs of image-caption (i.e. alignment of the two arrows for the penguin). Specifically, for image equivariance between sample  $i$  and  $j$ , for background change  $b_{k}$  to  $b_{l}$ , we compute:
$$
\operatorname {s i m} \left(v _ {b _ {k} \rightarrow b _ {l}} ^ {i}, v _ {b _ {k} \rightarrow b _ {l}} ^ {j}\right) = \frac {\left. v _ {b _ {k} \rightarrow b _ {l}} ^ {i} \right| _ {| | v _ {b _ {k} \rightarrow b _ {l}} ^ {i} | |} ^ {T} v _ {b _ {k} \rightarrow b _ {l}} ^ {j}}{| | v _ {b _ {k} \rightarrow b _ {l}} ^ {i} | | | | v _ {b _ {k} \rightarrow b _ {l}} ^ {j} | |} \tag {1}
$$
For text equivariance, we compute  $\text{sim}(u_{b_k \to b_l}^i, u_{b_k \to b_l}^j)$  while for across equivariance, we compute  $\text{sim}(v_{b_k \to b_l}^i, u_{b_k \to b_l}^i)$ . We report cosine similarity averaged over pairs and possible changes for each factor (higher value means higher equivariance, 1 is the maximum). Specifically, the image equivariance for background writes as
$$
\frac {1}{B (B - 1)} \sum_ {b _ {k}} \sum_ {b _ {l}} \frac {1}{N (N - 1)} \sum_ {i} \sum_ {j} s i m \left(v _ {b _ {k} \rightarrow b _ {l}} ^ {i}, v _ {b _ {k} \rightarrow b _ {l}} ^ {j}\right) \tag {2}
$$
where  $B$  is the number of possible backgrounds and  $N$  is the number of samples.
Figure 4: Measuring foundation models equivariance thanks to PUG: Animals. Left: Illustration of how to use PUG: Animals to compute equivariance. Right: Image and text equivariance is present with respect to background, while across modalities equivariance to background doesn't hold as much. See main text for detailed results.