MOTIVATION
For what purpose was the dataset created?	The 4 datasets we presented in this paper were created for representation learning research. PUG: Animals is a strong dataset for OOD research as well as for being able to better probe the representation of vision models. PUG: ImageNet was designed as an additional benchmark for ImageNet pretrained model to offer a better comprehension of vision models capabilities in term of robustness to specific factor of variations. PUG: SPAR showcase how synthetic data can be used to evaluate VLMs understanding while PUG: AR4T can be leveraged to fine-tune them.
Who created the dataset and on behalf of which entity?	This dataset was created by the FAIR team at Meta AI.
Who funded the creation of the dataset?	Meta.
COMPOSITION
What do the instances that comprise the dataset represent?	The instances represent images of animals in various environment for PUG: Animals. In contrast PUG: ImageNet contains 151 object classes (the full list is available in Appendix B.3). PUG: SPAR uses the sames assets as PUG: Animal while PUG: AR4T use the same objects as PUG: ImageNet.
How many instances are there in total?	PUG: Animals: 215 040 images; see Appendix B.2.PUG: ImageNet: 88,328 images; see Appendix B.3.PUG: SPAR: 43,560 images; see Appendix B.4.PUG: AR4T: 249,986 images for training and 23,216 test images; see Appendix B.5.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?	PUG: Animals contains all possible combination of factors of variations. In contrast PUG: ImageNet was sampled by changing only 1 factor at a time and is therefore a random sample of the distribution. Images in PUG: SPAR were sampled using all possible combination of factors of variations (with the exception that for the attributes the blue or grass animal is always on the left). Image-text pairs in PUG: AR4T were randomly sampled.
What data does each instance consist of?	For PUG: Animals, PUG: ImageNet, PUG: SPAR we release images along the factor of variation. For PUG: SPAR, we release the script to generate the captions from the factors of variations. For PUG: AR4T, we release images along with corresponding captions.
Is there a label or target associated with each instance?	Yes, a csv file. Each instance have a row in this csv files with all the factors of variation used to generate this image. For PUG: Animals, the csv files contains the following columns:
filename, world_name, character_name, character_scale, camera_yaw, character_texture
while for PUG: ImageNet, it contains:
filename, world_name, character_name, character_label, characterrotation_yaw, characterRotation-roll, characterRotation Pitch, character_scale, camera-roll, cameraPitch, camera_yaw, character_texture, scene_light.
For PUG: SPAR, the csv contains:
filename, world_name, character_name, character2_name, character1_pos, character2_pos, character_texture, character2_texture
For PUG: AR4T, the csv contains:
Relation, Actor1Category, Actor2Category, Actor1Name, Actor2Name, Actor1Location, Actor2Location, Actor1Rotation, Actor2Rotation, Actor1Scale, Actor2Scale, Actor1Texture, Actor2Texture, Actor1Attribute, Actor2Attribute, Camera Roll, Camera_Pitch, Camera_yaw, caption, alt_caption, Level, World.Name, filename, filename_neg, filepath
Is any information missing from individual instances?	No, all relevant information is included.
Are relationships between individual instances made explicit?	N/A.
Are there recommended data splits?	There is no specific split concerning PUG: Animals because this dataset should be used for OOD research. We primarily let the researchers choose their own held out or training/Validation/testing split to train their models. In contrast, PUG:ImageNet and PUG:SPAR should only be used as an additional test set. For PUG: AR4T, splits are described in Appendix B.5.
Are there any errors, sources of noise, or redundancies in the dataset?	We did not explicitly filter for occlusion, so some images may contain occluded views. PUG: Animals and PUG:SPAR are very clean and each animal is easily identifiable. In contrast, PUG: ImageNet and PUG: AR4T leverage assets from Sketchfab and the asset quality vary significantly.
Is the dataset self-contained, or does it link to or otherwise rely on external resources?	The dataset is self-contained however the assets that were used to build the dataset belongs to external sources which are listed in the github at https://github.com/facebookresearch/PUG.
Does the dataset contain data that might be considered confidential?	No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?	No.
COLLECTION
How was the data associated with each instance acquired?	The data (3D assets) were acquired through the Unreal Engine Marketplace https://www.unrealengine.com/ marketplace/en-US/store and Sketchfab https://sketchfab.com/. Assets were then incorporated into the Unreal Engine to generate realistic 3D scenes and corresponding images. The 3D assets were manually selected to ensure high quality.
What mechanisms or procedures were used to collect the data?	Manual human curation. Assets were manually collected.
If the dataset is a sample from a larger set, what was the sampling strategy?	For PUG: Animals and PUG: SPAR, all combinations are included. For PUG: ImageNet and PUG: AR4T, a random sample of possible combinations is provided.
Who was involved in the data collection process and how were they compensated?	Only the authors of this work were involved.
Over what timeframe was the data collected?	The data were collected between June 2022 and June 2023
Were any ethical review processes conducted?	No.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?	Third parties: Unreal Engine Marketplace https://www.unrealengine.com/marketplace/en-US/store and Sketchfab https://sketchfab.com/.
Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.	There is no personally identifiable information in our datasets as they are purely synthetic and contain no images of people. We purchased 3D assets from different marketplaces where required, however we did not explicitly contact the individual creators.
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.	N/A. See above.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).	N/A. See above.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.	No data about specific individuals is included in these data. See above.
PREPROCESSING
Was any preprocessing/cleaning/labeling of the data done?	N/A.
Was the “raw” data saved in addition to the preprocessed/cleaned/ labeled data?	N/A.
Is the software that was used to prepro cess/clean/ label the data available?	N/A.
USES
Has the dataset been used for any tasks already?	Yes, these data were used for the experiments that were presented in this paper.
Is there a repository that links to any or all papers or systems that use the dataset?	No.
What (other) tasks could the dataset be used for?	These Datasets could be used widely for evaluating and training neural networks. For example, assessing disentan- glement of models with respect to PUG: Animals factors of variation (e.g. with DCI metric Eastwood and Williams [2018]).
Is there anything about the composition of the dataset or the way it was col- lected and preprocessed/cleaned/ labeled that might impact future uses?	No.
Are there tasks for which the dataset should not be used?	These datasets should not be used for generative mod- elling purposes.
DISTRIBUTION
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?	Yes, the dataset will be publicly distributed.
How will the dataset will be distributed?	Tarball on a website.
Will the dataset be distributed under a copyright or other intellectual prop- erty (IP) license, and/or under applicable terms of use (ToU)?	The license of the dataset is cc-by-nc with the mention that these data should not be used for generative AI purposes.
Have any third parties imposed IP-based or other restrictions on the data associ- ated with the instances?	See EpicGames.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?	N/A
MAINTENANCE
Who will be supporting/hosting/maintaining the dataset?	Meta AI.
How can the owner/curator/manager of the dataset be contacted?	Please contact the corresponding author of this paper.
Is there an erratum?	No.
Will the dataset be updated?	Yes the dataset will be updated with versioning.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.	N/A.
Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.	It depends. If the dataset is updated because one of the asset creators has requested to remove their assets, we will not continue to host the dataset containing these assets. Only the newer version of the dataset which will not contain these assets will be available.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?	No mechanisms are in place yet, but they can contact the authors of this paper if they would like to contribute.