SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Character_Name : ["Goldfish", "Caribou", "Elephant", "Camel", "Penguin", "Cassowary", "Zebra", "Turtle", "Bear", "Beaver", "Capybara", "Crocodile", "Armadillo", "Cat", "Gecko", "Crow", "GiantAnteater", "GiantTortoise", "KomodoDragon", "Rhinoceros", "Dolphin", "EarlessSeal", "FruitBat", "Goat", "Hippopotamus", "Horse", "Impala", "Lion", "Orca", "Pig", "Rabbit", "Squirrel", "Tapir", "Wildbeest", "Wolf", "Anlylosaurus", "BlackRockFish", "Parasaurolophus", "PoisonDartFrog", "Spinosaurus", "Triceraptos", "Chicken", "HarpyEagle", "Ostrich", "Raven", "RedCrownedCrane", "Robin", "Seagull", "Secretarybird", "Shoebill", "Swan",
"Toucan", "Vulture", "Ammonite", "Ant", "Scorpion", "GoldBeetle", "Hornet", "SnowCrab", "Tarantula", "WhiteShark", "Tuna", "Arowana", "Ayu", "Betta", "Koi", "Pirarucu", "Salmon", "Cattle", "Jerboa"],
- Character_Scale: [0.7, 1.0, 1.3],
- Camera_Yaw : [0, 45, 225, 315],
- Character_Text: ["Default", "Sky", "Grass", "Asphalt"]
PUG: Animals is built by using all combinations of the factor of variation above. In Figure 7, we show random images from the PUG: animal dataset that highlight the diversity of this dataset.
Following Liu et al. [2023], we present a comparison in Table 5 with other datasets that are often used in OOD research:
Image Data	Colored MNIST
Arjovsky et al. [2020]	MNIST-R
Ghifary et al. [2015]	Waterbirds
Sagawa* et al. [2020]	Biased-Cars
Madan et al. [2022]	Nico++
Xingxuan Zhang [2022]	PUG: Animals
Ours
# Domains	3	6	2	-	10	64
# Categories	2	10	2	5	80	70
# Examples	-	6k	4.8k	450k	230k	215k
Shift Type	Color	Angle	Background	Views	Background	Back./Text./Size./View
Image Type	Digits	Digits	Birds	Synthetic Cars	Real Objects	Synthetic Animals
Table 5: Comparing PUG: Animals with other datasets traditionally used for OOD research. In contrast to other datasets that have variations across only a single domain, that have noisy annotations or that are to unrealistic, PUG: Animal over high quality images with reliable annotations across different domains such as the background, texture, size and view.
B.3 PUG: ImageNet
PUG: ImageNet contains 88,328 pre-rendered images using 724 assets representing 151 ImageNet classes with 64 backgrounds, 7 sizes, 10 textures, 18 different camera orientation, 18 different character orientation and 7 light intensity. Below is the values we used for each of these factors:
- World_Name : ["Egypt", "Desert", "AmusementPark", "ArcadeClub", "Arena", "Battleground", "Catacombs", "Tableland", "EuropeanStreet", "JunkYard", "OceanFloor", "Racetrack", "Ruins", "SciFiCity", "SciFiGarage", "SpaceIsland", "SpaceHangar", "SpatialStation", "TokyoDay", "TokyoNight", "TrainStation", "Bridge", "Beach", "BusStationInterior", "BusStationExterior", "Subway", "IndoorStairs", "Bar", "ScreeningCheckpoint", "Circus", "Apartment", "Hallway", "TrashRoom", "FuturisticSubway", "Footbridge", "BoxingRing", "Hangar", "Mansion", "ShoppingMall", "ConferenceRoom", "SpacePort", "VillageOutskirt", "VillageSquare", "Courtyard", "ElvenRuins", "Forge", "Library", "Museum", "Gallery", "ModernGallery", "Opera", "AncientAgora", "Restaurant", "RuralAustralia", "AustralianRoad", "ShadyRoad", "SaltFlats", "Castle", "StylizedEgypt", "Temple", "Snow", "Grass", "DryGrass", "Forest"],
- Character_Name : 724 Sketchfab assets (See github for the list)
- Character_Rotation_Yaw : [0, 45, 135, 180, 225, 270],
- Character_Rotation_Roll: [45, 90, 135, 180, 225, 270],
- Character_Rotation_Pitch: [45, 90, 135, 180, 225, 270],
- Character_Scale: [0.5, 0.6, 0.7, 0.8, 1.3, 1.6],
- Camera_Roll: [45, 90, 135, 180, 225, 270],
- Camera_Pitch: [240, 260, 280, 300, 320, 340],
- Camera_Yaw : [0, 45, 135, 180, 225, 270],
- Character_Text : ["Default", "Sky", "Green", "Gray", "Red", "Grass", "Color", "Black", "Curtain"],
- Scene_Light: ["255,255,255,0", "0,0,255,0", "0,255,0,0", "255,0,0", "0,255,255,0", "255,0,255,0", "255,255,0,0"] (The value for the lights are in RGBA format)
To generate PUG:ImageNet, we change only one factor at a time for each assets. When changing the background (World_Name), all the other factors (Camera/Object Orientation, Size, Texture, Light)
Figure 7: Random images taken from the PUG: Animals dataset.
are at 0 or at their default value. When changing the other factors (Camera/Object Orientation, Size, Texture), the background is set to "SaltFlats_0" (Which is the most basic background). When changing the light of the scene, we have used the environment "Opera".
The assets in PUG:ImageNet were selected base on 151 ImageNet classes which are listed below: ['BirdHouse', 'Chest', 'Bagel', 'WarPlane', 'Rocking_Chair', 'Bridge', 'Street_Sign', 'Cabbage', 'Pay_Phone', 'Butternut_Squash', 'JellyFish', 'Jack_O_Lantern', 'Bookcase', 'Stonewall', 'Punching_Bag', 'Toaster', 'Mushroom', 'Frog', 'Jeep', 'Television', 'Pineapple', 'Vacuum', 'Torch', 'Carousel', 'Desk', 'WineBottle', 'Wallet',
'Dining_Table', 'Military_uniform', 'Car_Wheel', 'Table_Lamb', 'Digital_Watch', 'Electric_Fan', 'Sweatshirt', 'Komodo_dragon', 'Racket', 'Cheeseburger', 'Can_Opener', 'Pomegranate', 'Convertible', 'Laptop', 'Chicken_hen', 'Wolf', 'Bulletproof vest', 'Shield', 'Bathtub', 'Throne', 'Lighter', 'Bycicle', 'Cofee_Mug', 'Motor_Scooter', 'Jean', 'Soccer_Ball', 'Vending-machine', 'Hatchet', 'Umbrella', 'Bear', 'Artichoke', 'Vase', 'Radiator', 'SpaceShuttle', 'Manhole_Cover', 'Polaroid_Camera', 'Traffic_Light', 'Radio', 'Soup_Bowl', 'Zucchini', 'Barrel', 'Tennis_Ball', 'Sunglasses', 'Microwave', 'Joystick', 'Aircraft_Carrier', 'Fox', 'Submarine', 'BasketBall', 'Running_Boe', 'Chain-saw', 'Piano', 'Crate', 'Loupe', 'Minivan', 'Shirt', 'Remote_controler', 'Airliner', 'Sock', 'Shovel', 'Mask', 'Tractor', 'Sandal', 'Wooden_Spoon', 'Drum', 'Goldfish', 'Gasmask', 'Mailbox', 'Volley_Ball', 'Banana', 'Penguin', 'Sliding_Door', 'Pool/Table', 'Burrito', 'Candle', 'Purse', 'Canoe', 'Typewriter_Keyboard', 'Espresso_maker', 'Carton', 'Park_Bench', 'Screen', 'African_crocodile', 'Cat', 'Hay', 'Elephant', 'WaterBottle', 'Modem', 'Palace', 'Ice_Cream', 'Washer', 'Sewing_Machine', 'HairDryer', 'Rabbit', 'Dishwasher', 'Bell_Pepper', 'Ambulance', 'French_Loaf', 'Refrigerator', 'Mouse', 'Obelisk', 'Starfish', 'Brocolli', 'Microphone', Great_white_shark, Power-drill, Locomotive, Perfume, Whale, Screwdriver, Dial_telephone, Backpack, Harmonica, Binocular, Skirt, Pizza, Cowboy_Hat, Computer_Keyboard, Kangarou,Baseball, Tile_Roof, Lawn_Mower, Safe, Cellular_telephone]
In Figure 8, we show random images taken from the PUG: ImageNet dataset.
B.4 PUG: SPAR
PUG: SPAR contains 43,560 pre-rendered images using 32 animals assets, 10 backgrounds, 4 positions and 4 textures. In contrast with the other PUG datasets, PUG: SPAR contain up to two animal in a single scene. For generating the PUG: SPAR dataset, we utilize the same subset of assets as PUG: Animals.
- World_Name: ['Desert', 'Arena', 'OceanFloor', 'Racetrack', 'TokyoDay', 'Circus', 'BoxingRing', 'AustralianRoad', 'SaltFlats', 'Museum'],
- Character_Name : ['Goldfish', 'Caribou', 'Elephant', 'Camel', 'Penguin', 'Zebra', 'Bear', 'Beaver', 'Cattle', 'Armadillo', 'Gecko', 'Crow', 'Scorpion', 'GiantTortoise', 'Tarantula', 'Rhinoceros', 'Dolphin', 'EarlessSeal', 'Goat', 'Hippopotamus', 'Horse', 'Impala', 'Lion', 'Orca', 'Pig', 'Rabbit', 'Squirrel', 'Chicken', 'WhiteShark', 'Anlylosaurus', 'BlackRockFish', 'PoisonDart-Frog'],
- Character_pos: ["Left/Right", "Bottom/Top"]
- Character_Text: ["Default", "Blue/Red", "Grass/Stone"]
In Figure 8, we show random images taken from the PUG: SPAR.
B.5 PUG: AR4T
Assets and Environments For generating the PUG: AR4T dataset, we utilize the same subset of Sketchfab assets as used in PUG: ImageNet. This leaves us with a set of 680 unique assets chosen from across 151 ImageNet categories, each manually inspected to quality control for photo-realism as much as possible. For this PUG dataset we are primarily concerned with object-object and object-attribute information, hence we utilize single camera and character orientation. Since Sketchfab assets differs widely in their scales, we first scaled down the longest edge of the asset bounding box to 150 pixels to normalize the order of magnitude of the asset dimensions, before any further scaling based on attributes. Next, we select a total of 26 unique environments as our background environments. The richness of some environments enables us to manually select different camera views in each environment as a novel environment view, and we generate a total of 28 visually unique backgrounds for our PUG: AR4T dataset. We provide each background with human-intelligible descriptive names for use in the dataset captions (Table 6). The PUG: AR4T dataset is composed with two subsets described in the following sections.
PUG: AR4T-Relations For generating the PUG: AR4T-Relations subset, we utilize the spatial relationships from Visual Genome that are not symmetric, as noted in the ARO benchmark Yuksekgonul et al. [2023]. The set of relationships used in PUG: AR4T-Relations is given in Table 7. It consists of three unary relations (at, in, inside) and ten binary relations (above, on, on top of, behind, in front of, below, beneath, under, to
Figure 8: Random images taken from the PUG: ImageNet dataset.
the left of, to the right of.) For each relation, the corresponding objects are picked randomly from the set of assets, and placed in the scene based on the co-ordinates given in Table 7. We add a random offset in the range [0-25] along each dimension for every individual object for every scene, so that the dataset does not contain shortcuts where the object locations can directly inform the underlying relation. The size of each asset is chosen randomly on a scale of 1-10, where 1 corresponds to 110 pixels and 10 to 200 pixels for the longest edge of the scaled asset.
Figure 9: Random images taken from the PUG: SPAR dataset.
The caption for a scene is generated using one of two templates based on whether the spatial relationship in the scene is unary or binary:
- Urinary Relation: "[ImageNet class label of asset 1] [relation] [human-intelligible background description]" e.g. Banana inside museum
Environment Name
(with camera view variant)	Descriptive Caption for PUG	Environment Name
(with camera view variant)	Descriptive Caption for PUG
Arena	arena	IceRoad	icy road
VillageOutskirt	village	Jungle	jungle
VillageSquare	village	Library	library
Battleground	battleground	Museum	museum
Beach	beach	OceanFloor	ocean floor
Bridge	bridge	AncientAgora	castle outskirts
Circus	circus	Racetrack	race track
Cliff	cliff	RuralAustralia	rural wilderness
Courtyard	courtyard	AustralianRoad	desert road
Egypt	egypt	SaltFlats	salt flats
ElvenRuins	ruins	SpaceIsland	space
EuropeanStreet	european street	StylizedEgypt	egypt
FightingArena	fighting arena	Temple	temple
Forge	forge	TrainStation	train station
Table 6: Environments and descriptive captions used in PUG: Attributes & Relations
Relation	Object 1 coordinates (wrt origin)	Object 2 coordinates (wrt origin)	Negative Relation
At	(0,0,0)	N/A	N/A
In	(0,0,0)	N/A	N/A
Inside	(0,0,0)	N/A	N/A
Above	(0,0,300)	(0,0,0)	Below, Beneath, Under
On top of	(0,0,300)	(0,0,0)	Below, Beneath, Under
On	(0,0,300)	(0,0,0)	Below, Beneath, Under
Below	(0,0,0)	(0,0,300)	Above, On Top Of, On
Beneath	(0,0,0)	(0,0,300)	Above, On Top Of, On
Under	(0,0,0)	(0,0,300)	Above, On Top Of, On
Behind	(150,0,0)	(-150,0,0)	In front of
In front of	(-150,0,0)	(150,0,0)	Behind
To the left of	(0,-150,0)	(0,150,0)	To the right of
To the right of	(0,150,0)	(0,-150,0)	To the left of
Table 7: Relations, corresponding asset locations wrt to camera origin, and corresponding hard negative relation used in PUG: AR4T
- Binary Relation: "[ImageNet class label of asset 1] [relation] [ImageNet class label of asset 2] [(random) unary relation] [human-intelligible background description]" e.g. Banana to the left of chair inside museum
For each binary relation, we also sample the corresponding hard negative scene and caption, by replacing the relation with its negative from Table 7 such that the semantic meaning of the scene and caption represents a switch from the original relation between objects. For example, the negative of "Banana to the left of chair inside museum" is given by "Banana to the right of chair inside museum".
We sample a total of 310 binary relation scenes (155 random + 155 negatives), and 310 unary relation scenes for each background, thus leading to a dataset of 112,840 image-caption samples (28 backgrounds x (310 pairs x 10 binary relations + 310 pairs x 3 unary relations)). The PUG:Relations dataset generation process described above is summarized as psueduocode in Algorithm 1.
Lastly, we split the dataset into 101,920 train and 10,920 test image-caption samples, such that the test set is balanced by background and relations (28 backgrounds x 13 relations x 30 samples). We also release the subset of training and test samples that only contain pairs of objects in scenes along with their hard negatives, which contains 78,400 training and 8,400 test samples, or 39,200 training pairs and 4,200 test pairs. This subset enables Winoground Thrush et al. [2022] style evaluation as well as training with hard visual negative mining for VLMs in future work. The PUG:Relations dataset generation process described above is summarized as psueduocode in Algorithm 1
PUG: AR4T-Attributes For PUG: AR4T-Attributes the selection of assets, relations between assets, and spatial locations of assets is done exactly as for PUG: AR4T-Relations. However, since the focus of this dataset is on the object-attribute pairs, the relations between objects are not represented in the corresponding scene caption in any form. The attribute for each object is chosen randomly from the set of 53 attributes described in Table 8. For size based attributes, the asset's material instance remains the same but its size is varied between 50 pixels (short, small, little, tiny) and
Algorithm 1 PUG: AR4T-Relations subset generation
{
    Urary = {At, In, Inside}
    Categories = {Set of ImageNet Class Labels}
    Environments = {Set of Unreal Environments}
    Relations = {Set of Relations}
    NegRelations = {Dictionary of relations as key, semantically negative relations as values}
    Assets = {Dictionary of Sketchfab assets, keys being ImageNet Class Labels}
    AssetLocations = {Function that returns locations of assets based on relation with a random offset}
    Dataset = φ
    for env in Env do
        for rel in Rel do
            ▷ For binary relations we also add the negative scene+caption to the dataset, thus the effective number of samples per background is  $2*15 = 30$ 
            if rel ∈ Urary then
                num_samples = 15
            else
                num_samples = 30
        end if
        for i = 1 to num_samples do
            cat1 = random.choice(Category)
            asset1 = random.choice(Assets[cat1])
            if rel∉Urary then
                cat2 = random.choice(Category)
                asset2 = random.choice(Assets[cat2])
            end if
            location1, location2 = AssetLocations(rel)
            scene1 = TorchMultiverse(asset1, asset2, location1, location2, env)
            if rel∉Urary then
                rel2 = random.choice(Unary)
                caption1 = cat1 + ', + rel + ', + cat2 + ', + rel2 + ', + env
                Dataset ∪ {scene1, caption1}
            ▷ Generate hard negative scene and caption
            scene1 = TorchMultiverse(asset1, asset2, location2, location1, env)
            caption2 = cat1 + ', + NegRelations[rel] + ', + cat2 + ', + rel2 + ', + env
            Dataset ∪ {scene2, caption2}
        else
            caption1 = cat1 + ', + rel + ', + env
            Dataset ∪ {scene1, caption1}
        end if
        end for
    end for
    end for
    return Dataset
200 pixels (big, long, tall, large). For all other attributes, the material instance of the object is changed in Unreal using Pytorch Multiverse.
The caption of the scene is generated using one of two templates based on whether the spatial relationship in the scene is unary or binary:
- Urny Relation: "[Attribute of asset 1] [ImageNet class label of asset 1] [relation] [human-intelligible background description]" e.g. Green banana inside museum
- Binary Relation: "[Attribute of asset 1][ImageNet class label of asset 1] and [Attribute of asset 2][ImageNet class label of asset 2] [(random) unary relation] [human-intelligible background description]" e.g. Green banana and large chair inside museum
For each binary relation, we also sample the corresponding hard negative scene and caption, by swapping the attributes between objects such that the semantic meaning of the scene and caption
Attribute Category	Attribute	Variants	Attribute Category	Attribute	Variants
Material	Brick	3	Color	Black	2
Metal	2	Blue	2
Wood	2	Yellow	2
Glass	2	White	2
Cloth	2	Green	2
Plastic	2	Gray	2
Rock	2	Brown	2
Size	Big	1	Red	2
Long	1	Silver	2
Tall	1	Orange	2
Large	1	Pink	2
Short	1	Gold	2
Small	1	Texture	Striped	2
Little	1	Dark	2
Tiny	1	Cloudy	2
Total=	53
Table 8: Attributes and corresponding number of variants used in PUG: AR4T-Attributes
represents a switch from the original object-attribute associations. For example, the negative of "Green banana and large chair inside museum" is given by "Large banana and green chair inside museum". For each of the 53 object attributes, we sample 2 scenes of it in conjugation with another object and attribute, and 2 scenes of the attribute in isolation in a scene. We repeat this for each possible background, thus leading to a dataset of 160,272 image-caption samples (28 backgrounds x (53 attributes x (53 attributes x 2 samples) + 2 samples)). The PUG:Attributes dataset generation process described above is summarized as psueduocode in Algorithm 2.
Lastly, we split the dataset into 147,976 train and 12,296 test images. For each attribute pair in a scene, we select 4 image-caption samples in the test set (4 samples x 53 attributes x 53 attributes = 11,236 sample). And we sample the remaining test samples by selecting 20 image-caption samples for each attribute in isolation in a scene (20 samples x 53 attributes = 1,060 samples). Similar to PUG-attributes, we also separately release the subset of training and test set with scenes and captions containing only pairs of scenes with their corresponding hard negatives, leading to 146,158 training samples and 11,236 test samples, or 73,0739 training and 5,618 test sample pairs.
Caption variants in PUG: AR4T In order to emphasize the idea that each scene can have multiple descriptive captions associated with it, we utilize a simple template to generate alternate captions for scenes with binary relations that are semantically consistent but linguistically different. During fine-tuning, the model randomly sees either the original caption or the alternate caption. The presence of alternate captions also prevents the VLM to learn shortcuts between the position of the object descriptions in captions and the underlying spatial relationship or object-attribute association.
- PUG: Relations "[ImageNet class label of asset 2] [negative relation] [ImageNet class label of asset 1] [(random) unary relation] [human-intelligible background description]" e.g. The alternate caption for 'Banana to the left of chair inside museum' is 'Chair to the right of banana inside museum'
- PUG: Attributes "[Attribute of asset 2][ImageNet class label of asset 2] and [Attribute of asset 1][ImageNet class label of asset 1] [(random) unary relation] [human-intelligible background description]" e.g. The alternate caption for 'Green banana and large chair inside museum' is 'Large chair and green banana inside museum'
Algorithm 2 PUG: AR4T-Attributes subset generation
Unary = {At, In, Inside}  
Categories = {Set of ImageNet Class Labels}  
Environments = {Set of Unreal Environments}  
Relations = {Set of Relations}  
Attributes = {Set of Attributes}  
Assets = {Dictionary of Sketchfab assets, keys being ImageNet Class Labels}  
AssetLocations = {Function that returns locations of assets based on relation with a random offset}  
Dataset =  $\phi$   
for att1 in Attributes do  
    for att2 in Attributes + [None] do  
        if att2 == None then  
            num_samples = 20  
        else  
            > For binary relations we also add the negative scene+caption to the dataset, thus the effective number of samples per background and attribute is  $2*2 = 4$   
            num_samples = 2  
    end if  
    for i = 1 to num_samples do env = random.choice(Environments)  
        if att2 == None then  
            rel = random.choice(Unary)  
        else  
            rel = random.choice(Relations - Unary)  
    end if  
    cat1 = random.choice(Categories)  
        asset1 = random.choice(Assets[cat1])  
        if att2 != None then  
            cat2 = random.choice(Categories)  
            asset2 = random.choice(Assets[cat2])  
    end if  
    location1, location2 = AssetLocations(rel)  
    scene1 = TorchMultiverse(asset1, asset2, location1, location2, att1, att2, env)  
    if rel∉Unary then  
        rel2 = random.choice(Unary)  
        caption1 = att1 + ’’ + cat1 + ’’ and ’’ + att2 + ’’ + cat2 + ’’ + rel2 + ’’ + env  
        Dataset ∪ {scene1, caption1}  
        > Generate hard negative scene and caption by switching object-attribute associations  
        scene2 = TorchMultiverse(asset1, asset2, location1, location2, att2, att1, env)  
        caption2 = att2 + ’’ + cat1 + ’’ and ’’ + att1 + ’’ + cat2 + ’’ + rel2 + ’’ + env  
        Dataset ∪ {scene2, caption2}  
    else  
        caption1 = att1 + ’’ + cat1 + ’’ + rel + ’’ + env  
        Dataset ∪ {scene1, caption1}  
    end if  
    end for  
    end for  
end for  
return Dataset
C Additional experimental details
C.1 Equivariance study details
In section 3.2, we used PUG: Animals to study how foundation vision-language models behave with respect to changes in factors of variations. We showed high image and text equivariance with respect to background, and text equivariance with respect to size and texture too. Here, we provide more details and results.
In our study, we use the following pretrained models:
- BLIP with ViT base backbone from the Huggingface transformers library [Wolf et al., 2020], trained on COCO dataset [Lin et al., 2014],
- NegCLIP from [Yuksekgonul et al., 2023],
- As in [Yuksekgonul et al., 2023] we use X-VLM pretrained on COCO dataset from https://github.com/zengyan-97/X-VLM,
- Flava with ViT-B/32 backbone from Huggingface transformers (https://huggingface.co/facebook/flava-full)
- CLIP models all come from OpenAI CLIP https://github.com/openai/CLIP. We use the versions with ResNet50, ResNet101, ViT-L/14, ViT-B/16, ViT-B/32.
We compute equivariance to each of the factors of variations. Since the text captions do not take into account camera and asset orientations, when we compute the equivariance with respect to the other factors we take only samples for a given orientation of the character and camera (0 for roll, pitch and yaw in both cases). Furthermore, in the text caption we replace sizes by three adjectives as follow: 0.7 is mapped to small, 1.0 to medium and 1.3 to big. Inspired by [Bouchacourt et al., 2021], we compute equivariance as the alignment between embedding difference vectors. That is, we compute embedding difference vectors  $z_{i} - z_{j}$  where  $z_{i}$  and  $z_{j}$  are the (normalized) embeddings of two images (or texts) of an object undergoing a given factor change. We then measure pairwise alignment as cosine similarity of embedding difference vectors (either between images pairs, text pairs or image-text pairs) corresponding to the same factor change. We report averaged cosine similarity of randomly paired vectors, and a higher value implies higher equivariance.
Note that a model can present image equivariance but no text equivariance (caption and image are not guaranteed to be encoded in the same vector), or have high equivariance across modalities but no image or text equivariance and vice-versa. In Figure 11 we report image equivariance with
(a) Image equivariance
(b) Text equivariance
(c) Across modalities equivariance
Figure 10: Measuring foundation models equivariance thanks to PUG: Animals: all three factors.