Visual Data-Type Understanding does not emerge from scaling Vision-Language Models
Authors: Vishaal Udandarao
Max F Burg
Samuel Albanie
Matthias Bethge
Venue: ICLR 2024 poster (2024)
Keywords: Data-Type Understanding
Vision-Language Models
Scaling
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce "Visual Data-Type Identification", a novel task for identifying visual distortions or perceptual changes to images, we find poor performance of VLMs across the board, and scaling models up does not improve performance significantly.

Abstract: Recent advances in the development of vision-language models (VLMs) are yielding remarkable success in recognizing visual semantic content, including impressive instances of compositional image understanding. Here, we introduce the novel task of Visual Data-Type Identification, a basic perceptual skill with implications for data curation (e.g., noisy data-removal from large datasets, domains pecific retrieval) and autonomous vision (e.g., distinguishing changing weather conditions from camera lens staining). We develop two datasets consisting of animal images altered across a diverse set of 27 visual data-types, spanning four broad categories. An extensive zero-shot evaluation of 39 VLMs, ranging from 100M to 80B parameters, shows a nuanced performance landscape. While VLMs are reasonably good at identifying certain stylistic data-types, such as cartoons and sketches, they struggle with simpler data-types arising from basic manipulations like image rotations or additive noise. Our findings reveal that (i) model scaling alone yields marginal gains for contrastively-trained models like CLIP, and (ii) there is a pronounced drop in performance for the largest auto-regressively trained VLMs like OpenFlamingo. This finding points to a blind spot in current frontier VLMs: they excel in recognizing semantic content but fail to acquire an
understanding of visual data-types through scaling. By analyzing the pre-training distributions of these models and incorporating data-type information into the captions during fine-tuning, we achieve a significant enhancement in performance. By exploring this previously uncharted task, we aim to set the stage for further advancing VLMs to equip them with visual data-type understanding.

Published as a conference paper at ICLR 2024
VISUAL DATA-TYPE UNDERSTANDING DOES NOT EMERGE FROM SCALING VISION-LANGUAGE MODELS
Vishaal Udandarao" ..2,+ Max F. Burg Samuel Albanie 4
Matthias Bethge
1 University of Cambridge, UK* 2 University of Tübingen, Tübingen AI Center, Germany
3 Institute of Computer Science and Campus Institute Data Science, University of Göttingen, Germany Equal contribution. Author ordering decided by coin flip. Joint senior authors. Correspondence: vu214 @ cam.ac.uk uk and max.burg @ bethgelab.org
ABSTRACT
Recent advances in the development of vision-language models (VLMs) are yield- ing remarkable success in recognizing visual semantic content, including impres- sive instances of compositional image understanding. Here, we introduce the novel task of Visual Data-Type Identification, basic perceptual skill with im- plications for data curation (e.g., noisy data-removal from large datasets, domain- specific retrieval) and autonomous vision (e.g., distinguishing changing weather conditions from camera lens staining). We develop two datasets consisting of an- imal images altered across a diverse set of 27 visual lata-types, spanning four broad categories. An extensive zero-shot evaluation of 39 VLMs, ranging from 100M to 80B parameters, shows a nuanced performance landscape. While VLMs are reasonably good at identifying certain stylistic dala-typex, such as cartoons and sketches, they struggle with simpler data-types arising from basic manipulations like image rotations or additive noise. Our findings reveal that (i) model scal- ing alone yields marginal gains for contrastively-train (ii) there is pronounced drop in performance for the largest auto-regressively trained VLMs like OpenFlamingo. This finding points to a blind spot in current frontier VLMs: they excel in recognizing semantic content but fail to acquire an understanding of visual data-types through scaling. By analyzing the pre-training distributions of these models and incorporating data-type information into the cap- tions during fine-tuning, we achieve significant enhancement in performance. By exploring this previously uncharted task, we aim set the stage for further advancing VLMS to equip them with visual data-type understanding.
models like CLIP, and
INTRODUCTION
Vision-Language Foundation Models (VLMs) (Bommasani et al., 2021) at the frontier of the machine learning ecosystem. Profiting from high-capacity transformer architectures (Vaswani al., 2017) and large-scale pre-training, these models excel at identifying the semantic content im- ages (Radford al., 2021; Pham et al., 2023; Jia et al., 2021). They also exhibit strong robustness to image distortions and perceptual changes as assessed on henchmarks like ImageNet-C (Hendrycks & Dietterich, 2019), ImageNet-Sketch (Wang et al., 2019), and ObjectNet (Barbu al., 2019). Tak- ing ImageNet-C as a concrete example, a classifier is tasked with correctly identifying a category (e.g.. stingray) in the presence of particular data transformation (e.g., defocus blur). Similarly, the other domains and perceptual transformations contained in ImageNet-C, ImageNet-Skeet -Skethh,, and ObjectNet can be seen as examples of different Visual Data- Data-Types obtained from ImageNet through applying image transformations that affect the appearance but not the content of the image.
The prevalent strategy in computer vision to cope with variable data-types is to use domain invariant classifiers, often achieved via data augmentation during training. An alternative strategy would be to retain the data-type specific information and explicitly model its composition with the semantic content of the image (Fig. IA). This constitutes symmetric split of the total image information into the complementary components of semantics and visual data-rypes (Granlund & Knutsson, 1995). Humans identification is an integral part of human perception (Oliva & Torralba, 2007; Ren et al., 2020; Bracci et al., 2023).
flexibly switch between these two complementary aspects and visual data-type

---

Published as a conference paper at ICLR 2024
A. Compositional understanding of Image information split into semantics and data-types
Data- Types
REFERENCE
Semantics
transform
EAGLE
Examples of data-type identification tasks in data curation
Ab Ab
AITLE
TIPOGIAPHIC Detacing Image turperkn
CA Cruadht specialoed cenguries
JPEG COMPRESS Detscting comprestion
VI.M Performonee
0.51
Charco
0,12
0.38
0:31
0.33
0.15
2:00
0.00
C. Examples of data-type identification tasks in autonomous vision
HIGH BRIGHTNESS
Recograing weather deiing
Recegolinng sarglaaa
inderpriiations
VIM Peeformince
0.89
0.0
0.08
0.01
2,09 -0.00
VUM legend:
OpenFlamingo
Figure 1: Data-type identification highly impacts vision tasks. Complementary standard se- mantic recognition tasks (A), data-rype identification targets recognising style and other contextual domain information. autonomous cars and agents. In all contexts, flexible recognition of data-types is paramount, yet, VLMMs exhibit poor performance on different data-types as illustrated by 4 select VL.Ms on data- types (highlighted in the boxes). Notably, there is one-size-fits-a model, underscoring the chal- lenge of universal data-type identification. The bar plots report the informedness metrics, for more details refer to Sec. 4.1.
applicable for many practical scenarios, e.g., (B) data curation, and (C)
The recent breakthrough of large language models (LLMs) to mimic human text understanding is reflected by remarkable compositional reasoning skills and flexibility to cope with arbitrary contexts and textual data-types. This suggests that VL.Ms could also gain an increasingly flexible, composi- tional image understanding to cope with arbitrary visual data-types by inheriting it from the use of such powerful LLMs. Therefore, we seek investigate to what extent the increasing robustness of VLMs to distribution shifts could be consequence of compositional data-type understanding.
The most likely alternative would be that the increasing robustness of VLMs originates from in- creasing domain invariance (Mintun et al., 2021). However, VLMs differ in two important ways from ImageNet-trained classification models of the last decade: (1) They are trained on much more data crawled from the internet making it difficult to judge whether test image is in-domain
2

---

Published as a conference paper at ICLR 2024
out-of-domain (Mintun et al., 2021; Fang et al., 2022; Nguyen al., 2022), and (2) Due to the compositional nature of language, training on image-text text-pairs could facilitate a compositional un- derstanding of images in VLMs. Both points drive performance yet, it is not easy to dissect their specific contributions. In addition, compositional understanding itself is property that needs to be learned and thus expected to gradually improve with the amount of training data and model scale (Wiedemer et al., 2023).
large set of visual benchmarks,
Here, we test the hypothesis that dataset robustness of VLMs could be consequence of compo- sitional data-type understanding by creating a carefully designed data-type identification task and investigating to what extent VLMs exhibit a compositional understanding of semantic context and image appearance. Data-type identification is a necessary condition for data-type understanding: If a VLM understands the data-type of an image, e.g., the blurring operation, needs to be able to identify it, independently from the particular image content.
Further, identifying the visual data-type of an image in addition to its semantic context is relevant in many real-world scenarios. For (7) data curation and filtering this is useful, for instance to exclude images of unwanted appearance from an image corpus (e.g., blurred samples), or to create a specific stylized domain generalization dataset (e.g., cartoons, sketches) (see Fig. IB). In the context of (2) autonomous vision (e.g., self-driving cars, household robots), knowing the data-type of camera- scenes is relevant to interpret the data and intervene accordingly: for example, adapting driving style or sensor sensitivity based on detecting changing weather conditions versus sun-glare (see Fig. 1C).
Rather than engineering narrow solutions for each of these problems individually, the flexibility of VLMs affords a general ability to cope with all possible conditions. A compositional understand- ing of data-types would be an attractive solution achieve this level of generality, and could be highly useful for practical challenges such as the long-tailed test-time distribution encountered in autonomous driving (Dosovitskiy et al., 2017; Makansi et al., 2021; Zhou et al., 2022). Due to the combinatorial number of possible conditions and the open-ended nature of perception for an au- tonomous agent, the importance of a compositional understanding of data-types extends to robotics at large to deal with variable conditions in households, agriculture, or healthcare.
In summary, our work aims to make progress on Data- Data-Type Identification; for this, we created two novel datasets containing images of animals, spanning 27 different data-types (see Fig. 2). On this data, we zero-shot benchmarked 39 state-of-the-art VLMs. with model sizes ranging from 100M to 80B parameters, across contrastive and LLM--ased VLMs. We find that scaling up model size does not yield sigmificant improvements. In particular, the largest auto-regressively trained VLMs perform significantly worse than their smaller contrastively-trainn trained counterparts like CLIP. By in- vestigating their performance across individual data-types, we found connections to structures in the pre-training data and vision embedding spaces of VLMs. Using this, show that performance on the novel data-type identification task can be enhanced by fine-tuning with carefully tailored data. Our findings highlight an important limitation in the training of current leading VLMs: while they clearly excel on recognizing semantic content, acquiring data-type identification skills does not emerge from simply scaling up but rather requires systematic change of the training data.
2 RELATED WORK
Stress-testing VLMs. Initial reports on the abilities of VLMs (e.g., in visual question answering)
were largely anecdotal. Very recently, there is a growing interest in systematic investigation of such capabilities, often entailing the creation of synthetic datasets tailored for specific evaluations (Yuk- sekgonul et al., 2022: Parcalabescu et al., 2021; Thrush et al., 2022: Hsich et al., 2023; Zhao al., 2022; Lewis et al., 2022; Yamada et al., 2022; Ma et al., 2023; Kamath et al., 2023; Marathe et al., 2023; Yarom et al., 2023; Bitton-Guetta et al., 2023; Bordes et al., 2023). Here, we too synthesize controlled dataset, but distinctively introduce the new task of Data-Type Identification, a basic perceptual skill that remains largely underexplored in previous work.
Distribution shifts, anomaly detection and domain adaptation. While many existing approaches study perceptually altered data, e.g., distribution shifts (Hendrycks et al., 2021: Taori et al., 2020; Schneider et al., 2020; Qiu et al., 2022; Rauber et al., 2017: Koh et al., 2021), domain adap- tation (Farahani et al., 2021; You et al., 2019; Wang & Deng, 2018), -uttt--disttibutiin detec- tion (Hendrycks & Gimpel, 2016; Yang et al., 2021; Liu et al., 2020), and anomaly detection (Roth
3

---

Published as a conference paper at ICLR 2024
Mi
Hal Caltood
Long Canthes
jegg pege Sanann
legend:
Geometric
Pixel
Style
"simple"
"comppee""
Figure 2: Proposed data-types. Example images from our SymeetiiTypellemmt dataset for each of our 27 data-types, spanning four categories: geometric, pixel, style, and semantic data-types.
al., 2022: Han al., 2022: Pang al., 2021), they often only determine the presence of an anomaly or shift without pinpointing its exact nature. In contrast, if an intervention to an anomaly necessary, we need to pinpoint its exact nature, which the goal of Data-Type Identification.
Very few previous works have touched upon this question in narrow scenarios. Some studied iden- tifying few specific perceptual data-types using convolutional neural networks (CNNs) binary classification setup, e.g., image mirroring (Lin et al., 2020) and cropping (Van Hoorick & Von- drick, 2021). Zhu et al. (2022) trained linear probes to understand predictability of domains like paintings or cartoons from the image features of pre-trained CNN. Paiss et al. (2023) investi- gated counting objects in VLMs (similar to our MULTISSME and MULTI_DIFFERE DIFFERENT data-types, Fig. 2). An al. (2023) showed that CLIP can reasonably infer a limited number of simple data-types in a binary classification setup and used this to improve CLIP's zero-shot semantic clas- sification. Rashtchian et al. (2023) used linear probes the image embedding spaces of vision-only and vision-language models, to identify perceptual manipulations on images, without accounting for their text encoders. Our Data-Type Identification framework subsumes all these setups in uni- fying approach: we investigate an extensive range of 27 data-rypes across broad perceptual and stylistic range for 39 VLMs, encompassing both cntrastivell-traime discriminative models and auto-regressively trained generative models. Our work therefore enables studying generalisation of VLMs on broad set of data-types.
THE TYPEIDENT DATASETS
To probe the effectiveness of VLMs in identifying data-types, created two novel datasets consist- ing of images of a single animal in a scene, spanning 27 data-types across 4 categories: geometric (e.g., left-rotation), pixel (e.g., applying Gaussian noise), style (e.g., creating cartoon-version), and semantic (e.g., replacing a single animal with multiple animals). Note that geometric and pixel data-types can be obtained from simple, well-defined transformations such pixel re-arrangement, linear filtering, or additive noise. In contrast, most transformations generating different style and semantic data-types from reference image distinctively rely on the use of more complex neural networks. For a complete list of all data-types studied, see Fig. 2 and refer to the Appendix.
Our first dataset, SyntheticTlypeldent, is constructed by first generating a set of 50 reference-images of animals using text-to-image model, with these images uniformly sampled across 10 animal species. Each generated reference-image was then altered with all our 27 data-type transformation functions, resulting in 1,350 evaluation samples (see Fig. 2 for an example of all data-type transfor- mations). For creating the geometric and pixel data-type images, we directly applied the correspond- ing point-wise image transformation function (c.g., adding Gaussian noise) the ifference-iaage To precisely control the transformations applied, we regenerated style and most semantic-lewe data- type images again using the same diffusion model.
4

---

Published as a conference paper at ICLR 2024
B
Meeu Informeddeess
0.5r
0.60
0
Modals
CLP zoont>
MutundTTypelentt
Preciggs
[-629 ge
10
10
To
0.05
1-60
LLakk
Mean Informedness
SymteciiTypeeeett
0.60
Ieformednee
LMM
Synthelcrypstieee
Suiing aving dite
0.10
1- 0,00 gei
E- 0004 gi
0.-
10'
10
10"
10
10t
101
10
Model Scale
Model Scale [S]
Iesiliion
Figure 3: (A) VLMs struggle with identifying data-types. Less recent, contrastively learned C- VLMs (e.g.. CLIP) outperform the much larger and more recent LMMs (e.g.. IDEFICS) despite the latter's strong language model priors. Scaling shows limited effect on performance. Chance-level performance is at O. (B) Weak scaling laws for VLMs. Power-law fits reveal that for achieving strong data-type identification (mean informedness>0..7 current VLMs would need to surpass a trillion parameters. This calls for an alternative strategy to just scaling up current VLMs.
For second dataset, NaturalTypeldent. manually curated 50 reference-imager from Kag- glcAnimallmages (Banerjee, 2023). We then followed the exact same procedure for creating data- type images from the reference-images. However, all generative steps were replaced by a refined, deduplicated web-retrieval step for mining style and semantic data-type images. This provides an in-the-wild, naturally occurring testbed, thereby complementing the precisely controlled Synther- icTypeldent dataset. Since we can procure appropriate images for only 25 data-types (we omit DUFTI_DIFFERENT and STRIPES). NaturalTypeldent only contains 1,250 samples.
Importantly, we manually verified both datasets to ensure that the target data-type for each image was the most prominent data-type reflected in it, enabling careful study between models without interference between data-types. For details about dataset creation refer to the Appendix.
BENCHMARKING VLMS ON DATA-TYPE IDENTIFICATION
4.1 EXPERIMENTAL SETUP
We evaluated 39 VLMS from model families, with sizes ranging from 100M to 80B parameters, across two groups: discriminative, contrastively-trained VLMS (e.g.. CLJP) which we refer to as C-VLMs, and generative, ruto-regressivvly trained VLMs (c.g., OpenFlamingo) which we refer to as large multi-modal models (LMMss) (Li, 2023). Specifically, from the C-VLM group evaluated CLIP (Radford et al., 2021), BLIP-2-ITM (Li et al., 2023c), and CoCa (Yu et al., 2022); in the LMM group tested Fromage (Koh al., 2023b), GILL (Koh et al., 2023a), Multimodal et al., 2023), OpenFlamingo (Awadalla et al., 2023), Otter (Li et al., 2023a), MPlugOwl (Ye et al., 2023), LLaVA (Liu et al., 2023a), BLIP-2-LLM (Li et al., 2023c). InstructBLIP (Dai et al., 2023), and IDEFICS (Laurençon et al., 2023). We tested all VLMs on correctly classifying the target data-type for each evaluation image, in a zero-shot manner. We evaluated C-VLMs by computing the cosine-similarity of the image embedding and the text embedding of the specific data-type de- scription, e.g., "A blurred ima ge of an ani mimml. " (see Appendix for full list). For fair comparison, we evaluated LMMs by log- likelihid scoring (Dai et al., 2023; Li et al., 2023b) each of the 27 data-type description texts, with the prompt: "< <<image>>>> Q: Describe the image. Seaatattpeeddesceee
(Gong
A: <data_type wwwpeeddesssiiioooeeee sponding text description for particular data-type. We quantified model performance using in- formedness, I = FPR;; data-type k, which in addition to the true positive rate (TPR, i.e., accuracy) accounts for the false positive rate (FPR). We summarized model performance as mean informedness across data-types, (Tk)). See Appendix for evaluation details.
replacing
by the corre-
5

---

Published as a conference paper at ICLR 2024
1.2
Mean informed across all models
Seple
E3DC
0.2
0.2
tatto
vnadetre
perool aloue
origann
difirest
embridder
custor
defocso Mlto
sertine mi
erolita
daprr stapes
plantor
tyrouge watll
sapp bo brightmnua
mitagg
putis anal reliutte
Jeea
enteoit
bro. wr.bhigghh costavtt
ruild rolute
eyeg, and Jeorpp
Jope cart
ludt retate
Pata-Types
Figure 4: Average-performance across data-types on SyutheticTypellee Typeldent. VLMs perform rea- sonably on style and semantic data-types (e.g., PENCIL.SKETCH CARTOON) and show weak results on pixel and geometric data-types (e.g., GAUSSIAN_NOISE, IIGIL.CNNTRATT Chance-level at O.
4.2 VLMs STRUGGLE WITH IDENTIFYING DATA-TYPES
Our evaluations reveal that all tested VLMs exhibit limited performance on both SyntheticTypeller and NaturalTypeldent (Fig. 3A). We found that C-VLMs performed better than LMMs, even though the latter more recent and orders of magnitude larger. The best C-VLM achieved mean informed- ness (0=40, 0.50) while its LMM counterpart achieved μ1, (0.22, 0.25) on SyntheticTypelleen and NaturalTypellent, respectively. As a control and for direct comparison, we also tested mod- els on animal identification on SyntheticTypelleer As expected, the performance on this semantic recognition task is very good, achieving mean informedness across models of 0.89. This confirms quantitatively that the performance on identifying data-types (detailed plots in Appendix) is sub- stantially worse than on object recognition. We further note three key findings from our evaluations:
LMMs, a downgrade? Surprisingly, LMMs consistently underperform C-VLMs. despite using LLMs text models, compared to the smaller text encoders C-VLMs. Notably, the largest LMM (IDEFICS, 80B parameters) substantially underperforms an orders-of-magnitude smaller CLIP- RN50 (100M parameters). The rich language grounding that L.LMs inherit from extensive real-world text training seemingly does not provide benefits for identifying data-types. This result challenges the prevailing notion that strong language model priors can improve fine-grained understanding in VLMs Rascante--onilla et al., 2023; Doveh et al., 2023; Yuksekgonul et al., 2022; Wang et al., 2023). We hypothesise two plausible causes for this performance drop to be studied detail by future work: (1) Weak aligument between the vision encoder and LLM might degrade the real- world symbolic grounding innate to each independently (Bavishi et al., 2023). (2) Discriminative- Generative gap might be play, i.e., discriminating between answers easier than generating one (Vapnik, 1999; Ng & Jordan, 2001). Both suggest that C-VLM contrastive objectives might better equip them for data-type identification than LMM auto-regressive objectives (Liu et al., 2023b).
Weak scaling behaviour. Interestingly, within the C-VLM and LMM groups, our results suggest weak scaling effects. We analysed this quantitatively by fitting power-law (Alabdulmohsin al., 2022: Henighan et al., 2020; Cherti et al., 2023) on the observed mean informedness vs. model scale relationship for CLIP (C-VLM) and IDEFICS (LMM), since they span the widest parameter sizes within model family. Fig. 3B confirms the weak scaling law, indicating a severe limitation for current VL.Ms: to achieve a performance practicable for data-type identification (11300.7), current models would need to surpass trillion parameters. This calls into question the effects of model scaling, and whether alternate strategies are required to enhance their performance.
Stark performance differences between simple and complex data-types. To get finer-grained understanding of the overall model performance (Fig. 4) we break-down the per-data-type averaged mean informedness across all models. We find that while VLMs reasonably good at identify- ing style and semantic data-types, they falter systematically on pixel and geometric data-types. For the majority of data-types even the best-performing models struggle to surpass chance-level perfor-

---

Published as a conference paper at ICLR 2024
tSNE Embeddings coloured by semantic concepts
t-NE Embeddings coloured by data types
I-NN Dimcassions
19
-30
-30
-to
20
-10
15
&NN Dimansion
t-NN Dimenston 1.
Figure 5: What does CLIP's image embedding space encode? CLIP-RN5O's image embeddings, colour-coded by ground-trune truth semantic concept (left) and data-type (right), reveal its pronounced affinity for recognising semantic concepts, while being largely invariant to data-type distinctions.
mance and no single model consistently outperforms others across a majority of data-types. Instead, multiple models each excel in identifying just few specific data-types. This reveals inherent biases in the pre- training procedures of VLMs, limiting the desired generality of foundation models.
UNDERSTANDING WHY VLMS UNDERPERFORM IN IDENTIFYING DATA-TYPES
We next investigate two plausible reasons for the sub-par performance of VLMs in identifing data- types: (1) their image embeddings lack data-type discriminative information, and (2) their pre- training datasets, despite the enormous sizes, lack sufficient data-type specific information, limiting models from learning data-type discriminative features. We probe both candidate reasons in detail, performing case study with CLIP, and find good evidence for both of them. Due to CLIP being prototypical C-VLM, and the widespread adoption of its vision encoders in LMMs, we suggest that our findings should be broadly applicable.
Reason 1: Peeking into CLIP's embedding space. We visualized the CLIP image embeddings of SyetheticTypeldent using t-SNE (Van der Maaten & Hinton, 2008). Colour-coding the embeddings by (1) the image's semantic concept, i.e., the animal type (Fig. 5 left), and (2) the image's 's target data- type (Fig. 5 right), uncovered an interesting dichotomy: while distinct embedding clusters emerge based on semantic concepts (animals), most data-types are not clearly demarcated (see Appendix for KNN and linear-probe analysis). This suggests that CLIP's vision encoder is somewhat invariant to data-types, despite it not being explicitly trained be (only auddom-resized cropping was used as training augeentugmtntnionn discussion in Appendix). As most C-VLMs and LMMs use CLIP image embeddings, this potentially explains the poor performance of all VLMS identifying data- types. We further note that the embeddings of only three data-types are closely clustered (TATTOO, PATCHLAND D.E.UULLEE and TYPOGRAPHIC), yet, these are precisely the embeddings which are not directly semantically distinguishable-thie suggests that CLIP might not encode semantic and data-type information compositionally but rather sacrifices one (data-type) over the other (seman- tics). This offers consistent explanation why CLIP models are so effectively robust at classifying semantic content (Fang et al., 2022: Shi al., 2023; Nguyen al., 2022: Santurkar et al., 2022; Ramanujan et al., 2023) but fail at solving the complementary problem of data-type identification.
Reason Peeking into VLM pre-training datasets. Fig. revealed that VLMs fare well on some complex data-types while falling short on simple ones. An intuitive explanation is pre-training dataset imbalance: an abundance of samples aligning with style data-types (e.g., CARTOON, PEN- CIL.SKETCII) and paucity of simple data-types (e.g., GAUSSIAN NOISE, LEFT ROTATE.. To confirm this quantitatively, we analysed LAION-2B-en CLIP's pre-training dataset. We first counted and retrieved all samples containing representative data-type keywords in the captions (e.g.. "blurry": : see Appendix for details and a semantic search- based analysis). As pure keyword- frequency might not account for mis-aligned image-caption pairs, we estimated alignment prob-
7

---

Published as a conference paper at ICLR 2024
ability- the fraction of retrieved samples where the image aptly captures the data-type concept- by manually labeling 100 random samples per data-type for data-type accuracy. Finally, we computed an abundancy score as the product of text-frequency and alignment probability. Correlating this abundancy score with averaged model performance across data-types revealed strong positive as- sociations (Spearman rank correlation, r=0.557 for SyntheticTypellent: r=0.489 for Natural'Iyp-- Ident). The association is even stronger on SyntheticTTpeeldet when correlating abundancy score with CLIP-model averaged performance (r=0.606), suggesting that the varying model performance across data-types can be explained by the constraints of their pre-training data distribution.
IMPROVING VLMS TO IDENTIFY DATA-TYPES
Fese-shot IIP-Adpperr with CLPP
Having understood some factors limiting the performance of VLMs, we experiment with methods using data-type information- rich samples to improve them. Here, we investigate CLIP (C-VLM) and Otter (LMM) as two representative models.
Mean Inlortsesteesss
0.3
6.1 FEW-SHOT TRAINING-FREE ADAPTATION DOES NOT HELP
0.0
eFowsshot Examples
Can few-shot examples boost performance without updating model weights, using in-context learning (Dong et al., 2022; Brown et al., 2020) or training-fret adapters (Zhang al., 2021; et al., 2022)? We answer next.
(a)
Jn In-conteat: learning with Otter
Udandarao
Mesci InCorrmeddenss
0.8
You
0.3
CLIP dap-er We test the TIP-Adapter (Zhang et al., 2021) framework with CLIP, using two few-shot example selec- tion strategies: Random (selecting examples with random animals) and SameAnimal (selecting examples with same animal as test im- age). We evaluate 1, 2, 4, 8, 16, 32, 48 shots with RN50 and ViT-L- 14 vision encoders. We found few-shot adaptation degrading per- formance across all settings (see Fig. 6a). This presumably origi- nates from TIP-Adapter leveraging semantic similarities in CLIIP's image embedding space, which lacks information to disambiguate between data-types (see Fig. 5). Hence, TIP-Adapter cannot capture any information discriminative across data-types but rather exploits semantic similarities b/w concepts which detrimental for our task.
flo-conteat Examples 15
(b)
Figure 6: Few-shot training- free adaptation methods fail. Both TIP-Adapter with CLIP (top) and in-context learning with Otter (bottom) fail to substantially improve VLM data-type identification.
Otter In-context Learning. We explored various in-context ex- ample selection strategies and found selecting n examples with one whose data-type matched the target of the test sample and other n-1 randomly worked the best-we evaluate 2=2,5,15 15 examples on the Random and SameAnimal strategies, using I.LaMA-7B (Tou- vron et al., 2023) or MPT-7B (MosaicMI,, 2023) as LLM baak-bbbbooness (see Appendix for details and in-context scaling results with LLaVA). Surprisingly, we found an initial uptick in performance with n- followed by a decline in-context examples increased (see Fig. 6b). We attribute this to Otter overfitting on its in-context examples, i.e., simply predicting a random data-type from within the in- context examples. Since chance-level performance also increases with fewer in-context examples, this could explain improved performance with n-2. We conclude that in-context learning does not enhance Otter's 's ability identify data-types.
Takeaways. Our empirical results strongly indicate that training-free few-shot approaches fail to enhance VLMs for identifying data-types, likely because VLMs lack data-type discriminative infor- mation in their embeddings, Rather, an intensive training procedure to infuse data-type knowledege might be more promising.
6.2 FINE-TUNING WITH APPROPRIATE DATA-MIXTURES IMPROVES PERFORMANCE
Data-mixtures. We created a specialised dataset, TeDaTy (Tachinn Data- Types,, incorporating data-type information into images and text-captions. We construct training images, sourced from COCO (Lin et al., 2014), ImageNet (Deng et al., 2009). PACS (Li et al., 2017), and Domain- Net (Peng et al., 2019), by applying our data-type transformation functions and adapting the captions
8

---

Published as a conference paper at ICLR 2024
Table 1: CLIP ViT-B-32 fine-tuning results on Typeldent datasets with different data-mixtures.
SyntheticTypeldent
NaturalTypeldent
Data-Mixture
Full
Freeze-Imann
Freeze-Text
Full
Freeze-Image
Freeze-Text
ID-I
OOD-I
ID-I OOD-I
ID-I 0OD-1
ID-I
OOD-I
ID-I
OOD-1
ID-I OOD-1
Zero-shot CLIP
|0.551 0.457
[0.451
0.457
00.451
0.457
0.440
0.473
0.440
0.473
0..40
0.473
COCO (control)
|0.451 0.468
[0.354
0.465
00.488
0.451
0.499
0.507
0.451
0.500
0.457
0.473
TeDaTy
0.669 0.392
007777
0,469
0,780
0.370
0.691
0.412
0.654
0.474
0.646
0.379
+ COCO
0.646
0.394
0.717
0.465
0.631
0.371
0.629
0,400
0.680
0.470
0.574
0.356
COCO + IN100K
0.600 0.383
0.700
0.469
0.586
0.354
0.557
0.381
0.634
0.456
0.471
0.323
Table Otter-LLaMA-7D fine-tuning results with different data-mixtures.
SyntheticTypeeleer
Natural/yypeleent
Data-Mixture
ID-I
OOD-1
ID-I
OOD-1
Zero-shot Otter
0.051
0.180
0.102
0.256
COCO (control)
0,020
0.246
0.085
0.315
TeDaTy
0.088
0.061
0.111
0.111
+ COCO
0,106
0.168
0.171
0.276
+ COCO + IN100k
0.120
0.166
0.166
0.261
accordingly, e.g., "This is a cartoon image of dog.". TeDaTy comprises 8 in-distribution (ID) data- types, holding out 19 for
(OOD) generalisation tests (see Appendix for details). To isolate effects of data-distributiions, we experiment with three data-mixtures: (1) TeDa'Ty, (2) CCOCO, and (3) TeDally+COOO++NNO0 (sub-sampled from ImageNet). We also fine- tune only on COCO as a control to disentangle gains from fine-tuning and specific data-mixtures.
Fine-tuning CLIP improved performance on the ID data-types for all TeDaTy mixtures (Tab. 1). However, COCO-only fine-tuning degraded ID-performance, highlighting the importance of incoporating key data-type information with TeDaTy. Freezing the vision-encoder while fine- tuning provided large ID-boosts and surprisingly even improved OOD. Freezing the text-encoder improved ID- ID-eerformance but degraded OOD D0D-performance, likely because of large gradients from only updating the vision-encoder. This corroborates previous CLIIP-uning studies (Zhai et el., 2022).
Results.
Transfer to Otter. To fine-tune Otter, kept the vision encoder frozen (best CLIP fine-tuning strategy) and tuned only the perceiver resampler, cross-attention and embedding layers. We found fine-tuning with all TeDa'Ty variants improved ID-performance up to two-fold, while preserving OOD-performance (see Tab. 2). Fine-tuning only with COCO degrades ID-performance, reinforcing the importance of a dataset that captures data-type knowledge.
Takeaways. Our results suggest that training with data-mixtures explicitly inducing data-type infor- mation is a promising direction for improving VLM data-type identification.
CONCLUSION
In this work, we introduced and motivated Data-Type Identification as a basic perceptual skill with general implications for visual foundation modeling. We created two novel datasets study model performance on this task, and released a third dataset tailored to fine-tune models to improve data- type identification. Our extensive zero-shot experiments across 39 VLMs revealed that they struggle to identify many data-types. Interestingly, scaling model size results only in minimal gains we traced this back to the structure in VLM embedding spaces and pre-training datasets, and suggest that studying weak alignment between image-encoders and LLMS (Bavishi et al., 2023) as well as the discriminative--eeeraiii gap (Vapnik, 1999; Ng & Jordan, 2001; Saunders et al., 2022) will be promising directions for future work (see for example Liu et al. (2023b)). We found that training- free few-shot methods do not improve performance, and that it is necessary to incorporate data-type information back into the training process. Taken together, study reveals important limitation of the desired generality of foundation models, and the dataset and insights presented in this paper set the stage for further advancing VL.Ms for visual data-type understanding.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We provide code and datasets to reproduce all experiments in the paper here: https: //g/tttubb....... com/bethger ab/ DataType Irrptiidaiiicaa vided comprehensive details on dataset creation in the Appendix. We specify the details of the 39 models tested along with their evaluation methods in the Appendix. For all our fine-tuning exper- iments, we used a fixed random seed for reproducibility. Further, we will release all our fine-tuned checkpoints and make public the WeightsAndBiases training logs for easy access.
For the TypeldeniDatasels,
have pro-
ACKNONLEDGEMENTS
The authors would like thank (in alphabetic order): Alexander S. Ecker, Çağatay Yıldız, Evgenia Rusak, Roland Zimmermann, Shyamgopal Karthik, Surabhi S. Nath, Susanne Keller and Thomas Klein, for helpful comments and feedback. The authors thank the International Max Planck Re- search School for Intelligent Systems (IMPRS-IS) for supporting VU and MFB. VU thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. SA is supported by a Newton Trust Grant. This work was supported by the German Research Founda- tion (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP4, project number: 276693517. MB is member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemmenschaat (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645.
REFERENCES
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-222 222112, 2022.
Bang An, Sicheng Zhu, Michacl-Andrei Panaitescu-Liess Chaithanya Kumar Mummadi, and Furong Huang. More context, less distraction: Visual classification by inferring and conditioning on contextual attributes. arXiv preprint arXiv:2308. 01 0/313, 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Opentlamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01339, 2023.
Sourav Banerjee.
Animal image dataset: 90 different animals, 2023.
URL
https://wwet kaggle com datasetss iamsouravbanerrrrrr.rrr jee/ mamaa--mmgeeddttasee (awpaassttt----ffffeeeeeee
1s. Accessed: 2023-07-10.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Fuyu-8b: A multimodal architecture for ai agents, 2023. URL https:
//www. adept ai/blog/ fuyu-8b. Bb. Accessed: 2023-11-18.
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a lisinugand-langgage bench- mark of synthetic and compositional images. arXiv preprint avXiv:2303.07277, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv: (xXi::2108.075505 2021.
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learn- ing. arXiv preprint arXiv:2308.03977 2023.
10

---

Published as a conference paper at ICLR 2024
Stefania Bracci, Jakob Mraz, Astrid Zeman, Gaëlle Leys, and Hans Op de Beeck. The representa- tional hierarchy in human and artificial visual systems in the presence of object-scene regularities. PLOS Computational Biology, 19(4):e 101 1086, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, et al. Going beyond nouns with vision language models using synthetic data. arXiv preprint arXiv:2303. 17590, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer 2023.
Vision and Pattern Recognition, pp.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- general-purpose purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.066 06500, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun. Jingjing Xu, and Zhifang Sui. survey for in-context learning. arXiv preprint arXiv:2301.00234 2022.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pp. 1-16. PMLR, 2017.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Amit Alfassy, Roei Herzig. Donghyun Kim, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. arXiv preprint 2305::2300.15599 2023.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In International Conference Machine Learning, pp. 6216-6234, PMLR, 2022.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pp. 877-894, 2021.
Leo Gao. Multiple choice normalization in Im evaluation, 2023. URL https://blon Accessed: 2023-11-18.
eleuther. http://aeee.ppeeeeeeeeeeeooooooo
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gnn gpt:: vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04799 2023.
Gösta Granlund and Hans Knutsson. Signal processing for computer vision, 1995.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35:32142-32159, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2024
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: arXiv: 122.1, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610.026 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1340-8349 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint (xXiv:2000.147000 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207. 125988, 2022.
arXiv preprint
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sug- arcrepe: Fixing hackable benchmarks for vision-language compositionality. arXiv preprint arXiv:2306. 14610, 2023.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 911:22, 2020.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and usion-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders are performance bottlenecks in contrastive vision-language models. arXiv preprint arXiv:2305. 14897, 2023.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305.1722 2023a.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. arXiv preprint arXiv:2301.13825, 2023b.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, and Victor Sanh. In- troducing idefics: An open reproduction of state-of-the-ar visual language model, 2023. URL https://hugginnete
co co/blog/////ooooooooooooooooooco iddffic. Accessed: 2023-09-18.
Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does clip bind concepts? probing com- positionality in large image models. arXiv preprint arXiv:2212. 10537, 2022.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03776, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal 1lms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
Chunyuan Li. Large multimodal models: Notes on evpr 2023 tutorial. arXiv:2306. 14895, 2023.
arXiv preprint
12

---

Published as a conference paper at ICLR 2024
Da Li, Yongxin Yang. Yi-Zhe Song. and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023c.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 22511-22521 2023d.
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Zhiqiu Lin, Jin Sun, Abe Davis, and Noah Snavely. Visual chirality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12295-12303, 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 000.:08844,888, 2023a.
Lizhao Liu, Xinyu Sun, Tianhang Xiang, Zhuangwei Zhuang, Liuren Yin, and Mingkui Tan. Contrastive vision-language alignment makes efficient instruction learner. arXiv preprint arXiv:2311.17945 2023b.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-o--distribution detec- tion. Advances in neural information processing systems, 33:21464-211777 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint
arXiv:2104.08786 2021.
Zixian Ma, Jerry Hong Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision- language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10910-10921, 2023.
Osama Makansi, Özgün Cicek, Yassine Marrakchi, and Thomas Brox. On exposing the challenging long tail in future prediction of traffic actors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13147-13157, 2021.
Aboli Marathe, Deva Ramanan, Rahee Walambe, and Ketan Kotecha. Wedge: A multi-weather autonomous driving dataset built from generative vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3317-3326 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXhv:2210.07182. 2022.
Agnieszka Mikolajezyk and Michat Grochowski. Data augmentation for improving deep learning in image classification problem. In 2018 international interdisciplinary PhD workshop (IIPhDW), pp. 122. IEEE, 2018
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and cor- ruptions in natural corruption robustness. Advances in Neural Information Processing Systems, 34:3571-3583, 2021.
NLP Team MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL https: / www. mosai cml cwm/bl og /mptt/pttt//tt 7b. Accessed: 2023-09-18.
13

---

Published as a conference paper at ICLR 2024
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances neural information processing systems, 14, 2001.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455-21469 2022.
Aude Oliva and Antonio Torralba. The role of context in object recognition. Trendy in cognitive sciences, 11(12):200-527, 2007.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. arXiv preprint arXiv:2302. /2066, 2023.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel, Deep learning for anomaly detection: review. ACM computing surveys (CSUR), 54())11-33 2021.
Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, lacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. arXiv preprint arXiv:21 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Luis Perez and Jason Wang. The effectiveness of data augmentation image classification using deep learning. arXiv preprint arXiv:1712.04621, 2017.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of op- tions in multiple-choice questions. arXiv preprint arXiv:2308. 1188,, 2023.
Hieu Pham, Zihang Dai. Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, pp. 126658, 2023.
Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Are multimodal models robust to image and text perturbations? arXiv preprint arXiv:22 2,004, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402-5415, 2021.
Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ludwig Schmidt, and Ali Farhadi. On the connection between pre-training data diversity and fine-tuning robustness. arXiv preprint 1233::2307.12532. 2023.
Cyrus Rashtchian, Charles Herrmann, Chun-Sung Ferng, Ayan Chakrabarti, Dilip Krishnan, De- qing Sun, Da-Cheng Juan, and Andrew Tomkins. Substance style: What does your image embedding know? arXiv preprint arXiv:2307.05610 2023.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox benchmark the robustness of machine learning models. arXiv preprint arXiv:1707.04131, 2017.
Sylvestre-Alvist Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint 103.v:2103.01946, 2021a.
14

---

Published as a conference paper at ICLR 2024
Aiivestre-AAviee Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 01129993529944 2021b.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using stamese bert- networks. arXiv preprint arXiv:1908 10084, 2019.
Mengye Ren, Michael Iuzzolino, Michael C Mozer, and Richard S Zemel. Wandering within a world: Online contextualized few-shot learning. arXiv preprint arXiv:2007.04546, 2020.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1-318-4338, 2022.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables shoo task generalization, 2022.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is caption worth a thousand images? a study on representation learning. In The Eleventh International Conference on Learning Representations, 2022.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint rxXiv:2206.0582,, 2022.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in neural information processing systems, 33:1 33:11539-11551 2020.
Zhouxing Shi, Nicholas Carlini, Ananth Balashankar, Ludwig Schmidt, Cho-Jui Hsieh, Alex Beutel, and Yao Qin. Effective robustness against natural distribution shifts for models with different training data. arXiv preprint arXiv:2302.01333 2023.
Connor Shorten and Taghi M Khoshgoftnar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.153: 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18855 18599, 2020.
Luke Taylor and Geoff Nitschke. Improving deep learning with generie data augmentation. In 2018
IEEE symposium series on computational intelligence (SSCI), pp. 1542-1547. IEEE, 2018
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 5238-5248, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
15

---

Published as a conference paper at ICLR 2024
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. arXiv preprint arXiv:2211.16198, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9111), 2008.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine learning, 109(2):373-440, 2020.
Basile Van Hoorick and Carl Vondrick. Dissecting image crops. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9741-9750, 2021.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural net- works, 10(5):988-999, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changxing Ding. Can linguistic knowledge improve multimodal alignment in vision-language pretraining? arXiv preprint arXiv:2308. 12898, 2023.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312: 135-153, 2018.
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Composi- tional generalization from first principles. arXiv preprint arXiv:2307.05596, 2023.
Yutaro Yamada, Yingtian Tang, and Ilker Yildirim. When are lemons purple? the concept association bias of clip. arXiv preprint arXiv:2212.120433 2022.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2 10. 11334, 2021.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint urXiv:2305.10400 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 1414,,880 2023.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2720-2729, 2019.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-wond words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part / 13, pp. 818-838 1338.33.0 Springer, 2014.
16

---

Published as a conference paper at ICLR 2024
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.0322 11,00330, 2021.
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entrope adversarial data augmen- tation for improved generalization and robustness. Advances Neural Information Processing Systems, 33:14435-1444 2020.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.0022/, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Weitao Zhou, Zhong Cao. Yunkang Xu, Nanshan Deng, Xiaoyu Liu, Kun Jiang, and Diange Yang. Long-tail prediction uncertainty aware trajectory plamning for self-driving vehicles. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 1275- 1282. IEEE, 2022.
Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. Ood-probe: A neural interpretation of out-of- domain generalization. arXiv preprint arXiv:2208. 12352, 2022.
A DETAILS ABOUT TYPEIDENT DATASETS
In this section, we provide additional details on our dataset generation process.
A.1 LIST OF TYPE- STUDIED
In the table below, we provide complete list of all the data-types we used in our study across the four coarse-grained categories.
Table 3: The 27 data-types used in our study.
Geometric
Pixel
Style
Semantic
LEFT_ROTATE
GAUSSIAN
NOISE NOISE DEFOCUS, BLUR SNOW
PENCIL.SKETCH CARTOON
TYPOGRAPHIC
RIGHT ROTATE
MULTI MAME
PATCHLAND RESHUFFLE BECGLLANDRREEHFFLLLE
EMBROIDERY
MULTI_DIFFERENT
CROPLAND ZOOM
HIGHL BRGGIBRIGISSES
PLUSHIE
TIGER STRIPES TIGER.STRIPSS
VERTICAL FLIP
LOW BRIGHTNESS
GRAFFITI
MIXUP
HIGH CONTRAST
TATTOO
CUTMIX
LOW_CONTRAST
SCULPTURE
JPEG_COMPRESS
ORIGAMI
A.2 ANIMALS USED IN OUR DATASETS
Our justification for selecting animal classes as the semantic concepts for our dataset was two-fold: (1) they are easy semantic concepts for most VLMs to identify, and (2) we can get a diverse set of animals ranging from mammals birds such that dataset is not too restricted to particular class. The 10 animal classes we used for creating our datasets are: DOG, EAGLE, ELEPHANT, HORSE, LION, OWT,, PANDA, PARROT, PENGUIN and TURTLE.
17

---

Published as a conference paper at ICLR 2024
A.3 SYNTHETICTYPELDEE CONSTRUCTION
For constructing the dataset, we first need set of 50 reference-images. For each of the 10 animal classes, we prompted ChatGPT for a of 5 different prompt descriptions use text-prompts for feeding into a text-to-image diffusion model. The prompt descriptions we used are:
1. DOG:
"A captivating, photorealistic 8k image of a (dog:1.3) relaxing in front of cozy fire- place in a log cabin during a snowy evening."
"A stunningly detailed 8k image of (dog:1.3) hiking through mountain range sunny day."
"A stunning, photorealistic 8k image of a (dog:1.3) sleeping peacefully under tree in lush garden."
"A natural and realistic 8k image of a (dog: lying on a carpet in front of a roaring fireplace cozy cabin."
"A breathtaking, high-resolution 8k image of a (.og:1.3) peacefully sitting at a camp- site in front of roaring bonfire under starry sky."
2. EAGLE:
"An awe-inspiring 8k image of an eagle perched rocky ledge overlooking vast desert landscape."
"A stunning, high-resolution 8k image of eagle soaring majestically over a snow- capped mountain range."
"An exquisite, high-quality 8k image of an eagle soaring in a clear blue sky over mountain range."
"A captivating, photo-realistic 8k image an eagle perched on a tree branch over- looking a stunning waterfall."
"An ultra- ultra-high-re high-resolution 8k image of an cagle perched on a branch, watching atten- tively for prey in dense forest."
3. ELEPHANT:
"A vivid image of an elephant carrying logs in the midst of busy local village."
"An 8k image of an elephant enjoying mud bath the middle of the serene savan- nah."
"A stunningly detailed, high-resolution image of (elephant:1.3) taking refreshing dip in cool, clear river."
"An oissoutogg image of an elephant crossing river with tall grass swaying in the background."
"A detailed, high-quality 8k image of (elephant:1.3) trumpeting loudly, while its herd is crossing a river flowing through beautiful and serene landscape."
4. HORSE:
"A vibrant 8k image of a (horse: galloping through a shallow lake with a mountain range in the background."
"An enchanting 8k image a majestic horse standing under blossoming cherry tree with tranquil pond and distant mountains in the background."
"An ultra-realistic 8k image of majestic black horse grazing gently in a lush green meadow surrounded by a dense forest."
"A high-quality, photorealistic 8k image of (horse: galloping freely deserted beach at sunset."
"A beautiful, realistic 8k image of a thorse:1.3) grazing in a green meadow with moun- tains the background."
5. LION:
"A majestic 8k image of (lion:1.3) standing tall on a rocky outcrop overlooking a vast desert landscape."
18

---

Published as a conference paper at ICLR 2024
"An impressive, photorealistic 8k image of (lion:1.3), gazing majestically into the camera against a backdrop of pale blue sky."
"A captivating. high-resolution 8k image of a (lion:1.3) resting on a rock in front of scenic waterfall."
"A stunning, photorealistic 8k image of (lion:1.3) relaxing in the shade of a tree on a hot summer day.."
"A dramatic 8k image of a (lion:1.33 1.3) roaring fiercely with backdrop of stormy sky.."
6. owL:
"A stunning, photo-realistic 8k image of an owl (species TBD) swooping down to catch its prey during the night in forest.,"
"An incredibly lifelike 8k image of (great horned owl:1.3) perched on tree branch dense fog.."
"A beautiful and naturalistic 8k image of a (screech owl:1.3) nesting in a tree hole in forest.,"
"A captivating 8k image of owl staring intently into the camera dimly lit forest."
"An intricately detailed 8k image of an owl perched on a barren, windswept cliff in stormy sky."
7. PANDA
"An dutra-high-resolution 8k image of (panda:1.3) lazily lying on tree branch, basking in the sun's rays."
"An impressive, high-qualin, 8k image of a panda (with panda colors: 1.3) standing in front of gorgeous waterfall in a tropical jungle."
"An ultra-realistic, photorealistic 8k image of a panda (1.3) playing with a ball
grassy meadow under sunny sky."
photorealistic 8k image of panda (1.3) standing tall on its hind legs and reaching for juicy bunch of bamboo shoots."
"An attention-grabbit
"A picturesque 8k image of a panda (1.0) stargazing while sitting on a grassy hilltop rural valley."
8. PARROT:
"An alsseltigh- 8k image of (parrot:
1 1.3) enjoying some refreshing mist on hot day near a waterfall in mountainous forest."
"A realistic 8k image of a (parrot:1.3) perched on a tree branch, with a lush jungle background."
"An enchanting and realistic 8k image of parrot taking refreshing bird bath in
small stream, surrounded by wildflowers."
"A breathtaking 8k image of a (parrot:1.3) perched on wire fence with a stunning mountain range in the background."
"A realistic 8k image of (parrot:1.3) playing with ropes and swings in well- furnished living room."
9. PENGUIN
"A detailed, high-quality 8k image of penguin waddling along rocky beach with waves crashing in the background."
"A stunning, high-resolution image of (penguin:1.3) waddling on a snowy beach, with snow-capped mountains in the background."
captivating, photorealistic 8k image (penguin:1.3) glancing over its shoulder, watching for predators amidst the Antarctic wilderness."
"A breathtaking 8k image of (penguin:1.3 standing on rocky outcrop, overlooking the vast ocean with icebergs the backdrop."
"An exquisite, high-quality 8k image of (penguin:1.3) waddling on a snowy moun- tain top against clear sky background."
10. TURTL.F
"A stunning, high-quality 8k image of a (turtle:1.3) basking in the sun on a sandy beach."
19

---

Published as a conference paper at ICLR 2024
"An impressive, photorealistic 8k image of (turtle:1.3) climbing up steep moun- tain."
"A beautiful, photorealistic 8k image of a (turtle:1.3) crawling through lush green foliage in tropical rainforest."
"An impressive, photorealistic 8k image of a (turtle:1.3) crawling on a log in murky swamp."
"A stunning, high-quality 8k image of (turtle:1.3) basking in the sun on a rocky beach."
We then used the Kandinsky-2.1 text-to-image diffusion model generate 50 synthetic reference- images (100 diffusion steps, guidance-scale (Ho Salimans, 2022), 768x768 output resolution) using the prompts specified above. We transformed these reference-images into our final data- type images by leveraging pointwise transformation functions (for pixel and geometric data-types) or re-generating them using Kandinsky2.1 with the same random-seed but with a slight modification of the prompt to capture the specific data-type information. This results in 1,350 total evaluation describe the specific functions and the prompt modifications applied for transforming the reference-images to data-type images across the four coarse-grained data-type categories below:
samples across 27 data-types.
A. Geometrie
LEFT ROTATE:
We rotate every reference-image the left (90%).
RIGHT ROTATE:
We rotate every pesrrence-image to the right (2709).
PATCH AND RESHUFFLE: We patchify the image into a 5 x 5 grid, and then randomly shuffle them spatially.
CROP AND ZOOM: We use a zoom-factor of 2 for randomly zooming into a reference- image.
VERTICAL.FLIP. We flip every reference-image vertically.
We mix every reference-image with another random reference-image with a mixing coefficient of 0.35.
MIXUP:
On every reference-image, we paste randomly cropped patch from another random reference-image.
CUTMIX:
B. Pixel
GAUSSIAN NOISE: To every reference-image, we add gaussian-noise such that its pixel-variance increased 1.4x.
DEFOCUS_BLI _BLU:: We iteratively blur the reference-image with a disk-shaped filter un- til it reaches target blur level of 1.4x the estimated initial blurriness level (estimated using normalized maximum laplacian variance).
SNOW: We apply a randomized snow effect to the reference-image. by first generating a snow layer, followed by motion blur, and then overlaying back on the original image.
HIGH BRIGHTNESS: We increase the brightness of each reference-image multiplica- tively by 1.5x in the HSV colour space.
LOW BRIGHTNESS: We reduce the brightness each reference-image multiplica- tively by 1.5x in the HSV colour space.
HIGH.CCNTRAST: We increase the contrast of each reference-image by scaling the mean-normalized pixel histograms 1.5x.
LOW.CONTRAST: We decrease the contrast of each reference-image by scaling the mean-normalized pixel histograms 0.5x.
IPEG.COMPRESS: We iteratively jpeg-compress the reference-image until its peak- signal-to-noke ratio is 26.
C. Style
SENCIL.SKETCH: We regenerate images using the eeference-prompee and replacing "high-resolution image/8k image" with "hand-drawn sketch/peneil drawing/black- and-white doodle" in them, followed by manual curation. The end data-type images look like pencil sketches of animals.
20

---

Published as a conference paper at ICLR 2024
We regenerate images using the reference-prompe and replacing "high- resolution image/8k image" with "cartoon drawing/cartoonn/hiltert cartoon" in them, followed by manual curation. The end data-type images look like cartoon ani- mals.
CARTOON:
EMBROIDERY: We regenerate images using the eefrence-prompee and replacing "high-resolution image/8k image" with "knitted embroidered pattern/knitted embroi- derylembroidered shawl" in them, followed by manual curation. The end data-type images look like embroideries of animals.
PLUSHIE: We regenerate images using the reference-prompts and replacing "high- resolution image/8k image" with "plushie" in them, followed by manual curation. The end data-type images look like plushie toys of animals.
GRAFFITI: We regenerate images using the reference-prompee and replacing "high- resolution image/8k image" with "spray-painted graffiti/graffiti art/modern-style graf- fiti" in them, followed by manual curation. The end data-type images look like graffitis of animals.
Wegenerr
TATTOO: We regenerate images using the reference- prommpts and replacing "high- resolution image/8k image" with "body tattoo/hand tattoo/leg tattoo" in them, fol- lowed by manual curation. The end data-type images look like tattoos of animals.
SCULPTURE: We regenerate images using the reference-prompe and replacing "high- resolution image/8k image" with "marble/stone/bronze sculpture/statue" in them, fol- lowed by manual curation. The end data-type images look like animal sculptures.
ORIGAMT: We regenerate images using the reference-prompts and replacing "high- resolution image/8k image" with "origami/origami toy/origami drawing" in them, fol- lowed by manual curation. The end data-type images look like origami animals.
D. Semantic
TYPOGRAPHIC: We paste random text a random position on top of every reference-image
MULTI SAME: We regenerate images using the reference-promptt and replacing "a {animal]" with "pack/hted/grouppteam of [animals]" " in them, followed by manual curation. The end data-type images contain multiple animals instead of single one.
MULTI MULTI-DIFFE DIFFERENT: We use GLIGEN (Li al., 2023d) to in-paint tiger into each reference-image. The end data-type images contain two animals, a tiger and the orig- inal animal in the reference-image.
TIGER_STRIPES, We regenerate images using the reference- proomctte and add one of these prompts to the them: ["with the distinctive stripes of a (tiger:0.9) on its body", "displaying the unique stripes of a (tiger:0.9) on its face and limbs", "with the eye- catching stripes of (tiger:0.9) on its body". "bearing the prominent stripes of (tiger:0.9) on face and limbs", "having the stunning and distinctive stripes of (liger:0.9) on its body", "with skin containing the characteristic stripes of a (tiger:0.9) on its face and limbs"), followed by manual curation. The end data-type images con- tain animals with tiger-striped skin features on them.
A.4 NATURALT CONSTRUCTION
Here, we manually curated 50 reference-images from the KaggleAnimallmagee dataset (Baner- jee, 2023) across the same animal categories before. For creating images for all data-types in the geometric and pixel categories, and the TYPOGRAPHIC data-type in the semantic category, we applied the same transformation functions detailed previously on the curated reference-images. For MULTI DIFFERENT, we manually curated images from KagglcAnimallmages where there was more than single animal the image. For sourcing data-type images for the style categories, we followed a 3-step procedure: (1) we first searched google images with data-type specific animal- prompts, e.g., "a cartoon dog". "a graffiti
lion" etc. We applied a time-filter on the search to only retrieve images post January 2022, to ensure that the retrieved images are not contained in LAION- 2B-en, CLIP's pre- trainininng dataset, (2) we manually curated 100 images per data-type across ani- mals ensuring data quality and diversity, and (3) de-duplicated our final set of 100 images per data-type against LAION-2B-en with an image indexed-searee in LAION-5B and the PHash algo- rithm. This procedure resulted in curated of 50 images for all the style data-types. We how-
21

---

Published as a conference paper at ICLR 2024
were unable to find effective method for sourcing data-type images for and TIGER SIGER.STRIPES, hence our Natura/Typeldent dataset only contains images from data-types (1,250 samples).
B DETAILS ABOUT MODEL EVALUATION
We first enlist all the tested models and then describe the individual evaluation strategies used for C-VLMs and LMMs.
B.1 MODELS TESTED
A. C-VLMs
CLIP. We tested 9 models in total using the OpenCLIP repository: LeIP-ResNNt50- openai, LuP-ReeNNt101-openee
CLIP-ViT---33-oppena, CLIP-ViT-D-32-laion2b
CLIP-ViT-B-I6-2D, CLIP-ViT-L-14-2b, CLIP-ViT-H-14-2b, -LI--v and CLIP-ViT-bigG-14422.
CoCa. We tested 2 models using the repsttrPrrpooitory
repository: CoCa-
and CoCa-ViT-L--le ViT-L-14--2.
BLIP-2-ITM. We tested 2 imnge_.tex.maatchnggg models using the LAVIS repository: pretrain and pretrain.
B. LMMs
Fromage.
We used the open-source implementation provided by the authors. Mult We used the open-source implementation provided by the authors.
OpenFlamingo. We used the OpenFllaminoo-9B-IH model released with the Otter
repository.
Otter: We tested models using the Otter repository: OTTER-Imag--LLaMAZBLLA InContext and MitiaamOOOEER-mmaaMPPPPP
GILL. We used the open-source implementation provided by the authors.
MPlugOwl. We tested 4 models using the MPlugOwl open-source implementation:
mplug-oowl-lamaa-7-pp mplug-ooll-llmaa-7. mplug-owi-llamaa-7bbf and mplug-
owl-bloomz-7bb -a-uumm----muuulluguuud
LLaVA. We tested models using the LLaVA open-source implementation: LLaVA- 7B-v0, LLaVA-13B-v0 and LLa LLL.AALggninngMPP--BB-----e
BLIP-2LLIM. We tested 3 "blip2.15" models: pretrain_flantSx: netrrinnflanttxxl and fittrainn and 2 "blip2.opt" models: pretrain.opt2.77 7b and pre- train opt6.77 7b, using the LAVIS repository.
InstructBLIP. We tested 2 "blip2.15 -__ntrrutt models: flant5xl flant5xxl, and models: vicuna7b and vicunal3b. using the LAVIS repository. IDEFICS. We tested models using the open-source huggingface release by the au- thors: idefics-9b, idefics-9h fifess-bb-instucee instrutt idefics-80b and Seefics-80b-intrun
"blip2..iiuma
B.2 DATA-TYPE TEXT-PROMPTS USED FOR EVALUATION
For evaluating C- C-VLMs by cosine-similarity scoring, and LMMs by log-likelihood scoring, we used default, fixed set of 27 data-type text prompts detailed below:
GAUSSIAN NOISE: "This is noisy image of an animal."
DEFOCUS RBLOC "This is a blurred image of an animal."
SNOW: "This is a snowy image of an animal."
HIGH BRIGHTNESS: "This a bright image of an animal."
LOW BRIGHTNESS:: "This is dark image an animal."
HIGH.CONTRAST: "This is high contrast image of an animal."
LOW CONTRAST: "This is low contrast image of animal."
22

---

Published as a conference paper at ICLR 2024
JPEG.COMPRESS: "This is a jpeg compressed image of animal."
JOOTAE "This is a left-rotated image of an animal."
RIGHT ROTATTE "This is right-rotated image of an animal."
PATCHLAND REICHANNDRESHUFFLLL. "This a patched and reshuffled image an animal."
CROP AND 200M: "This is randomly cropped and zoomed image of an animal."
VERTICAI._F.II: "This is vertically flipped image of animal."
MIXUP: "This is image of animal mixed with another image."
CUTMIX: "This is an image of an animal with one patch replaced by another image."
PENCILSKKETCH: "This is pencil sketch of an animal."
CARTOON: "This is a cartoon animal."
EMBROIDERY: "This is an embroidered animal."
PLUSIIE: "This plushie animal."
GRAFFITI: "This is graffiti of an animal."
TATTOO: "This is tattoo of an animal."
SCULPTURE: "This is sculpture of an animal."
ORIGAMI: "This is an origami animal."
TYPOGRAPHIC "This an image an animal with some text written on top."
MULTI_SAME: "This an image of multiple animals."
MULTIEDIFFERET: "This is image of tiger and an animal."
: "This is an image of an animal with tiger stripes."
B.3 DATA-TYPE PROMPT-VARIATION FOR TESTING EFFECTS OF DIFFERENT PROMPTS ON RESULTS
To further test how much of effect different prompting styles have results, two alternate prompt variations for testing our models too. We enlist the alternate sets of prompts used here. The first prompt variation list is:
MAUSSIAN_NOISE. "This is grainy image of an animal."
DEFOCUS.BLUR, "This is a blurry image of an animal."
SNOW: "This is an image of an animal the snow."
IIGGILBRIGHTTESS: "This image an animal is bright."
LOW BRIGHTNESS: "This image of animal dim."
HIGH_CONTRAST: "The contrast in this animal image is high."
LOW.CONTRAST: "The contrast in this animal image is low."
JPEG COMPRESS: "This is an image of an animal in jpeg format."
LEFT ROTATE: "This is a counter-clockwise rotated image an animal."
RIGHT_ROTANE "This is clockwise rotated image of an animal."
BATCHLAND.RESHFFIE "This scrambled image of an animal."
CROP_AND DNP.AAND.ZOOM "This image close-up crop of animal."
VERTICAL.FLIP: "This is an upside-down image of animal."
MIXUP: "This is mixed image of two animals."
CUTMIX: "This is an image of an animal with patch swapped from another image."
PENCIL.SKETCH: "This is hand-draw sketch of an animal."
CARTOON: "This is an animated animal."
EMBROIDERY: "This looks like animal embroidery."
23

---

Published as a conference paper at ICLR 2024
PLUSHIE: "This is stuffed toy animal."
GRAFFITI: "This graffiti art an animal."
TATTOO: "This is an inked image of an animal skin."
SCULPTURE: "This is a statue of an animal."
ORIGAMI: "This is paper-folded (origami) animal."
TYPOGRAPHIC: "This is an image of an animal with some text overlaid."
MULTI-SAME "This image shows several of the same animals."
MULTIEDIFFERENT: "This image shows both tiger and another type of animal."
TIGER STRIPES: "This animal seems to have the pattern of tiger stripes."
The second prompt variation list is:
GAUSSIAN_NOISE "This an image of an animal with some graininess."
DEFOCUS BLUR: "This is out-of-focus image of animal."
SNOW: "This is an image of an animal seen amidst a snowy backdrop."
IIGII BRIGHTTEE:::::: "This an image of an animal, the image is bright."
LOW BRIGHHTNESSSS: "This is an image of an animal, the image is dark."
HIGH CONTRAST:: "This is an image of animal, the image has high contrast."
LOW "This is an image of an animal, the image has low contrast."
JPEG COMPRESS: "This is a compressed image an animal."
LEFT_ROTATE: "This is an image of an animal, the image is rotated 90 degrees to the left."
RIGIT ROTATE: "This is an image of an animal, the image is rotated 90 degrees to the right."
BEHHLANDDRRESUFFLE
"This a jumbled image of an animal."
CROP ZOOM: "This is an image of an animal, the image is cropped and resized."
VERTICAL.FLIP: "This an image an animal, the image is flipped upside-down."
MIXUP: "This is image of animal with mixup augmentation."
CUTMIX: "This is an image of an animal with cutmix augmentation."
PENCIL.SKETCI: "This is sketch of an animal."
CARTOON: "This is a rendition of an animal in a cartoon style."
EMBROIDERY: "This is an animal embroidery piece."
PLUSHIE: "This is soft toy animal."
GRAFFITI: "This an image of animal street art."
TATTOO: "This is an image of an animal in tattoo style."
SCULPTURE: "This is an image of an animal, the image looks like a sculpture."
ORIGAMI: "This is an image of an animal crafted out of paper folding (origami)."
TYPOGRAPHIC: "This is an image an animal, the image has some text superimposed."
MULTI SAME:
MAME: "This is an image of multiple same animals."
MULTI DIFFERENT: "This is an image of an animal and tiger."
TIGER STRIPES: "This is an image of an animal patterned with tiger-like markings."
24

---

Published as a conference paper at ICLR 2024
B.4 DETAILS LOG-LIKELIHOOD SCORING FOR LMMS
To compute the zero-shot performance of C-VLMs, we computed the cosine similarity match- ing score of each image with each of the 27 data-type text prompts {D1. ......, D2r3 enumerated in Appendix B.2) and predicted the data type with the highest score, i.c. predicted data type=arg (I..2( cos-simi cros_sim(I.enc(// ennnc(,)). Since LMMs are auto-regressively trained, evaluating them in the same way is impossible. For the most fair comparison of both model classes, we instead compute log- ligeliiiioooos, of the prompt containing the image with the appended data-type text prompts D; to the image. Specifically, for each im- age I. we compute the log-like ligeeikk under the LMM, for the query prompt P i="<images Describe the image. A: <D i>"" where <D i> is replaced by the specific data-type text prompt. More concretely, for particular data-type prompt Pá,, we tokenize the prompt P.i, pass these input tokens into the model, and retrieve the logits for each of the input tokens. Then, sum up the log- prebbbbbbilities of each input token, and use that as the final aggregated bog-robbybylity L.i for the particular data-type. We repeat this process for all data-types, and then predict the data- type with the highest log- pogorbbbbiylity i.e., predi wtt.deddddaaaaaaeeeeeeeeeeeaa a.ype.ay m (I....27) ( L.i. Note that the default query prompt P differs across different models depending on the particular prompts used for instruction tuning, and match our prompting strategy accordingly (sce Appendix B.5).
Q:
This log-likelihood evaluation strategy is common for evaluating LLMs on multiple choice ques- tions (Gao, 2023; Brown et al., 2020; Sanh et al., 2022) and LMMs on classification tasks (Dai al., 2023; Awadalla et al., 2023). Further, as enumerated in (Gao, 2023), we tried different length nor- malisation strategies while computing the log- likeliihoood scores, but observed significant changes to the
we used the standard log-likelihood scoring procedure outlined above.
For concreteness, also showcase the exact code snippets for computing these log-probabilities for three models, LLaVA, IDEFICS and Otter.
The below functions for computing
are defined within model class where we defined modol, tokenizer other auxiliory variables. The itself instance of alavat//amFForrausaal3 defined as:
self.tokenises AutoTokent pram_ prettrained (mode) _naee)
self.model - LlavatilmaaoorCusaaa Iront. proorprretzaane Imodel_namee 1ow_opu Bow_opp_menn_usseeeTuu
sorch_dtype=torch www.sorrcc...off6..... sse_sache=True) .euda()
CLIP imggerrcessorr wtw.errccssoo....ccoo...o.. =orch_dtypostorchh flotettoooo......66.
def Log_lik_scorte (self, images, prompt, option):
11
Compute log-likelihood score model given input image, text prompt, answer option
13
14
15
= prompt
17
if self wse.f.mmuus...m gs qs manga_token_lent DEFAUNT_TM, T___TT__________ TOKEN else:
CI_LL_IXSSAARRRTA MALSTARR_TTHAA DEFAULT BOING_PAACCHTUUAEAE
19
20
CAn" DEFAULT
self, www.m_mooeeeeeeeeeeeeeeeoeeoeoee
21
*vl* self.pretzained.uuk (owe conv mode "llava_v1" elif "mpt" " self.pratrained,1cm
25
25
conv mode *mpt_mmp mp modal
26
else:
27
conv_mode "multimods1"
28
conv poyy_templtes (eeav [coav_mode]] mode].ccoy. [)
30
wow..append_meeeeee message (cenv. rcles ccles[0], ga)
31
conv.append, massage (oonv, roles [1], None)
32
i This is the Zinal promot that
the detault text prompt to probe with.
33
proupt coav.get. conv.get_promp:
34
35
image_tenan self, wimagg_prrossseeemmm proppppaaoo.prrrrooeeessoooooaas (imagen, deturn_tessorss/ge pixel_values
36
37
target prompt prompt ()r, format (option) (on)
25

---

Published as a conference paper at ICLR 2024
39
inputs self. torkniizee ((target_prompt))
40
41
input_ torch. ae_teno (inputs. input_ids)) ccda
42
attention_mask torch. sorch.as_tene te_teesor te attennttteettt mask) cuda
44
with torch, inferenn mode :):
outputs self.model. wor.mel....rrrr....... forword
46
"nppt_ pout_idd-imput__dh
47
Labels=input_ide
48
49
.h)l ()
51
return -ostputs.
53
54
def
(self, image, text_slasses):
56
Rero-shot classify the image with the given text alaas crensformed prompts .*.
58
log_scores
60
61
prompt_to_use 'Describe the image.'
63
for gt_P in text_claases
input_images image
65
input_ppompp - prempt_to_use
model_out Log_lik_sccrer tinput_images, ixput_prcopt, gF_p) log_scores append (model_out)
67
68
pred ip.ergmax(log__coo axis=-1)
70
Listing LLaVA log-likelihood scoring
functions computing log-likel:hooda are defined within model class where we have defined a model, tokenizer and other auxi laxiliary variables. The model itself ia instance of Idaficssooriiiionoeet2eee defined as:
self.model -
pecoo_ppeeraaaedeaiiiae ussegeeeeesseeeeeeeee device_mep- auto') self self.procesace Autorrocessor. promeererzaed (checkpoint)
def log [ig_lie ailkackooeaasaeeee context_inputa prompt, option):
***
11
Compute log likelihood score under the model given an input image, text prompt, and answer option
12
13
14
context_with witht_wtth_ae wtt__ttt__aswwe_iiiuuuue self, processor (prompt I: 11 [prompt I 1] (ormaorma optian)), ], return tetum_ttnssossapph debug=Fs 1sa) to toe (self.ded devvvcce)
context_with_ane witleet_wiihhennsee__nnlulss labela"; context_will wtt_eesteeitttssse___eaee__ee
Input_ida""; colone self.model (*Accrtext_ wnswer imputs) loss. (item()
(self. device)
17
return -loss
18
19
def wwt.eeeeeeeeoooooooeeeeooeeeeeee log_acores =
image, text_classes):
20
22
proapt_to_use 'Describe the image." prompt_ta_use
24
"Maer:",
25
image,
26
"() "n'nnnsssstttt"" (promps_to_use),
27
29
Lmpge_images image
30
context_pronge prompt_to_pae
31
32
context_inpute processooessor (context__poomeeeeeee return__eneors- debug=Faiee)..o
self.device,
33
gt_p text_clesses:
35
model_out = sals. follfloo_lid lia___ik_ooseersoeeeeeeeeooeeeoe (contott_aatt prompt_to_use, gt_p)
log_acores.appent (mode) _ott)
26

---

Published as a conference paper at ICLR 2024
38
pred (gog_axloo__ooroor axiga-1)
return pred
Listing 2: IDEFICS log-likelihood scoring
The belox functions for computing log-likel-hoods are defined within model. class where we defined model, tokenizer and other auxi Hiry variables. The model itself instance of hon.:oorrcontttoooaaeeeeeee
defined aa:
self.model 0 htt.raaatttoottooreeeeeeeaeeaa prroopprerraineeee (PRETRATNED modee prretraeddd device_nap="autoo
salf.tokenizer self.model wtx.ttxx___ccce sexc_tckenizerr
prr.eeeeroooaoor transformera. wtt.mmeeaaa..../aaee....eee wtw.rrre.cttinnggerrooessoo
mel..model.eval1
self mff..model.cuda
11
12
def find_oub list (self, sl,
14
Wiility function find sub-liat given list, used for computing Likellhood of answez tokens only without computing likelihood prompt context
16
18
19
results = (] sll len(sl) for ind in for 1, enumerate(l) If =* (0])
20
21
ir 1tind
sll] results. appendii sil -1
23
return results
25
26
def
vision_x, prompt, opticn, prompt_tokensl:
**n
28
Compute sggllikelhhod score under the model given an Input image, text pronpt, and anawer option
29
31
target_text (prompt) (opcion)"
32
lang_x self model) teex_ttoken zer (etttttgttttt "eturn_tensors=age"")
34
outputs self.medel:
36
vision_x-hone
=ang_x=lang_x[_
("inn___((io___ddd"""" [dd=iooo_dd"") (),
38
iutention_mase waa____skllan__iiilee ("tteet woweennioommmss..oomm
39
clear soaar_ooddiiooee___ lay_aaderaaaaaoolll... bie_cachee__veeion__eee
41
43
probs = torch.softmax poutputs..ooita... dins-1) .detach)) proba probsli, :-1, :]
45
input_ids lang_x (ii_i(("i_t__de""" [t, L:l.ouded
46
gen_proba torch. (probs, 2, input_ida(:,
48
probs -
input_pentence, input_ zip (ingut_ida, pen_probs)
50
idxea salf. find_aub____________ list iprompt__okeme cerpu__senennaddeee () cppu n1mpy
;olis:!1)
51
input_probe paput_prcbs [iexes (torch (nppt_probb)) .item
52
54
return probs[0]
56
def wot._ppreectoooooofooooooooo model fself, Image, text_olasses): Log_acores - []
57
58
59
prompt_to_use *cimage> C: Describe the image, A:*
61
imput_images [image]
63
media_token_id self.tokenizer (erliierrr(riiin ("<inage>", Bdd_speciall_oken--taul "input_ids"]
visicn_x wwwpocesoor.rppppoeccoe (input_images return_ return_tensors= "pixel_valuse
66
self image wwwiiimaaaeeeeeeeeeeeoo
1. u.unquuezee (1) unsqueeze
67
pre-oache features for currert sampla here
27

---

Published as a conference paper at ICLR 2024
self.model._encode htt...o.l....ooood....eeotttttd.
70
71
prompt_toxen
72
self. (prempt_to_use, edi_special teturs"lensors="mpp") |* imput_ids ravell ttllist
74
75
gt_ text_ text_claases:
76
model _out oout sel "all.log_ 1k soorrss prompt_to_use, st_P, prompt_tokers) 1og_scores appeodd (model__ttt
77
78
79
pred np ap.ergmax (logaaesoo_ axism-l)
return
Listing 3: Otter log-likelihood scoring
B.5 LMM-SPECIRIC EVALUATION DETAILS
By default, for fair comparison with C-VLMss evaluate LMMs using log- likelihood scoring with the prompt: "<image> Q: Describe the image. A: where bata.type.descripiie.
substituted with each of the prompts from the section above. However, some LMMs have specific prompting styles, which we incorporate ensure correct evaluation, follows:
Mul.
<BOS> Bellw is an instruction that describes a task. Write response that appro- priately completes the request.
### Image: <image>
### Instruction: Describe the image. ### Response:
MPlugOwl.
The following is a conversation between a curious human and Al assistant. The assistant gives helpful, detailed, and polite answers the user's questions.
Human: <image>
Human: Describe the image. AI:
L.LVA-7/13BB-0
You are LLaVA, a large language and vision assistant trained by UW Madison WAIV Lab. You are able understand the visual content that the user provides, and assist the user with variety of tasks using natural language. Follow the instructions carefully and explain your answers in detail.
##HHuma:: Hi!
###Assistamt: Hi there! How can I help you today?
###Human: Describe the image. iiti-iiiissttttt. stanssttttttmmppm
##sssiittnt:
LLVVA-MPT-7B
28

---

Published as a conference paper at ICLR 2024
Prompt variation
Averaged-mean informedness across models
Default
0.228
Alternate
0.191
Alternate
0.197
Table 4: The default prompts used obtain the best average performance across models.
im start system:
You are VL, a large language and vision assistant trained by UW Madison WAIV Lab.
You are able understand the visual content that the user provides, and assist the user with variety of tasks using natural language.
You should follow the instructions carefully and explain your answers in detail.
im_end-3
im sol.start->
user: Describe the image.
<-im start- im patch->< im end->
assistant:
C PROMPT VARIATION EXPERIMENTS
For our main zero-shot probing results in Sec. 4.2, used set of default, fixed prompts for each data-type (as listed in Appendix B.2). To further test if our results are sensitive to variations in different prompting styles, also prompt variation experiments two alternative sets of data-type prompts (as listed in Appendix B.3). In Fig. 7, we showcase the results with the three different prompting styles-we observe largely the same qualitative trends across the three different prompting styles, with the C-VLMs still outperforning LMMs. We do however note that the absolute averaged mean-informedness values across models is quite different (see Tab. 4) suggesting that different prompting styles do make slight difference for the absolute mean informedness values, however the overall results (C-VLMs outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) remain the same.
subset of our models with
Results with default prompts
Results with prompt variation
Results with prompt variation
Mean Informedness
Moan Informednese
Moan Informednese
G.
0.4
0,2
0.0
0.01
0.0
18
Model Scale
Model
Model Scale
Imilitons prrmmn))
Crillions paramsi
Cmillions of paramsi
Figure 7: Prompt sensitivity for zero-shot results. The overall qualitative trends remain the same even with different prompt variations for testing the zero-shot models, for both C-VLMs and LMMs.
Despite showcasing results with three different prompt variations, still inconclusive we are using the optimal set of prompts for probing the VLMs. To further extend this analysis, the method from VisDesc (Menon Vondrick, 2022) to generate a large-set of descriptors per data-type by prompting GPT-3 (Brown al., 2020) with the following:
29

---

Published as a conference paper at ICLR 2024
Q: What useful visual features for distinguishing lemur in photo?
A: There are several useful visual features to tell there is lemur in photo:
four-limbed primate
black, grey, white, brown, or red-brown wet and hairless nose with curved nostrils
long tail
large eyes
furry bodies
clawed hands and feet
Q: What are useful visual features for distinguishing (category name) in a photo?
A: There several useful visual features to tell there is (category name) in photo:
where (categozy-name) replaced by a description of each of our data-types.
We then use these several descriptor prompts for our zero-shot analysis. For probing C-VLMs, we directly average the text embeddings corresponding and compute similarities with the test image (exactly the method used in VisDesc). Similarly, for LMMs, we compute the log likelihoodss of each of the text descriptors per-data-type and average them. We then take the data-type with the maximum averaged log- log-likelihood as the prediction for that test sample'.
the different text descriptors per-data-type,
Results with visual descriptions
Mean Informedness
0.8
CLIP HLIF2-IM
LLaVA-11.5 InstructSLIP
C-VLMs
Otter
0.6
0.4
0.2
0.0
102
103
104
Model Scale
(millions of params)
Figure 8: Results with visual descriptor prompts from VisDesc
Fig. 8 shows our zero-shot results with these enhanced descriptor prompts. With this probing method, we again do not see any significant deviations in trends from the default probing prompts we used. We therefore conclude that despite different prompts having some effect on the final results, our language- zero-shot probing results are fairly robust across prompting methods.
D ANIMAL IDENTIFICATION EXPERIMENTS
To ensure that the results we see in Fig. 3 are indeed informative and not an artefact of incorrect evaluation, we run control experiment where we tested models on identifying animals Synther- icTypeldent. Due to time-constraints, we omit the IDEFICS models from this analysis. In Fig. 9,
This log-likelihood averaging has also been used previously for evaluating OpenFlamingo
30

---

Published as a conference paper at ICLR 2024
plot the mean informedness across the 10 animal classes, for all models. Compared to main results on data-type identification ( Fig. 3), all models, including the LMMs, are extremely good at identifying the animal in the images. This suggests that VLMs are more adept at discriminating the semantic concepts within an image over identifying data-types. This result also connects with our emhedding space analysis in Fig. 5.
1.0
Mean Informednese
0.8
0.6
0.4
0.2
OpenFlamingo
BLIPZ-LLM
CoCa
Octar
TrstructBLIP
BLIP2-FTY ITM
GILL
CVLMs
0.0
Promage
LMMs
MM-GPT
LLaVA
chance
-0.2
102
103
104
Model Scale
(millions of params)
Figure 9: Performance in identifying animals on SyntheticTyppeldeis
E DISCUSSION ON DATA- DATA-AUGGENTTTION AND CONNECTIONS TO DATA-TYPE IDENTIFICATIO
Data-augmentation is standard technique used for improving generalisation (Raileanu al., 2021; Mikolajczyk & Grochowski, 2018; Perez& & Wang, 2017). reducing overfitting (Taylor & Nitschke, 2018; Shorten & Khoshgoftaar, 2019), and enhancing robustness distribution shifts (Rebulfi et al., 2021b; Zhong et al., 2020; Zhao et al., 2020; Rebuffi et al., 2021a), in deep neural net- works. Almost all the ImageNet-trained CNNs of the last decade (Krizhevsky al., 2012; Si- monyan & Zisserman, 2014; Zeiler & Fergus, 2014; He et al., 2016; Szegedy et al., 2015) have used some form data-augmentation for improving their generalisation performance. Further, the fields of self-supervised (Jaiswal et al., 2020) and semi-supervised (Van Engelen & Hoos, 2020) vi- sual representation learning almost solely rely on data-augmentation techniques for capturing their supervision signals. A prime example, SimCLR (Chen et al., 2020), relies using simple data- augmentation techniques like flipping, rotating or cropping for procuring its positive image samples for the instance-basee contrastive loss. Therefore, these networks rely makes them insensitive to capturing data-type information. Hence, we can expect them to underper- form in identifying data-types since the information is never captured in the embeddings by account of this invariance prior. However, we note that CLIP does not use any data-augmentation while training other than random-resized cropping. Similarly, BLIP-2 and CoCa only use random resized cropping and horizontal flipping (not a part of our data-types) as augmentations. Therefore, this suggests that despite models not being explicitly subjected learning invariances, they learn to be invariant to understanding data-types (as we show in Fig. 5). We further showed in Sec. 5 that this emergent insensitivity to data-types can traced back to the lack of data-type rich pre-training data. Therefore, current models- explicitly or implicitly-do not capture information required for distinguishing data-types, and we make a strong case why this important for the worldworld. Through data-rype identification framework, call for rethinking the current prac- tices of training VLMs that induce invariances them, and show why it is an important practical problem.
invariance prior that
31

---

Published as a conference paper at ICLR 2024
Animal-class
Data-type
0.96
0.29
01.96
0.30
10
0.94
0.33
50
0.97
0.40
Table 5: K-NN results on SyntheticTypeldeee We present the mean informedness of k-nearest- neighbours classifiers on the SyntheticTypp Typeldentte dataset for predicting the animal-class or the data- type for each of our samples based on CLIP's image-feature embeddings.
F QUANTIFYING LINEAR SEPARABILITY OF ANIMAL- AND DATA-TYPES IN CLIP'S IMAGE-SPACE
To quantify how invariant CLIP-RN5O's vision encoder is to identifying animal-classes and data- types (see Fig. 5), we run k-nearest-neighhours experiments on the CLIP--NNO image embeddings as features and animal-classes / data-types as labels. While the KNN classifier performed quite well on identifying the animal in the image, performance significantly dropped in identifying the data-type (see Tab. 5). Similarly, linear probe (multinomial logistic regression with LBFGS opti- mizer akin to what was used for CLIP-RN5O's original linear probing results (Radford et al., 2021)) showed high linear separability of animal classes (mean informedness=0.99, whereas data-types were less linearly separable (mean informedness=0..4)) These values were obtained on the train set only, we want to study linear separability of the data points and generalisation. The number of 1,350 data-points in our dataset is similar to the dimensionality of CLIP-RN50's feature space (1,024 dimensions), which might in part account for the higher per- formance of the logistic regression classifier, confounding our results slightly. Overall, the worse performance of these classifiers when applied on data-types compared to animal-types suggest CLIP is somewhat invariant to data-types compared to identifying animal classes.
interested in test
G ANALYSING THE PRE-TRAINING DISTRIBUTION
For conducting the analysis in Sec. 5, we first searched the entire LAION-2B-en text index for consider a hit a keyword (or its associated term) is verbatim present in the text caption of the candidate sample's text caption. We here enumerate the different search keywords we used for this analysis for all data-types. The vertical bar ("l") denotes a logical OR in regex syntax, indicating any of the keywrods separated by "I" can be matched.
keyword-hits for each data-type.
GAUSSIAN NOISE: "noisy"
DEFOCUS.BLUR: "blurred"
SNOW: "snowy"
HIGHLBRIGHTNESS: "bright"
LOW BRIGHTNESS: "dark"
HIGILCONTRAST: "high contrast"
LOW CONTRAST: "low contrast"
JPEG.COMPRESS: "jpeglipeg compreseddjeeg-coppssset""
LEFT_ROTATE: "left-rotatedllet rotated"
RIGHT ROTATE: "right-rotated rotated"
PATCH_AND.RESHFFIT "patched and reshuffled patched & reshuffled paached
CROP.AND.ZOOM: "croppee and zoomed cropped & zoomed"
VERTICAI.FLIIP "vertically flipped flipped"
MIXUP: "mixed with|mixup"
CUTMIX: "patch replaced|cutmix"
32

---

Published as a conference paper at ICLR 2024
PENCIL.SKETCH:
"pencil sketch"
CARTOON: "cartoon"
EMBROIDERY: "embroidered"
PLUSHIE: "plushic"
GRAFFITI: "graffiti"
TATTOO: "tattoo"
SCULPTURE: "sculpture"
ORIGAMI: "origami"
TYPOGRAPHIC: "text written"
MULT "multiple"
DIFFEENI "tiger and a"
VIGER_STRIPER "with tiger stripes"
We also provide the raw values of (1) the text- frequency in LAION-2B-en (2) the alignment proba- bility, and (3) the abundancy score, obtained for each data-type in Tab. 6. Note that since we compute the Spearman rank correlations between the informedness values per data-type and the abundancy score, the scales of the scores do not matter. We further provide the complete set rank-correlations obtained between the abundancy scores and the performance (mean informedness) across all models as well as only across CL.IP models, in Tab. 7. Finally, in Fig. 10 we provide qualitative examples of retrieved samples for different data-types, sorted by average model performance that particu- lar data-type (from Fig. 4) i.e., average perforapee decreases from left to right-a a careful visual inspection gives us a sense of the strong association we from the correlation scores-the more performant particular data-type is, the more aligned the retrieved samples are to the data-type concept.
33

---

Published as a conference paper at ICLR 2024
Table 6: Raw text-frequency, alignment probability and abundancy scores data-types.
Category
Data-Type
Text-frequency
Alignment probability
Abundancy score
LEFT ROTATE
8.62 10-9
0.294
2.54 10-a
ROTATE
1.16 x 10
0.375
4.36 x 10~9
PATCHLAND PAICHAAND.EEHUFFLE RESHUFFLLE
5.04 10
0.040
2.04 10
Geometric
CROPLAND_ZOON
4.31 x 10-9
1.000
4.31 x 10
VERTICAL FLIP
1.95 x 10-5
0.141
2.75 10
MIXUP
2.98 10
0.020
6.02 x 10
CUTMIX
1.59 x 10
0.286
4.56 x 10
GAUSSIAN NOISE
3.01 x 10
0.131
3.95 x 10 10-6
DEFOCUS.BLUR
2.37 10
0.810
1.92 x 10
SNOW
2.73 x 10
0.677
1.85 x 10
Pixel
BRIGHHTIG
2.36 10
0.649
1.53 10
LOW BRIGHTNESS
1.68 x 10
0.310
1.45 x 10
HIGH. CONTRAST
1.18 10
0.570
6.72 10 10
LOW_CONTRAST
1.77 x 10
0.454
8.05 x 10
JPEG_COMPRESS
1.87 x 10
0.828
1.55 x 10
PENCIL.SKETCH
1.85 x 10
0.950
1.76 10
CARTOON
2.45 x 10
0.622
1.53 x 10
EMBROIDERY
1.97 x 10
0.697
1.37 x 10
Style
PLUSHIE
10
0.750
2.30 X 10
GRAFFITI
2.50 x 10
0.617
1.54 X 10
TATTOO
1.43 10
0.515
7.38 10
SCULPTURE
6.64 x 10
0.745
4.95 x 10
ORIGAMI
2.68 x 10
0.727
1.95 10
TYPOGRAPHIE
3.33 10-6
0.030
1.01 10
Semantic
MULTI SAME
7.03 x 10
0.490
3.44 x 10
MULFIDIIFEREET
7.66 10
0.245
1.88 10-7
TIGFR_STRIPES
1.29 x 10
0.823
1.07 X 10
Table 7: Abundancy-scores are strongly correlated with informedness.
Dataset
All models, r=
CLIP models, y=
SyntheticTypelden
0.557
0.606
NaturalTypeldent
0.489
0.415
34

---

Published as conference paper at ICLR 2024
35
Figure 10: Sample images per data-type from the retrieved set of correct data-type examples.

---

Published as a conference paper at ICLR 2024
G.1 REPLACING THE BASED SEARCH WITH SEMANTIC-SMMILARTT SEARCH
For the main correlation analysis, we used simple text-based search in the pre- pre-traniin distri- bution's ' text captions. However, using semantic similarity metrics instead of simple exact text- matching metric might help in creating a more comprehensive vocabulary for each data type (e.g., BAICHLAND.RESHUFFR can also be represented by terms like collage, mosaic etc).
To probe this hypothesis, we used a semantic search based on the sentence wwwrennce-raastteee connsformers//o Mall-Minine L6-v2 text embedding model. We make use of this model as is both wigght and has been optimised to work effectively for sentence similarity matching in large corpora (Reimers & Gurevych, 2019). Specifically, we encode the text model to get their embeddings. We also encode all the text captions in the LAION-2B-en pre-training dataset (note that due to compute constraints approx. 30% of the pre-training data). For each data-type, we then match all the text captions from the LAION-2B-en dataset that have cosine similarity larger than 0.85 with the data-type description embedding. We chose 0.85 as our matching threshold by manually inspecting similarity scores on curated set of sentences containing data-type descriptions below 0.85 we noted that the concepts within two sentences no longer matched. Once we obtain the matching pre-training samples per data-type, we follow the same procedure as in Sec. 5 to obtain abundancy scores. With this new semantic search method, we get high rank correlation between abundancy scores and averaged model informedness, r=0.798 for SyntheticTTxellden.. For CLIP-only based model average informedness, the rank correlation is similarly very high: r=0.776. This result provides further evidence that the lack of such data-type rich samples in the pre-training dataset is a strong reason for the poor performance of VLMs on data-type identification.
descriptions of all 27 data-types using the all-MiniLM--G
only run this analysis on 36 out of the 127 parquet files available i.e., on
H TRAINING-FREE ADAPTATION EXPERIMENTS DETAILS
We augment the main text by providing more details about few-shot example selection for both methods, and an added interpretation of results
results here.
H.1 CLIP TIP-ADAPTER
We used two strategies for selecting few-shot examples for adaptation: Random: for each test sam- ple, we selected few-shot examples for each data-type randomly from across all the images from our test set for that data-type. Concretely, for a n-shot experiment, if test sample is left-rotated image of a dog, for each of the 27 data-types we randomly picked n examples corresponding to that data-type. Since we have 10 animals and 5 examples per animal in test set, can pick the n examples per data-type from 50 images for that data-type available in our test set, and (2) SameAnimal: for each test sample, we selected few-shot examples for each data-type such that all the few-shot examples contained the exact same animal as in the test sample. Concretely, if a test sample is a left-rotated image of a dog, for each of the 27 data-type we randomly picked examples corresponding to that data-type while ensuring that the picked samples always contain the same an- imal as the test sample. Since we have 10 animals and examples per animal in our test set, we can pick the n examples per data-type from 5 images for that data-type available in our test set (hence why our SameAnimal few-shot adaptation plot cuts off at few-shot examples in Fig. 6a).
H.2 OTTER IN-CONTEXT LEARNING
Motivated by previous works (Lu et al., 2021; Pezeshkpour & Hruschka. 2023) demonstrating sev- eral biases of in-context learning in LLMs, we experimented with multiple in-context example se- lection strategies. We present the two best performing strategies here: (1) Random: For selecting n in-context examples for a test sample, we randomly select n-1 examples from the test set while always ensuring one in-context example that corresponds to the target data-type of the test sample. Concretely, if the test sample a left-rotated image of dog, we select n-1 random examples from the test set, and select one left-rotated image (the animal in the image need not be dog) as an additional in-context example-tte this ensures that for a particular target data-type, Otter always has an example that describes that particular data-type, and (2) SameAnimal: For selecting in-context
36

---

Published as a conference paper at ICLR 2024
examples for a test sample, the exact same procedure the Random strategy, while addi- tionally ensuring that each in-context example is an image of the same animal as the test sample. For our left-rotated dog example, would ensure that all the selected in-context examples would be images of dogs.
H.3 LLAVA IN-CONTEXT LEARNING
We asked if in-context learning on our data-type identification task would improve with larger model sizes. Thus, repeated the same experiments as for Otter (see Sec. 6.1, Fig. 6b, and Appendix H.2) for the larger ALLVA-113 13B and LLaVA-7B as comparison. Interestingly, both model versions un- derperformed Otte-LL and OtterMPP77B, and surprisingly, LLaVA- even underper- formed its smaller LLa LLaVA-BB version (see Fig. 11).
In-context learning with Llava
Modeis
25-1Aara-78
Mean informedness
0.20
Models
Llava 78 1iam
0.15
Incontes:: Types Strrennine
Rantirn
0.10
0.05
0.00
#In-context examples
Figure 11: In-context learning results for LLaVA-7B and LLaVA-13B. On our data-type iden- tification task, both models underperformed in-context learning with Otter, and the larger LLaVA model underperformed it's bigger variant.
H.4 EXPLAINING CLIP TIP-ADAPTER'S FAILURE
Why does providing few-shot target examples that should prime the model better for data-type derstanding degrade performance over even the zero-shot model? We try to explain this a bit further here. Adp-ter utilises semantic similarities in CLIP's image embedding space for adaptation. However, as we' ve shown from Fig. 5, the image embedding space does not contain enough infor- mation disambiguate between data-types, and hence the adaptation method is unable capture any discriminative information for classifying between data-types; it rather exploits semantic sim- ilarities between concepts which is actually detrimental for our task. An intuitive example would be: say we are classifying image of a noisy dog. The target data-type is GAUSSIAN _NNOSS. For the adaptation, we would have picked random few-shot examples that contain the target data-type samples as well as other samples. Since TIP-Adapter utilises semantic similarity over matching data-types, could assign higher image- inage-image similarity scores for a data-type where all its few- shot examples dogs. whereas the target data-type shw-shot examples only contain other animals. This can therefore systematically degrade performance.
I DETAILS ABOUT TEDATY CONSTRUCTION
We constructed TeDaTy by sourcing training images from ImageNet (Deng et al., 2009), PACS (Li et al., 2017), and DomainNet (Peng et al., 2019). TeDaTy uses in-distribution data- enlist them here: GAUSSIAN_NOISE DEFOCUS_BLUR, types (out of all 27) for fine-tuning, WwW..AN...EEFLLEEEE PENCIL.SKETCH, CARTOON and TY- LEFT.ROTATE, RIGHT_ROTATE,
POGRAPHIC. Our motivation for selecting these data-types was two-fold: (1) wanted to have uniform representation of data-types across the four broad geometric, pixel, style, and seman- tic categories, and (2) we wanted to have few data-types that models were already reason-
37

---

Published as a conference paper at ICLR 2024
ably performant (PENCIL.KKETCH, CARTOON) and others that the models completely fail on (GAUSSIAN.NOISE, PATCHLAND.ESHUURLLE).
RESHUFFLE).
Constructing the training samples for GAUSSIAN_NOISE, DEFOCUS-BLUR, LEFT-ROTATE,
RIGHT ROTATE, PATCH AND RESHUFFLE and TYPOGRAPHIC straightforward we first subsam- pled 100k-sized subset (100 training samples per class) of ImageNet's training dataset (which call IN100k), and then applied the transformation functions corresponding to each of the data-types accordingly. For CARTOON, we acquired training samples by combining the training datasets of PACS-cartoon and DomainNet-clipart, For PENCIL.SKETCII, we adapted the training dataset of DomainNet-sketch as Further, for each data-type transfored image, we ensured that the caption for that image incorporates the data-type information in it. For example. a CARTOON image of dog acquired from the PACS-cartonn subset would have the caption as "This is a cartoon image of dog.". Similarly, a LEFT_ROTATE, image of a tench from the ImageNet subset would have the caption, "This is a left-rotated photo of a tench.". We provide few sample visualisations of the specific training images along with their captions used in Fig. 12.
J FINE-TUNING DETAILS
J.1 CLIP
For fine-tuning CLIP, we used the CLIP ViT-B-32 model for all experiments using the OpenCLIP repository. We used the standard contrastive loss for fine-tuning all our models for 5 epochs with the A (Loshchilov & Hutter, 2017) optimiser, using 50 steps of warmup, cosine-annent annielinn learning schedule and batch-size of 512. For each experiment, we swept over 5 different learning-rates (1e-6, 5e-6, 6, 1e-5, 5e-5, le-4). We used a single NVIDIA A100-40GB GPU for all our CLIP fine-tuning experiments. Our longest running experiments finished in just under 70 minutes.
J.2 OTTER
For Otter fine-tuning experiments, we froze the vision encoder, and only updated the perceiver resampler module, the cross-attentins layers in the LLM-encoder, and the input/output embeddings. We fine-tuned the model on all data-type mixtures for 9 epochs with the Adam optimizer, learning rate of 1e-5, batch size of 128 and cosine-annealing learning-rate schedule, using accelerate-FSDD in mixed precision (bfloat1] 6). We conducted our experiments on 6 NVIDIA A A100-40GB GPUs and our longest running experiments finished in under 40 hours.
K
IMAGE RIGHTS AND ATTRIBUTION
The images in Fig. were sourced from various origins. We here attribute their original sources along with their corresponding licenses. We thank the original creators for availing these images for public use.
A. All dog and eagle images are sourced from our NaturalTypeldent dataset. datastt.
B.
https//hww..oodhouuueep.... best--mulii-purrpoe-cleeeesss
TYPOGRAPHIC
image:
CARTOON httppp//www..aggaa.caaaattttttta
cartoon-classsifccatiio
JPEG COMPRESS image: httpp://www.shhtteeebhoo.oooooon
(license: https: httppp/.....oo../ecccccccccc.ccr
C.
SNOw image: driving-inh inviing-in-thee -ee--iittthh-wwww-
http:///epprraarroooogg.e//////3
HIGH BRIGHTNESS image: http:///www..aafaaaaaaaaaaaaaaaa
briving-in--te-r
38

---

Published as a conference paper at ICLR 2024
Apaired PalLEs pide A.g.
320
149
24
59
39
24
KATy AUgEEr Sauuing Oratra Caikan
TEA TIME S88
$8
1UH ITINNIORE
100
150
207
120
340
An meg Meace 7e seoeedd
24
209
(b)
Apecched suloPiec
369
125
103
359
191
150
381
254
209
24
159
293
Page
20
49
390
150
204
(d)
Figure 12: Random samples from the TeDaTy dataset.
LEFT
ROTATE
image:
https://jane-athome
long- living-room room/
39


Reviews:
[{"review_id": "stDvyWCGHw", "reviewer": "Reviewer_aN8Q", "summary": "The paper introduced a new task: visual data-type identification for vision foundation models. This task builds on earlier literature on robustness and domain adaptation of ImageNet models, but tailored for vision foundation models. This task has practical importance for data curation and data cleaning. The authors have conducted extensive experiments (detailed in the \"Strengths\" section below) and found that scaling model size results in minimal gain, and training with data-type information is a promising direction.", "questions": "- For Section 4.1, when assessing the accuracy of LMMs, aren't there more than one correct answer for data type description? For example, \"pencil sketch\" could be \"pencil drawing\", \"black and white drawing\" etc and \"Origami style\" could be \"low-poly style\" etc. How did you deal with these?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting taskification of data-type identification. I agree with the usefulness for downstream applications such as data curation and data cleaning.\n- Introduced TypeIdent dataset spanning 27 data types across 4 categories (geometric, pixel, semantic, and style)\n- Extensive experiments ranging from initial evaluation using 13 model families, error analysis using embeddings and looking into CLIP's pre-training dataset, in-context learning, and fine-tuning with newly created dataset TeDaTy, which incorporates data-type information into image-text pairs.\n- Interesting findings such as: scaling model size results in minimal gain, in-context learning (using 7B models) doesn't improve the performance of data-type identification much.", "weaknesses": "-  The size of the model (i.e. 7B) used for in-context learning experiments might be too small to test the capability of in-context learning. In-context learning with larger models might work so I think it would be better if the authors could clarify this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZhMa5bpZ1", "reviewer": "Reviewer_mQDi", "summary": "This research introduces the novel task of Visual Data-Type Identification, which involves identifying the visual data-type of images and holds practical value in data curation and autonomous vision systems. Two datasets were created, featuring animal images modified to represent 27 different visual data-types, and 39 VLMs were extensively evaluated. While VLMs perform well with certain stylistic data-types like cartoons and sketches, they face challenges with simpler data-types. Importantly, the study emphasizes that merely scaling up models is insufficient for improving performance, especially for the largest auto-regressively trained VLMs. By incorporating data-type information and pre-training analysis, the study achieved notable performance enhancements, setting the stage for advancing VLMs with visual data-type understanding.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is clearly written and easy to follow.\n2) The method introduces a novel task of Visual Data-Type Identification. This task involves recognizing the visual data-type of an image, such as whether an image has been altered, and how it has been changed. This concept is relatively unexplored in the field of vision-language models.\n3) The researchers created two datasets containing animal images altered to represent 27 different visual data-types, spanning a wide range of categories. This diversity in the datasets allows for a more comprehensive evaluation of VLMs' performance. They conduct an extensive evaluation of 39 VLMs, covering a wide range of model sizes, from small to extremely large. This comprehensive evaluation provides insights into how different VLMs perform in the context of Visual Data-Type Identification.\n4) The study identifies a limitation in existing VLMs. While these models excel at recognizing semantic content, they struggle to understand visual data-types, even when scaled up. This finding highlights the need for a more systematic approach to data-type understanding.\n5) The method demonstrates a way to significantly enhance VLM performance by incorporating data-type information and pre-training analysis. This innovative approach improves the models' capability to understand visual data-types.", "weaknesses": "In page 6, the authors identify that LMMs consistently underperform C-VLMs, despite using\nLLMs as text models, compared to the smaller text encoders in C-VLMs. The authors propose two potential factors for this difference, namely, \"weak alignment\" and the \"discriminative-generative gap.\" However, it is suggested that these factors appear to be more like observations rather than fully explored reasons. It is recommended that further investigations are necessary to gain a deeper understanding of these performance differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjgNAMcTeo", "reviewer": "Reviewer_GzDz", "summary": "This papers explores the task of visual data type identification to better understand abilities of vision-language models (VLMs). Data-type identification here refers to cases like distinguishing between a natural image and a left-rotation / blurred version of the natural image. The authors construct two datasets containing 27 different datatypes (under 4 categories of geometric, pixel, style, semantic) with one dataset synthetic and the other natural. The synthetic uses generative models and image manipulations for creating all its images. The natural uses web-crawled images. Considering VLMs under contrastive (C-VLMs) and generative (LMMs) categories, they evaluate 39 different VLMs on the task of distinguishing each of these different datatypes. Their results indicate weak performance of both categories, with (the generally larger, newer) LMMs inferior to C-VLMs. Interesting analysis is presented including analysis of feature spaces and fine-tuning these models with data-type based supervision (which gives good results).", "questions": "1. KNN + Linear Probing\n* For C-VLMs, can the analysis in Figure-5 be done with KNN evaluation and linear probing? This will give a clear understanding of how linearly separable the visual features are for each datatype. If they are separable, the idea that it is a misalignment with text modality will be clear, and this could be an interesting contribution. If they are not, maybe even SVM on features could be done as a final test to verify their lack of separation. \n* While t-SNE embeddings are visually appealing, they can often be misleading in my personal experience. \n\n2. VLM pre-training datasets\n* While the text based search is good, it maybe better to use semantic similarity metrics (maybe word vectors, scores like METEOR) to create a more comprehensive vocabulary for each data type (e.g. PATCH AND RESHUFFLE can have more words like collage, mosaic). This could retrieve more images similar to the setup. \n\n3. The reasoning for weakness\n* Given how simple supervision helps a lot, the weaknesses could be attributed to a language-image mismatch in these models. Particularly, if the train set rarely contains these language terms. Can any simple experiments directly negate / verify this hypothesis?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. Valuable dataset contribution \n2. Interesting problem setup \n3. Useful framework for creating more data-type datasets \n\n4. Assuming correct, interesting and extensive analysis\n\nThe authors have performed extensive work evaluating a range of models and extracting multiple insights.", "weaknesses": "1. Method weak with respect to language (leading to possibly wrong analysis)\n\n* A core component of experiments is language based classification (into one of 27 data-types). However, the authors simply use a short description for each data-type (as in appendix) on language side. This could simply mean that the model is not being prompted correctly (and the sensitivity of VLMs to prompting is well-known). If this is the case, the assertions of weaknesses (i.e. inability to distinguish data-types) is not generic. It is only for the selected set of prompts used in paper. \n* For simple verification, I took 10 random images, applied the patch&reshuffle operation, and passed both original and augmented to the online LLava API (https://llava.hliu.cc - this is one of the models evaluated in paper / API uses newer ckpt). While it did not generate the expected term \"patch&reshuffle\", it generated outputs for the augmented images different to original, consistently involving terms like \"collage, mosaic, collage art style\" (words somewhat synonymous to augmentation) which indicate some awareness of the model to this patch&reshuffle operation. The original images were not described with these keywords. However, according to Figure 4 in paper, the best LMM (LLava is one of those evaluated) has 0 informedness about patch&reshuffle. For this case, either the metric of informedness or the evaluation protocol does not well-represent the actual abilities of evaluated models like LLava.   \n\n\n2. Missing LMM evaluation details\n\n* \"For a fair comparison, we evaluated LMMs by log-likelihood scoring\" - please explain this in detail in the main text (or at least appendix) without directing reader to other papers (in fact the directed papers also re-direct to different papers) . This evaluation is crucial to understand all the reported analysis. Please explain it clearly. \n\n\nWhile the paper possibly contains some flaws in their method / evaluation, I urge the authors to provide clear responses (especially in case I am mistaken). The dataset and evaluation framework contribution alone can be useful to the community. Also, the direction explored is highly meaningful and valuable.\n\n---\n*Post Rebuttal* \nThe authors provide extensive additional analysis to disprove concerns I raised, and also strengthen the paper detailing all information related to evaluation. Therefore, my final rating has been raised (5 to 8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)