SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weijie  $\mathbf{T u}^{1}$  Weijian Deng Tom Gedeon2,3
<sup>1</sup>The Australian National University <sup>2</sup>Curtin University <sup>3</sup>University of O'Buda
firstname_lastname@anu.edu.au firstname_lastname@curtin.edu.au
Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable generalization capabilities across multiple challenging distribution shifts. However, there is still much to be explored in terms of their robustness to the variations of specific visual factors. In real-world applications, reliable and safe systems must consider other safety objectives beyond classification accuracy, such as predictive uncertainty. Yet, the effectiveness of CLIP models on such safety-related features is less-explored. Driven by the above, this work comprehensively investigates the safety objectives of CLIP models, specifically focusing on three key properties: resilience to visual factor variations, calibrated uncertainty estimations, and the ability to detect anomalous inputs. To this end, we study 83 CLIP models and 127 ImageNet classifiers. They are diverse in architecture, (pre)training distribution and training strategies. We consider 10 visual factors (e.g., shape and pattern), 5 types of out-of-distribution data, and 8 natural and challenging test conditions with different shift types, such as texture, style, and perturbation shifts. Our study has unveiled several previously unknown insights into CLIP models. For instance, they are not consistently more calibrated than other ImageNet models, which contradicts existing findings. Additionally, our analysis underscores the significance of training source design by showcasing its profound influence on the three safety-related properties. We believe our comprehensive study can shed light on and help guide the development of more robust and reliable CLIP models.
By leveraging natural language supervision, CLIP has made significant progress in enhancing the zero-shot capabilities of models, unleashing their potential for remarkable out-of-distribution generalization performance [1, 2]. For example, CLIP models perform zero-shot classification without being explicitly trained on the target dataset, and they exhibit strong robustness to challenging natural distributional shifts [3-7]. Understanding such behavior of CLIP models is critical for advancing the next generation of image-text foundation models. Current research on this topic has explored various aspects of CLIP models, including dataset creation [8], reproducible scaling law [9], fine-tuning approaches [10], and training distribution [11].
In this work, we conduct an in-depth analysis of the safety-related properties of CLIP models, with a particular emphasis on their robustness and reliability across diverse testing environments. Specifically, we delve into the three critical safety-related properties, namely: 1) robustness to visual factors, to assess whether CLIP models can maintain robust when encountering varying visual factors, such as pose, size, color, lighting, and occlusions; 2) out-of-distribution (OOD) detection, to evaluate the capability of CLIP models to detect instances with labels that are not part of the training distribution; and 3) predictive uncertainty, to investigate whether CLIP models can provide calibrated predictions that accurately reflect their uncertainty in different testing conditions.
Our research offers a comprehensive examination of advantages and drawbacks of CLIP models across various critical facets. Building on prior research that has highlighted robustness of CLIP, we further study their performance across specific factors such as pose and lighting. Additionally, although CLIP models have demonstrated their efficacy in detecting OOD data [12], it is still uncertain whether this ability remains consistent when the training distribution, fine-tuning datasets, model size, and architecture are altered. Moreover, beyond classification accuracy, it is also important to evaluate whether CLIP models offer reliable uncertainty estimation across various distributions.
In light of the aforementioned questions, we evaluate 51 zero-shot CLIP models with varying visual encoder architectures, training sources, and dataset sizes, as well as 32 ImageNet fine-tuned CLIP models. To establish a baseline, we compare these models against 127 ImageNet models without language-image pre-training. We examine 10 visual factors variations present in the ImageNet validation set [13], including object pose, lighting, and background, to assess models' visual factors-level robustness. As for OOD detection, we employ ImageNet as an in-distribution (ID) set following [12] and test on 5 types of OOD scenarios. Furthermore, to investigate the predictive uncertainty, we use a set of canonical ImageNet distributions, such as texture, style, and perturbation shifts. Below we present key observations and insights obtained from our study:
- CLIP models are generally more robust than ImageNet classifiers on 6 visual factors. However, they can be less robust on factors like object pose; In addition, training distribution plays an important role in CLIP robustness against visual factors (Section 4.1).
- CLIP models are biased towards shape when making predictions. However, we have also found that this bias diminishes after fine-tuning on ImageNet and becomes similar to other ImageNet models that are pre-trained on more data. (Section 4.2).
- When trained on the same source, classification accuracy of CLIP models correlates with their OOD detection performance (Section 5).
- CLIP models are not always more calibrated than other ImageNet models, which contradicts existing findings [14]. Our research highlights the impact of training data distribution and quantity on these observations (Section 6).
- Compared to other groups of models, CLIP maintains reasonably good uncertainty estimates under distribution shifts after ID calibration with temperature scaling. (Section 6).
Robustness focuses on investigating the resilience of machine learning models to various forms of distribution shift at test time. To this end, a commonly used approach is to introduce artificial transformations onto images, such as, style transfer [15], corruptions and perturbations [16, 17]. Moreover, many real-world datasets are introduced to assess model robustness under a natural distributional shift [3-7]. For instance, Idrissi et al. [13] propose ImageNet-X by relabelling ImageNet validation set to provide detailed labels for naturally occurring factors such as different pose, background and lighting, to identify models' underlying failure patterns.
OOD detection targets at identifying test data that do not belong to any of classes modeled in training distribution [18-20]. A large number of methods are proposed for deep learning models, including generative model-based methods [21-28] and discriminative model-based methods [29, 18, 30, 31, 20, 32]. For example, maximum softmax probability [18] is used as the metric to detect OOD samples. Moreover, the above approaches mainly study OOD detection for a task-specific model using only visual information. In contrast, as CLIP models enjoy popularity, zero-shot OOD detection [12], is proposed, where the objective becomes filtering out input from the task of disinterest.
Predictive uncertainty aims to classify images with calibrated prediction probabilities so as to match the empirical frequency of correctness [33, 34]. Several works improve uncertainty estimations through post-hoc calibration on validation sets [33, 34]. Moreover, some other works show calibration can be improved by directly applying methods, such as ensembling [35] and pre-training [36]. Ovadia et al. [37] point out that calibration methods become less effective under distribution shift. Minderer et al. [14] suggest that CLIP models are well-calibrated given its accuracy. Based on these observations, this work comprehensively studies the quality of predictive uncertainty given by CLIP.
3 Experimental Setup
3.1 Models of Interest
Contrastive language-image pre-training models: We use 51 zero-shot CLIP models (CLIP) and 32 ImageNet fine-tuned CLIP models (CLIP-FT). They have different visual encoders, including slightly modified ResNet [38], ConvNeXt [39], and ViT [40]. There are two training sources (LAION [41] and WIT [1]) and multiple sizes of training datasets from 80 million to 2 billion. For the CLIP-FT models, the vision tower of CLIP is fine-tuned on ImageNet-1K. We consider two fine-tuning procedures, one directly fine-tuned on ImageNet-1K [42], and the other first fine-tuned on ImageNet-12K, a subset of ImageNet-22K before fine-tuning on ImageNet-1K. Unless specified, we use the default prompt template by [1] for zero-shot CLIP models.
Compared models: we use 127 ImageNet models with various architectures, including Convolutional Neural Networks (e.g., ResNet [38] and ConvNeXt [39]), Vision Transformers (e.g., ViT [40] and Swin [43]) and all-MLP architectures [44, 45] (e.g., MLP-Mixer [45]). Following [46], we divide them into three categories: (i) Standard Models. This group consists of models supervised on the ImageNet training set. (ii) Contrastive learning models. This category contains 8 models pretrained by contrastive learning. There are 6 training algorithms investigated, including InsDis [47], MoCo [48], SimCLR [49]; (iii) Pre-trained on more data. This group contains models pre-trained on a significantly larger dataset (e.g., ImageNet-21K) than the ImageNet training set. All the above models, including CLIP, are publicly available on TIMM [50] and OpenCLIP [51]
3.2 Test Sets and Metrics
Robustness. We first pinpoint failure patterns of models by testing on ImageNet-X [13], which is relabelling of ImageNet validation by 16 naturally occurring factors. This work mainly considers 10 factors labelled with a sufficient number of test samples: Pose, Background, Pattern, Color, Smaller, Shape, Partial View, Subcategory, Texture and Larger. The metric is accuracy, and high is better. We evaluate on cue-conflict stimuli and Stylized-ImageNet [15] to measure model bias towards shape.
OOD Detection. We use large-scale OOD detection benchmark which is build up on ImageNet: in-distribution (ID) ImageNet v.s. {iNaturalist [52], SUN [53], PLACES [54], TEXTURE [55] and ImageNet-O [7]} (OOD). The metrics are the area under the receiver operating characteristic curve (AUROC) and the higher is better; false positive rate (FPR@95) when true positive rate is at  $95\%$  and a lower score means better performance.
Calibration. We study ID and OOD datasets, where ImageNet validation is ID dataset and OOD datasets are: ImageNet-V2 [3], ImageNet-Rendition [5], ImageNet-Adversarial [7], ImageNet-Sketch [4], ObjectNet [6] and ImageNet-Vid-Robust [56]. Metrics are estimated calibration error (ECE) [57] and negative log likelihood (NLL). A lower ECE or NLL indicates better calibration.
3.3 Analytical Methodology
In our endeavor to understand the underlying factors that influence the performance of CLIP models, we delve into six primary aspects: 1) training distribution, evaluating the effect of data source; 2) model architecture, looking into the potential effects of different structural choices on model performance; 3) dataset quantity, probing the interplay between the amount of data available for training and the model's efficiency; 4) contrastive loss, understanding its specific role in training dynamics 5) fine-tuning, and 6) test-time prompt, assessing the impact of prompts during the evaluation on model outputs. We follow the analytical methodology of seminal work [46] and a series of following works like [8, 11, 58]) to study the influential factor. Specifically, within the performance trends observed across all models, any factor causing a deviation from these trends is recognized as influential. Notably, in our research, we mainly emphasize and discuss such influential factors within each facet of our investigation.
4 Visual Factor-Level Robustness
The unprecedented robustness of CLIP models has spurred intense research efforts to identify the underlying factors responsible for their performance under distribution shifts. Recent studies provide
Figure 1: The models performance on the subset of ImageNet-X annotated with a given visual factor (y-axis) to their overall accuracy on the whole ImageNet-X (x-axis). Each point represents a model. The x-axis and y-axis are probit transformed following [46]. The black dashed line represents the ideal robust models whose performance on each visual factor is the same as the overall performance. The blue straight lines are fit with robust linear regression [59]. We include models supervised on ImageNet-1K, pre-trained on more data, contrastive learning models, CLIP models trained on two data distributions and their fine-tuned counterparts. We find that CLIP are generally more robust on six out of ten factors, but are less robust against Pose than other groups of models.
valuable insights on the design of training source [11, 8]. Our research builds upon previous findings on the robustness of CLIP models and focuses on the potential failure types of the model. Instead of solely measuring overall accuracy across distributions, we investigate the behavior of CLIP models when faced with varying visual factors such as Pose and Background.
4.1 CLIP Models Generally Exhibit Better Factor-Level Robustness Than Other Models
Factor-level effective robustness. In our study, we extend the concept of overall effective robustness [46] to visual factor-level. Specifically, it measures a model's ability to achieve higher accuracy on the subset annotated by a specific visual factor compared to what is expected based on its overall accuracy on ImageNet-X. Figure 1 displays the accuracy on the subset annotated by a specific visual factor relative to the overall accuracy on ImageNet-X.
CLIP models are generally more robust than other ImageNet models on 6 out of 10 visual factors. Figure 1 highlights several insights into the factor-level robustness of CLIP models. First, we find that CLIP models are more robust than other models on six out of ten visual factors, including Subcategory, Smaller, Color, Shape, Texture, and Larger. Specifically, CLIP models exhibit higher factor-level effective robustness than other models on each of these factors. Second, we observe that CLIP models are less robust than other models on Pose and Partial View factors. Third, CLIP models show a similar trend to other models on the Background factor. In addition, Idrissi et al. [13] observe that data augmentations can improve robustness to related factors, but with spill-over effects to unrelated factors. We speculate that the data augmentations used for training CLIP models may introduce the similar effects.
Training distributions lead to different trends in CLIP models. The choice of training distribution impacts factor-level robustness of CLIP models. Specifically, we find that training on different datasets (i.e., LAION and WIT) forms distinct trends on each visual factor for CLIP, and there is no single training source that always leads to higher factor-level robustness than another. For instance,
Figure 2: Shape bias analysis of CLIP, CLIP fine-tuned (CLIP-FT), models pretrained on more data (Pretrain), and standard models. Large points mean larger models within the group. We observe that CLIP models are more shape-biased.
Source	Backbone	Shape bias	IN-Val	SIN
LAION	ViT/H-14 (336/224)	0.42 /0.51	0.89 /0.88	0.28 /0.32
ViT/L-14 (336/224)	0.41 /0.47	0.88 /0.88	0.27 /0.31
ViT/B-16 (384/224)	0.35 /0.43	0.87 /0.86	0.23 /0.25
ViT/B-32 (384/224)	0.33 /0.45	0.85 /0.83	0.21 /0.22
ConvNeXt-B (384/224)	0.31 /0.38	0.87 /0.86	0.17 /0.21
WIT	ViT/L-14 (336/224)	0.39 /0.45	0.88 /0.88	0.24 /0.30
ViT/B-16 (384/224)	0.35 /0.41	0.87 /0.86	0.22 /0.23
Table 1: The influence of input resolution when finetuning CLIP on Shape bias and ImageNet-Val(idation) and Stylized ImageNet (SIN) accuracy. The higher value in a model pair is in bold. With same backbone, the model fine-tuned with a larger input resolution is more accurate on IN-Val but less shape-biased and less accurate on SIN.
we observe that CLIP models trained on LAION demonstrate higher robustness on Shape factor than those trained on WIT, while this reverses for Background and Pose factors. The results show a mixed observation on Large factor. Furthermore, we further point out that CLIP models trained on different subsets of LAION i.e., LAINON-80M, LAION-400M, and LAION-2B) follow the same trend. The above observations highlight the importance of the choice of training source in determining not only the overall accuracy but also the factor-level behaviors of CLIP models. This suggests that visual factor-level robustness should be considered when designing the training source for CLIP models.
CLIP fine-tuned models perform slightly better than more data pre-trained models. We compared CLIP fine-tuned models (CLIP-FT) with other models that are pre-trained on more data and find that CLIP-FT shows improvement in overall accuracy and robustness on visual factors of Subcategory, Shape, and Pattern. However, no additional robustness gain is observed on other factors. Additionally, CLIP-FT models outperform zero-shot CLIP on variations such as Pattern and Partial View, indicating their superiority in handling visual factors. It would be intriguing to explore fine-tuning techniques that maintain or improve the factor-level robustness of zero-shot CLIP.
4.2 Texture Bias v.s. Shape Bias
CLIP exhibits a shape bias. We conduct experiments using the cue-conflict stimuli dataset [15] to investigate the presence of shape bias in the model's predictions. Shape bias, in this context, refers to the proportion of accurate predictions made based on object shapes. Figure 2 presents a visualization of the shape bias exhibited by the models, which are grouped according to their training methods (zero-shot, CLIP finetuning, more data pre-trained, and standard training) and architecture (transformer versus CNN). Our findings indicate that, among the four training methods, CLIP models are more likely to make predictions based on shape compared to the other three groups. Furthermore, while the transformer is reported to have a stronger shape bias than CNN [60, 61], we observe that CLIP using CNN as the vision encoder also exhibit a strong shape bias.
Model size solely does not explain the shape bias of CLIP. We further observe that larger CLIP models do not necessarily have higher shape bias than smaller-size ones. For example, both trained on LAION-80M, CLIP-ViT/L-14 has 0.54 shape bias, which is 0.09 lower than CLIP-ViT/B-32. This implies that the shape bias of CLIP models cannot be attributed solely to model size. Based on the above observations, we speculate that the shape bias of CLIP may be attributed to its objective, which involves training the model to associate text and image pairs.
CLIP models have a tendency towards texture bias after fine-tuning. Our study reveals that shape bias in CLIP weakens after fine-tuning on ImageNet. Moreover, the fine-tuned CLIP models exhibit a shape bias comparable to models that are pre-trained on larger datasets. This finding is consistent when using transformer and CNN as visual encoder. Moreover, these results illustrate that fine-tuning discards the shape-biased property of zero-shot CLIP, which may affect model robustness [62, 15].
Larger input image resolution during fine-tuning of CLIP results in a stronger bias towards texture. In Table 1, we observe that an input resolution during fine-tuning is a important factor to shape bias: increasing the input resolution during fine-tuning leads to better performance on ImageNet
Figure 3: OOD sample identification capability of models vs. ID dataset classification accuracy. The OOD detection ability is measured by AUROC  $(\uparrow)$  and FPR@95  $(\downarrow)$ . Each point represents a model. We plot the results on iNaturalist, SUN, PLACES, TEXTURE and ImageNet-O. The blue straight lines are fit with robust linear regression [59]. We observe that training distribution has a greater impact than training dataset quantity on the OOD detection performance of CLIP. Moreover, after additionally fine-tuning on ImageNet-12K, CLIP are generally better at detecting OOD samples than those directly fine-tuned on ImageNet-1K.
validation but also results in more texture-biased models with lower accuracy on Stylized-ImageNet. Across seven pairs of experiments and two training sources, we observe this pattern consistently. Given that input resolution is a crucial model dimension [63-65], it would be interesting to study its effects on shape bias beyond classification accuracy when devising scaling strategies.
5 Out-of-Distribution Detection
Zero-shot CLIP allows for a flexible definition of in-distribution (ID) classes without re-training the model. Namely, they can conduct zero-shot OOD detection. The current findings suggest that zero-shot CLIP models are competitive with other state-of-the-art models [12, 66]. Based on this finding, we conduct an extensive analysis to determine whether the purported benefits persist across various training sources, subsets, and network architectures. In the experiments, for zero-shot CLIP models, we utilize maximum concept matching [12] to detect OOD data. For models that are trained or fine-tuned on ImageNet-1K, we employ maximum softmax score [18] for OOD detection.
For CLIP models from the same source, their ID accuracy correlates with OOD detection performance. Our study includes CLIP models trained on two sources (WIT and LAION). Given the same training source, our study, conducted across five challenging OOD scenarios, reveals a strong correlation between the ID accuracy of zero-shot CLIP models and their OOD detection performance (measured by AUROC and FPR@95). This observation suggests that the zero-shot classification accuracy of CLIP models on ID data can serve as a reliable indicator of their OOD detection performance. In contrast, such a trend is not as strong for standard models and more data-pre-trained models. Additionally, CLIP-FT models fine-tuned on ImageNet-1K do not exhibit such a clear correlation.
Training source influences the trend of CLIP. Upon closer examination of the training distribution, we have observed that the correlation trend between ID accuracy and OOD detection performance is
largely dependent on the training source. As illustrated in Figure 3, our research shows two distinct trends between CLIP models trained on WIT and those trained on LAION. Moreover, with the same ID accuracy, CLIP models trained on WIT exhibit superior OOD detection performance compared to their counterparts trained on LAION on three OOD scenarios. This further indicates the importance of training source selection for CLIP. When developing dataset curation methods, it is valuable to investigate the influence of training sources on OOD detection performance.
Fine-tuning procedure influences the OOD detection ability of CLIP. First, we point out that fine-tuning enhances classification performance of CLIP, but this improvement does not necessarily translate to better OOD detection accuracy. Some CLIP-FT models even achieve worse OOD detection performance than Zero-shot CLIP models. Our analysis of CLIP-FT reveals a distinction between two groups of CLIP-FT, based on their fine-tuning procedures: the first group is fine-tuned solely on ImageNet-1K, while the second group undergoes additional fine-tuning on ImageNet-12K. We observe that this additional fine-tuning procedure has a substantial impact on the model's ability to detect OOD examples. As depicted in Figure 3, despite not leading to an improvement in classification accuracy, CLIP-FT models with additional fine-tuning on ImageNet-12K show better OOD detection performance across all OOD scenarios. As future work, it is valuable to investigate this observation further and explore alternative fine-tuning procedures that yield improved OOD detection performance. Moreover, exploring impacts of fine-tuning datasets other than ImageNet-1K and ImageNet-12K would be another interesting direction.
6 Prediction Uncertainty