SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ImageNet benchmark score to the text features in the Unified baseline did not improve the  $L_{1}$  compared to the text-only baseline. This is expected as the imagenet performance cannot be used to predict the performance on a different dataset. Fig. 4 shows the predicted vs. ground-truth accuracy. Our approach had a  $R^2$  score (or coefficient of determination) of 0.55, showing significant room for improvement in accuracy prediction. To show that changing the LLM does not affect these results, we re-ran our experiments with a different LLM and report the results in Sup. Sec. C.3
4.3 Insights into VLM Behavior
In this section, we visualize the dependence of the text-derived features on the pre-training datasets and model architectures while averaging them across the different datasets (see Fig. 5).
Model Size. From studying Fig. 5, we can we can identify a clear trend of Fisher criterion and Silhouette score improving with model size, while Class Dispersion score and Synonym Consistency score degrade with model size. Silhouette score quantifies the degree of inter-class overlap or the degree of overlap between different classes in the embedding space. As the model size of the visual encoder increases, the embeddings from different classes become more and more orthogonal, decreasing the inter-class overlap. Fisher criterion quantifies the degree of granularity a model perceives the target datasets to be. As model size decreases, Fisher criterion decreases, or the degree of perceived granularity increases. Class Dispersion score quantifies the degree of intra-class dispersion, or how similar embeddings of the same class are. Specifically, as we increase model size, Class Dispersion score decreases, and therefore the class embeddings become more varied, effectively expanding the class cone radius. Synonym Consistency score quantified the closeness of a class to its synonyms and behaved similarly to Fisher criterion.
Pre-training Dataset. When studying the effect of pre-training dataset size, it is clear that there is a positive correlation between pre-training dataset size and all of the metrics when comparing models of the same size. As the pre-training dataset increases, the intra-class similarity increases more rapidly than the inter-class similarity, hence effectively different classes are more seperated. Specifically, Fisher criterion and Silhouette score increase, or the degree of perceived granularity decreases, and embeddings from different classes become less orthogonal, increasing the inter-class overlap. As the pre-training dataset size increases, Class Dispersion score increases and the intra-class dispersion is more condensed, leading to a smaller effective radius of a class dataset cone. Interestingly, larger models are more affected by the increase in dataset size (as seen by the large slope of ViT-L compared to ViT-B) - which could explain previous works' observation that larger models benefit more when trained on larger datasets [Fang et al., 2022].
Figure 5: Analyzing Score Trends. Average text scores dependence on pre-training datasets and model architecture on our text-derived scores. (left) scores quantifying inter-class similarity (right) scores quantifying intra-class similarity. ResNet  $(\blacksquare)$  and ConvNext  $(\times)$  based models are grouped separately to evaluate their effect on the score trends.
Model Architecture. Pre-training datasets and model architectures significantly influence each other. ResNets and ViTs, for instance, consistently demonstrated differing behaviors and appeared to reside at distinct points on the class separation-dispersion trade-off curve. In particular, ResNets displayed lower Class Dispersion score and Silhouette score, indicating challenges in encoding instances of the same class within the feature space compared to ViTs. This may account for ResNets' superior performance on datasets with low visual variation, like MNIST; as the visual variation is relatively low, we would not expect the Class Dispersion score to be the limiting factor in model performance, making them less affected by this aspect of the dataset. Intriguingly, ConvNEXT models exhibited characteristics more in line with ViT-base models than ResNet-based ones. What leads to variation between WIT and L400m remains unclear, necessitating further investigation.
Vision-Language Models. The field of vision-language models (VLMs) has witnessed significant progress in recent years, particularly with the introduction of contrastive pre-trained VLMs such as CLIP [Radford et al., 2021]. These models leverage large-scale datasets of aligned image-caption pairs to obtain shared embedding spaces that capture rich visual and textual features. The learned image and text encoders from these VLMs have demonstrated impressive feature extraction capabilities and even set state-of-the-art zero-shot performances. However, the performance of VLMs can vary significantly across different datasets, especially when there exists a domain, content, or distribution shift [Fang et al., 2022]. As the number of model architectures & pre-training datasets [Illharco et al., 2021, Schuhmann et al., 2022] increase, it is challenging to select a pre-trained VLM, as good performance on existing benchmarks does not always translate to the downstream task. Therefore, there is a need to develop strategies that can estimate VLM performance on a new task without requiring an exhaustive evaluation of these models using the target dataset.
The Cross-Modality Transferability Phenomenon: Text as a Proxy For Images. While these VLMs aim to project representations from different modalities into a shared embedding space, Liang et al. [2022] found that corresponding image and text pairs don't completely overlap in the embedding space. Instead, a "modality gap" exists between the image embeddings and text embeddings subspace. Subsequently, Zhang et al. [2023] has found that this gap can be approximated as an orthogonal constant between true pairs of image and text and is, therefore, parallel to the decision boundaries for a given modality. This suggests that cross-modality transferability - using one modality as input to the other's classifier - is possible for these contrastively pre-trained VLMs. Several studies have demonstrated the utility of the cross-modality transferability phenomenon in different tasks. For instance, Domino leveraged the cross-modal embeddings to identify error slices and generate natural language descriptions of the error slices [Eyuboglu et al., 2022]. Similarly, Jain et al. [2022]
used these embeddings to discover model failure directions in the multi-modal embedding space. Meanwhile, Zhang et al. [2023] proposed the DrML, which diagnoses and rectifies vision classifiers using natural language inputs. In this study, we also use text as a proxy for images, but for the novel task of ranking and estimating VLM performance.
Unsupervised Model Selection. Unsupervised model selection was recently introduced by Sun et al. [2021], to select the best model for a new target domain without utilizing labeled data. Their work only considered domain (and not content) shifts and proposed constructing a proxy dataset that captures/closely approximates this shift. This proxy dataset is constructed by minimizing different dataset statistics using several labeled datasets. Evaluating models on this proxy set performs well for model selection/ranking. However, such a strategy is limiting in the setting of evaluating VLMs - the size of these models and their pre-training datasets makes it too computationally expensive to achieve the desired goal of evaluating model performance on any downstream task.
Unsupervised Accuracy Estimation. Unsupervised or label-free accuracy estimation aims to estimate classifier model performance with only access to the unlabeled test set of a new task. Platanios et al. [2017, 2016] proposed strategies to apply probabilistic modeling approaches, such as the probabilistic logic or Bayesian modeling, to analyze and aggregate predictions from multiple classifiers. Other works approach this task by fitting models on feature statistics of the target dataset [Risser-Maroix and Chamand, 2023]. Some studies evaluated model agreement, where many classifiers are used on the target dataset, and the degree of agreement was correlated with model performance [Chen et al., 2021, Jiang et al., 2022] Other approaches for unsupervised accuracy estimation include training a neural network on the weight distribution statistics [Unterthiner et al., 2020] or composing a meta-dataset with available datasets, such that the meta-dataset matched some target dataset statistics [Deng and Zheng, 2021]. Some have attempted to craft embedding-based scores, trying to quantify the separability of clusters in the embeddings spaces [Pandy et al., 2022, Ding et al., 2022]. All these methods assume access to the unlabeled dataset of the target task. Instead, our method only requires text descriptions of the novel task to estimate the model's performance.
In this work, we introduce a new problem setting and task LOVM, which aims to select the best-performing VLMs for a downstream vision task by only using its textual description. To demonstrate the feasibility of such a task, we show how large language models, in combination with the cross-modal transferability phenomenon, can be leveraged for such a task. We exhaustively test these methods on the proposed LOVM benchmark, consisting of 35 VLMs and 23 benchmark datasets. Our findings validate the viability of our proposed LOVM task, with unified (both text scores and ImageNet benchmarking) baselines outperforming the ImageNet benchmarking baseline. This suggests that text-based model selection methods (i.e., LOVM methods) provide additional benefits to baseline selection based on a model's performance on ImageNet. Furthermore, we found that the granularity-based scores influence performance prediction and modal ranking more greatly. These findings bolster the research direction of developing methods for VLM selections using only text.
Our proposed LOVM benchmark aims to foster this research direction. We see two promising avenues for future research: (i) improving text-based classification correlation with ground-truth accuracy by either text generation, evaluation metrics, or cross-modal transferability, and (ii) introducing new granularity and transferability scores to the text-only paradigm. Namely, we anticipate the development of methods improving over our proposed baselines presented in Tab. 2. Our work aims to facilitate future research in this area and provide a more accurate and reliable means of comparing pre-trained VLMs, accelerating their utilization in downstream applications.
For a discussion about broader and potential negative societal impacts please see App. Sec E.
Acknowledgments. We gratefully acknowledge the computational credits provided by Google Cloud Platform through Stanford's HAI Institute for Human-Centered Artificial Intelligence. We also thank the Knight-Hennessy Scholars Foundation for generously funding Orr Zohar.