Diagnosing and Rectifying Vision Models using Language
Authors: Yuhui Zhang
Jeff Z. HaoChen
Shih-Cheng Huang
Kuan-Chieh Wang
James Zou
Serena Yeung
Venue: ICLR 2023 poster (2023)
Keywords: model diagnosis
multi-modal contrastive learning
vision and language

Abstract: Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by leveraging the rich information in large-scale image-caption datasets. Our work highlights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language. The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation. Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behaviors, without requiring any visual data. Through a combination of theoretical explanation and empirical verification, we present conditions under which classifiers trained on embeddings from one modality can be equivalently applied to embeddings from another modality. On a range of image datasets with known error slices, we demonstrate that our method can effectively identify the error slices and influential attributes, and can further use language to rectify failure modes of the classifier.

Published as a conference paper at ICLR 2023
DIAGNOSING AND RECTIFYING VISION MODELS USING LANGUAGE
Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, Serema Yeung Stanford University, Stanford, CA 94305, USA [yuhuiz, Shaochen, mschuang, wangkual, jamesz, syyeunglestarford.edu
ABSTRACT
Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by lever- aging the rich information in large-scale image-caption datasets. Our work high- lights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language. The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation. Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behav- iors, without requiring any visual data. Through combination of theoretical planation and empirical verification, we present conditions under which classifiers trained on embeddings from one modality can be equivalently applied to embed- dings from another modality. On a range of image datasets with known error slices, we demonstrate that our method can effectively identify the error slices and influential attributes, and can further use language to rectify failure modes of the classifier.
INTRODUCTION
Recent models trained using modiil contrastive learning have leveraged large-scale datasets of aligned image-caption pairs obtain shared embedding spaces that capture rich visual and textual features. The learned image and text encoders resulting from multi-modal contrastive learning have been demonstrated to be effective feature extractors that can be used to train strong single-modaligy classifiers (Radford et al.) |02 Jia et al. 2021 Yuan et al. 2021)) I.. In this work, we show how visual classification models obtained through multi-moda contrastive learning, described above, offer significant additional advantage: the ability to use language to probe and diagnose the behavior of the vision models.
Model diagnosis aims to gain a systematic and comprehensive understanding of when and why mod- els fail. This is critical quality assurance process prevent unexpected and catastrophic failures of models in high-stake settings. A growing body of work has proposed methods for addressing this need. For example, error slice discovery methods aim to find subsets of inputs with similar charac- teristics where the model performs significantly worse (d' Eon et al. 2022 Eyuboglu et al.) 2022). Interpretability methods aim understand the black-box process model prediction and thus the reasons why models fail for certain inputs (Ribeiro et al., 2016 Lundberg & Lee 2017 Koh et al., 2020). In addition, model diagnosis is relevant model auditing, an important topic that also deals with identifying model failures and sensitive attributes (Raji et al., 22020), and has broad societal impact terms of Al accountability and integration (Buolamwini & Gebru 2018: Mitchell et al., 2019 Gebru al. 2021
While these prior efforts have made progress in vision model diagnosis, they all suffer from a critical Achilles" heel - susceptibility to lack of visual data. Curated training and test sets from the same data distribution are typically used to develop vision models. Even models achieve perfect per- formance these datasets, their performance
degrade drastically when deployed ta-the-wildd due to distribution shifts (Koh et al. 202T;: Wiles et al.] (022). Yet most existing model diagnosis methods require visual examples of failure modes (e.g., present in the test set) to discover them. As

---

Published as a conference paper at ICLR 2023
Discover Error Slices
Image Embedding
Training
Widertivv in
Landbrrd in water Acc:
Waterbed Landbid?
Classifier
Identify Influential Attributes
+5% Ocean
+7% Lofe
infuemtial Altrbutes for Watechind
Westem kiko.
Cantrol octon
Bamboo -1%%
Text
Rectify Model Misbehaviors
Embedding
ZITAI
Gastvall Ermn
Diagnosing & Rectifying
Forsst
Landbird
Aoc: 50%
App:
Figure 1: Overview of our approach, DrML, that diagnoses and rectifies vision models using language. Our approach leverages the shared image and text representation space learned by multi- modal contrastive learning. We find that classifiers trained on embeddings from one modality be equivalently applied to embeddings from another modality, despite the fact that embeddings from these two modalities are distinctly separated. This cross-modal transferability phenomenon enables us to diagnose vision model by training it on the image embedding space and probing it with text embeddings. The use of language allows us to generate large set of diverse and novel inputs to discover error slices, identify influential attributes, and rectify model misbehaviors.
a result, using these methods is reliant on efforts to collect large-enough datasets to cover all data distributions and potential failure modes of interest, which often impractical or infeasible.
The goal of our work is to circumvent this need to collect test data representing all data distributions of interest, and instead use natural language input to diagnose vision classifiers. It is often easier to generate set of diverse natural language inputs by combining known attributes and prompt gen- erators than to collect a set of image inputs representing the same desired concepts. We observe that vision classifiers trained image embeddings from shared image-text embedding space sug- gest the possibility of leveraging text embeddings as a proxy for image embeddings. Multi-modal contrastive losses are frrqueetly used to learn such shared embedding spaces. However, while these losses encourage image and text embeddings to be closer for aligned pairs than for mismatched pairs, there is guarantee that in practice, using text embeddings on the image embeddings will result in the same predictions. In this work, we first verify that text inputs indeed work as good proxies to image inputs trained space obtained through contrastive learning. We refer to this as cross-modal transferability.
input into vision classifier trained
shared image-text embedding
Based on the phenomenon of cross-modal transferability, we then present DrML for Diagnosing and Rectifying Vision Models using Language. We show that DrML can use language to diagnose vision models in two different ways: discovering error slices including concepts for which we have no visual data, and identifying attributes that have the greatest impact on model predictions. Finally, we present method that uses language to rectify undesirable behaviors without requiring the collection of more visual data. Figure illlssratts our framework for diagnosing and rectifying vision models using language. three image datasets representing the three most common types of model failure modes, demonstrate that DrML can effectively identify error slices and influential attributes, and can further rectify these model failure modes using language.
In summary, contributions are:
We present a theoretical explanation of when cross-modal transferability happens (Section 2.1 and empirically verify that the assumptions required by the analysis sis is true in practice across range of multi- modal contrastive models and datasets (Section (32).
2. We propose DrMI., a framework for diagnosing vision models using natural language, includ- ing error slice discovery and influential attribute identification. We empirically validate DrML. by simulating common types of failure modes using the Waterbirds (Sagawa et al. 2020), Fair- Face (Karkkainen & Joo, 2021). and VSpites (Matthey et al.) 2017) datasets, and show the effectiveness of our method in identifying known error slices and influential attributes.
3. We further demonstrate that DrML can rectify undesirable model behaviors and improve model performance with respect to the identified error slices and influential attributes, by fine-tuning the vision classifier using text embeddings constructed from the diagnosis process.
2

---

Published as a conference paper at ICLR 2023
2 APPROACH
We first define basic notations used in this paper. Given a pre-trained multi-modal contrastive model, along with image X X' text Y E y as input,
obtain their fg-normalized embeddings or y from the image encoder fx : X' R or the text encoder fy y R respectively, where d is the dimension of the shared multi-modaa embedding space. We can build classifiers h : R H C such as linear layer or multi-layer perception on the shared embedding space to predict the label E C given image embedding or text embedding. We focus the case of vision classifiers trained using image embeddings.
2.1 TEXT EMBEDDINGS AS PROXIES FOR IMAGE EMBEDDINGS
The core of our work hinges the ability to use text as proxy for image inputs, thereby enabling us to use language to diagnose vision models. Here we describe our approach to analyze if this is feasible in practice
text inputs good proxies for images in contrastive representation space?
Cross-modal Transferability. To answer the question, we first define cross-modal transfcrability. Let PD be the joint data distribution over image-text pairs. For X,Y PD. we denote (X) and y = (Y) the corresponding image and text embeddings respectively. We say that a vision classifier h achieves cross-modal transferability when it outputs similar predictions æ and y. In other words, the difference across the prediction pair is small:
Em, ((()) = 0,
where D).,)) measures the difference between predictions, e.g. the 0-1 loss D(u, v) 1uze-
Modality Gap. While intuition suggests that embeddings of a matched image-caption pair should be close, recent work shows instead that the embeddings are approximately clustered per modality Liang et al. (2022). They refer to the distance between these clusters as the modality gap. We define individual-level modality gap I as the difference between image and text embeddings for single pair, and the class-level gap ge the average difference between image and text embeddings for given class c E C. Formally, the modality gap definitions are written as:
g
y and g. H Ye: where
xx = Ex Ex
y. Ey Ex-ro(y)[[f(()))
Modality Gap Geometry. We take a closer look at the modality gap geometry across a range of multi-modal contrastive models and datasets, presented in detail in Section 3.2 and empirically find that the following hold true:
1. The modality gap between corresponding image and text embeddings be approximated by a constant vector, particularly at the class level. We verify this by computing distributions over |[g|| (magnitude) and [g]l) (direction).
2. The modality gap is orthogonal to the span of image embeddings and text embeddings, and image embeddings and text embeddings have zero mean the subspace orthogonal to the modality gap. We verify this by computing distributions over cos(æ [x] Eg[[] (or-
thogonality) and Eg - x g 9g',, (center), where g |g[g// and E [d]. The
subscript denotes indexing the i-th dimension of the vector.
Cross-modal Transferability under Modality Gap. The above findings with respect to the geome- try of the modality gap indicate that the classifier input between training and cross-modal evaluation only differs in a constant g. i.e., h(x:) x hiy + g). Intuitively, since the modality gap g is an orthogonal constant the span of embeddings. the weight matrix of the learned classifier should also be orthogonal to Hence the prediction of the classifier is not affected by g. This intuition explains why observe strong cross-modal transferability under modality gap in practice, across different multi-modal contrastive models trained on different datasets. These results are presented in Section In the following Proposition [.1]] we further theoretically prove that linear classifier trained with a regularized quadratic loss is guaranteed to be orthogonal to the modality gap and hence achieves cross-modal transferability. The formal statement and proof are in Appendix A.2
Proposition 2.1 (Informal version of Proposition A.1). Suppose there exists a gap vector g E I" such that every pair of image embedding and caption embedding y satisfies g x y. the gap
3

---

Published as a conference paper at ICLR 2023
g is orthogonal the span of image embeddings. and the image embeddings have zero mean in the subspace orthogonal to g. Then, any linear function minimizing a regularized quadratic loss on image embeddings achieves the same loss text embeddings, enabling cross-modal transferabilityy
Cross-modal Transferability by Closing the Modality Gap. The observation that the modality gap approximates constant provides another perspective to achieve cross-modal transferabilityy by closing the modality gap so that there is no inconsistency when feeding embeddings from another modality. We propose simple technique to close the modality gap so that the gap becomes zero. During training, instead of feeding æ to the model h, we feed it with æ [x] During cross-modal evaluation, we feed y Ey[v] instead of y. With this strategy, we close the gap and observe additional improvements in cross moosl transferability compared to training with the gap.
2.2
DIAGNOSING VISION MODELS USING LANGUAGE
Having established that text embeddings can be good proxies for image embeddings (Section a and
3.2 we now describe DrML, which uses natural language inputs for diagnosing vision classifers.
Discovering Error Slices through Language. Deep learning models often make systematic errors on subgroups of inputs with similar attributes, referred as error slices and formally defined as:
S (S Xe(e(S) e(X)),
where a test set of images and e(.) is the model's error rate on the set of input images. However, collecting large enough test set that covers different image distributions is a fundamental challenge. The collected test set often only covers a small percentage of model failure modes (i.e., error slices) in the wild. In contrast, language inputs easy to generate.
Our proposed method, DrML., is capable of discovering error slices through language inputs. DrML. works follows:
1. We define an attribute set A related to the task.
2. Given specific attribute subset F S A, we different prompt generators
p P : 2 Y map attribute combinations to text inputs.
In this way, we can combine a wide range of attributes with different prompts to collect diverse and novel set of text inputs. The generated text set y is typically much more diverse than the available image test set X, allowing the discovery of more comprehensive and unseen error slices.
Importantly. DrML has distinctive benefits over the typical approach of using an image test set. First, DrML. only requires minimal effort to define meaningful set of attributes to generate the input set, circumventing the human cost of data collection. Second, the combination of defined attributes naturally defines human- -ittran-itterretable data slices, whereas image-based slice discovery methods do not directly provide a text summary of the error slice.
Identifying Influential Attributes through Language. Interpreting what attributes influence model predictions is crucial for understanding why models fail. Since language is directly inter- pretable by humans, perform counterfactual analysis using language to understand which at- tributes or concepts most impact model predictions. With defined as the attribute set, we aim to identify a subset of attributes that significantly influences model predictions to any given class
A_ {a (a) > 0),
where s-(-) is the influence of an attribute to class c. We measure the influence by Shapley value, widely-used interpretation tool in machine learning (Lundberg & Lee| 2017, Ghorbani & Zou 2019), which computes average prediction change with the presence and absence of this attribute:
|F]|(IA| - |FI 1)!
(a)
Σ TEAl(a)
(P(())))
AAI!
where pa(-) is the average predicted probability of class
set of inputs with certain attributes.
With natural language, we can casily compose a large set of inputs with and without that attribute and feed them to the model to calculate the influence. For example, to compute the influence of attribute
4

---

Published as a conference paper at ICLR 2023
"ocean" class "waterbird", the ocean" and "A photo of species", and compute the average difference of the model predicted probabilities of "waterbird". Note that it is particularly challenging identify influential attributes using image inputs because requires an extensive collection of images with attribute annotations.
generate various text inputs such as "A photo of species
Connection. Discovering error slices and identifying influential attributes important comple- mentary applications, with the same ultimate goal finds specific subgroups about when the model fails, while attributes provide abstract explanations of why the model fails. Mcanwhile, figuring out influential attributes helps discover error slices, because attributes provide information about the space of potential error slices, and vice versa.
diagnosing the model. Error slice discovery
2.3 RECTIFYING VISION MODELS USING LANGUAGE
In addition to discovering errors during model diagnosis, how to rectify these errors is a practical but challenging problem, which is seldomly addressed in existing works about the model diagnosis. Our finding about cross-modal transferability enables us to rectify undesirable behaviors of vision classifiers through language. Here we propose a simple solution where we generate additional data that the model fails using language and continue training the model on these synthesized data.
Given the error slices {S Xle(S) > e(X)) discovered, we aim to rectify model perfor- mance on these error slices by minimizing [S]. For each S E S defined by list of attributes, we generate a large set natural language inputs related this slice V/s through attribute composi- tion and prompt manipulation (Appendix BB and continue training the model on these text inputs V/s. We continue training the model using the same hyperparameters as the model is trained on corresponding images, since we have proved that texts are effective proxies of images. This simple strategy significantly improves model performances on corresponding image error slices with mini- mal impact on other data, and has distinct advantage that no visual data is required for rectification.
3 EXPERIMENTS
In this section, we first demonstrate that text embeddings are good proxies for image embeddings in multi-modal contrastive representation space (Section 3.2) Based that, demonstrate how DrML successfully discovers error slices (Section 3.3), identifies influential attributes (Section 3.4..
and further rectifies model misbehaviors on three datasets (Section 3.5
3.1 EXPERIMENTAL DETAILS
Model Architecture. We use CLIP (Rafford et al. 2021) as the shared multi-modal embedding space. For classifiers built CLIP's embeddings, we linear layers and multi-layer perceptrons.
Datasets. For cross-modality transferability (Section we use the MS-COCO dataset (Lin et al, 2014), which includes both captions and object annotations for each image. The task is mabti-label classification problem of predicting the presence of 80 objects based on images or captions. For model diagnosis and rectification, we simulate the three common types of model failures. For spuri- ous correlation, we use the Waterbirds dataset (Sagawa et al. (2000) which asks a model to classify a given bird image a waterbird or landbird. The training data contains a spurious correlation between bird species and backgrounds
95% of waterbirds appear in the water, and 95% of land- birds appear on the land. For underrepresented data, we use FairFaces (Karkkainen Joo (02T) which contains face images from 9 age groups and 7 race groups The task is gender classification. To simulate the underrepresentation
minority groups, we sample races in proportion the demo- graphics of the state of Montana for our training set. For unseen data, we use dSprites (Matthey) et al. (017)) which contains images of shapes with different colors, sizes, and positions. The task to classify the shape in image. To simulate errors caused by unseen data, only use images with orange triangles or green squares during training. More details are shown in the Appendix B
3.2 ARE TEXT EMBEDDINGS GOOD PROXIES FOR IMAGES EMBEDDINGS?
We have provided theoretical explanations in Section 2.1 that classifier's boundary is transferable across modalities if the modality gap satisfies certain geometric conditions. Here we first verify these conditions and then show empirically that closing the modality gap can improve transferability.
5

---

Published as a conference paper at ICLR 2023
Magnitude
Direction
Orthog- onality
Center
Model
Individual
Class
Individual
Class
CLIPCOCO CoCO
2021
1.18 0.03
0.88 1 0.04
0.70 0.06
0.94 1: 0.04
0.00 t 0.06
0.00 i: 0.02
CLIP Imagehor
1.00 1 0.02
0.83 1 0.05
0.00 0.06
1 0.03
ConVIRT 2022
1.22 0.10
0.67 0.09
0.02 0.
0.00 = 0.02
VideoCLIP 2021
1.35 + 0.03
0.79 0.04
0.00 0.06
0.00 + 0.02
CLASP 2021
1.33 + 0.04
0.79 0.12
0.03 0.13
0.00 1 0.02
Table 1: Geometry analysis of modality gap for various multi-modal contrastive represen- tation spaces. The modality gap approximates a constant vector, indicated by the magnitude and direction distributions, Modality gap also orthogonal the span of embeddings from two modal- ities, and embeddings" centers for both two modalities are zero vectors in the subspace orthogonal to the gap, indicated by the orthogonality and center distributions. Based on our theoretical anal- ysis, these findings suggest that cross moosll transferability is widely established in multi-modal contrastive learning. - connects mean and standard deviation. Detailed distributions in Figure
Modality Gup
Model
Evaluation Image mF]I
Evaluation on Text
Consistency:
Loss.
MFL,
Loss;
mF1,
MFL,
Random
0.6939
0,0655
0,0443
0.6938
0.0696
0.0437
0.8644
Linear MLP
0,0501
0.7276
0.6790
0.1188
0.5642
0.5429
0,9637
Default
0,0480
0.7523
0.7158
0.0858
0.6350
0.6135
0.9789
Closing
Lincar
0.0498
0,7280
0,6777
0.0719
0.6554
0.6168
0,9842
MLP
0.6503
0.6358
0,9806
Table 2: Cross-modal transferability in multi-modal contrastive representation learning. We train a classifier using CLIP's image embeddings and test the trained classifier using text embed- dings on the MS-COCO multi-label classification dataset. Despite the modality gap, classification boundaries learned from one modality transferable to another modality. Closing the modality gap further improves pross-modall transfcrability without harm to in-modal evaluation. Notations: mF1 Miero F1, MF1 Macro F1, Random A randomly initialized linear classifier.
Modality Gap Geometry. In Table D we first show that the modality gap can be well approxi- mately by a constant vector. For instance, on MS-COCO, the class-level gaps between image and text embeddings extracted from CLIP (ViT-B/32) have almost the same magnitude (0.88 + 0.04) and direction (cosine similarity 0.94 + 0.04). We then show that the modality gap is orthogonal to the span of image embeddings and text embeddings, and embeddings have zero mean in the subypace or- thogonal to modality gap. This is supported by the near-zero means with low standard deviations in "orthogonality" and "center" columns. Our findings here show that the assumptions required by our theory of cross-modal transferability (Section 2.1 I) hold true in practice across various datasets and contrastive multi-modal models, suggesting that cross-modal transferability should be pervasive phenomenon in multi multi-modal contrastive learning.
Cross-modal Transferability. Table 2 shows the image-to-text transfer results on the MS-COCO validation set. Based on our theory, we indeed find that cross-modality transferability is possi- ble regardless of the modality gap. For instance, we find that an classifier capable of achieving 67.90% maero F1 score can maintain 54.29% macro F1 score us- ing text embeddings as inputs, and the consistency between predictions using images and texts is 96.37%. Similarly, text-to-image transfer is also possible, which is shown in Appendix Table 7 While there exists slight degradation in performance under cross-modal evaluation, the difference in performance is relatively small, and the cross modal transfer performance is much higher than random classification. The same finding is observed when using multi-layer perceptrons that learn non-linear features. As shown in the bottom half of Table 2 closing the modality gap further im- proves modal transferability. The linear classifier achieves 9. 12%, 7.39%, and 2.05% absolute improvements on micro F1, macro Fl, and prediction consistency for image-to-text transfer without harm to in-modality evaluation. The improvements using MLP are smaller but consistent.
weweeedinggttrainnnneeee linear
Are Generated Language Prompts Good Predictors of Error Slices? Here we further investigate whether our generated language prompts are good predictors of the error rate of a given data slice. We do so by looking at the correlation between performances on generated prompts and correspond- ing image slices. A strong correlation indicates that we can perform error slice discovery using text as proxies, which circumvents the challenges of collecting image data.
6

---

Published as a conference paper at ICLR 2023
Waterbirds
FairFace
dSprites
Method
Spearman
Pearson
Spearman
Pearson
Spearman
Pearson
(Buse): Gen Inage (Prob)
0.5822
0.5608
0.3884
0.3361
0.3059
0.3119
(Base): 20 Images (Proh)
0.6034
0.5938
0.4288
0.4557
0.5309
(1): Gemerate Text for Slice
0.4167
0.4355
0,0801
0.0957
0.6278
0.6723
(2): (1)+ Use Label Prohubility
0.5899
0.5773
0.2065
0.1760
0.7071
0.7481
(3): (2) + Prompt Engineering
0.6462
0.6721
0.5669
0.7024
0.6998
0,7595
(Ours): (3)+ + Prompt Easemble
0.6465
0.6776
0.5614
0.7227
0.7028
0.7918
Correlation analysis of model performance on image and text slices. Correlation can be improved by using label probability instead of label accuracy on text predictions, generating better text through prompt engineering and ensemble. Our approach outperforms the baseline text- to-image generation model by large margin. The best near-best results are bolded.
Table 3:
We treat each attribute subset F A as slice. For each slice, we generate set of text inputs D'y using prompt generators d and select all the images Xy with attributes F. We compute the Spearman and Pearson correlation between model performances on Vir and Xy. Table shows strong correlation between image and text slices. Furthermore, correlation can be improved by: 1)
using the average probability of the label on text predictions instead of accuracy, 2) generating better text inputs via prompt engineering which composes attributes into more fluent sentence, and 3)
prompt ensemble that uses different prompts to generate more diverse inputs (details in Appendix B
As baselines for comparison, we use the state-of-the- art text-to-image generation model (Rombach) et al. 2022) 1 + X' to generate a set of (we use 1 or 20 in our experiment) images X3= from text prompts y/ and compute correlations between X and Xy. Our method outperforms this bascline by a large margin and does not utilize significant computational time and cost typically required for the image generation process. Samples of the generated image samples are shown in Appendix C Even while significant progress has been made in text-to-image generation, generating high-fidelle -iddeiilitt images that maintain the original semantics is still challenging.
In summary, combining the empirical findings presented in this section and the theoretical results in Section 2. show that text inputs can good proxies for image inputs, enabling to diagnose vision classifiers using generated language prompts.
3.3 DISCOVERED ERROR SLICES
The strong correlation between the performances text and image slices allows to confidently run image slice discovery algorithm using text inputs. In this study, we use simple error slice discovery method of sorting slices by their performances. We further marginalize attributes by merging similar slices into larger slices. In Table 4 we summarize the most essential discovered error slices by our language-bass basse approach on the three datasets, each representing one of the three typical model failure patterns under distribution shifts (Wiles et al. 2022).
For Waterbird, the top identified error slices are waterbirds in land and landbirds water, which correctly corresponded to errors are caused by spurious correlations present in the dataset. For Fair- Face, the African American population is among the top identified error slice, which also reflects their underrepresentation in our training set. For dSpites our method correctly identifies green tri- angles and orange square as critical error slices. Additionally, pink triangle slices are also correctly identified, since they were never seen in the training data. By using images to verify our discov- ered slices, our method not only correctly identifies the most critical error slices but also accurately predicts the slice performances on images.
In Appendix DOMINO (Eyuboglu et al. 2022). When evaluated using datasets with the same distribution as the training set, DOMINO can only discover slices present in the dataset, and could not discovery errors caused by distribution shifts.
we report results from the state-of-the-art slice discovery baseline
3.4 IDENTIFIED INFLUENTIAL ATTRIBUTES
In Table 5 we report the most influential attributes to a specific class on the same three datasets. These attributes provide high-quality interpretation of how models predict and why they fail. For
7

---

Published as a conference paper at ICLR 2023
Waterbirds
FairFace
dSpritesV
Slice
Text
Image
Slice
Text
Image
Slice
Text
Image
Whird in L
0,3258
0.3233
Black
0.9134
0.8997
Green triangle
0.1641
0.0616
L.irrd
0.9268
0,9446
Orange square
Whird in W
0.9306
0.9549
Asian
0.9305
0.938
Pink triangle
0.5044
0.9861
Lbird in L 0.9957
0.9979
White
0.9427
0.9597
Red triangle 0.5651
0.9954
Table 4: Discovered error slices using language. With the images used for validation, our method succeeds in discovering important error slices (bolded) and accurately predicts model performances on image slices. Notations: Image model accuracy using real image inputs, Text-predicted accu- racy using text inputs as a proxy, W water, L. land.
Waterbirds (waterbird)
FairFace (female)
dSpritesV (triangle)
Attribute
Influence
Altribute
Influence
Attribute
Influence
0.3062
Very old
0,0229
Orange
0.3736
Place
Forest broadleaf
Lake natural
-0.1540
0.0713
Age
Young
Little
-0.0079
0.0161
Color
Red
.0.4700
Bamboo forest
-0.1931
Infant
-0,0171
Green
-0.3521
Table 5: Identified influential attributes using language. We show top 2 most positively and
negatively influential attributes, which provide insights into how models predict and why they fail.
Waterbirds Rectily
FairFace
Slice
Original
Lonly
JTT
GDRO
Slice
Original
Rectily
Lonly
JIT
GDRO
Whird in
0.3233
0.5564
0,5639
0.5865
0.7368
Black
0.8997
0.9075
0.8920
0.8920
0.9017
Lbird W
0.6524
0.8271
0.7747
0.7468
Asian
0.9290
0.9390
0,9384
Whird in W
0.9549
0.9700
0,9023
0.9248
0.9474
Indian
0.9446
0.9439
0.9301
0.9406
0.9453
Uhird I.
0.9893
0.9443
0.9764
0.9443
White
0.9626
0.9424
0.9602
0.9588
Table Rectified model performances on discovered error slices. We continue training models on language inputs corresponding error slices (bolded) and observe significant performance im- provements on these slices. GDRO not directly comparable because attribute annotations required.
example, one of the most influential attributes for waterbird classification "ocean" with an influ- ence value of 0.3062, indicating that the model predicted probability of waterbird increases by 0.3 on average when "ocean" is present in a bird image. Since the attribute "place" should not affect pre- dictions, this shows obvious eтrоr of the model. Similar findings apply the attribute "color" for dSpites V. But what is more interesting is that the color "pink" is never seen during training but will bias the model predict "square" with 0.1 increased probability. On FairFace, no attribute is found to significantly influence model prediction; thus, no obvious spurious correlations were learned.
3.5 RECTIFEED MODEL MISBEHAVIORS
In Table 6 we report performances of original models and rectified models. On both Waterbirds and FairFace dataset, our simple method of continue training the model on text inputs significantly improves model performances on error slices with minor influences on other slices. We also perform ablation by only training the model on all the language inputs from scratch (Lonly), and find that continuing to train the pre-trained image model achieves better results, but even training only with language can also work reasonably.
Our approach rectifies model misbehaviors caused by spurious correlation and underrepresented data by correcting the data bias. Another series of methods to tackle these errors are robust training tech- niques, such GDRO (Sagawa et al. 2020) and JTT (Liu et al.) 2021). which explicitly optimizes each slice's performance during training. Our method outperforms ITT. While GDRO performs similarly to ours, it requires attribute annotations on images, which is highly time-consuming and cost-prohibitive for most real-world applications. Moreover, GDRO and JTT cannot fix errors on unseen data, while ours can, because our rectifying process requires no visual data.
4 RELATED WORK & DISCUSSION
Multi-modal Contrastive Learning. Many recent works in vision- vision-language contrastive learning, such as CLIP (Radford et al., 2021). ALIGN (Jia et al. 202T), and Florence (Yuan et al. 2021
8

---

Published as a conference paper at ICLR 2023
have leveraged large image-can maage-caption datasets to obtain embedding spaces that capture rich visual and textual features. As a result, the learned image and text encoders are demonstrated to be strong uni-modal classifiers. In this work, show how vision models obtained through multi-modal contrastive learning offer another significant advantage - model diagnosis and rectification.
Multi-modal Contrastive Representation Space Geometry. Although multi-modal contrastive learning minimizes the distance between embeddings for matched pairs, prior work has shown that embeddings from two modalities are distinctively separated in the embedding space, which is re- ferred to as modality gap (Liang et al., 2022). In this work, we further analyze the modality gap geometry and connect it the cross- modal transferability phenomenon. Our finding is related to several recent works built on multi-modal contrastive representation spaces, such as DALL-E (Ramesh et al., 2022), ClipCap (Mokady et al., (021), and other works (Cohen et al. 2022 Gal et 2002). They found that trained models can directly take cross-modal embeddings but worse than same-modal embeddings. We not only explain this but provide a straightforward solution to im-
prove transferability, which can be applied to all future works built upon multi-modal embeddings.
Slice Discovery. Many recent works aim to understand model systematic errors by finding subsets of inputs with similar characteristics where the model performs significantly worse. This is referred to as slice discovery (Chung et al. 2019; Singla et al. 2021 Eon et al. 2022, Eyuboglu et al. 2022 Jain et al. 2022a). However, these algorithms fail to address the most fundamental challenge for slice discovery - the lack of data. These works are only able to find errors that exist in the dataset. Our work circumvents the data challenge by performing slice discovery on the text space.
Interpretation. Many model interpretation methods have been proposed, including attribution- based (Ribeiro et al. [016 Lundberg & Lee 2017 (Shrikumar et al. (017] and concept-based (Ghor- bani et al. 2019b Koh et al. (020)). While these methods help in understanding the model prediction
process, the outputs are complicated for humans understand and inconsistent across models and
algorithms @Chorbani et al. 2019a] Jain et al. 20226; Joshi et al. (021). Others require modi- fications in model architectures or complex post- proceproesssings (Ghorbani et al. [019b Koh et al. 2020). In contrast, language is inherently understandable by humans and simple to construct. In this work, we interpret the model prediction process by identifying the most influential attributes using language, which provides us meaningful interpretations without pre-processing post-processing.
Algorithm Fairness. Ensuring algorithmic fairness is key to avoiding potential harm to our soci- ety (Hovy& & Spruit 2016; Zou& & Schiebinger! 2018). Methods for improving the fairness of ma- chine learning algorithms is an ongoing active area of work (Bolukbasi et al., 2016, Sagawa et al.) 2020 (Sohoni et al.) 22200; Ramaswamy et al.) 2021 Liu et al. (222)). Among these, notable solu- tion is to correct data bias, as model bias stems from data bias. In this work, we show that language can be used correct data bias by generating additional data, hence improving model faimess.
Limitations. While our work introduces a novel and effective approach for diagnosing and rectify- ing visual classifiers, there are additionally important areas for future work. First, since assume vision classifiers are built using an image-text embedding space trained through multi-modal con- trastive learning, our method can also inherit limitations from the contrastive model and pre-training dataset. For example, although we aim to leverage large and ceneral-purpose image-caption datasets in pre-training, the encoders may still not appropriately embed off--iiiiitittiionn examples far from what the contrastive model was trained on. Misaligned or inaccurate pre-training data can also af- feet encoder quality. Additionally, it is challenging to diagnose low-level visual attributes that are difficult to describe in words, such as texture or object orientation (Leclerc et al. (021). We con- sider these fruitful directions for future work. Our method will also benefit from improvements in multi-modal contrastive pre-training as these methods are improved.
CONCLUSION
Our work reveals a valuable advantage of using vision classifiers built on top of multi-modal em- bedding spaces learned through contrastive learning the ability to diagnose and rectify the vision classifiers using natural language inputs. We first use a combination of theoretical analysis and ex- perimental findings to verify that cross-modal transferability exists; namely, that text inputs can act as good proxies for image inputs. This then allows us to propose and validate a framework for diag- nosing and rectifying vision classifiers using natural language inputs. Our work suggests promising new directions both for achieving reliable and trustworthy computer vision models, and for the use of cross-modal transferability in other problem domains.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
One of the main contributions of our work is an approach for diagnosing and rectifying vision clas- sifiers trained using embeddings from multi-modal contrastive model. We showcase experimental results on identifying error slices and influential attributes. For example, our method can detect failures caused by the lack of representation of certain races in the training set. In our FairFace experiments, the prediction of gender (i.e., the label "female") given an image was affected by race (e.g., the race "black"). We further show that we can rectify this behavior using our approach. Hence, we see our work as contribution to the broader community concerned with model accountability and model auditing, and improving the responsible integration of Al into society.
However, it is also important to be aware of potential negative impacts brought about by our findings. One can imagine an adversary who extends our approach and uses it to their advantage, perhaps reinforcing racial or gender biases by fine-tuning vision model using biased language prompts. Our work also inherits limitations from the contrastive model and pre-training datasets used to obtain the image and text encoders, described in the Discussion section of paper. We hope that this statement raises awareness both of the importance of better model diagnosis and rectification methods and of future directions of work to address limitations and potential negative impacts.
REPRODUCIBILITY STATEMENT
We provide open-source implementation of work httts::/githhii com/ yubui 201 2h15/drml The implementations will enable researchers to reproduce all the ex- periments described here well run their own analyses on additional multi-modal models and datasets.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Tolga Bolukbasi, Kai- Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman to homemaker? debiasing word embeddings. Conference on Neural Information Processing Systems (NeurlPS), 2016.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Conference on Fairness, Accountability, and Transparency (FAccT), 2018.
Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. Slice finder: Automated data slicing for model validation. In International Conference Data Engi- neering (ICDE), 2019.
Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. arXiv preprint arXiv:2204.01694 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Greg d'Eon, Jason d'Eon, James R Wright, and Kevin Leyton-Browy The spotlight: A general method for discovering systematic errors in deep learning models. In Conference on Fairness, Accountability, and Transparency (FAceT), 2022.
EleutherAl. Clasp: Contrastive language aminoacid sequence pretraining, 2021. URL https:
/tgithub.com//iicit
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, lean-Benoil Delbrouck, Christopher Lee- Messer, Jared Dunnmon, James Zou, and Christopher Re. Domino: Discovering systematic erross with moos embeddings. In International Conference on Learning Representations (ICLR), 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208.01618, 2022.
Timnit Gebru, Jamie Morgenstem, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64 (12):86-92, 2021.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning (ICML), 2019.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In
AAAI Conference on Artificial Intelligence (AAAT), 2019a.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Conference on Neural Information Processing Systems (NeurIPS), 2019b.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Dirk Hovy and Shannon Spruit. The social impact of natural language processing. In Annual Meeting of the Association for Computational Linguistics (ACL.) 2016.
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. arXiv preprint orXiv:2206.1475: 2022a.
Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Alek- sander Madry. Missingness bias in model debugging. International Conference on Learning Representations (ICLR), 2022b.
11

---

Published as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- -ainuala representation learning with noisy text supervision. In International Conference Machine Learning (ICML), 2021.
Gargi Joshi, Rahee Walambe, and Ketan Kotecha. review on explainability in multimodal deep neural nets. IEEE Access, 9:59800-5999 59821, 2021.
Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Winter Conference on Applications of Computer Vision (WACV), 2021.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning (ICML). 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang. et al. 3db: A framework for debugging computer vision models. arXiv preprint arXiv:2106. 03805, 2021.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap in multi-modal contrastive representation learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), 2014.
Evan 7. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. International Conference on Machine Learning (ICML), 2021.
Scott M Lundherg and Su-In Lee. A unified approach to interpreting model predictions. In Confer- ence on Neural Information Processing Systems (NeurIPS), 2017.
Loic Matthey, Trina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Dis- entanglement testing sprites dataset, 2017. URL https://gitaub.com com deepmind/ esprites-dataset.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Conference on Fairness, Accountability, and Transparency (FAccT), 2019.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint wxXiv:2111.0973.1 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML). 2021.
Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountabil- ity gap: Defining an end-to-end framework for internal algorithmic auditing. In Conference on Fairness, Accountability, and Transparency (FAccT), 2020.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky, Fair attribute classification through latent space de- biasiig In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In International Conference Knowledge Discovery and Data Mining (KDD), 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2020.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning (ICML), 2017.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subelass left behind: Fine-grained robustness in coarse-grained classification problems. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvis Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations (ICLR), 2022.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-tre pre-training for zero-shot video-text understanding. In Conference on Empirical Methods in Natural Language Process- ing (EMNLP), 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 2021.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher Manning, and Curtis Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learn- ing for Healthcare (MLHC), 2022.
James Zou and Londa Schiebinger. Design ai so that it's fair. Nature, 559(7714):324-326, 2018.
13

---

Published as a conference paper at ICLR 2023
OVERVIEW OF APPENDIX
In this appendix, we supplement additional details of theory, datasets, experiments, and basclines.
In Appendix A we provide more details about the modality gap geometry, a theoretical proof of mross-modal transferability given the modality gap, additional cross-modal trans- ferability results on MS-COCO and ImageNet.
In Appendix B
we provide details of four datasets (MS-COCO, Waterbirds, FairFace, and dSpritesV) V) used in our experiments, including data preprocessing, attributes, and prompts. We also provide the model and experimental details.
In Appendix C image generation for model diagnosis, which sometimes fails to generate fidelity images given text prompts. Second, we present the baseline method for slice discovery using DOMINO, which fails when error slices are absent the dataset.
we provide two baseline methods. First, we present the result using text-to-
A CROSS-MODAL TRANSFERABILITY
A.1 MODALITY GAP GEOMETRY
Figure shows the modality gap phenomenon in various multi-modal contrastive learning models, where inputs from two modalities are embedded at arm's 's length in their shared representation space. This phenomenon is caused by the combined effect of model initialization and optimization. Deep neural networks have the cone effect- - encoders will only map inputs to small cone of the entire representation space. Therefore, two cones will be created for a multi-modal model with two en- coders. As a sequence, the modality gap occurs at the initialization stage. During optimization, the contrastive loss will preserve the gap due mismatched data (Liang al. 2022
CLIP
VideoCLIP keara Vicao
COnVIRT
CLASP AAT.--DDOSEAAAAA
agu-
Meciaal Eetge-
UNAP
23
URAPP
0.5
0.4
625
SVD
610
-32
-25
-16
-02
0.5
06
67
-1.
-25
-03
-28
-04
-32
Figure 2: Modality gap for multi-modal contrastive learning. Embeddings from two modalities are visualized using UMAP and SVD. Figure credit: Liang et al. (2022).
Figure 3 shows four statistics that reveal important properties of the modality gap geometry.
The modality gap approximates constant vector, particularly at the class level. We verify this by computing distributions over ||g|| (magnitude) and cos(g. E,[g]) (direction). is the gap between embeddings of paired data from two modalities.
The modality gap is orthogonal to the span of embeddings, and embeddings have mean in the subspace orthogonal to the modality gap. We verify this by computing distribu- tions over cos(æ - [x] [g]] (orthogonality) and [x x g''g'' (center), where g' Eg[g |/lg//|BB[9ll and i [d] denoting i-th dimension of the vector.
Based on our theoretical analysis in the next section, these findings suggest that cross cross-mode modal trans- ferability is widely established in multi-modal contrastive learning.
14

---

Published as a conference paper at ICLR 2023
CUPP COCO
0.03
Denery
1.3
3,2 MagiTuuee
1.
0.6 Direction
-02
6.0
62
-0.3 Centering
Orthogonaltyy
CUP CCCO
MMurr: 2,59 0.04
Mears 0.94, Ste
Rean: 0.000.05
Mears 3td: 0.03
12
10
10
Mag
Direction
Orthogonality
Centering
ImagaNet Class
Muum: 2.00 500 0.02
Moar: 9.85, S18 0.00
Rean: 0 OL, Sidt 0:05
Muari 0.00, 30d 0.03
20
10
0% 50
10
3.49
1.85
0.7
0.9
-0.4
-03
Direction
Orthogonaiiyy
ConVERT
Nean: (22 Sts: 0.10
Mean:: OJST. 0.09
8.02. S:d:
Meant 0.00, Sttl 0.02
3.5
30
Derdky
05
0.3
0.0
Magritude
Oiroction
Orthogorakky
Corfoe mg
Peean 3,25, 0,03
Mear: 05
Iean: 0,50 Sttt 0.06
Hear: -0.00, 7adi
7.5
10
5-
2.5
0.3
Magnttue
Dirccion
Ortrogorallty
Contaring ng
Mear: 1.33,
Newn Sdd 0.12
M4r 0.50, 3MI 0.33
Moan:: 80, gee 0.02
3.32
0.5
-0.3 Contaring
Magrituae
Diecctio
Orihogomally
Figure 3: Geometry analysis of modality gap for various multi-modal contrastive represen- tation spaces. The modality gap approximates a constant vector, indicated by the magnitude and direction distributions. Modality gap is also orthogonal to the span of embeddings from two modal- ities, and embeddings' centers for both two modalities are zero vectors in the subspace orthogonal to the gap, indicated by the orthogonality and centering distributions.
15

---

Published as a conference paper at ICLR 2023
A.2 THEORETICAL PROOF FOR CROSS-MODAL TRANSPERABILITY
In this section, we expand and formally discuss what is in section 2. the intriguing cross-modal transferability phenomenon. We explain why the modality gap in the multi-modal representation space does not prevent cross-modal transferability because of the unique geometry of the modality gap.
We theoretically explain
For class c E [[C]]. let e, E {0, 1)(c) be a one-hot vector such that the e-th dimension is 1 and other dimensions are O. We define the following balanced target label vector é. ee Eg[ee] where the expectation is over the distribution of classes on the image domain.
We consider learning a linear function hw(u) = W u, where W E klld is the weight matrix and u E I the image or text embedding. Given hw (u) and label e, we consider the following quadratic loss:
Gqund (hw (u), (1lw(u) ēollé-
The following proposition shows that when the gap between image and caption embeddings is the same for all image-caption pairs and is orthogonal to the embedding span for each modality, linear model trained to minimize the quadratic loss modality transfers the other modality without loss of accuracy.
Proposition A.1. Suppose there exists a gap vector g R. such that every pair of image embedding and caption embedding y satisfies
Suppose the gap g is orthogonal to the span of image features (i.e., g g" x' for two image embeddings and x' ). and the image features have zero mean in the subspace orthogonal to g (i.e., [1, (x)] = O where llg(x) projects the vector the subspace orthogonal g). Then, for any > 0 and linear function hw (u) that minimizes the regularized quadratic loss quad (a:), c)] A|| have that
x
hw(x) hw (v)
Thus, cross-modal transferability happens.
Proof of Proposition A./ Since g² g" x' for all image features and x', find E TR. such that æ = [T) + rg. Notice that
Ex,cl (/e (2)(
Bx,([ll W - E.1
= =||E_[ Wx] E. + E,, ē.) [E,[W=] [e,
= =I[TE,, W x:] - F. 212 + E (x) e..
[[]))]( 7 Wg Ell +
1.1)
Since [x]]] 0 and [_[ē]]= = 0, the first term reduces to - W Notice that the second term the loss decomposition only involves W's components that are orthogonal to g. Thus the minimization of the second term is independent of the minimization of the first term. As result, any W that minimizes the regularized quadratic loss must satisfy Wg =0
For pair of image and text features æ, y, since ---- y = g and Wg g 0, we have hw (x) = how (z), which finishes the proof.
A.3 ADDITIONAL CROSS-MODAL TRANSFERABILITY RESULTS
MS-COCO. In the main paper, we only report image-to-text transfer, where we train classifier on image embeddings and test on text embeddings. Here we report the full results, including text- to-image transfer, in T
ImageNet. In the main paper, report cross- modal transferability on the MS-COCO dataset. Here we report cross-mos modal transferability results using the ImageNet dataset (Deng et al. 2009). We split ImageNet validation set into 40K 10K images for training evaluation. We apply OpenAl
16

---

Published as a conference paper at ICLR 2023
In-Modal Evaluation
Cross-Modal Evaluation
Consist
Model
Transfer
M
Loss;
mF1,
MFI,
M
Loss;
mF1,
MF1,
ency;
Random
I
0.6939
0.0655
0.0443
T 0.6938
0.0696
0.0437
0.8644
Default Modality Gap
T
0.0501
0.7276
0.6790
T
0.1188
0,5642
0.5429
0,9637
Linear
T +
T
0.0580
0.6983
0.6631
I 0.1572
0.5320
0.4833
0,9527
I T
I
0.0480
0.7523
0.7158
T 0.0888
0,6350
0.6135
0,9789
MLP
T I
T
0.0572
0.7119
0.6826
I
0.0750
0.6359
0.5929
0.9795
Closing
Modality Gap
I T
I
0.0498
0.7280
0.6777
T
0.0719
0.6554
0.6168
0.9842
Linear
T-I
0.0578
0.6988
0.6628
I
0.0660
0,5782
0.4767
0,9858
I
0.0483
0.7495
0.7130
T
0.0885
0.6503
0.6358
0.9806
MLP
T
T
0.0573
0.7073
0.6763
I
0.0685
0.6603
0.6173
0.9801
Table Cross-modal transferability in multi-modal contrastive representation learning. We train a classifier using CLIP's image embeddings and test the trained classifier using text embed- dings, vice versa, on the MS-COCO multi-label classification dataset. Despite the modality gap, classification boundaries learned from one modality are transferable to another modality. Closing the modality gap further improves cross-modal transferability without harm to in-modal evaluation. Notations: Z Image, T Text, M Modality, mF1 Mícro F1, MFI Macro F1, Random A randomly initialized linear model.
CLIP's 80 prompts 1000 ImageNet class names and get 80K texts, and we split them into 64K / 16K for training evaluation. All the experimental settings are the same as MS-COCO experiments. Results are shown in Table
Again, despite the modality gap, we find that the classification boundaries learned from one modality are transferable to the other modality. When linear classifier is trained on image embeddings and achieves 70.86% image classification accuracy, directly feeding the text embeddings to the trained classifier achieves 85.24% accuracy. The transfer from text to image is much worse than from image to text, because the texts used are generated from prompts and thus lack diversity to train a classifier with good decision boundarics. Closing the modality gap improves the transferability in most cases.
Image-to-Test Text MLP
Text-to-Image Linear
Split
Linear
MLP
Default Modality Gap 0.7086
In-Modal Evaluation
0.6687
0.9974 0.4953
0,9951
Cross-Modal Evaluation
0.8524
0.7758
0.4552
Closing Modality Gap
In-Modal Evaluation 0.7048
0.6683
0.9978 0.5050
0.9949
Cross-Modal Evaluation
0.8754
0.7892
0.4438
Table Cross-modal transferability in multi-modal contrastive representation learning using the ImageNet dataset. We split ImageNet validation set 50K images to 40K / 10K for training and evaluation. Texts are generated using OpenAl's 80 prompts multiply by 1000 class names.
A.4 THEORETICAL INTUITION FOR MODALITY GAP AND CROSS-MODAL TRAASFERABILLITY
In this section, provide theoretical insights about the intriguing modality gap and cross-modal transferability phenomenon. We show that after optimizing multi-modal contrastive loss, there is a modality gap between image embeddings and text embeddings, and there is a linear classifier trained on image embeddings that is guaranteed to generalize to text embeddings regardless of the modality gap.
17

---

Published as a conference paper at ICLR 2023
Basic Notations. Given image and text 2, image encoder f(-) and text encoder g(-) map them the shared D-dimensional representation space. We denote f(x) Rº as image embedding and g(=) E R as text embedding. Given N images and M texts, we denote the concatenated image embedding matrix as F' E g gNxD and concatenated text embedding matrix as G E R MXD
which is shown in the following equation:
f(
92212
==
ER NxD
G=
E ER M
(1)
L((#N)
Lo(=Ma)
Image-Text Connection Graph. Given an image I and a text 2, we denote the probability of (z, being an image-text pair p(x, 2). With the probability definition, we have P(x, 1. Given N images and M texts, we denote the probability matrix P R N Note that the probability matrix P a sparse matrix with most elements zero, because most image-text pairs mismatched and cannot be collected (e.g.. a cat image with dog caption). P can be viewed as adjacency matrix the vertices of the graph and their connection probabilities the edges. The adjacency matrix of this graph can be written as the following equation:
bi-particle graph ((x, 2). {p(x, =)}), where the images and texts
21)1,21) ... p(x1, 2M) ]
P=
(2)
(1)N,21) P(NN,
We have the following theorem which shows the connection between multi-modal contrastive learn- ing with the partitioning of the connection graph defined above. This result can be viewed simple generalization of the results in HaoChen et al. (2021) to the multi-modal setting.
Theorem (Equivalence of Multi-modal Contrastive Loss and Graph Partitioning). With the mild assumption that every image and every text pair has equal presence probability Vr Ps
Σ, p(2,(22 2) N. V2 Pz = E.p(a 2) = minimizing the multi-modal contrastive loss in
Equation
is equivalent to minimizing P|P FG 11::
= -22E [5(x) g())] NME P. [(+))) 9(x))
(3)
Proof of Theorem
1
The following equation proves Theorem
min P
Σ 2)))) g(=)+ [((()""(())) {),,)))
min
Σ
2)((x,))(())) g (((())"(=))²
= min
(4)
2,5
2,22
min
Σ
22((x, 2) 2)((x)) g(x) NM
NM (f(x)" ((x))
2,5
[(x)) s())] + NME Ps [f(x)" o(=))²
= =mn -2E
Connection to CLIP Contrastive Loss. Given N images and N texts, with the assumption Vi,j p(x,,2,) N contrastive loss in Equation
1 [[ j]. the CLIP contrastive loss is shown in Equation 5 and is very similar the
18

---

Published as a conference paper at ICLR 2023
CCUP
exp (Jxz)) (((m;)""((())) s(z1))
exp (f(x)))'(()))
1 N Σ
log Σ, oxp (f::) ) 9(2,))
log
g(z)))
1 Σ
-2f(x;)) g(²,) + log [ exp (s(x;) g log Σ
exp (f(x)) s((x))
N
)
log Σ exp s(())) 1 N Σ log Σ (xp()())) [[(({))'((z)))) ] -2 [ [(x)) g(z)))((>>))))))))))
N
i
2 [s(x)* s(=)] P..-P. log Σ exp(((x)) a(=')) log Σ exp((((x))* g(=))
Let us now consider the modality gap phenomenon in the above mentioned contrastive learned representation space.
Proposition 1 (Modality Gap). After optimizing the multi-modal contrastive loss defined in Equa- tion 3 image embedding and text embeddings will be separated in the shared presentation space, causing the modality gap phenomenon.
Proof of Proposition 1
Since optimizing the multi-modal contrastive loss defined in Equation 3
is equivalent to minimizing ||P-FG" achieving the minima of P 112 does not indicate image embeddings F and text embeddings G are close in the embedding space. For instance, since for any scalar > O, we can scale F by factor of e and G by factor of 1/c without changing the contrastive loss, we know that there exist many solutions that both achieve the minimal loss and also exhibit modality gap.
Now we consider the cross-moos modal transferability phenomenon on downstream classification prob- lems using the above mentioned contrastive learned representations. We first introduce the following notations for the downstream task's ' labels.
Labels of Images and Texts. Given an image xr with the label Vr E [[C], where C is the number of classes, we denote its one-hot representation as Cys [[1[va 1]. i].... [[v, = CNJ". Given N
images, we denote the image label matrix Y, [[ eyxx {0, 1) NxC We use similar notations V/ ey., Yg for the text.
T
We make the following assumption which says the label of a text can be predicted by the labels of the images that it is paired with.
Assumption 1. With the definition of connection probablity matrix P, image label matrix Y,, and text label matrix Y,, we have pa ===11 -Y
M
Intuition of [ssumption In most realistic settings, the connection between an image and a text 2(x,2) > 0 if y.r = ys and p(z,2) 2) 0 if yı # yz- Therefore, Vz En p(x, 2)c,, p(x,2)e,, M Ey. The following equation proves the matrix form:
21)
2(NN,2))
Σ, Mar 21 M.r,,220 1
pT Y,
1
Y,
(
P(EN, 2M)
E,p(a,, 2M)
M
M
CyeN
(6)
Proposition 2 (Cross-modal Transferability). After optimizing the multi-modal contrastive loss in- troduced in Equation 3 with infinite encoder dimension D 00, we can find a linear classifier that only uses image representations but is transferable to text representations. More specifically, the weight of the linear layer is M F Yg R DXC which can be viewed as a class-mean mean classifier.
19

---

Published as a conference paper at ICLR 2023
Proof of Proposition After optimizing the contrastive loss, have P FG (Theorem 1 With p'y, MY (Assumption I. we have G(MF"Y, Y,) M(FG") MP Y, Y.
The following equation explains why M MF y is class-meat classifier:
T
=M(((a) ... f(xN)]
M [[. = =] E.I(:)[[v c]]
M
Intuitively, the class-mean linear head is very similar to the trained linear head. For instance, learning the linear head with one step of gradient descent starting from zero initialization would recover the class-mean linear head. We study the class-mean linear head here because it's more amenable theoretical analysis. This provides intuition why cross- modal transferability can be achieved regardless of the modality gap.
B DATASETS AND EXPERIMENTAL DETAILS
In this section, we report details of four datasets: MS-COCO (in et al. 2014). Waterbirds (Sagawa al.) 2020). FairFace (Karkkainen & Joo 2021). and dSprites (Matthey al. 2017). and details of two major experiments.
B. DATA PRE-PROCESSIAE
MS-COCO. We follow the standard MS-COCO dataset split, which includes 118K / 5K images for training/ validation. Each image is annotated with multiple objects from 80 categories and five human-written captions. We randomly select one caption from five captions. Therefore, we have 118K / 5K image-caption pairs with multiple labels for training validation.
Waterbirds. We follow the standard Waterbirds dataset split, which includes 4.8K / 1.2K images for training/ /alidatioo. Data samples be viewed in Figure 6 and
FairFace. We resample the training set using the demographics from the state of Montana, which includes 92.8% White, 6.4% Indian, 0.5% Asian, and 0.3% Black. The final dataset contains 17K / 11K images for training validation. Data samples can be viewed in Figure 6 andd]
dSprites We use our own scripts to reproduce variant of the dSprites dataset and name VS We use six colors (red, pink, orange, green, cyan, blue), four locations (upper left, upper right, lower left, lower right), and three sizes (small, medium, large) to create triangles and squares with a scale ranging from 0.8 to 1.2. Each attribute is uniformly sampled, and synthesize 10K images. We only use 80% orange triangle and 80% green square for training. Finally, has 1.3K / 8. 3.7K images for training/ validation. Data samples can be viewed in Figure 6 bland]]
B.2 ATTRIBUTES
Here we list the known attributes during the data collection of the three datasets. We do not cherry- pick attributes and just use these attributes for our experiments.
Waterbirds.
Two attributes are used: species (200 values) and places (4 places).
FairFace.
Three attributes are used: races (7 values), ages (9 values), and genders (2 values).
dSpritesV.
Three attributes are used: colors (6 values), size (3 values), and shapes (2 values).
B.3 PROMPT ENGINEERING
We use the prompt engineering techniques proposed CLIP (Rafford et al. 202T) for our experi- ments.
20

---

Published as a conference paper at ICLR 2023
Waterbirds. We use "(species), {place)." the raw prompt, and "a photo of a {species) in the {place)." as the engineered prompt. Therefore, we can generate 200 x 4 800 text inputs.
FairFace. We use "(age adjective), [race). {gender)." as the raw prompt, and "a photo of a (race} fage adjective) (gender)." the engineered prompt. Therefore, we can generate 7 x 126 text imputs. The age udjectives are infant (0-2), little (3-9), teenage (10-19), young (20-29), adult (30-39), middle-aged (40-49), senior (50-59), elderly (60-69), and very old (more than 70).
VSpritesV. We use "I "(size), (color), (shape}." as the raw prompt, and "[size) (color) {shape)." the engineered prompt. Therefore, we can generate x 6 x 2 36 text inputs.
B.4 PROMPT ENSEMBLE
We use OpenAl CLIP's 80 prompts (Radford et al. 2021 to augment text inputs by 80 times. Parts of them are shown in Figure
B.5 EXPERIMENTAL DETAILS
Model Details. Unless explicitly stated, we use CLIP (ViT-B/32) for all experiments, encoding images and texts in the same 512-dimensional space. Linear layer maps input dimension 512 the number of classes. Multi-layer perception uses the hidden size as 512.
Cross-modal Transferability Training Details. For each image-caption pair, we use CLIP's im- age and text encoder (Radford al. 202T) to get image embedding and text embedding. We do not use image augmentation techniques during training and inference. We train the linear model or multi-layer perception for 25 epochs using the Adam optimizer with a fixed learning rate of 0.001. During training, CLIP's image and text encoder are fixed. We pick the best model based on the lowest validation loss on the training modality.
Classifiers Training Details. For each image in the dataset, we use CLIP's image encoder (Rad- Ford et al. 202T) to get its image embedding. We do use image augmentation techniques during training and inference. We train the linear model or multi-layer perception for 25 epochs using the Adam optimizer with fixed learning rate of 0.001. During training, CLIP's image encoder is fixed. We pick the best model based on the lowest validation loss images.
Model Rectification Training Details. For each text from error slices generated by attribute com- position and prompt engineering, we use CLIP's text encoder (Radford et al. (021) to get its text embedding. We continue training the pre-trained linear model or multi-layer perception for 10 epochs using the Adam optimizer with fixed learning rate of 0.001. During training, CLIP's text encoder fixed. We pick the best model based on the lowest validation loss texts.
B.6 DISCUSSION: Is USING ATTRIBUTES AN APPROPRIATE CHOICE FOR MODEL. DIAGNOSIS?
In this work, heavily rely on attributes for model diagnosis and rectification. A natural question is what is the rationale for using attributes in these processes, and how much will attribute selection affect the validity of our method.
We first clarify that in our experiments, we did not cherry-pick attributes and just used the known attributes from the data curation process of the three datasets. More broadly, we believe attributes are useful for model diagnosis, because it is not hard to define meaningful set of attributes given a specific task; there are many known attributes given any dataset, and sometimes there may be specific attributes of interest for which we wish to test model vulnerability. For example, for a self- driving car application, we can easily come up with attributes such as weather, traffics, pedestrians, buildings, etc. For any class in ImageNet classification, such guitar, it is straightforward to think about its color, material, location, size, etc. We agree that an initial set of chosen attributes may not be perfect for reflecting all the essential emrrs,, and better attribute selection reveal more model
21

---

Published as a conference paper at ICLR 2023
vulnerabilities. Therefore, this process can be improved with human involvement to iteratively design better attributes based on the model feedback, which can be useful for future works.
While attribute selection is important and may require human involvement, method is still very useful because we provide an casy way to test the model under many cases. Like software testing, there is generally framework for any task without any prior knowledge. We have already significantly reduced the diagnosis cost compared to previous works. Previous works all assume large collection of labeled images is available for model testing, which is unrealistic given the extreme difficulty of collecting diverse image inputs and the cost of data annotation. Our method instead provides a way test sensitive attributes for which you may even have no image data. For any specific task, it is always much easier to come up with a meaningful set of attributes and then generate a large collection of novel text inputs by combining different attributes than collecting corresponding images, thanks to the easy-to-ma pasy-to-manije manipulate and compositional nature of the language modality. More importantly, the combination of defined attributes naturally defines numan-interpretaal based slice discovery methods do not directly provide a text summary of the error slice.
free lunch for model diagnosis, and it is impossible to design a general diagnosis
data slices, whereas image-
Finally, we hope to clarify that one of the main contributions of our work is to theoretically and em- pirically demonstrate a pervasive phenomenon in multi-modal contrastive learning transferability, which allows texts to be effective proxies for images. Our method performance, in terms of correlation strength between model performances on images and corresponding texts, is independent of attribute selection. It is just volvement in this process. Moreover, it is possible to collect a large set of text inputs in a different way to diagnose vision models instead of using the attribute-based combination. For example, one may be able to prompt large language models such as GPT-3 in a few-shot fashion to generate a large set of descriptions of certain classes, and then feed these inputs into vision models for diag- nosis. We leave this to future work. Overall, diagnosing vision models using text modality is much more desirable than image modality, because language enables us easily generate realistic and diverse inputs with better control and manipulation.
moodll
errors can be discovered with more human in-
C BASELINES
C.I LANGUAGE-BASED VISION MODEL DIAGNOSIS BASELINE: TEXT-TO-IMAGE GENERATION
In Figure 5 we show the baseline method to diagnose vision models using language - text-to- to-image generation. The method generates real images to test models using the text-to-image generation model.
From the results, we can understand why this baseline is worse than our method, which does not require explicitly generating images. While significant progress has been made in the text-to-image generation field, state-of-the-art text-to-image generation models (Rombach et al. 2022) still fail to generate fidelity images.
In addition, text-to-image generation is computationally expensive, requiring thousands of more computations than our approach.
C.2 ERROR SLICE DISCOVERY BASELINE: DOMINO
In Figure we show the discovered error slices using the baseline slice discovery method, DOMINO EEuboglu et al., 2022).
In real-world applications, it is unrealistic to assume large set of labeled images from different distributions is available. Therefore, the most critical challenge for slice discovery work, we circumvent the data challenge by using language to synthesize extensive test examples.
is data. In this
C.3 MODEL RECTIFICATION BASELINES: GDRO AND JTT
Our approach rectifies model misbehaviors by correcting the data bias. Another series of methods to tackle these errors robust training techniques, such as GDRO (Sagawa al. 2020) and JTT (Liu
22

---

Published as a conference paper at ICLR 2023
al. 2021). which explicitly optimizes each slice's performance during training. Compared to them, one of the distinct advantages of our approach is that we do not require any visual data during the rectification process. Both GDRO and JTT require image data present in the training set. There- fore, they cannot fix errors on unseen data. GDRO even requires attribute annotations on images, which is highly time-consuming and cost-prohibitive for most real-world applications. Moreover, our method can also be combined with robust training techniques when image data and attribute annotations are available, and leave this to future work.
Here we provide implementation details of GDRO and JTT:
We We reproduce GDRO on our datasets by adopting the official GDRO loss implementation to our code base. We use all the same hyperparameters they use in the paper, where important hyperparameters include l2 penalty strength a 0.2 and group adjustment 7 0.1. We train linear classifier for 25 epochs using the Adam optimizer with a fixed learning rate of 0.001. During training, CLIP's image encoder fixed. We pick the best model based on the lowest validation loss.
GDRO.
JTT. We reproduce JTT the same hyperparameters they use in the paper. We also perform a hyperparameter search on the upsampling weight Aup E (5, 20, 50), which is very important hyperparameter based the paper. The best Aup is 20 for Waterbirds and 5 for FairFace. We train linear classifier for 25 epochs using the Adam optimizer with a fixed learning rate of 0.001 for round and round During training, CLIP's s image encoder is fixed. We pick the best model based on the lowest validation loss.
datasets by implementing the algorithm by ourselves. We use all
23

---

Published as a conference paper at ICLR 2023
=
lambda c: f"a bad photo of a (c).",
lambda f"a photo of many (c).".
lambda c: f"a sculpture of a (c).",
lambda c: f"a photo of the hard to see (c).",
lambda c: f"a low resolution photo of the (c).",
lambda c: f"a rendering of {c].",
lambda c: #"graffiti of (c).",
lambda c: s"a bad photo of the (c).",
lambda c: I"a cropped photo of the (c).",
lambda C: f"a tattoo of a (c).",
lambda f"the embroidered (c).",
lambda c: f"a photo of hard to see (c).",
lambda c: f"a bright photo of (c).",
lambda c: f"a photo of a clean {c).",
lambda c: f"a photo of dirty {c).",
lambda c: f"a dark photo of the (c).",
lambda c: s"a drawing of (c).",
lambda c: i"a photo of my {c).",
lambda c: :"the plastic (c).",
lambda C: f"a photo of the ccol (c).",
lambda c: f"a close-up photo of a (c).",
lambda c: f"a black and white photo of the (c).",
lambda c: f"a painting of the (c).",
lambda c: f"a painting of (c).",
lambda c: f"a pixelated photo of the (c)."
lambda c: f"a sculpture of the (c).",
lambda c: f"a bright photo of the (c).",
lambda c: f"a cropped photo of a (c).",
lambda c: f"a plastic (c).",
lambda c: f"a photo of the dirty (c).",
lambda c: f"a jpeg corrupted photo of a (c).",
lambda c: i"a blurry photo of the (c).",
lambda c: t"a photo of the (c).",
lambda c: t"a good photo of the (c).",
lambda c: f"a rendering of the (c).", ",
lambda c: f"a {c} in a video game.",
lambda c: f"a photo of one (c).",
lambda c: f"a doodle of a (c).",
lambda c: f"a close-up phote of the (c).",
lambda c: f"a photo of a (c).",
lambda c: f"the origami (c).",
lambda c: g"the (c) in a video game.",
lambda c: f"a sketch of a (c).",
lambda c: f"a doodle
the (c).", lambda c: f"a origami (c).",
lambda c: f"a low resolution photo of a (c)."
lambda c: #"the toy (c).",
lambda c: f"a rendition of the (c).", ."
lambda c: g"a photo of the clean (c).",
lambda c: f"a photo of a large (c).",
lambda c: f"a rendition of a (c).",
lambda C: f"a photo of nice (c).",
lambda c: É"a photo of weird (c) "
lambda c: :"a blurry photo of (c).",
lambda c: f"a cartoon (c).",
lambda c: g"art of a (c).",
lambda c: t"a sketch of the (c).",
lambda e: g"a embroidered (c).",
lambda e: s"a pixelated photo of a (c).",
]
Figure 4: A subset of the 80 prompts form OpenAl we use to augment text inputs for prompt ensembling.
24

---

Published as a conference paper at ICLR 2023
Text prompt
Well generated examples
Poorly generated examples
photo a Chuck Widow in the lake naturall"
photo Shiny Cowbird in the bamboo forest."
Well-g generated examples (columns 2-3) are realistic and correctly reflect the bird species and background location described in the text prompts. Poorly generated images either do not include the bird species (column 4) are noticeably unnatural (column 5). Specifically, the top image in column 5 includes bird with lobster claws as head a rather unusual phenomenon the world
Text prompt
Well generated examples
Poorly generated examples
'The face of a white adult woman.
'The face of east assian teenage boy."
Well- generateerd examples (columns 2-3) realistic and correctly reflect the ethnicity and age group described text prompts. Poorly generated images include more than one person (column 4), do not include the person's 's face (columa 5 bottom), or are noticeably unrealistic (column 5 top). For instance, the top image in column 5 includes woman with arms protruding out of her chest, which is rare in the real world.
Text prompt
Well generated examples
Poorly generated examples
"Large green triangle."
'Small red square."
Liteng DEL
Well-generated examples (columns 2-3) correctly reflect the shape and color described in the text prompt. Poorly generated images either include incorrect shapes (column 4-5 top) or are misinterprcted due to polysemy (column 4-5 bottom). For instance, the phrase "red square" was misinterpreted by model historic site in Moscow.
Figure 5: Text-to-image generation results on Waterbirds, FairFace, and dSpites using the state- of-the-art generation model (Rombach et al. 2022.
25

---

Published as a conference paper at ICLR 2023
Images With the Correct Class Prediction Probability in Each Slice
Top DOMINO Generated Text Descriptions For Each Slice
photo of the coast guard
DOMINO Discovered Top Worst Performing Slices
photo coast
photo coast guard
photo horned
Slice
2. photo the forest
3. photo of redwood
1. photo the foresl
Slice
2. photo of horned ow photo
For in-distribution Waterbird, the text descriptions generated from DOMINO do not elucidate which attributes are causing misclassifications. addition, some the descriptions are incorrectly generated. instance, "coast guards" are not present in any the photos Slice 1.
Top Images With the Lowest Correct Class Prediction Probabillyy Each Slice
Top DOMINO Generated Text Descriptions For Each Slice
1. photo an orphan girl
Top Worstteerrrriiig Slices
Slice
2. photo girl
photo girl
DOMINO Discovered
photo
birtdday
Slice
photo sightionth
birlidyy
3. phoso of young girl
a photo mystarious
Slice
2. a photo of compallnng man
3. photo of an undisclased
For in in-distribution Fairface, DOMINO is unable to discover the correct error slices, Instead of slices for minority groups, DOMINO discovered slices for the prevalent ethnic group in the dataset. Furthermore, the generated descriptions failed to include the key attribute race.
Top Images With the Lowest Correct Class Prediction Probabiltt
Top DOMINO Canerated Text Descriptions
1. photo the slogan appears
Top 3 Worst Pwrtarreertrmmiii Performing Sloee
Slice
photo the photo
DOMINO Discovered
photo outtolldrrs
photo lodge
3. photo masonic
photo height difference
Slice
2. photo plumber
3. photo the alphabet
For in-distribution VS DOMINO failed to discover the most critical error slices slices with spurious correlations (orange squares & green triangles) and slices with unseen data (pink triangle). The generated descriptions also do reflect images in the slices.
Discovered error slices in-distribution Waterbirds, FairFace, and dSpitesV datasets using the ofte-oof-the-an slice discovery method DOMINO (Eyuboglu et al., 2022). DOMINO is unable to discover most of the top error slices. The generated text descriptions often do reflect images in the slice accurately.
Figure 6:
26

---

Published as a conference paper at ICLR 2023
Images With the Correct Class Prediction Probability in Each Slice
Top DOMINO Generated Text Descriptions For Each Slice
photo the forest photoo mammooth
Top Worst Performing Slices
DOMINO Discovered
photo
Slice
2. photo of forest
3. photo the woodland
a photo of the forest
Slice
2. photo a homed owi
For cut-of-distribution Waterbird, DOMINO unable generate text that clearly defines the of misclassification slice. Moreover, some of descriptions not accurately describe images. For example, "mammoths" are never included in any images.
Top Images With the Lowest Correct Class Prediction Probabillyy Each Slice
Top DOMINO Generated Text Descriptions For Each Slice
1. photo of boy photo smell photo younger brother
Top Worst Pertorteerorriing Slices
Slice
DOMINO Discovered
orphan
Slice
girl
3. photo a creale girl
photo her eighleenth birthday
Slice
2. photo our little girl photo lottle
For out-of-distribution Fairface, DOMINO is able discover the generated descriptions failed to provide any insights why the model fail for these slices. Specifically, the generated descriptions not include keywords for the race attribute.
slices. However, the
Top Images With the Lowest Correct Class Prediction Probabiltty
Top DOMINO Canerated Text Descriptions
1. photo national
Top 3 Worst Portoreertmmmiinn Performing Sloee
Slice
goographic society photo
photo the upgrade logo
DOMINO Discovered
photo
photo orange
photo nickelodson lago
photo the
Slice
2. photo the emerald neckiace
3. photo
VSprites DOMINO is able to discover slices with spurious correlation (orange squares & green triangles). However, the generated descriptions do include attributes that contribute the spurious correlation. Additionally, DOMINO not discover slices with unseen data (pink triangles).
For
Figure Discovered error slices on out-of-distribution Waterbirds, FairFace, and dSpites datasets using the state-of-the-an slice discovery method DOMINO (Eyuboglu et al.) 2022). DOMINO was able capture some, but not all, error slices. Furthermore, artificially generating nut-of-distribution data for evaluation remains challenging in real-world settings.
27


Reviews:
[{"review_id": "hHsNjKtALw", "reviewer": "Reviewer_Wtjn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed approach exploits an empirical property of the latent space of multimodal embedding (essentially image & text) revealed in (Liang et al., 2022): the fact that the two modalities are embedded in different parts in the latent space, inducing a modality gap. The paper extends the results by empirically stating that the two embedded modalities are correlated by a quasi constant shift, easing cross-modal transferability, and that the data manifolds are orthogonal to the modality gap.The multi-modal correlation is used to analyze the quality of vision based prediction on three tasks: discovering hard examples described by sets of attributes (error slices), scoring attributes by their impact on performance using Shapley  values, and by fine tuning a given model using text data augmentation. \n", "summary_of_the_review": "The paper pursues the analysis of the modality gap initiated by (Liang et al., 2022) and discusses another finding: the almost constant distribution of this modality gap and its geometry with respect to the data manifolds. It proposes three original tasks that exploit the results. They are, however, mostly empirical, and could have been verified with better statistical justification; the analysis of the tasks is also not very thorough.\n", "strength_and_weaknesses": "== Strengths ==\n- The paper is well written and easy to follow.\n- Well justified exploitation of multimodal embedding and of their empirical properties \n- Application on three rather original tasks using cross-modal transferability\n\n== Weaknesses ==\n- The findings (structure of the embeddings) are empirical and not justified by the (contrastive) learning scheme.\n- The three tasks are more illustrations of the multimodal embedding properties than complete studies: they are not fully developed or analyzed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a5JMb1FpJL", "reviewer": "Reviewer_LGzN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explains why cross-modal transferability happens even if there is a modality gap. Authors have shown this empirically on various datasets and have also theoretically proved it as well. By proving this, they demonstrate that text can be a good proxy for images. Based on this assumption, they introduce a new framework, DrML, which can discover error slices, identify influential attributes and correct misbehavior of the model.\n\n", "summary_of_the_review": "This paper demonstrates their assumptions about cross-modal transferability in several ways, and experiments are also conducted on several datasets. Also, they introduce a new framework, DrML, that can rectify the model’s misbehavior. In sum, I think the proposed idea looks novel and may shed new light on diagnosing models. However, there are some questionable points when understanding this paper. I think it could be accepted if these are improved.\n", "strength_and_weaknesses": "**Strength**\n- They prove that the phenomenon of pervasive cross-modal transferability occurs when the modality gap meets certain conditions by empirically and theoretically.\n- It was made human-interpretable by explaining the model's error using languages as well as images.\n- The proposed framework further rectifies the behavior of the models. \n\n**Weakness**\n- The explanation of the process of correcting the misbehavior of the model using language may need more detail.\n- Few parts of the proposed framework are not justified well (Details are in Clarity 2.)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_ywwTeUNIPZ", "reviewer": "Reviewer_yEKt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on observation in Liang el al., 2022 on the modality gap, the paper further proves that the modality gap in multi-modal settings has not influence on the prediction of classifiers, and thus enables cross-modal transferability. The papers further shows that texts can be used as a proxy for image inputs, and provides a method (DrML) to use natural language to identify data subsets (error slices) and the most influential attribute. Their approach is validated on 3 benchmark datasets and shows at some certain level being more convenient than given baselines since this method can diagnose models without requiring diverge visual data collection/creation. The paper further presents a method to rectify undesirable model behaviours based on generating a larger language dataset related to the error slice and continue model training on these.", "summary_of_the_review": "The paper introduces a novel idea of how to use natural language to diagnose vision classifiers. This method can be understood easily by practitioners, and addresses the lack of out-of-distribution data in one modality. However, there are still some concerns w.r.t. to the method and comparison to SoA.\nTo the best of my knowledge, this is the first work being able to use natural language to discover error slice of vision classifier and extending language data instead of vision data to correct data bias problem. Other works do research on multi-modal and the phenomenon of modality gap but not emphasize on purpose of diagnosing model. My biggest concern is that author does not provide any proper reasons how they could pick the attribute set. The choice of the attribute set may affect the method performance significantly. In their experiments on the effectiveness of the rectifying method, the paper misses the comparison to related work, they use only one other method (GDRO), while stronger baselines are available.\n\nEdited after author rebuttal: \nI appreciate the authors effort on addressing my concerns and their improvements of the paper. Specifically, the authors added a comparison to SoA (experiments) and a discussion on attribute selection (in the appendix). I adjusted my rating. I would appreciate a short note on the manual need for attribute selection in the main body of the paper. ", "strength_and_weaknesses": "Strengths\n\n[S1] Ideas are presented with full examples and explanation then it makes paper easy to follow.\n\n[S2] The main contribution of the paper is the theoretical and empirical verification  of the ability to use text as a proxy for image inputs.\n\n[S3] The novelty of the idea to use natural language (which is deemed to be inherently interpretable) to diagnose machine learning models and at the same time addressing the problem of lacking diverge real life vision data.\n\nWeaknesses\n\n[W1] Prior knowledge needed to select a suitable attribute set A. \n\n[W2] Missing comparison to SoA. \n\nDetails\n- The way of choosing attribute set A initially could be a concern. It seems that the degree of diversity and size of the generated text input set partially depends on the chosen of attribute set A. Without any prior knowledge of spurious correlation within the dataset, it would be hard to choose a really 'related' set A. For example, in the dataset WaterBird, it is widely known the spurious correlation is the bird class and the background, then the attribute was picked are 'bird species’ and ‘places’. In the case of unknown prior knowledge, one might fail to point out the expensive attributes.\n- It is hard to judge whether the method has ‘diagnosed’ vision classifiers thoroughly. The authors compare to GDRO which (requires annotations) and  focuses on increasing the worst-group accuracy, which is not addressed by Rectify. Only generating more data on error slice and keep continue training is not expected to help when a model is already biased. There are other works that the authors reference, such as Just Train Twice (Liu et al, 2021), which also do not require annotations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y034h9pHHLV", "reviewer": "Reviewer_n4R4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "summary_of_the_review": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "strength_and_weaknesses": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster