Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift
Authors: Yihao Xue
Siddharth Joshi
Dang Nguyen
Baharan Mirzasoleiman
Venue: ICLR 2024 poster (2024)
Keywords: multi-model contrastive learning
contrastive learning theory
representation learning
theory
robustness
distribution shift
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP, have achieved a remarkable success in learning representations that are robust against distribution shift and generalize to new domains. Despite the empirical success, the mechanism behind learning such generalizable representations is not understood. In this work, we rigorously analyze this problem and 
uncover two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which allows the model to learn features with a high variance, and \emph{inter-class feature sharing}, where annotated details in one class help learning other classes better. Both mechanisms prevent spurious features that are over-represented in the training data to overshadow the generalizable core features. This yields superior zero-shot classification accuracy under distribution shift. Furthermore, we theoretically demonstrate the benefits of using rich captions on robustness and explore the effect of annotating different types of details in the captions. We validate our theoretical findings through experiments, including a well-designed synthetic experiment and an experiment involving training CLIP models on MSCOCO/Conceptual Captions and evaluating them on shifted ImageNets.

Published as a conference paper at ICLR 2024
UNDERSTANDING THE ROBUSTNESS OF MULTI-MODAL CONTRASTIVE LEARNING TO DISTRIBUTION SHIFT
Yihao Xue, Siddharth Joshi, Dang Nguyen, Baharan Mirzasoleiman
Department Computer Science, University of California, Los Angeles
y yihacxuelg.uue. ucla edu, s joshiis 8048ca uul a edu, nguyentuuahaiidaaggggmaaa com, baharanêcs www.ccllll... edu
ABSTRACT
Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP (Radford et al., (202]), have achieved remarkable success in learning represen- tations that are robust against distribution shift and generalize to new domains. Despite the empirical success, the mechanism behind learning such generalizable representations is not understood. In this work, we rigorously analyze this problem and uncover two mechanisms behind MMCL's robustness: intra-class contrasting, which allows the model to learn features with a high variance, and inter-class fea- sharing, where annotated details in class help learning other classes better. Both mechanisms prevent spurious features that are over-represented in the training data to overshadow the generalizable core features. This yields superior zero- shot classification accuracy under distribution shift. Furthermore, we theoretically demonstrate the benefits of using rich captions on robustness and explore the effect of annotating different types of details in the captions. We validate our theoretical findings through experiments, including well-designed synthetic experiment and experiment involving training CL.IP models on MSCOCO (Lin et al. /C14 ceptual Captions (Sharma et al. 2018 and evaluating them on shifted ImageNets.
INTRODUCTION
Learning classifiers that generalize under distribution shifts and across various domains has long been a challenge in machine learning. A key reason is that modern models are highly susceptible to learning simple, domain-dependent spurious features in the training data instead of more complex
generalizable features (Zhu et al.) 2016 (Sagawa et al.) 2019: Xiao et al. 2020 Taori et al. (020). Recently, Multimodal Contrastive Learning (MMCL) has demonstrated significant robustness in zero-shot image classification. It trains vision and language encoders using contrastive loss to align representations of paired images and text, while pushing apart the representations of images and texts from different pairs (e.g., CLIP (Radford et al. 2021), ALIGN (Jia al. 202T))
Radford et al. 2021 have shown that models trained with CLIP, an exemplar of MMCL algorithms, (OOD) generalization (c.f. their Fig 13). Specifically, CLIP- trained zero-shot classifiers achieve higher OOD accuracy compared to classifiers with equivalent In-Distribution (ID) accuracy that are trained using various supervised learning techniques, including existing robust methods. Interestingly, the advantage of CLIP diminishes if any label supervision is introduced, e.g., through linear probing with labeled data on CLIP's image encoder (c.f. Fig 14 of (Radford et al. 202Th). Both findings suggest that the zero-shot classifier produced by CLIP is more robust distribution shifts than any classifier trained with label supervision.
exhibit better
However, comprehensive understanding of the reasons behind does not yet exist in the literature. Existing theoretical studies have only examined MMCL's in-distribution generalization (Nakada et al. 2023 Zhang et al.) 2023a), but have not explored its OOD-robustness. In this paper, we aim to explain how CLIP, and more broadly MMCL, produces zero-shot classifier with superior robustness, while demystifying the contributions of MMCL loss and image captions. This study is conducted by comparing the zero-shot classifier learned through MMCL with classifiers learned via supervised learning. The latter is arguably representative of all other non-zero-shot methods of classifier training, as supervised learning is essentially involved (e.g., end-to-end supervised learning, and linear probing on representations learned by unsuperriieed/sefff-upeeveee algorithms).

---

Published as a conference paper at ICLR 2024
Specifically, we demonstrate that the MMCL loss accompanied by rich image captions enables at least two mechanisms providing robustness to zero-shot classification. (1) the intra-class contrasting between image-text pairs within the same latent class enables MMCL to easily learn generalizable features with a high variance, when they are annotated by text. In contrast, SL. is highly prone to learning simple spurious features instead of the more generalizable features with higher variance (Sagawa et al. 2020). For example if the majority of cow images with diverse shapes and colors appear on simple green grass background, SL learns the grass to predict 'cow', but MMCL successfully learns the cow; (2) the inter-class feature sharing enabled by MMCL loss allows learning information about a class that only exists and is annotated in other classes. For example, all the images of the tree class have green leaves but an image in the wolf class has a tree without leaves in its background, MMCL can disassociate the green leaf from the tree class. In contrast, SL cannot leverage this information and learns the green leaves as an indistinguishable part of 'tree'. Hence, it fails to classify trees without green leaves in the test data. Both mechanisms together enable MMCL to learn representations that are robust against distribution shift and generalize to unseen domains.
Furthermore, to emphasize the importance of captions, we analyze the effect of varying caption richness and show that rich captions are essential for achieving robustness. As extreme case, if the captions are merely labels, no gains in robustness can be achieved.
We further support our theoretical findings with experiments, including well-designed synthetic experiment and experiments on real datasets, including MSCOCO, Conceptual Captions, and shifted versions of ImageNet. The results demonstrate the crucial roles of the MMCL loss function and rich captions in achieving robustness, further validating our theoretical findings.
2 RELATED WORKS
Distribution shift. There is a long line of work on dealing with different types of distribution shift.
This includes sub-population shift and domain generalization among others, where distribution of sub-populations in training and test data is different, and some sub-populations may be
underrepresented or missing in the training data (Cai et al. 2021: Yang et al. [002]] Santurkar
cet al. 2020), (Gulrajani & Lopez-Paz 2020 Joshi et al. 2023 Zhang et al. 2023b Hu et al. 20200]
Fahrbach et al. (023), or hybrid of both Koh al. (021)) Another line of research focuses on
evaluating models on natural variations in the source of data collection. with the precise category
of shift typically unformalized or unknown. For example, a dataset that contains art and cartoon renditions of ImageNet classes (Hendrycks et al. (02Ta), and other variations of ImageNet (Barbu) jtt al. 2019; Recht et al. 2019 Shankar et al. (021). Despite the diversity of settings, extensive studies (Sagawa al. 2019 2020 Xiao et al. 2020 [lyas et al. (019) revealed common theme across these subfields: deep learning models often rely heavily on spurious correlations that are specific to the training data but do not hold in general, e.g.. those between certain object classes and backgrounds/textures in the image (Zhu et al. 2016 Geirhos et al.) 2018).
Multi-modal (contrastive) learning. Learning better representations based on multiple modalities has been a long pursuit (Ngiam et al. 2011 Srivastava & Salakhutdinoy 2012). Numerous methods for learning joint
representations (Li et al., 2019 Lu et al. 2019 Tan & Bansal 2019 Li et al. 2020; Yao et al. 2021) have emerged. Among them, MMCL (Radford et al. 2021 Jia al. 2021 Mu et al. 2022 Goel et al. 2022 Pham al. (023) has stood out by achieving SOTA performance in various tasks. Notably, (Radford et al. (021)) showed that MMCL on large image- text datasets achieves a significant improvement in robustness to distribution shift. The empirical investigations of (Fang et al. (022) suggests that this is only attributed to the large diverse image training data, with MMCL loss and text supervision contributing little. We show provably that is not only the diverse image data that contributes to superior robustness of MMCL. Indeed, MMCL. loss and richness of text annotations are crucial factors.
3 A FRAMEWORK FOR COMPARING MMCL AND SL
In this section, we present a general framework for comparing MMCL and SL, along with the corresponding notations. We start by modeling the multimodal data, and then formalize the MMCL and SL pipelines and their evaluation for robustness to distribution shift. We will formulate and analyze specific types of distribution shift in the next section.
2

---

Published as a conference paper at ICLR 2024
3.1 MODELING MULTIMODAL DATA
To model multimodal data, it is essential to capture the fact that inputs from different modalities can represent the same abstract notion. For instance, both text and an image can represent 'a cow on grass'. We define underlying, feature vectors to model this abstract notion, and model each input in specific modality as projection of the underlying feature vector onto that modality's input space.
Underlying feature. There is an underlying feature space shared among different modalities, where abstract notions reside. We model this as a vector space IR', where each vector termed an underlying feature vecior (e.g.. 'a cow on grass"), )) and each element within the vector is referred to as an underlying feature (e.g.. 'cow").
Latent classes and labels. Each z is associated with latent class, represented by a label y. We note that the labels are only used by SL but not by MMCL.
Inputs in each modality. Each input example in modality is an instantiation of an abstract notion. We model this as a projection from an underlying feature vector to another space where this modality's inputs live. Formally, let M represent modality. Given underlying feature vector a corresponding input aM in this modality is generated as: z æM = D DM (z) + EM. where JM (2) E RR' is a random vector that depends on It can be interpreted as possibly distorted version of the original feature vector z. Note that setting Ma (z) = z implies no distortion in the features when represented in
o3 ld The matrix DM E RR this modality. M E Rd is random noise drawn from (0,
(dar > 1) is a matrix with orthonormal columns that can be interpreted as a dictionary matrix. It captures the transformation from the lower dimensional feature space the higher dimensional input space. Different modalities can have different D MM matrices, reflecting the idea that the same underlying feature may be instantiated differently in each modality (e.g.. colors are represented differently in images and texts). Modeling modalities as above is consistent with (Nakada et al.) (023))
In this paper, for clarity and illustration, we focus on the popular vision and language modalities. We let I denote the modality for images, and T denote the modality for texts. However, we note that our framework and results directly apply to other modalities.
Distribution shift. We define two joint distributions between underlying features and latent classes: 9t, representing the 'ground-truth' in the real world, and goT', from which our training data are drawn. We let 90r exhibit spurious correlations between certain features and latent classes which do not hold in the ground-truth g0* This setup captures the underlying reason for the performance drop observed in various types of distribution shift scenarios, as we will discuss in Section 2
3.2 MOLTI-MODAL CONTRASTIVE LEARNING (MMCL)
Unlike traditional supervised learning, MMCL does not see the input data's latent classes, but is instead given pairs of inputs from two modalities and aims to learn the correspondence between them.
Training dataset. The training dataset comprises n image-text pairs, denoted as { D where for each index i, both I and æτ are generated based on the same underlying feature vector z, In practice, the texts are usually captions accompanying the images. The feature vectors J are drawn from the training distribution T
Linear encoders. The encoders for modalities I and T are denoted gr Rd R and Gr : IR dr > RR respectively. We consider linear models for the encoders, given by (z) W i and gT (x) W 72, where W, , E R xp and WT E R xp with p 2 / are the corresponding encoder parameters. Linear encoders are employed widely in previous studies of MMCL (Nakada
al. 2023 Ren & Li 2023) and general feature learning (Jing et al. 202T Tian al. 2021 Ji et al.
2021 Wu et al. 2022: Tian 2022 Xue et al.) (023). to facilitate the analysis. In Section
6 we will
empirically confirm that our findings extend to lonelinear models.
Representation learning with MMCL. MMCL learns representations for both modalities in a shared latent space. We consider the linearized contrastive loss function from Nakada et al. (003)::
LMMCL (W,,wr) )) 2n(n - 1)
1
ΣΣ
Ssy
si)) + 2n(n 1)
Σ
Σ ($ sa) & P W W 12
jti
jti
where $;j = (z),))) (T( = (W ,)) W WETj
j is the similarity (measured by inner product) between representations of an image and text. This loss encourages the model to align each
3

---

Published as a conference paper at ICLR 2024
image-text pair by increasing their representation similarity (stz) and contrast between images and texts that are not paired together by reducing their representation similarity (s1y, i * j). The last term is a regularization term with p > 0. The linear loss and its uni-modal counterpart are widely used in analysis of CL., as they closely captures the dynamics of popular contrastive losses (Ji et al. 2021
Tian 2022 Nakada al., 2023), such as CLIP,
will experimentally confirm in Section 6
Prompts for zero-shot classification. We test the model's capability in zero-shot classification, where text prompt is created for each label (e.g.. 'a photo of dog'). and the prediction determined by the prompt with the highest representation similarity with the given image. To formalize this, we define the prompt Py for each latent class y as Py Dr where Zy == ( ge [z|y]] That is, the prompt is 'the center of all underlying feature vectors with label in the true distribution' represented modality T. This closely resembles real world practices where the representation of
multiple texts with engineered templates like 'a bad photo of O),, , a good photo of a are averaged (Radford et al. 202T).
Robustness evaluation. Given two encoders gr and gT with parameters W, and WT, respectively, we evaluate the zero-shot performance on the true distribution 3t. Given an image 2r, the prediction ((x)) = arg maxy 91 (21) gr (Py)) The test accuracy, denoted by Ace (W,, Wr),
MMCL (,,, Wr) (E Diiu)) (1 (0(z)) =)], Acc ge+
where (.) denotes the indicator function.
Relation to feature cross-covariance. We utilize the connection between the cross-covariance between images and captions, and the MMCL objective for our analysis.
Definition 3.1. We define C as the cross-covariance between images' and texts' feature vectors
C = EZerTr, ms((), (r(z) [1 (2) When pz(-) and pr(()) both are identity, T is the covariance of the feature vector.
Lemma 3.2 (Informal). Given image with feature μt' and text with feature μ², the similarity (inner product of representations) between them, computed using encoders trained on the training Cij plpu".
set, is: similarity score = pt CTT E
That is, the image-text similarity is a weighted sum of products between the features in image and text inputs. The weights determined by the feature cross-covariance matrix of training data, whose i j-th element is the covariance between feature i in images and feature j in texts.
Importance of zero-shot. We emphasize that using zero-shot classification instead of training a linear classifier on the representations is crucial for achieving robustness in MMCL. The latter essentially involves SL, which falls short for the same reasons as shown our analysis for SL in Section 4
3.3 SUPERVISED LEARNING (SL)
Standard SL. has access to each input's s label and the inputs are from single modality (i.e., images). Let
be the training dataset with n inputs æ1,; and their labels yı. we train linear model f(z) = W x with weights W E xa, where q for binary classification and q = #classes for multi-class classification. We consider minimizing logistic loss for binary classification, and Cross-Entropy loss for multiclass classification, with gradient descent at a sufficiently small step size.
Robustness evaluation. Given a model with weights W. the accuracy, denoted by Ace ge. (W), is evaluated on the true distribiution g as Acc (W) (=.) = [1 = y)], [ T æı}; for multi-class classification, with [ [Tæzl, denoting the j-th element in the vector W xT.
where (zI) = si x1,) for binary classification, and ((x) = arg max,
4
TWO MECHANISMS BEHIND THE ROBUSTNESS OF MMCL
Next, we explore two scenarios illustrating MMCL's superior robustness to distribution shift, compared to SL. First, we consider the scenario where generalizable core feature has a higher variance than domain- -opendede spurious feature. Then, we consider the data distribution where each latent class has core feature, that co-occurs with strong spurious feature in the training data. These features can occur in other latent classes as well, independently of each other. For clarity, set (2) 2, (2) 2, Vz E RR in this section.
4

---

Published as a conference paper at ICLR 2024
4.1 ROBUSTNESS VIA INTSS CONTRASTING
We start by analyzing the first scenario which illustrates how MMCL can learn features that are chal- lenging for SL learn. Consider the case where the majority or all images of a 'cow' appear on 'grass' Here, grass is a spurious feature with high correlation with cow. Grass is often a simple green surface without high variation. But, cows can vary lot in their appearance. This makes cows more difficult to learn than grass. Below, we will formalize this scenario and demonstrate that SL learns the spurious feature (grass) but MMCL learns the generalizable feature (cow) and obtains a superior robustness.
4.1.1 DISTRIBUTION OF FEATURES
The following definition simulates the aforementioned scenario.
Definition 4.1 (Data Model I). In both 3 and glv, each label is uniformly drawn from {-1,1} and the corresponding feature vector z E R2 is generated as = [Zcores Zspul IT where Zcone ~ NNy, d represents the core feature that contains information of the label y, and Zapu ~ N(a, oyuu): In the true distribution Pr, is uniformly drawn from {-1,1} and is independent of the label y. making the feature Zypu irrelevant to the label. However, in the training distribution TTT, there is strong correlation between a and y, S.I. Pr(a = y) Popu where 1 2 PPpa > 1/2.
Recall from Section 3.I that the inputs in each modality are generated based on feature vectors. In SL, where we have only one modality. the situation becomes equivalent to the one analyzed in (Sagawal eet al. 2020). Similar variants are studied in (Wald et al., 2021 Aubin et al. [021]] Yao et al. 2022) to investigate distribution shift and oft-of-domain generalization. Despite its simplicity, this setup reflects key aspects of general distribution shift. Here, Zcore is the core feature and Zspu is the spurious
feature, such as 'grass' in the aforementioned example, or texture/backgrounds in ImageNet.
We assume that the core feature has a larger variance than the spurious feature, indicated by the values of core and spu- This detailed in below, along with some additional assumptions.
Assumption 4.2. The gap between the variances of the core and spurious features is significant: o(1).
Core (1), Ceore 2 1 and spu = ( ( The spurious correlation is large: Pipu
We consider the high-dimensional (overparamettrizee) setting where = (1),, di = (n) and dr (()). The noise levels are not too large: de, O(log n) and øg. O(log n).
4.1.2 COMPARING ROBUSTNESS OF SI. AND MMCI.
Under Assumption 4.2 SL tends to associate labels mostly with the spurious feature, they appear to be more stable and reliable for prediction compared to the core feature. This results in low accuracy when tested on the ground-truth distribution, as demonstrated in the following theorem.
Theorem 4.3 (Theorem from (Sagawa et al. 2020)). Let represent the model trained using SL as described in Section 3.3 Assuming that Assumption 4.2 holds, and n and di are sufficiently large, with high probability, the accuracy W' on the true distribution satisfies Acc SL (W*) < 2/3. Ad- ditionally, the model's 's test accuracy on examples where a # y is S 1/3, worse than random chance.
Next, we examine MMCL. From Lemma 3.2 we know that the similarity between an image
1 + o 2Psp
1
with feature and text with feature z' is approximately [Zcore Eapa]
2pppu - 1 + 0 spm
spu
showing that the variance of features ensures that image and text features, that share the underlying
core features, have higher similarity score. Furthermore, if we let z' be the feature Zn- TV< 0 in label y corresponding prompt Py', we deduce that the similarity to the prompt is approximately (1 + o )y' Zoore + (2pspu - 1) 1)y' Zspa- Here, the core feature carries more weight when the variance is large. In essence, since the MMCL loss contrasts images and unpaired texts in the same latent classes, learning features that have high variance is encouraged; this is in contrast with SL, where features that have low variance are preferred. With the above observation, after bounding the effect of noise, we arrive at the following theorem (with proof in Appendix C.2).
Theorem 4.4. Let Wi and W;; be the weights of the encoders trained using MMCL as described in Section 3.2 Under Assumption 4.2 with high probability of at least O poly(n)) 1 1 o(1),
'The theorem holds under more relaxed assumptions about the variances and spurious correlation level; see details in Appendix C.2. but here we use Assumption 4.2 to keep consistency with Theorem 4.3
5

---

Published as a conference paper at ICLR 2024
the encoders achieve the following zero-shot accuracy the true distribution
1
(W W Ae
MMCL
21 ① (K1)
T (K2)) - o(1), 2 2
K2 o1 p + 12p o and 4 denotes the CDF of the standard normal distribution, Meanwhile, the model's test accuracy examples where a # y is lower bounded by ((11) ( ((1).
where K1 (wn) o (2Pgg 1p
20m 22 Con
29 o.
AWL Corollary 4.5. With core 1, for sufficiently large n,, A. (W;, (7) 2 81%, Moreover, in this case, no model can achieve an accuracy higher than 85%.
This, compared with Theorem 4.3
demonstrates that MMCL can outperform SL by a large margin, and comes close achieving the best possible accuracy of 85%.
Additionally, in terms of performance on examples where the spurious correlation does not hold, i.e., y, it's evident that MMCL excels. As Theorem 4.3 shows, SL's accuracy is even worse than random chance. In contrast, Theorem 4.4 demonstrates that MMCL consistently performs better than random chance. It maintains random chance even in the worst-case scenario, as indicated by @ ((Ka) S 1
2. owing to 2ps 2 dir < 0. When cone = 1, it achieves an accuracy of 69%.
4.2 ROBUSTNESS VIA INTER-CLASS CLASS FEATURE SHARING
Next, consider the second scenario and demonstrate how MMCL benefits from annotated details in some latent classes to disassociate spurious correlations in other latent classes, while SL fails to grasp these details. For example, typical images of 'tree' have green leaves. However, trees in in the background of images of "wolf" or 'ski resort' may appear without leaves. SL, which only observes the labels, tends to overlook the trees without leaves as they do not contribute learning "wolf' and "ski resort', thus incorrectly correlating trees with the color green. In contrast, in MMCL, if the trees without leaves are annotated in the captions, the model can disassociate the green leaves from tree.
4.2.1 DISTRIBUTION OF FEATURES
We first present the underlying feature distributions and then compare MMCL's robustness with SL.
Definition 4.6 (Data Model 2). True distribution 3r. We have 2m latent classes in total, with labels 1..... 2m. For each label y. we define unique alias (k, c): = [(y + 1)/2), and c 1 if y is odd, and c c= -1 if is even. The label is sampled uniformly. Let B E (0. Given a label alias (ks, c), the corresponding feature vector z = [21, 22, 22m) generated as:
Vj S m. Vj m,
if k then zj if j = + m then 2, ~ ~(({-a+aa)
if + then 2j [[{-3,+83
if # k + then 2j ~ U((-Box, Box))
where USS) denotes the uniform distribution over set S.
Training distribution TTr The training distribution similar to the true distribution, but with Zk+m always equal to co, making it appear as if the l + m-th coordinate also indicates the label.
Here, each feature vector in latent class (k, c) (e.g.. 'tree') has core feature at coordinate k (characteristics of tree) and a spurious feature coordinate k + m that correlates with the latent class in the training distribution but not the true distribution (e.g., the color green). With a large such a spurious feature has a larger magnitude than the true feature, making it easier to be learned. There also other features at different coordinates that do not correlate with the label; these features are weaker (indicated by 3 < 1) so that they do not change the latent class. One observation that examples in latent class other than (k, c) would show correlation between the k-th and k + m-th features, hinting at their independence from each other (e.g., trees are not necessarily green). We will show that unlike SL. MMCL. can leverage such hint to obtain superior robustness.
4.2.2 COMPARING ROBUSTNESS OF SL AND MMCL
The theorem below demonstrates that SL achieves a low accuracy under distribution shift when the spurious feature is strong, i.e., when is large.

---

Published as a conference paper at ICLR 2024
Theorem 4.7. Assuming that the input noise in each modality is zero, i.e. 0e.I 0g. = 0. and all possible feature vectors in DIr uniformly appear in the training dataset. Let W be the model trained using SL as described in 3.. The accuracy on the true distribution has the following
upper bound: Acc (W*) - 50%- + [I+a")[-p)ン-3 For example, if = 10 and B = 1/3, then
Acc (W*) - 60%.
Next, we will examine how MMCL leverages the information about independence of core and spurious features in each latent class, which is hidden in other latent classes. First, recall the conclusion in Lemma 3.2 and obtain that the similarity between an image with features 2 and a text with features z'
1)6² I
a I
is given by x
z'. The fact that 3 only appears on the diagonal
(m-
I²I, Im
m and not on off-diagonal elements shows that the occurrence of core feature of given latent class in other latent classes increases the weight for same-feature products between images and texts rather than different- feature products. For example, trees without green leaves in classes other than tree increase the covariance between texts and images of tree, but do not contribute to the correlation between tree and green. Hence appearance of green in any image has limited impact on its similarity to text describing a tree. More precisely, when computing the similarity between a given image and
the prompt for a tree, a weight (1) is assigned to 'the true characteristic of tree' and weight m is assigned to 'green'. Here, a larger /3 leads to more weight placed on the core feature, highlighting how MMCL utilizes shared features between classes to enhance robustness. This insight leads us to the following theorem demonstrating the superior performance of MMCL under distribution shift.
Theorem 4.8. Under the same assumption as in Theorem 4.7 Let W and W be the weights of a1++))) 1 B3?
encoders trained using MMCL as described in Section Then long as 32-m > 1-8
the model has 100% zero-shot accuracy on the true distribution, i.e., Acc go (W") W)) 100%.
We also observe that the features were not shared between classes, i.e., 3 = 0, it would be impossible for the model to achieve such performance. This once again emphasizes the role of shared features. Important Consideration about Robustness An important question is whether the improvement in accuracy under distribution shift is solely due to MMCL's improvement in in-distribution generalization. In Appendix [ we demonstrate that we control for in-distribution generalization in both theoretical examples. Specifically, in Data Model 1, SL has slightly better in-distribution accuracy, while in Data Model 2, both SI. and MMCL achieve 100% in-distribution accuracy. Thus, MMCL's improvement solely results from enhanced robustness, and in fact, both relative and effective robustness as defined in Taori et al. (2020).
5 UNDERSTANDING THE BENEFIT OF RICH IMAGE CAPTIONS
In Section we assumed that both (-( and (r())) are identity, implying that the captions mentioned everything depicted in the image. However, in practice, captions often serve as annotations or illustrations accompanying the image, with certain details omitted. Empirical evidence suggests that rich captions are generally beneficial (Santurkar et al. 2022 Nguyen et al., (023), but it remains unclear if richness of captions can affect robustness and, if so, how. In this section, we theoretically investigate this question by varying how much and what information is mentioned in captions. Specifically, we keep (1(-) (-) as an identity function, while let μr (z) represent a masked version of the original feature vector 2, where some information may not be reflected in caption.
Benefits of mentioning variations in the core features. Recall that in Section 4.1 utilizing Data Model (Definition 4.1 we showed that MMCL can learn large-variance core features better than SL, resulting in less reliance on the spurious feature. Now, we use the same data model to explore what happens if the feature variance is not fully reflected in the captions. For example, when the caption only contains the word 'cows' or "grass', without describing their appearance.
Definition 5.1 (Feature masking in data model (Definition 4.Ip) Given a feature vector z
+ Vssu (Zsp (Eppu a) with veore drawn from Bermoulli(T.col and thap drawn from Bernoulli (..u). Both Teore and Tpu are E 10.
Zcore
with corresponding y and a, we let (z) y/ + Veore (Zcore - y)
Zapu
Asssumptions are made to simplify the analysis, but our analysis can be readily extended to show that same conclusions holds with high probability in broader settings with sufficient sample size and reasonable noise level.
7

---

Published as a conference paper at ICLR 2024
Here, Zcore y and a represent the variations in the and spurious features, both of which Gaussian random variables by Definition[4. [.I] This implies that the captions capture these variations with probabilities Toore and TTpu respectively. When πT =0, caption ignores all the details and treats all features of the same kind as a single entity. The following theorem shows the effects of Teore; suu Theorem 5.2. With data from data model I and ftt defined in Definition 5.1 with high probability, the model trained using MMCL has a test accuracy on examples where the spurious correlation does not hold (i.e., a + y) given by ( of this accuracy increases as Tcore increases and is independent of sspu-
V(L+ one F of ) o(1). The non-negligible part
2p-2 20-22-mn- "
The theorem reveals that the model exhibits less reliance the spurious correlation when the caption mentions the variance in the core feature (e.g., appearance of the cow in each specific image). Additionally, notice that mentioning variance in the spurious feature has minimal effect on the robustness, as it does not impact the correlation with the core feature.
Mentioning more features benefits robustness. Next, we utilize data model 2 to explore the effect of mentioning more features in the captions.
Definition 5.3 (Feature masking in data model 2 (Definition 4.6)). For a feature vector z with label (k, c). let (z) = + ④ 2, where +b = [wh- v/7] with t/ = and 4/ ~ (Brnoulli(T) foroouull((iffrrrr for j + k.
Here, the caption always mentions the feature indicating the latent class, while other features mentioned with a probability π. Note that π = O corresponds to the setting where the caption is just the same the label. The following theorem demonstrates that the model can achieve robustness only when the caption sufficiently mentions features that are not directly related to the image's ' latent class.
Theorem 5.4. With data model 2 and pt defined in Definition 5.3 let W and W be the weights of encoders trained using MMCL. Then the model's 's accuracy on the true distribution satisfies AMCL AM W7) = 100% if ā, and Acc MMCL (Wi, W S 50% if π < ã, where
T = (1+-9)0 1+.
(1-3)0-(m-))
As explained in Section 4.2 even if certain features do not directly indicate labels for a class, they
can still help learn relationships between features (for example, not all trees are green), and this knowledge can be valuable for other classes. However, if these features are missing from the captions, they contribute less to the cross-covariance matrix used by the model for predictions (Lemma (.2). In the extreme case where 0, captions reduce to labels used by SL, and robustness does not improve.
6 EXPERIMENTS
0
4
8
7
Semi-synthetic Experiment. We conduct a carefully designed semi-synthetic binary clas- sification experiment to showcase MMCL's ro- bustness and the significance of rich captions. The task is to distinguish digits 0 to 4 (class 1) from digits 5 to 9 (class 2). In the training set, MNIST [Degg 2012) digits are placed on colored backgrounds, including three types of blue and three types of red. As illustrated Figure 4 for digits 0-4, 99.5% of images have randomly selected shades of blue as the background, while the remaining 0.5% have random red backgrounds. The same applies to digits 5-9, but with blue and red swapped. for all images. Therefore, digits represent the core feature, while colors serve as the spurious feature whose correlation with classes only exist in the training data. Captions are simulated as vectors, where the first coordinate contains digit information and the second contains color information.
[-4.5 -1.5, -11
[-05, -15,.1
[B.5, -1.5,
[25, 155...]
MTeey Rig
(25,-155...
12.5, 2.5...
12.5, 0.5
25...
simulated captions
Figure Construction of captions.
the test set, backgrounds are randomly chosen
Both features exhibit variance; for example, there are four variations of digits between 0 and 4 and three variations of blue backgrounds. We use Teore and Tpp to control the specificity of the captions, determining how much the caption mentions the variance in each feature. Being 'specific" means mentioning the exact value (e.g., specifying particular shade of blue), while 'not specific" means re- ferring to a value that represents an entire category (e.g., using the mean value for three shades of blue to represent any blue). Figure [ shows an example. For more details, please refer to Appendix G.I
We plot the OOD accuracy in Figure 2
while varying the values of meure and T We observe: (1) With sufficiently rich captions (high Weone), MMCL exhibits better robustness than SL. (horizontal
8

---

Published as a conference paper at ICLR 2024
#500C0
MSCDOD
199
75.0-
60p Accunary IN)
00D Accurniyy (h3
carcor
72.5
000 accurscy
70.0
68.0
vorynn Roo
000
02.5
Ror vorying fee
00.0
supanssed
Acouracy (%)
60 12
10
ID Acouracy (%)
30
10
50
0.0 64 D6 Ba
Rooe of Rag
020 Accureny (N))
Concertual Captions
Concoptual Captions
mtder quice
179
Concept ail Captions
vila dlas und.
Figure 2: OOD accuracy on the semi-synthetic data. large Toore
000
000
is crucial for ensuring MMCL's superior robust- ness compared to SL, but the value of TTp has min- imal effect.
Accuracy (8)
102
Accurecy (6)
Acturaey OS)
(a) MMCL vs SL.
(b) Caption richness
(c) Intra-class contr.
Figure 3: (a) MMCL is more robust than SL. (b) Caption richness and (c) intra-class contrasting contribute to robustness. Note that (c) is in a different setup than (a)(b), detailed in Appendices G.2 and G.3
line). (2) A high Teore. indicating that the captions mentioning the variance of the core feature, is essential for achieving robustness, as reducing Turr significantly hurts the robustness. (3) In contrast, spu has minimal effect on robustness. It's worth noting that (2) and (3) directly validate the conclusions from Theorem 5.2 Additional discussion be found in Appendix G.T
Robustness on real data. We further corohorate our conclusions with experiments on MSCOCO (Lin et al.] 2014) and Conceptual Captions (Sharma al. 2018). We train models on these two datasets, and evaluate them on six shifted versions of ImageNets. See experimental details in Appendices G.2 and G.3).
MMCL is more OOD-robust than SL. We employ the widely used CLIP loss for MMCL and CE loss for SL compare the robustness of the resulting models. Due computational resource constraints, we adopt the simplified training setting from (Ren & Li 2023), training a 3-layer MLP on top of frozen pretrained encoders. Following (Toori et al. 2020). we plot the ID-ODD accuracy relationship by varying the model size (width of the MLP). Fig 3a shows that models trained with MMCL. demonstrate superior robustness on both datasets. Note that although the ID accuracy of CLIP is lower than that of SL, resulting in seemingly comparable OOD accuracies, we anticipate the actual advantage of CLIP to become more pronounced as the dataset size scales, similar to the original dataset used in (Radford al. 2021). We also provide results regarding with other algorithms, including SimCLR and SupCon, and other backbone architectures in Appendix G..4
Richness of captions is critical in achieving robustness. To demonstrate the impact of caption richness on robustness, we train an alternative version of CLIP wherein the captions are simplified to be the same the labels. As depicted in Fig 3b. this modification leads to diminished robustness, which corroborates the theoretical conclusions in Section 5
Intra-class contrasting contributes robustness. To illustrate the mechanism theoretically presented in Section [ we modify the CLIP loss to exclude pairs from the denominator they are from the same class, thereby eliminating contrasting between images and texts of the same class. In this experiment, unlike the previous ones, we train the encoders from scratch and obtain different ID-OOD accuracy pairs by varying the training data size. This is because the effect of intra-class contrasting was not evident in the 3-layer MLP setting, likely due to the small model's 's limited capacity rendering it less sensitive to modifications in the loss. In Fig we observe that removing intra-class contrasting from the loss compromises robustness, confirming the importance of intra-class contrasting.
CONCLUSION
In this work, we provided the first theoretical explanation for MMCL's enhanced OOD robustness compared to SL. We showed conclusively that this robustness is attributed to aspects of the MMCL loss function, i.e (1) intra-class contrasting (2) inter-class feature sharing, as well as the nature of multi--oddl data i.e. (3) richness of captions. We confirmed our theoretical results using both synthetic and real-world experiments. Our findings could inspire the development of improved loss functions and data curation practices to further enhance MMCL's robustness.

---

Published as a conference paper at ICLR 2024
Acknowledgements This research is partially supported by the National Science Foundation CA- REER Award 2146492 and Cisco Systems.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
Benjamin Aubin, Agnieszka Stowik, Martin Arjovsky, Leon Bottou, and David Lopez-Paz. Linear unit-tests for invariance discovery. arXiv preprint arXiv:2102.10867 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei. A theory of label propagation for subpopulation shift. In International Conference on Machine Learning, pp. 1170-188, PMLR, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. February 2020. doi: 10. (0446550/axx.22202.0570999
URL htttps:::///zxe org/ abs/2002. 05709v3
Ching-Yos Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased Contrastive Learning. In Advances Neural Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020. URL
neuripss...ss cc/ Abst- .ttml
paper/2020/hash. 63c3dacc 7b wtt.ccaaaae4222111999444
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916):11-1144 2012.
Matthew Fahrbach, Adel Javanmard, Vahab Mirrokni, and Pratik Worah. Learning rate schedules in the presence of distribution shift. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 9523-9546. PMIR, 23-29 Jul 2023. URL htttss://proceeeeeeess ngs ml w.w.rr.rrssss..... 02. Fahrbach23a html
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In International Conference on Machine Learning, pp. 6216-6234. PMLR, 2022.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 122311, 2018
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-6719, 2022.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ittr--disttibution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3340-83499 2021a.
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:22118 18-22133, 2020.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. arXiv preprint arXiv:2 10.00273, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 199044916 PMLR, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348. 2021.
Siddharth Joshi, Yu Yang. Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman, Towards mitigating spurious correlations in the wild: A benchmark and more realistic dataset, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wit wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint urXiv:/908.0.3. 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- -nd--annualgege tasks. Advances in neural information processing systems, 32, 2019.
Stephen J Montgomery-Smitth Mathematical Society, 109(2):517522 1990.
The distribution of rademacher sums. Proceedings of the American
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European Conference Computer Vision, pp. 529-544. Springer, 2022.
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. arXiv preprint 0rX2v22302.066232, 2023.
Edwin G Ng, Bo Pang, Piyush Sharma, and Radu Soricut. Understanding guided image captioning performance across domains. arXiv preprint arXiv:2012.02339, 2020.
11

---

Published as a conference paper at ICLR 2024
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-1I), pp. 689-696, 2011.
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. arXiv preprint arXiv:2307. 10350, 2023.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 555:126658, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pp. 5389-5400, PMLR, 2019.
Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint arXhh:2304.003711 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115
(3):211-254) 2015.
doi: 10.1////s -63-015-0816-..
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterrization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008..4888 2020.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is cap- tion worth thousand images? controlled study for representation learning. arXiv preprint arXiv:2207.07635, 2022.
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202. 14037, 2022.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9661-9669, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2556-2565, 2018
Galen R Shorack and GR Shorack. Probability for statisticians, volume 951. Springer, 2000.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1): 2822-2878, 2018
Nitish Srivastava and Russ R Salakhutdinov. Multimodal learning with deep boltzmann machines. Advances in neural information processing systems, 25, 2012.
12

---

Published as a conference paper at ICLR 2024
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modalit, encoder representations from trans- formers. arXiv preprint arXiv:190..07990 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 11118888318999 2020.
Yuandong Tian. Understanding deep contrastive learning via coordinate-wise optimization. Advances in Neural Information Processing Systems, 35:19511-19522, 2022.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference Machine Learning, pp. 10268-10278 PMLR, 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Mach. Learn. Res., 22:281-1, 2021a.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning. multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206 PMLR, 2021b.
Martin J Wainwright. Hiigh-dimensional statistics: non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. Advances in neural information processing systems, 34:2215-2227, 2021.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9933, PMLR, 2020.
Per-Åke Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12:99-111, 1972.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference Machine Learning, pp. 11112-11122. PMLR, 2021.
Ruijia Wo, Linjun Zhang, and T Tony Cai. Sparse topic modeling: Computational efficiency, near- optimal algorithms, and statistical inference. Journal of the American Statistical Association, pp. 1-13, 2022.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.
Yihao Xue, Siddharth Joshi, Eric Gan, Pin- Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. arXiv preprint arXiv:2305. 16536, 2023.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzych Ghassemi. Change is hard: A closer look at subpopulation shift. arXiv preprint 2rXiv:2302.1225:: 2023.
Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpola- tion. arXiv preprint arXiv:2106.0 02695, 222.2.00.00065, 2021.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. arXiv preprint 2464722306 2023a.
13

---

Published as a conference paper at ICLR 2024
Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16036-16047, 2023b.
Zhuotun Zhu, Lingxi Xie, and Alan L Yuille. Object recognition with and without objects. arXiv preprint arXiv: 161 1.06566, 2016.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORK
Uni-modal contrastive learning. There have been both empirical Chen et al. (2020); Chuang et al.
(2020); Khosla
(0200) and theoretical studies about CL Wang & Isola (2020) Tosh al.
(202 la by: Arora et al. 2019); HaoChen et al. (2021); Wen & Li 2) et al. 2021]; Saunshi et al.
(2022); Xue al. 2023). We will demonstrate for the first time how the contrastive aspect of the
loss can benefit OOD robustness, while noting that this advantage only exists when equipped with
multi-modality and zero-shot learning, which are present MMCL but not in unimodal CL.
B PRELIMINARIES
B. MINIMIZER OF MMCL LOSS
Nakada al. 2023) has shown the equivalence between minimizing linear MMCL and SVD. We reiterate this for reference in our proof. As defined in Section 3.2 our loss function is
(W
1
Siy 58) 2n(n
Σ
Σ
(89s 8:)
W WT 1
2m(n 1)
1)
jyt
jyh
which can be rewritten as matrix factorization objective
L(W, s, Wr) (WWW)WWF) P PI W Wrl.:
(2)
where S is the cross-covariance matrix
Σ Ea S= 1 n - n(n 1) id1
(3)
The following directly follows from Eckart- Young- theorem: let Ei=1 Au,, with A 2 A2 2 ... Ad > be the SVD S, and let W;. W be the minimizer of the loss, then
Wi W
uur
B.2 MINIMIZER OF SL LOSS
We note that in both Data Model 1 and Data Model 2, under the assumptions we've ve made, the training data are separable, or separable with high probability. As shown (Soudry et al. (2018), minimizing the logistic loss Cross-Entropy losses with linear model at sufficiently small step size converges to the solution for a hard-margin SVM. Therefore, in our analysis for SL, we equivalently examine this solution.
C ANALYSIS FOR DATA MODEL
C. ANALYSIS FOR SL
Here, we briefly explain how Theorem 4.4 is derived from Theorem 1 in (Sagawa et al. 2020). Firstly, we provide the following version, which is direct translation from (Sagawa al., 2020) but with our notations. First let Amaa denote the number of examples with = in the training set and
Mmaj denote the number of examples with y in the training set. Define o o/2 = o dp
02 +
de d, o2 = (d-22)
2001 2 2 1, 12 I 16 lol 100m 12 Omin 100, there exists do such that for all d > do with high probability over draws of the training data
Theorem C.1. For any Pmmsi 2 1
S
6002 and
spu
Test examples where + y achieved by W 2/3.
15

---

Published as a conference paper at ICLR 2024
It is now easy to that the quantities in Assumption 4.2 can satisfy the conditions in the theorem above when they become sufficiently asymptotic. Note that the above statement is about test error on examples where # y, which accounts for half of the entire distribution 30* the accuracy on the entire distribution 3 is at most 2 x 100% + x (1 2/3) 2/3.
C.2 ANALYSIS FOR MMCL
We note that Theorem 4.4 holds under a more relaxed assumption, which the following anlaysis based
Assumption C.2. The gap between the variances of the core and featurs is significant: core spm (1) and spu = 0(1). can be any value between and 1. We consider the high-dimensional (overparameterize)) setting where w11,, ds s2(n) and dr (2().. The noise levels are not too large: de. = O(log n) and og, = nl)
y) + C1,
We define the following notations. We write each 2 as 24 =
where C1,5 and Ç2.6 are
a; {2.5
11 [2.1)
Gaussian variables according to our definition. We also let C:
Let 3 - S2,,5.,..... Snl.
E
E2..,E ET ST,,,.. FT.J..
Additionally, we let F
U1, U2, ................................ 01 02 ............ an
C.2.1 USEFUL CONCENTRATION BOUNDS
Lemma C.3 MontggomerySSmit (1990). Let {ar)==1 be set of random Rademacher variables, then with probability least the Jollowing holds
21((1)))
2=1
Lemma C.4 (Mills' ratio. Exercise 6.1 in Shorack & Shorack 2000 Let v be Gaussian random variable drawn from N(, 1). Then for all O,
1
<
(T(u2N) À 2m 2m
X + 2m
Corollary C.5. Let (x,) be a set of random variables independently drawn from (0, o²), then with probability at least O poly(n)) the following holds
log n xal Olov
n
Lemma C.6 (Restatement of Theorem 6.1 from WWiinwright 2019 Let X E Ruxd be a random matrix with x its i-th Each I; is drawn i..dd from N (O,E). E). Then with probability at least 1 8, the maximal singular value of X, denoted as mmax (X) satisfies the following
T mmax (X) S /ntymax (VE)(1+ V 2m16(1
Tr(E).
a
Lemma C.7 (Gaussian covariance estimation, Example 6.3 from (Wainwright) 2019)). Let {u,) be set of vectors E IR' independently drawn from (0, E), and ler Σ -im V.v then with probability at least 2e -88 /22
Σ
E11
d m
2
28 (V 22
Σ
C.2.2 CONCENTRATION OF THE PROSS-COVARIANCE
Concentrations in Low-dimensional Underlying Feature Space:
16

---

Published as a conference paper at ICLR 2024
Lemma C.8. With probability at least 8 the following holds
In In(1/8)
a,y: (2p-1)1 - V
Proof. By Hoeffding's inequality.
Lemma C.9. With probability at least
Ol pplly(m)
the following holds
To
log
S:S2
1/2 S O
o xpm
care
Proof. By Lemma 0.7
Lemma C.10. With probability least - O (ply((n)
log
y/ a2
S. 1l S Oo
Proof. By Lemma C.5
Lemma C.11. With probability least 1 - O
poly(n)
1
1 + 0²
2p -
log
2,2
2p
+ 7
112 S
O
(5)
1=1
Proof. Write 1 Eim 2 as the following
- 2,2 =
2
yaa,
Σ
Saly
a, +
Σ S.S..
12
72
n
Then invoking Lemmas
C.8
C.9
and
C.
10
completes the proof.
Lemma C.12. With probability at least - O
poly(n)
log
2, 11 S 00
n
n
Proof. Note that each of the two elements in - 2, be the of the mean of independent Rademacher variables and the mean of n independent Gaussian variables. Combining Lemmas C.3 and C.5 completes the proof.
Lemma C.13. With probability least
0 ppol((n)
(polyy(m))
1
Σ ig
+ 02
2p
log O(v
n
2:2
nnn
1)
2,2
2p
1 +
1
n
Proof.
1
n
1
72
1
n(n 1) Σ 2,2
2
2
2,2
2:
2j +
71
n2n2
1)
71
T2n1
1)
n
Σ 2,2
2
Z: n
2j +
1 1m
n(n
1)
1=1
j=1
1=1
17

---

Published as a conference paper at ICLR 2024
Note that by Lemma C.12 the norm of the second term on the RHS is O log ). and by Lemma C.11 the norm of the third term is O H( Combining these results and applying Lemma C.1 the first term yields:
11
1 + o core
2p
log n O
72
2
n(n 1)
2
2p
spa
2
2
n
Concentrations in High-dimensional Input Space:
Corollary C.14. By applying Lemma C.6 we can conclude that the following statements hold with probability of at least - 35
221n(1/8)
13 I Syno nor (1
+ V 02are + opu
n
2mm(1/0)
E ll2 Soc
+
+ 16.1
di
nd,
21n(1/0),
B ll2
Sogrr(
+
+ σF,T
dr
ndr
Lemma C.15. With probability at least 0
poly(n)
Er,ll S 0
and
ST.
ST.il
S 50(-0 - Vn
Proof. This can be obtained by recognizing that Σi= Er, can be treated as single sample from (0, of ndy La,) and by applying Lemma C.6 with pn]y similar argument applies to DEmE ÊT,
Lemma C.16. With probability at least - O )
s D,
1 + core
2p
DT
<0
GTT + Jg. +
0g.10
2p -
Vn
spu
TE.A
log
og Veavllogr + dg.
+0
n
log
+ +0
n
where S is defined in Equation
Proof.
Firstly let's write S as
S
1
Σx
(6)
n1
n((n
1
D D
D.T.
Σ
D Diz,2 D... + R
n1
n(n 1)
itj
18

---

Published as a conference paper at ICLR 2024
where
n R
D E + E1,22 DJ- + E1,57,))
D(DiZiE) + E1,22] DJ. + (1,55T
T i=1
n(n - it
Di 2 & - E1,zz DJ + (1
8
D Eiz + E1,22 DJ + E1.ET
( - 1)
1=1
Let's rewrite Rı
1 (Di(F+3) +3) +B,(F (3) +3) DJ + E
R1
n
Then
[
- 11 1
[|)]|a([[F|] + |311)) + |3 [ + 131 Drllz + ||
Note that ||D, || Dr ||2 since they have orthonormal columns. Additionally, observe
that ||F||2 ||
/2n. By combining these and applying Corollary C.14 with O poly(a))
we obtain the following
[Ril|2 50( J OgT + 0g,1
Vn
n
=0( OET + t
+ S.T
VT
n
Next, we rewrite R2 as
R2
12
D
21
+
E1
- Σ 2j)
D r
+
-
E1
Σ
Ex.e) TT.
n n-1
1=1
j=l
i=1
2m1
Then applying Lemmas C.I2 12 and C.15
yields
EEA S.AVlogn- + G.BVlog 1g.
|Rg l2 s O
(8)
n
Additionally we observe that
1
Σ D. n(n - 1)
1
Σ
Diz DJ. D
1 2,
1
Σ
2
D.F.
T2
1
n(n 1)
itj
it
Therefore by Lemma
2 DJ
DiZIZ DJ D
+
-
0
D
n(n 1) bt1
+
o spa
2
log n
DD 22 Dr
-0(1 log
(9)
Then, combining Equations
g 6 yields
Dj
+ o²
2
2p
D
50(
OET oE,
IE
2p
1 +
Vn n
qm
2
JE.A Fg,Aylog + ag log + d
+0
log 7
+0(
19

---

Published as a conference paper at ICLR 2024
C.2.3 PERTURBATION IN SVD
Lemma C.17. G* RRdixdr is matrix whose SVD is Ei=1 Au where A > )2
A = (()), 22 = (1) and \ - 22 ((1). G G + E where |Bll2 S Let Lim1 Xrúão be the SVD of G. If ((1), then |E A G² - O(ve), where 2 S p S
Proof. Weyl's Theorem tells us that DN - l1y S 12 - A2l < and Ax < for 2 3. Now, let's define 8 min( (}1 X2, A23. By applying Wedin's Theorem (Wedin 1972) with the singular values partitioned into { }or and fer we have sin 0(u1, ú1) < (()). Similarly, by applying Wedin's
Theorem with the singular values partitioned into {>2] and {A. we have sin (u2, U2) < (
Considering that 5 = ④(1), ), we further have sin ((u1. ú1) I O()) and sin (u2, ú2) & O(c). Similar conclusions hold for 0, well.
Now, for i = 1, let's examine the difference between it and uy:
u -2(1 cos (u, ū::)
-2(1 /1 sin 6 uu ú,))
sin f Ui; ú;)
-2
1 1 /1 sin Ouu,, ú;)
=O(sin 0(u, ūi)) <0(e).
Therefore, |ū, || < O(v2). Similarly, we can deduce that | -, S O(ve). By some
algebraic calculations, we further have | \ -G' ((ve). Now, if p 2 3,
previously established that | for 2 Consequently, | X,ūū 1|l = max max {1:) S..
Thus, E λ,ūū G E A,ū T G 112 + Ei= λ,ū 1le S O(Ve).
O
For convenience, let's define
co = GGT + agg
0C.0% a..vlog dg,BVloon Tg. + logn
+
Vn
n
n
n
Corollary C.18. The minimizer satisfies the following with a probability of least 1 O poly(n))
[ W
.DI
+
2
2p 2p-1
D 12 S O(vo))
2p
1 02
P
C.2.4 ZERO-SHOT CLASSIFICATION
In the following analysis, will examine the zero-shot accuracy in event where
D + 2
2p-1 1 D S o(
[ W
2p
1 mpp
P
It important note that such event oceurs with probability of at least O (ploy(m) by
Corollary C.18
+ + C2
Let D,
+ Er, be test input satisfies y 1, -1
By Lemma C.6 with probability at least 1 O polly(n))
x Oloogn) n.)
(10)
20

---

Published as a conference paper at ICLR 2024
Recall that the prompls are Py = Dr
y (
for y = -1, 1. Then
2 W
(v) W TT
1
D1
1 oane
2p -
DT (v)
> | .T
O(VED)
(11)
2p
1 + [
p
0 O V logn).
p
1 + uwee
2p
D Djaf (2)
Now let's look at x
D
2p- 1
1 2
i D1
+ 0
2p -
D (2) = v + (1 + 2orr)) + + (2) - -1) (12)
2p
o spu
In order for the model to make correct predictions for this example, we need xj W 17;17 (1) >
x , Wi
Wi Wizh"') (-1)
Based on Equation
11
we can establish the following sufficient condition:
1 D
x
2p- 1
+ a 2
core
1 1 d 2
2p
D a
(1)
( n)
spu
1
Dr
+ core
2p -
DJat (-1)
1 5
60 log
2p 1
+ Apm
By substituting Equation 12 into the above expressions, we obtain:
(1 + (1)(1 + domr)) (21 O(veologn) 0.
(13)
Let 61 denote last term on the LHS, i.e., O(
By recognizing that (1 + 20nc)) + (2 - 1) is a variable follows the Gaussian distribution
A (o, (1 + )o + (2p - ppu)) we can derive the following probability:
Pr ((1 + (1) + + + ( (2) 1)) 61 > 0)
(14)
2p-2 2 o E1
Pr (0
v-N(0.1)
(1 + od 120 + (2p 1) 1)2
2p- 2 - o core + 61
=1 Φ
(1 + 0 20 + (2p 1)2022 spu
where (-)) denotes the CDF of the standard normal distribution.
Therefore, conclude that, in order for the model to make correct predictions, the failure
probability is bounded by ( V(I+owu) Paan (2p- plus the probability for Equation 10 to not
hold. Thus, the error rate on test examples where y 1,a -1, denoted by Err(1,-1) 1)+ is bounded by:
Err
W W
2p - - + E1
+ 1
(1 + 2 )20 (2p 1)
poly
2p- 2 22
+
(1 + a
2
2 2
+ (2p 10
(1 + o 2o? + (2p - 1) spu
poly(n)
core
spu
2p-
2 2
1
+
(1 mome
2
(2p 2
(oly(n)
spu
Note that Equation ① is obtained by taking the first order Taylor approximation for +.
21

---

Published as a conference paper at ICLR 2024
We also derive that (-11) is bounded in the same way as above. Similarly, obtain
2pp o
1 + (1 poly(n)) (15)
(rT( = Err (1,,)) & Φ
(1 + o ) - (2p 1) spu
Converting error rate to accuracy yields Theorem be lower bounded with the same non-negligible term.
Note that the error rate can For example, Err(-1.-1) 2
22p-0
(1).
(1+ Ho (2p 1)
C.3 MMCL WITH FEATURE MASKING
With feature masking, the proof is almost the same as above, but we just need to realize that the only change the covariance of features, thus we would get
[ T W
- D P
1 Tecee core
2p -
D
12 S 1 -
2p 2p-1
1 Tspu Tspu spu
Then going through the same steps in Section C.2.4 yields Theorem 5.2
D ANALYSIS FOR DATA MODEL
D.1 ANALYSIS FOR SL
For a vector v, we use víj] to denote its j-th element. W.O.L.G., let D, E dl x be the first 1-th columns of an identity matrix (since by definition D, has orthonormal columns and can always apply a change of basis).
By definition, the solution of SVM, denoted by W satisfies
W arg
min
s.t. w. 21 w. 2 1, v(æ1,y) and y' + in the training set. W
2mm
(16)
j W2 W2mh. where W2 -2k-14 (1+ k-th element is (I (1=8)(++03) its (k m))th element is [1a) and its other elements are zero. It is easy check that W satisfies the condition w D1 i u 2 for any (21,y) and y' * y from the training set, and
Construct W-
||W|| -2m (1-9)(++a7) az
Definition D.1. Define S as the set of classes such that V(k e) S (k, e) is the alias of y), the following holds: 3 example (I1, y) from 'r, such that z (k + m))th element is -ac, and the margin maximizer on training data can make the correct prediction on this example, i.e..
xr]x4m -ac and arg max w xr y.
(17)
Define : |81/m Next, we are going to show that the assumption contradiction.
8 (1+a")1-9p 99- would lead
Assume S 2 em. For each (k, E S, let a be the imput of an example satisfying the condition in
equation 17 Then xt's k-tth element is c, and (I: + m)-th element is -ac. We construct an example
27 whose k-th element (k: + m)-th element is ac, and the remaining elements are the opposite of the corresponding elements in x
From assumption
17 have
Vj # y, w 27 w x7 > 0.
(18)
Note that 77 shows in the training set. By the condition for the SVM solution, we have
Vj # y,
x w 27 2 1.
(19)
22

---

Published as a conference paper at ICLR 2024
Now, for any j # y, let's compute a lower bound for ||w5 w. vector w ---wy wj can be written
as azj + bay + v.,, where Ta is vector orthogonal to both 27 and + By equations 18 and we have
Gia + c2b > 0 C2a + czb 21 ,
(20)
where
CI = | 2 = (1 + a²) (1 (m - 1)0²))
(21)
C2 _ 3 = 1 a² 2 -mmm- 1)0² + a²)
(22)
63 = | 2| = (1 + a²) (1 + (m - 1))3²)
(23)
Remember that we want to lower bound the following quantity
| w wIll
a + cab² 222abb + |1
(24)
given the constraints in 20 By equations 21 to 23 (2c2) ---444400 c3 < 0. Then cia +cyb+ 202ab = D always an ellipse or circle centered the origin in the a-b coordinate system, where a larger D means a larger radius. Also, given that > 0, C2 < 0, c3 0, by plotting the feasible area, we can observe that D achieves its minimum when the intersection point of lines a + c2b and C2a + cgb = is exactly at the ellipse (or circle). Now we can solve for the minimum of Cia cyb² + 2czab:
c1a² + cyb² + 20zab 2
C103 c2
Then by equation 24 we have
W -ww]2 C1 C3 C2
Then we get the following lower bound for ||w, ||2 + [/w
( (/w5)) || + ||w 11)
ww + ||00]112
W - w1)1)
1 c1
(25)
2 C103 c₂
1
(1+ + a²) (1+ (m -1).2)
8 2
1 + (m- 11)32 + (m - 1).32
1
(26)
Now since there least em different such y's and for each y we can pick a distinct get that the sum of the squared norms of the corresponding weights is at least em/8. Then we introduce the following lemma. Lemma D.2. For any + j. the following holds for the margin maximizer W
|| + || a1+ 3)
Proof. Consider any y, whose aliases are (ks c) and (K', e') and y # y' Firstly, recall the condition Equation 16 which is
w y I1 -w; x1 2 1, Vas with label
23

---

Published as a conference paper at ICLR 2024
we also have w x wy x 2 1, Ve'y with label y/. Note that assume all examples occur in the training data in the theorem. We let be example whose k' element is 3 and k' + m-th element $a, and let æ', be an example whose k-th element is 3 and & + m-th element is Ba, and other elements the same in øy. Note that such examples exist. Then we have
By the condition in Equation 16
| 112 =
(1 + a1 + (m 1))
(1 + (22 (m 2)
Similar to the steps from equations
19
25 we can solve that
| 1| + |/ 11?
|
(1+ a²) 3)2"
Note that this hold for any y + of which completes the proof.
Combining equation 26 and Lemma D.2 yields || 1 N + (1 - c)m (1- Then the
assumption [1+aa](1 8y--8 yields |W*1| > [W]| which contradicts the fact that | is the
solution which separates the data with smallest norm. Therefore S (1+a")(1-8)-- Then we can
bound the accuracy on g0r by
100% + X 2
em 50%
c S 50% +
2
2 2m
4
(1 + a2) B)²
D.2 ANALYSIS FOR MMCL
Here we analyze with the feature masking defined in Definition 5.3 To obtain Theorem 4.8 need to set π = 1.
we just
As shown in Section B.1 the minimizer can be expressed in terms of SVM of S. We can calculate that
1=x(m 1)82 Im
S = CD]
m
Dr:
(27)
a
where C' is some constant.
Then, at test time, the similarity between an input 21 = D from class y and a prompt P.: Drdles
from class (k', y²) denoted by Sim, is given by
+ Tmm - 1),32
+ a
Sim = Cz
ex
ek++m
m
m
There are two cases to consider. (k, (k',')
1
m
a²
Sim = C(
B2
+
(28)
m
m
m
k = ko
Sima
-C(
m
+
a²
(29)
m
m
m
W
T = 13 a + m
1
Sim, = C
(30)
m To achieve 100% accuracy, we need Sim, > Simm, Sim > Simg. which yields,
m
a²
π ( - 1)
(31)
B
and
(1 - 3)8 am 1) (1+ 3)a² 1+3.
(32)
Note that if inequality 32 holds, then inequality 31 holds as well. Therefore only need inequality 32 which completes Theorem 5.4
We note that Theorem 4.8 is obtained by setting 1.
24

---

Published as a conference paper at ICLR 2024
E ANALYSIS FOR IN-DISTRIBUTION ACCURACY
E DATA MODEL
We provide brief explanation about the in-distribution accuracy, starting from SL.
W.L.O.G., we can let D, be the first columns of the identity matrix. Then the core and spurious features show in the first elements of the input. Let eeore and Upu denote first two elements of the SL loss minimizer W Rd,, respectively.
In the following, for simplicity, we assume (1) core [su 9Q(1): (2) dsso > O. However, it's worth noting that both of these assumptions can be shown hold with high probability through more involved analysis.
As shown in Proposition
Sagawa al. 2020). the following holds
(1+ d' - s'su Weore
+ (0.006)
(33)
00 T + spsu
2 spm
where < 1/2000, d' < 1/1000, 7 9/10. After some calculation we have R aligmment with noise is bounded.
Bags merer
1.51. also
Lemma in (Sagawa et al. 2020) shows that
20 n2
W I < s oo + Mmin +
0
(34)
E
where u = 1.3125, 2.61
Zcure Eapu
Given test example (xy, v), where xt Dr
Er, we have
W x [Woore
Wspu
Zcore Essu
[W
(35)
By considering that W f is a Gaussian variable and applying Lemma C.4
we have
log di
W = 0 O(I 1l0g.1
(36)
d1
which further gives
W"E/1= log di dj
(37)
by equation 34 Thus with sufficiently large d, IW E, constant, the prediction is dominated by [xéçorc Wspul
(1)) Given that || [[[Deore (ispu]] is at least
Zcare
Zupu
Let's begin by considering test examples where y. By following similar steps in equations T3 = o(1). In the scenario where
and 14 we can get the accuracy such examples $
1+R
Vo
R
Joore and Spu 0, given that 1.51, the above accuracy is (22.51) 99.4% + o(1). Given - o(1), the overall
that in the in-distribution test data, such examples occur with probability in-distribution accuracy is (2.51) 99.4% o1).
For MMCL, from Section
C.2.4 the accuracy on examples where = is
-2p-od ?
Φ (l Poo (2p- 1
o(1). Considering that Pspa - o(1), the in-distribution ac-
22p-6r 22.
curacy is also Φ
V(1+aam Po 2nm +(2p a o(1). In the case where ooore 1, app 0, the above accuracy is 4 + o(1) = 93.93% + o(1).
Therefore that SL has slightly higher in-distribution accuracy.
25

---

Published as a conference paper at ICLR 2024
E.2 DATA MODEL
For data model 2, it is evident that SL can achieve 100% in-distribution accuracy, as under our assumption, every possible example shows in the training set and they are seperable. For MMCL, every example in geTr also shows in y The fact that it achieves 100% accuracy g have already proved, implies a 100% in-distribution accuracy.
F COMPARISON BETWEEN SUPCON, SELF- SELF-SUPERVISED-CL AND MMCL
F.1 A DETAILED DISCUSSION
While the original analysis in the main paper has already thoroughly demonstrated the mechanisms leading to MMCL's robustness, an in-depth comparison between SupCon, Self- Self-Supervised-CL (e.g.. SimCLR), and MMCL offers an alternative interpretation of the same findings. We hope this can further illustrate the roles of two crucial elements in MMCL: contrasting between individual examples and multimodality.
SupCon < Seff-Supervised-CLE Role of contrasting between individual examples. Let's compare the representations learned by two different unimodal representation learning techniques: SupCon and Self-Supervised-CL. Although their loss functions are quite similar, Self-Supervised-CC contrasts any two different examples, while SupCon contrasts only those with different labels. We have the following important conclusions: (1) Self-SSupevised-CC closely resembles MMCL but within single modality. Consequently, its learned representations exhibit similar structure to MMCL's representations in that modality. This includes learning large-variance features and features shared between classes. (2) In contrast, SupCon's representations exacerbate the issue of spurious correlations because it maps both core and spurious features to the same direction in representations, making them entangled. We theoretically demonstrate this in Theorems and F2 the next subsection.
Self-Suppevised-CL < MMCL: Role of multi-modality. Now, one may ask, given that Self- Supervised-CC CL achieves good representation structures, can it achieve the same level of robustness as MMCL? Well, (1) No, because seff-Supervised-CC solely learns unimodal (e.g., image) representa- tions. To enable classification, we rely supervised learning on these representations, which, we have already shown, not robust. (2) Yes, only if one could align representations of language and image modalities for vero-shot classification (bypassing supervised learning) after separately learning representations in each modality with Self-Supervised-CL However, this essentially falls into the category of MMCL because MMCL precisely performs these two tasks contrasting individual examples and aligning between modalities -iiulttanoously. An alternative way to think about this is that even if we adopt MMCL's image representations, training linear classifier on the representations instead of conducting zero-shot classification would lead reduced robustness. Evidence for this can be found in Figures 14 and 15 in the CLIP paper (Radford 2021), where zero-shot classification outperforms logistic regression/few-sher learning on representations.
F.2 THEORETICAL ANALYSIS FOR SUPCON
We consider the following supervised contrastive loss, which is naturally analogous to the MMCL loss used in the main paper, and is akin the linear loss widely adopted in theoretical CL papers in the literature Ji et al. 20221;; Nakada et al. (2023).
CsupCon (W)
1 Σ
( (o(x1,)) N
Gll NIWTWII. W
N
All
where || represents that examples i and j are from the same class, k A represents that examples i and & are from different classes, N = Hi 1 dt and N, = [k: le X i))- We consider linear model g(z,) )- = Wx, with W E Rexd The examples are drawn from the training distribution Tr. within the image modality.
26

---

Published as a conference paper at ICLR 2024
F.2.1 DATA MODEL 1, SUPCON
Since SupCon solely learns representations, to enable classification, need add classifier / on top of the encoder g. we consider a linear classifier ((v) = B u where B E RR. The entire model, consisting of both the encoder and the added classifier, represented as I(g(z1)) prediction (z is given by sign(((g(z))) The test accuracy on the true distribution is then denoted as Acc go. B) = E y #===Duu()+EE) (1)(z)) v)].
B Wx. Its
The following theorem demonstrates that SupCon is not robust to distribution shifts, resulting in random chance accuracy across the entire true distribution 3 and making almost entirely wrong predictions examples where & y. To simplify the analysis consider minimizing the population loss noiseless setting. It's important to note that similar results general cases can be obtained using concentration bounds similar to those in Sections C.2.2 and C.2.3
Theorem F.1. Consider Assumption 4.2 and additionally consider the noiseless setting where T1.T = and let n + cc. Let W be the minimizer of the SupCon loss, then no linear classifier can separate the learned representutions of the two classes well. More specifically, vB,
Acc (W*,8) S 50% + o(1).
Meanwhile, the model's 's test accuracy on examples where a # is o(1,, ie. approaching zero.
Proof. First, we define
S SupCon 2 1
Σ
x(0)
Σ
vE(-1,1)
where ((v) = n/2 Ex,, from class I1 denotes mean of examples from class y. Based on our assumption, easily calculate S SupCon follows
2pspu (2pspu 1)2 DT.
S Suppon = PPsp 1
Let Σd Auuuui represent the eigen decomposition of SSupCon- Similar the explanation provided
in Section B.I by rewriting LSupCon as a matrix factorization objective and applying the Eckart- Young-Mirsky theorem, we obtain the minimizer of the loss follows:
w-T W
Nuay
We can casily compute the eigen vectors and eigen values for SuppCon A1
(2Pspu
V (2pirs 1)2+1 2pyy
1)2 1, 11 Dr
and da
Ad 0. Therefore, W
(2paps 179+1
/ 12p 1)2 +1 Pl
DD where P can be any p x 1 vector with / (2ppps 1)241
2Pr
(2prse - 11241
norm 1. Consequently
2((2psp 1)2 Pl
2P
W
DJ x
(2Pspa 1)2
(2pspu
1)2
2((2psp - 1)2 + Pl
2Pspa
(2pspu 1)2 2.
(38)
(2popu 1)2
For any B, the test accuracy is given by
Acc. (W*, B) Pr(yf W' "ol''TW''' >
By equation 38 and some calculation we get
Acc x- (WC..WW.F. Pr(u + U2 -1 - (2p 1)ay)
27

---

Published as a conference paper at ICLR 2024
where
VI -cooe
1/2 =y Ep
and each can considered as a Gaussian random variable with zero mean, independent from each other. Therefore
Pr(11 P2 -1 - (2papu 1)ay) Pr(m1 + 122 -ppoo)
Pr(ui + 2(Pspu 1)).
Considering
Ui 4-2
(O, 1), we have
V + (2pyp 1019p-11'00 1)
2pspu
2(1 Pspa)
gc (W'B)
(
+ $
2
0
(2p
1)
0 + (2pspu 1)20 ? spi
spu
1
2pppp
+
) + o(1)
because Pspu = -(1)
2
Liue
(2pppo
1)20
<50% + o())
Additionally, the accuracy on examples where + y given by p
2(1-P(u)
o(1).
V Vo + (2p 1)=0
ie., almost zero.
F.3 DATA MODEL 2, SUPCON
Theorem F22. Under the same assumption in Theorems 4.7 and 4.8 Let W be the minimizer of the Lumpoon loss. If we train a multi-class linear classifier f( Bf with with B2nxy using Cross- Entropy loss on the learned representations (i.e., given by W*x1), ) the test accuracy over the true distribution is 50%. Formally, let B be the trained linear classifier Acc (W*. B') 50%. More- over, no linear classifier can achieve accuracy better than 75% i.e., YB, Ace (W* B) 75%.
Proof. First, define
S SupCoon
1
(1) 2f (v)T
2m
where x((² examples from class y es denotes the k-th standard basis in IR" Similar to the previous subsection, by rewriting the objective and applying the Eckart- Young-Mirsky theorem, we obtain
Ez, from class II = |ce ace; IT for any with alias (k, Here
W W
Auuu
where Ei N is the eigen decomposition of SupCon The eigenvalues and vectors can be calculated as follows:
A 2m a1+ u: = Dr( VI a
2
a
TTT
Vi E [m]
VI a
A 0, Vi m.
Therefore
1
2(1 + 2 W = 2m -
where P E RRm can be any matrix with orthonormal columns. Consequently, for any example in class (k, c).
W x
2(( + a²)
c
+ sign(z(k + ml) v1 + a²
a²
Pec-
(39)
2m
v1 a
28

---

Published as a conference paper at ICLR 2024
On the training distribution, sign(=[k + ml) is always Therefore, the linear classifier trained the representations of the training data would be (recall the relation between CE loss and SVD in Section
B.1:
Ita"
2m 1
2 I e|
B
PT.
2m-1 1
Ita C2
2m-1
1
Ita e22
On the true distribution, B can make correct predictions for examples that also show during training. However, for any example æ, from class y with alias (k: c). its z satisfy that z|k + m] ca, can be observed that x the overall accuracy is 50%.
x [y], leading to incorrect predictions. Therefore, c) > (B'W*a 2;
Next, we analyze the case for arbitrary B. For any two classes (k, -1) and (k, 1) in the true distribution, we can group them into four groups denoted by (essig(e((++ml)> i.e. based on the combinations of e and sign(z(k: + ml). Since a 1, we have
a V1+a² V1+a² V1+a²
a V1+a² V1+a2 + v1+a² < VI+a²
a²
a² v1+
we conclude that the four groups lie on the same line. Moreover, going from G-1.- to G1.1, we pass through G1 first and then G-1.1. Given this order, no linear model can classify more than three of the four groups correctly. Since this is true for any pair of classes (k, 11 and (k, 1), it follows that no model can perform better than 75% accuracy on the entire distribution.
Combining this with equation 39
G EXPERIMENTAL DETAILS
G.1 THE SEMI-SYNTHETIC EXPERIMENT
Image generation. Firstly, we define three types of blue: dark blue, medium blue, and light blue, well as three types of red: dark red, medium red, and light red. Then, we modify images in the MNIST dataset. To generate the training data, for each image with digit between 0 and 4, there is 99.5% probability that the background will be colored with random shade of blue from the three types; otherwise, it will be colored with random shade of red. Similarly, for each image with digit between 5 and 9, there a 99.5% probability that the background will be colored with a random shade of red from the three types; otherwise, will be colored with a random shade of blue. The task to classify whether the digit in an image falls between 0-4 or 5-9. To generate the test data, the color is uniformly selected from all six colors for each image.
Caption generation. We simulate captions using vectors, Each caption is 200-dimensional vector generated + &, where NN, 2000 I200) and [a, ....0.... with a, b generated as follows.
First, we define dictionary that assigns value to each color and digit:
DICT = (0: -4.5, 1: -3.5, 2: -2.5, 3: -15,, 4: -0.5, \
5: 0.5, 6: 1.5, 7: 2.5, 8: 3.5, 9: 4.5 11
"dark blue": -2.5, "medium blue -1.5, "light blue ": -0.5 1,,1 S
"light red". : 0.5. "medium red": 1.5, "dark red ": 2.5
29

---

Published as a conference paper at ICLR 2024
core:
spurious:
99.5%
0
1
2
3
4
0.5%
0.5%
5
6
7
8
99.5%
Figure 4: Construction images
We also define the following values as the means for each category:
meanp
DICT 225
d=1
means.9
DICT(d] 2.5
DICrr blue"] DICTII meddium blue"] DICTT blue"] 1.5
meanhbue
DdCrk" red"] DICT[" red"] DICT("light red"]
mean
1.5
3
For an image with digit d and color c, the corresponding and are given by:
DICT(d):
with probability Teure with probability Teoore
mean044 d E (0,1,2.3,4) else means-9,
DICT(C]
with probability Tspu
mean if is kind of blue else mean,ed-
with probability 1 Tspu-
Training details. For MMCL, we choose LeNet with an output dimension of 128 our vision encoder. For the 'language' model, since the captions are represented as vectors, we employ a linear model with an output dimension of 128. We use the CLIP loss, but without normalization when computing similarity. We use momentum SGD as the optimizer with a learning rate of 0.01, weight decay of 0.001, momentum of 0.9, batch size of 128. The model is trained for 100 epochs. For SL, we train LeNet using momentum SGD with a learning rate of 0.01, weight decay of 0.001, momentum of 0.9, batch size of 128, for 40 epoch minimize the Cross-Entrope loss.
In distribution accuracy. To measure the in-distribution test accuracy, we evaluate the models on dataset constructed in the same way the training data but with images from the MNIST test set. [egure5]preeenn the results, showing that supervised learning achieves the highest in-distribution accuracy. This indicates that the improvement out-of-distribution accuracy shown in Figure 2 can only be attributed to MMCL's superior robustness. In other words, MMCL enhances both effective robustness and relative robustness, as defined in (Taori al. 2020).
G.2 EXPERIMENTS ON MS COCO
Datasets. The MSCOCO dataset Lin et al. (2014), Microsoft Common Objects in COntext, is comprehensive computer vision dataset comprising diverse images annotated with object masks and captions. We train model MSCOCO and test different variants of ImageNet Russakovsky ctt al.] (2015) as follows. ImageNetIK, a subset the ImageNet dataset, contains over a million high-resolution images for image classification. Recc et al. (209)) addresses biases and
30

---

Published as a conference paper at ICLR 2024
100
Nsp 1. varying More
ID accuracy (%)
90
Toree 1, varying Rapu
80
supervised
60-
50
0.0
1.0 0.2
0.4
0.8
Meare or Nspu
Figure 5: In-distribution test accuracy evaluated on a dataset constructed in the same way as the training data but with images from the MNIST testset.
offers balanced category distribution. ImageNet Sketch Wang et al. (2019) features hand-drawn object sketches. while ImageNet-A - Hendrycks et al. (2021b) serves as an adversarial benchmark. ImageNet-R HeageNee-RHeendrckkss [Hendrycks al.] (2021a) offers real-world, unfiltered images, and ObjectNet Barbu et al.) (2019) emphasizes robustness with objects in varied, cluttered scenes.
Settings. (1) Training 3-layer MI.ss on top of frozen encoders (Figures) 3a and 3b). Due to the limited time and compute resource constraints, we consider simplified setting similar to the one used in Ren Li (2023). We use the pre-trained ResNet50 of CLIP, followed by a 3-layer fully connected network (MLP) with batch norm between layers for the image encoder part. During training, we freeze the ResNe150 part and solely train the 3-layer MLP. Similarly, for the text part, employ separate MLP with the same size on top of the pre- preitrei Transformer of CLIP and only train the MLP. The hidden and the output dimensions
this MLP are selected from the set {16, 32, 64, 128, 256, 512, 768, 1024, 2048, 4096, 8192) to produce different accuracy pairs. We train for 100 epochs with a batch size of 256. Other hyperparameters are set to their default values. Each experiment is run on I NVIDIA A6000. (2) Training the entire model (Figure Be). We use the official Pytorch implementation of CyCLIP Goel et al. (2022 3 We train CLIP (with ResNet50 as the image encoder and Transformer as the text encoder) on MSCOCO for 100 epochs with a batch size of 512. Other hyperparameters are set to their default values. Each experiment is run on 2 NVIDIA A6000.
Evaluation. Following the measurement proposed in Taori et al. (2020), which has become standard way of measuring robustness, we plot OOD accuracy (the average of the zero-shot accuracies on 6 shifted versions of ImageNet) against ID) accuracy (the zero-shot accuracy on the validation set of MSCOCO). In setting (1), to obtain different ID-OOD accuracy pairs, we train MLPs with 11 different widths described above. In Setting (2), to obtain different ID-OOD accuracy pairs, train models on 25% 50%, 75%, and 100% the original dataset. Note that, for ImageNet variants, only sample subset of classes (320 out of 1000) that can be mapped the 80 MSCOCO classes.
G.3 EXPERIMENTS ON CONCEPTUAL CAPTIONS
Datasets. Conceptual Caption (CC3M) Sharma et al. (2018) consists of around 3 million image- caption pairs which are collected and processed from Internet wehpages. The dataset is divided into Training, Validation, and Test splits. The Training split includes 3,318,333 pairs in which subset of 2,007,528 has machine-generater labels Ng et al. (2020). Utilizing the image labels in the subset of CC3M's Training split, we filter a subset of CC3M training images whose predicted labels belong to ImageNet classes with confidence score of at least 0.6. To mitigate the effect of class imbalance, we further select classes with at least 100 but no more than 10,000 samples. The resulting subset consists of 296,801 images corresponding to 316 classes of ImageNet. To create the training and validation datasets, we split the subset with the 7:3 ratio a stratified fashion.
Settings. (1) Training 3-layer MLPs on top of frozen encoders (Figures 3a and 3b). We use the same settings in Appendix G.2 except for the batch size which we increase 1024 because the training set
31

---

Published as a conference paper at ICLR 2024
Top1 accuracy
CLIP
OOD Accuracy (%)
50
SupCon
SimCLR
40
SL
25
50
100
ID Accuracy (%)
Figure 6: Comparison of robustness (OOD-ID relation) CC3M between different learning ap- proaches when changing the model width.
larger. (2) Training the entire model (Figure Be). We use the same settings as training MSCOCO in
Appendix G.2 We cannot increase the batch size in this case due to the GPU memory constraint.
Evaluation. We train different models on the training set and perform zero-shot 316-class classifi-
cation on the validation set for CLIP. Similar to Appendix G.2) all models are evaluated on the test sets ImageNet variants, and the average accuracy across these datasets is considered the OOD accuracy.
G.4 ADDITIONAL EXPERIMENTAL RESULTS
Comparison between different learning algorithms. In this experiment, we compare CLIP with SL., SupCon, and SimCLR. We train the same MLP as in CLIP on the pre-computed embeddings of images but with different losses for SL, SupCon, and SimCLR. The image embeddings are computed using the pre-trained CLIP image encoder. For a fair comparison, we also train them for 100 epochs with batch size of 1024. Other parameters are set to the same values the settings of CLIP. For SupCon and SimCLR, following the standard linear evaluation procedure, discard the projection head and additionally train linear classifier the representations to perform classification. The model width (i.e., the hidden and output dimensions of MLP) varies the (16, 32, 64, 128, 256, 512, 768, 1024, 2048, 4096, 8192]. Figure illustrates that training CLIP yields better robustness than other learning algorithms in terms of zero-shot accuracy. This result well aligned with our analysis in Appendix F Interestingly, the robustness of SL, SupCon, and SimCLR is almost similar, resulting in nearly identical slopes.
Comparison between CLIP and SL with different image encoder's architectures. In this experiment, we change the architecture of the CLIP image encoder. We utilize four different architectures including ResNet50, ResNet101, ResNet50x4, and ViT-B-32 whose pre-trained weights are available. Figure Jillustrtts that training CLIP on CC3M yields better robustness than training SL across different encoders' architectures.
32

---

Published as a conference paper at ICLR 2024
Top1 accuracy
OOD Accuracy (%)
100
CLIP-RN101
SL-N11
CLIP RN5004
SLRNS Du4
80
SLAIT-8-32
20
20
40
60
80
100
ID Accuracy (%)
Figure 7: Comparison of robustness (OOD-ID relation) on CC3M between CLIP and SL when changing the image encoder's 's architecture. The solid lines represent CLIP models while the dashed ones represent SL models.
33


Reviews:
[{"review_id": "6qtUocwFAZ", "reviewer": "Reviewer_1D1C", "summary": "This paper explores the mechanisms behind learning generalizable representations and robustness to distribution shift of existing multi-modal contrastive learning (MMCL) approaches such as CLIP. The theoretical findings attribute the superior robustness to MMCL loss and rich text annotations, specifically to the two mechanisms - (1) intra-class contrasting between image-text pairs, and (2) inter-class feature sharing enabled by MMCL loss. The theoretical findings are validated through experiments using synthetic dataset based on MNIST and robustness study on MS COCo/ImageNet shifted versions.", "questions": "- What are the six versions of shifted ImageNet used for evaluation? Is the entire test set from ImageNet-A, ImagetNet-Sketch, ImageNet-v2, ImageNet-R and ObjectNet are used for evaluation?\n- Why not consider domain generalization benchmarks such as domainbed [Gulrajani 2020] for distribution shift robustness evaluation?\n- I encourage authors to consider including prior works in the contrastive learning paradigm in the related works section for the readers to get the broader perspective on contrastive learning.\n\n[Gulrajani 2020] Gulrajani, Ishaan, and David Lopez-Paz. \"In Search of Lost Domain Generalization.\" International Conference on Learning Representations. 2020.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents a theoretical perspective of the multi-modal contrastive learning (MMCL) method's robustness to distribution shift, supporting the empirical findings from prior works such as CLIP. The work shows insights of using rich text annotations and explains how MMCL loss helps to learn representations with superior robustness against distribution shifts.", "weaknesses": "- The paper generalizes training with cross-entropy loss as the supervised learning, but there is class of supervised learning approaches based on contrastive learning/loss [Khosla 2020]. It is not a fair comparison to say the study is about supervised learning and MMCL\n- The study is based on a linearized contrastive loss function, so it not clear what is the effect of other variations of contrastive loss functions. Also, there is no supporting validation on if the robustness is due to single modality v/s multi-modality or cross-entropy v/s contrastive loss.\n- Lack of details on the experimental study. I suggest authors provide details on experimental setup and distribution shift study/datasets (at least in the appendix, if not able to provide all details in main manuscript)\n\n[Khosla 2020] Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in neural information processing systems 33 (2020): 18661-18673.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmbHJIWoCz", "reviewer": "Reviewer_id3b", "summary": "This paper is a study on the distribution-shift robustness of multi-modal contrastive learning (MMCL) compared with supervised learning (SL). The authors provide formal definitions of MMCL and SL, and define robustness evaluation metrics for each of them. They provide both theoretical and empirical results in their paper.", "questions": "- Is there a way to qualitatively describe the derived or observed conclusions, beyond just \"at this robustness is attributed to aspects of the MMCL loss function, i.e. (1) intra-class contrasting (2) inter-class feature sharing, as well as the nature of multi-modal data i.e. (3) richness of captions\"?\n-The semi-synthetic experiment seems not make much sense, as the task is too simple and the data is limited. Could you please explain the reason for using this dataset?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The results are shown in both theoretical proofs and empirical experiments, providing a thorough analysis of MMCL and SL.\n- For the theoretical parts, the conclusions look well-supported.\n- The conclusions are useful in developing new MMCL methods, in the direction of loss function and data filtering and representation.\n\nI am not a machine learning theoretical researcher, and I do not major in this direction. Therefore, I am unable to check the theoretical analysis carefully.", "weaknesses": "- There should be more points in Fig.2 to better show the conclusion. Given just 4~5 points, the observation is not very convincing. In Fig.2(b), the linear approximation for \"label\" is a little far-fetched.\n- Only one dataset, namely MSCOCO, is used in \"robustness on real data\", making the results less convincing. More datasets should be used here.\n- For the empirical parts, the authors did not mention which backbone (e.g., ResNet50 or ViT-base) they used for the evaluation. This might significantly affect the resulting accuracy. It is also interesting to investigate the performance difference between different backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCd7rq0wog", "reviewer": "Reviewer_8jub", "summary": "This paper investigates multimodal contrastive learning(MMCL) mechanism that leads to its superior robustness in zero-shot classification/Out-of-distribution tasks. The paper attributes such benefit to MMCL loss that features 1) intra-class contrasting, which enables the model to learn features with high variance (such as core features) than relying on the spurious features that usually have smaller variance (easier to learn as shortcut); 2) inter-class feature sharing such that captions/features in a different class that contains useful information about another class can be learned. Both experiments on synthetic datasets and real-world datasets are performed to validate the theoretical analysis.", "questions": "1. In assumption 4.2, is it always true that spurious features have a smaller variance? Sometimes the background features are treated as spurious features and there can be backgrounds that have high variance. How is $\\sigma_{spu} = O(1/\\sqrt{logn})$ determined?\n2. In section 4.2.1, more explanation on the setup and use of notation might help with understanding. Such as what are the indications of $\\beta$ and $\\alpha$? What is the meaning and usage of c? And what does coordinate k mean?\n3. Does rich details in the captions correlate with a high variance of features?\n4. In the experiments section, how is the different accuracy on In-domain distribution obtained?\n5. As the dataset scale is very different, the accuracy on ImageNet is quite small. Is CLIP trained from scratch on MSCOCO? Usually, CLIP models pre-trained on MSCOCO are evaluated on Flick30K. Are the numbers in Figure 2 reported as the average of ImageNet and its variations? To represent ID and OOD distribution, maybe it makes more sense to pre-train on ImageNet original version and evaluate on domain-shifted versions? In Figure.2b, the accuracy drops a lot as the label approach accuracy increases, does this indicate overfitting on the labels?\n6. One comma missing on page 4, the second last line. \"independently of each other\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this paper investigates an interesting problem from a theoretical perspective and provides insights on the mechanism of multimodal contrastive learning and why it learns more robust features compared to supervised learning. \n\nThe analytical framework is sound, and the exposition is straightforward. \n\nThe conclusions drawn make sense and explain why rich text description is desired and why MMCL tends to be more robust to spurious features compared to supervised learning.\n\nThe experiments section includes both synthetic and real-world datasets, and the setup is interesting and validates the theoretical findings.", "weaknesses": "For the theoretical analysis part, section 4.6 could benefit from more clarification on the notation and its indications.\n\nFor the experiments part, the training dataset (MSCOCO) scale is relatively smaller compared to testing (ImageNet and its variations).\n\nOverall I think this is a sound paper and I have a few questions listed in the Questions section.\n\nSome related works studying multimodal learning and how each modality could interact with and complement each other could be considered to be included in related works such as:\n\nMind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning\n\nAre Multimodal Transformers Robust to Missing Modality?\n\nUnderstanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning\n\nInvestigating why contrastive learning benefits robustness against label noise", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BdridQOsWQ", "reviewer": "Reviewer_RF8u", "summary": "This study delves into the mechanisms behind the success of multimodal contrastive learning (MMCL), particularly in models like CLIP. It identifies two key mechanisms contributing to robustness: intra-class contrasting and inter-class feature sharing. These mechanisms help the model avoid over-reliance on misleading features and enhance its generalization capabilities, particularly in zero-shot classification tasks. The paper also highlights the positive impact of rich, detailed captions for improving model robustness.", "questions": "Please address the questions in the above weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Task importance: This work addresses essential aspects of multimodal contrastive learning, pivotal for AI's real-world performance.\n\n2. Theoretical analysis: The study underscores the necessity of moving beyond surface-level results and engaging in deep theoretical scrutiny to truly understand the mechanisms that drive successful outcomes in contrastive learning.\n\n3. Interesting discoveries: The research brings to light interesting insights, particularly around intra-class contrasting and inter-class feature sharing,", "weaknesses": "1. Lack of distinction from traditional approaches: The research does not sufficiently differentiate multimodal contrastive learning (MMCL) from traditional single-modality contrastive learning. It's unclear what unique perspectives MMCL brings compared to these established analyses for conventional contrastive learning. Clarifying this distinction is crucial for understanding the specific contributions and innovations of MMCL.\n\n2. Assumptions about feature learning: The paper posits that intra-class contrasting allows for the learning of high-variance features, leading to the acquisition of generalizable core features beneficial for out-of-distribution (OOD) generalization, especially when annotated with text. However, it does not adequately explain or justify why this effect occurs. A more thorough explanation is needed to understand how intra-class contrasting specifically contributes to OOD generalization.\n\n3. Need for more intuitive theoretical explanations: The theoretical analysis could benefit from more intuitive explanations, examples, or visual aids. As it stands, the theoretical aspects may be challenging for readers to fully grasp, especially those unfamiliar with complex concepts in contrastive learning. Simplifying these explanations could make the research more accessible and understandable.\n\n4. Limitations of experiments: The credibility of the observations is somewhat compromised by the scale and nature of the experiments conducted. The use of small-scale or toy experiments raises questions about the findings' applicability in more complex, real-world scenarios. Future research would benefit from more extensive, large-scale experiments to test the theories in environments that more closely mimic real-world applications.\n\n5. Implications for method design of contrastive learning: The study falls short in outlining how the observations made could influence future designs of contrastive learning methods. While it discusses the mechanisms of MMCL and their benefits, there is a gap in guidance on applying these insights to the practical construction or improvement of next-generation contrastive learning models. Providing clearer, more detailed pathways from observation to application would enhance the paper's utility for future research and development in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)