Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Authors: Zixiang Chen
Yihe Deng
Yuanzhi Li
Quanquan Gu
Venue: ICLR 2024 poster (2024)
Keywords: Representation Learning
Multimodal learning
Learning Theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the substantial practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. In addition, we conduct empirical evaluations on real data to back
up our theory. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.

Published as a conference paper at ICLR 2024
UNDERSTANDING TRANSFERABLE REPRESENTATION LEARNING AND ZERO-SHOT TRANSFER IN CLIP
Zixiang Chen Yihe Deng"", Yuanzhi Li", Quanquan Gu $ *Department of Computer Science, University of California, Los Angeles MMachnne Learning Department, Carnegie Mellon University, Pittsburgh (chenzx19, yihedeng) ) @cs... ucla. yuanzhi Machiilaaan Gandrew.. cmu. edu, qguêcs www.auclac...oo edu
ABSTRACT
Multi-modal learning has become increasingly popular due to its ability to lever- age information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and cxhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how fea- tures from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-ther the--rrettttt methods on benchmark datasets.
INTRODUCTION
Multi-mode learning (Ngiam al. 2011) integrates information from variety of data types, resulting in Al systems that are both robust and precise. Recently, CLIP (Radford et al.) 2021) emerged as milestone work that leverages vision-language contrastive pretraining to jointly learn image and text embeddings, using the vast amounts of image-text data available on the web. During the training process, CLIP considers image-text data that appear together as positive pairs and other combinations as negative pairs. The goal is to maximize the embedding similarity for the positive pairs while minimizing for the negative pairs. Remarkably, this approach has achieved significant success in zero-shot transfer (Lei Ba et al.) 2015), indicating the model's ability to handle great variety of tasks without prior exposure to any of their training data. Inspired by CLIP's groundbreaking
zero-shot capabilities, subsequent studies (Yao et al. 2022 Li et al.] 2022, Mu et al. 2022 Goel ct al., 2022; Zhai et al., 22022 Alayrac et al. (0222 emerged with the primary objective of further enhancing CLIP's zero-shot performance. Despite the empirical success of CLIP in zero-shot transfer, the theoretical understanding of how it works remains elusive. An intriguing inquiry is thus: How does CLIP learn representations that are transferable to the various downstream tasks?
This paper delves into the mechanisms through which CLIP learns transferable representations (i.e., embeddings) and demonstrates how such representations ensure successful zero-shot transfer for downstream tasks. We begin with identifying several challenges associated with the theoretical analysis of the transfer mechanism in CLIP: (1) alignment between different modalities, (2) unique features in different feature domains, and (3) sparsity of shared features across domains. In particular, unlike unimodal contrastive learning where the embedding function is shared, CLIP employs different embedding functions f and g for different modalities. This difference poses the alignment challenge specific to muuli-modda learning. Secondly, the feature domains lie in different spaces and may lack a one-to-one mapping. Some features are shared, while others are unique. Take Figure ] as an example. The attribute "stop sign" is shared feature in both the image and the text. However, the "blue sky" and "white cloud" are examples of unique features in the images that are not evident in the caption. This misalignment causes bad alignment at initialization. Lastly, the shared features in multi- contrastive learning (e.g., objects) can be sparse, compared to the unique features (e.g., textures, colors). Consequently, certain image-text combinations, despite not being paired, may still have
"Equal contribution.

---

Published as a conference paper at ICLR 2024
0.8220
Seif Standling Features
Shared Features
The stop sign is located corner.
Modality
STOP
Modality
0.7549
0.7132
Modality #1
stop sign front brick building.
STOP
Modality 42
Transterable Space
0.7428
Figure 1: Illustration of the Challenges. Left: The feature domains are different and not one-to-one mapping. We need to learn transferrable features while preserving the shared features. Right: The image-text data show in the same batch have similar shared features since the shared features sparse (here "stop sign"). The learned similarities between each image-text pair are very close.
shared features, suggesting they should be treated positive pairs. This challenges the traditional view of considering image-text data not paired together as negative pairs. To tackle the above challenges, we present our theoretical result for transferable representation learning in CLIP and summarize our contributions follows.
We theoretically examine transferable representation learning in CLIP. Our analysis shows that if near-optimal network is obtained on the training data, features from different modalities become aligned, enabling zero-shot learning if appropriate prompts are issued. We also demonstrate that, interestingly, contrastive learning with sparse features may lead to unexpected positive pairs. Therefore, we need to take into careful consideration. Moreover, while previous studies typically require very large batch size for training, our theoretical framework applies to small batches. Building upon our general theoretical findings, we delve deeper into specific cases, providing more comprehensive theoretical insights. We illustrate how multi-modal learning aligns different features and reveal when the learned features obtained by CLIP can outperform those obtained through naive square loss. By comparing CLIP loss and square loss, we formally established that CLIP is an effective learning objective for zero-shot transfer tasks, whereas square loss does not.
We conduct experiments on real data confirm our theoretical predictions. Furthermore, inspired by our theoretical findings, we propose a new regularization technique for CLIP that effectively leads improved zero-shot performance. Empirical results confirm that the proposed regularization can effectively improve the zero-shot performance across various tasks.
Notation. We use lowercase letters, lowercase boldface letters, and uppercase boldface letters to denote sealars, vectors, and matrices, respectively. For a vector x, we use ||||2 to denote its Euclidean norm. For a matrix W, we use |W] || to denote its Frobenius norm. Given two sequences {xn} and {yn}. we denote Zn = O(yn) if |ral I Cilyn for some absolute positive constant C1. Zn (((,) if [z,]]
C2lvnl for some absolute positive constant C2. and In. +(y,) if Cslynl S | S Calynul for some absolute constants C3, C2 O. We also use ((-) to hide logarithmic factors of in O(-). Additionally, we denote In, = poly(yn) if xx, Ovy ) for some positive constant D. and J. = polylog(y,.) if xr, = poly(log(us))) (log(yn)). We also denote by J, = o(yn) if lim Tn. In/Un Un = 0. Finally we use [N] to denote the index set (1.....N .N. In the function space, let (f) denote the ball of radius r centered at f. with the metrics ||. lloo A set C' the covering of function class FF with radius if and only if F C (r The covering number of JF with radius r the minimum cardinality any covering of F. denoted as (FFr)).
RELATED WORK
HisionLLanguage Pre- Fre-Training. While labeled data are expensive and relatively scarce, images
paired with text descriptions are available in much larger volumes Thomee al.) (016). Conse-
quently, numerous studies (Gomez et al., 2017 Sariyildiz et al. 2020 Desai & Johnson [021 Zhang
al. 2022; Liang et al.) 2023) have focused on leveraging free-form natural language supervision
to learn visual representations. Recently, CLIP (Radford et al. 2021] and ALIGN (Jia et al. 1021
have emerged as prominent works extending contrastive learning to the vision-language pre- pre-training
framework. Built upon CLIP's success, several studies (Pham et al. 202 Gao al. 2022 Saito
et al. 2022) have refined CLIP's contrastive methodology to better learn from web-scale image-text
data. Notably, UniCL (Yang et al. 2022) additionally incorporates image-label data, enabling the
2

---

Published as a conference paper at ICLR 2024
identification of broader range of positive pairs. FILIP Yao al. 2022) introduces fine- grained contrastive loss tailored for transformer architectures. DeCLIP (Li et al. 2022) and SLIP (Mu
et al.
2022) additionally incorporate eeddaityy self-supervised learning. CyCLIP GGoel et al.
2022) introduces two regularizing terms enforcing cross-modal and in-modal consistency. LiT (Zhai
et al.
2022) and Flamingo (Alayrac al. 2002) consider training from pre-trained single-modality
models. In our empirical validation of theoretical findings, we employ the same setting and train from pre-trained image and text encoders.
Theory of self-supervised learning. Numerous studies have been conducted to understand unimodal contrastive learning, a widely used self sulf-suuee learning approach rooted in data augmentation
(Saunshi et al.| 2019: Tsai et al., 20200 Mitrovic et al. 2020 Tian et al. 2020: Wang & Isola 2020
Chen et al.] 2021] Wang & Liu 2021; Tosh et al. 202 HaoChen et al. 2021 Wen & Li 202T
Saunshi et 2022). In multimodal learning, theoretical explanation has been explored in several
studies (Zadeh et al., 2020 Huang et al. [021]] Lee et al. 20200 Nakada et al. (023). These works
have established that multimodal learning can surpass unimodal learning in terms of performance. For instance, [ee et al. (2020) employed square loss prediction to learn image representations
under certain conditional independence assumptions, offering generalization performance guarantees. Mcanwhile, Nakada et al. (2023) examined CLIP within specific linear representation settings and emphasized its correlation with singular value decomposition (SVD). We note that, these related works have not considered the zero-shot transfer mechanism and thus can't 't adequately explain the zero-shot transfer capability of CLIP.
PROBLEM SETTING AND PRELIMINARIES
3.1 DATA DISTRIBUTION
In our paper, focus on the setting where the image and the text y conditionally independent given the shared feature z.
Assumption 3.1. Let (x, be generated from the joint distribution Dxxy- We assume z to be a shared feature of x, satisfying x y/z, and further denote (x, y, that follows the joint distribution D Dxxyxz x with marginal distributions Dxx2, Dy xz. We further assume to be a discrete and sparse random variable z E V = (VI,..., VK} with P& == IP = vk).
Intuitively speaking, the shared feature z in the above assumption may denote set of shared topics or keywords underlying image x and text y. We can consider the following simple example to understand it. Let = [0, 1, 1] represent the existence of topics "chair" and "table" and the absence of topics "car" and "train". Then, x and y are generated given z such that they both include "chair" and "table", yet with different unique features and noises.
Remark 3.2. The assumption of conditional independence is frequently made in the analysis of self-supervised learning (Saunshi et al. [219 [ee et al.) (2021) and dimension reduction algorithms (Fukumizu et al.] [2044] 2009). Under the premise that x, y are conditionally independent (CI) given z, it can be posited that any additional patterns found within x z and y z should be interpreted as unique features. Notably, in the absence of discrete and sparse constraints, a suitable z can always be found, given that one could simply assign z x or z = y. From the generative model's point of view, Assumption 3.1 naively holds when the data are from some generator with x = Ti E) and
y T2(z, 5) where E $L Llz.
3.2 LEARNING VIA CONTRASTIVE LOSS
CLIP is trained on millions of image and text pairs. Formally, we assume the data set S is drawn from the distribution Dxxy defined in Assumption 3.1 The CLIP architecture has three main components:
(i) an image encoder network g that can encode the image x into the embedding g(x) E RRd: (ii) a text encoder network h that can encode the text y into an embedding vector h(y) E Rd; and (iii) a score function f( y) = sim(g, h) that measures the similarity between the image x and the text y given their embeddings g, h (e.g.. f(x, y)
h(y)))
During the training, we will sample batch of image-captions pairs S' = fx,, vil S. The contrastive objective in CLIP aims to align the image representation g(x) and text representations h(y) by minimizing the following loss function,
Ls (f.T) = B
1
log iES'
Σ
EjeS' exp ({(3),,)/7))
exp (xx..
1 BBBL
Σ YES'
-
log
exp (S(x, Ejes exp V:)/T)
1
Σ
log
Σ JES
exp ([(()) ((x))
B
iES
3

---

Published as a conference paper at ICLR 2024
1 B
Σ
Σ log &ES'
Σ exp exppp1((eee) (
(3.1)
JES'
where 7 >> 0 is a temperature parameter. The training loss Lir over single epoch can be viewed as the empirical version of the following population loss,
E
log
Σ :E(B3]
exp ([I(x),)t) - -(x))
+ E
log
Σ :E(B)
exp (1
(3.2)
where the expectation is taken with respect all B random pairs ye) i.i.d. sampled from Dxxy- Therefore, CLIP learns the score function ʃ with the corresponding representations g and h by minimizing LnD" ()) In fact, we can divide the training dataset S into n batches Sk The following theorem shows that the empirical loss Bs(f. == (1/n) Exew (f. concentrates on the population loss when is large enough.
Theorem 3.3. Suppose
(0, 1) and n 2 e M log B) log (og(2N(F,c/8MM) then with probability at least 1 - 6, we have
[)((f.T) LDe Se
for all function F and [l S M, where NNF c) is the covering number of F.
Theorem 3.3 shows that the generalization gap (f.
(f. r) approaches zero as the number of batches n increase. practice, the batch size limited by the GPU's memory and is smaller than
the number of batches (or the number of training examples). Therefore, instead of letting the batch [0000|Pham size go to infinity like in prior studies (Wang & Isola [2020] [Pham et al.) (021), we keep the batch
size B as constant in (3.2) and Theorem 3.3 to enable the analysis of CLIP even for small batches.
Pham al. (202T) also provided the generalization gap for CLIP. However, their result for B - oo and loss function without the log term, i.e., exp (J(xi, (.)r)))))//// ( (Ejes' exp (a)x)))))/T))
4 TRANSFERRABLE REPRESENTATION LEARNING
The key idea of CLIP is to pull the embeddings of positive image-text pairs together while pushing the embeddings of negative pairs apart. For the data pair (x,y') y') generated with x ~ Dxlz: Y' ~ Dyln' (x,y') is a positive pair if z = z' and a negative pair if z & z'. The reason is that when z = z', the joint distribution of (x, y') is the same the joint distribution of (x. mutually independent given the latent variable z. Next, we will show that the learning objective (3.2) will lead to the distinguishable representation of different latent variables under certain assumptions.
Dxxy'= since y
Assumption 4.1 ((a, 3, CoCplleette)s There exists a score function f bounded by 1 (i.e., If"l & with f sim(g", h") satisfying the following properties,
For any + z', let D Dxja. Dy 2/)::3 D y' Dyln- With probability at least 1 a, we have (xx'x) S (x y) 7 and (x) (x.
Let (x, D xxxxxz, assume (xy))) [Var<j y)) [xxx)) (f S S.
In simple terms, Assumption 4. is made on the data distribution to allow the existence of good encoding functions g" and h" Specifically, the first bullet guarantees that the data with different the underlying shared feature, is well distinguishable with margin 7. If the data from different z does not satisfy this condition, the majority of the diagonal term f() in (3.I) can be smaller than the off- diagonal term f yt). In other words, all encoding functions may yield higher similarity score for negative pairs than positive pairs, which is not favored by the mechanism second bullet requires the similarity score within each underlying shared feature not vary too much, which is naturally satisfied if the learned embeddings g(x), h(y) are consistent and do not vary too much given the same In the following theorem, establish the result that CLIP model trained to convergence exhibits desirable properties in representation learning.
CLIP. The
Theorem 4.2. Suppose Assumption 4.1 hold and we can find an approximate minimum f EEF with respect to the temperature 7 such that f bounded by M and
Los (f, S LDg (f", .) t
(4.1)
Then the following results hold:
4

---

Published as a conference paper at ICLR 2024
1. For (x,z) Dxxxs {y& ~ Dyives k [K]], yet EKE[K] 1 valyk. we have
E
log Σ
exp ([](x,
-[xx -
sc'.
(4.2)
KE(K]
2. For (y ~ Dyxui
P [K]], x=tx== =EE 11 (2
vR)XE, we have
E
log Σ
f(x", exp ([f(xx.)) -(x") ((xxx)))//) (x11/1) sc'
(4.3)
KE[K]
3. For (x, ~ Dxxyxxs variance B(y Var (fx, + B(xxx) (a 1 16M2(.
where é' = ( CF +2 (xxBBexppp/000 -/T) and C = Õ(), CB Ō(max Pr /B).
Remark 4.3. Theorem 4.2 establishes soft margin between CLIP's learned embeddings data of different z's. For instance, an image x has a shared feature z = V1, we have its accurate description it follows that log (EEtK: exp ([f(x,
y EkEjKI 1(z va)ya Y1. From
(x)))))/+)) is small. This can only occur when yx) < y1) for all 2 2, i.e. the trained model always yield higher similarity score for this image-text pair compared to all other texts generated on different topics. This outcome aligns with the expectation that image-text pairs with the same shared feature will yield the highest similarity score.
Remark 4.4 (Choice of temperature parameter). When the data is well separated (i.e., a, B 0). smaller temperature will invariably lead to a smaller and, consequently, better performance. In practice, is typically set to be 0.01, sufficiently small value that ensures the term exp(-7/T) is less than 0. 0.0000454 for 7 = 0.1. However, when the data is nonseparable (i.e., a and B3 exceed 0), a balance must be struck between the terms related t. As consequence, should not be too small. reasonable choice would be r = O(+/ log (B/c)).
Remark 4.5 (Batch size). While we do not demand an increasing batch size B, our analysis does suggest preference for larger batch sizes, they can reduce the constant C3 and consequently c'.
ZERO-SHOT TRANSFER
In this section, we will discuss why the embeddings learned by CLIP in Section enable zero-shot transfer learning tasks. In the zero-shot transfer task, we have K prompts {yk, k E [K]} where Yk Dylve- For new image x generated from Dx, we want to predict the label the shared feature z in x. For example, if x has shared feature vI, then the label of should be 1. As suggested by |Rafford et al. (202I). we calculate the similarity score between x and the prompts yı and pick the indices for top-r scores the labels of The following corollary provides the guarantee of zero-shot transfer learning for CLIP.
Image data
Automobio
"A flight was traveling when the animal got free tuesday night*
Image encoder
Bedd
M(x
Dear
Dog
Frog
Horse
"hiss fast food!"
Ship
glyx
y
Truck
Image-text data
Text encoder
Prompt: "A photo
Pre-training
y
Figure 2: Illustration of zero-shot transfer learning. With encoders jointly pre-train pre-trained on the image-text dataset, zero-shot transfer is done hy issuing prompts according to all potential labels of the task. With similarity score computed between the image embedding and all prompt embeddings, the label that resulted in highest similarity is the prediction.
5

---

Published as a conference paper at ICLR 2024
Corollary 5.1. Suppose the result of Theorem 4.2 holds for the learned similarity function f. Then we calculate the similarity score y(x, yx) for all k E [K] and pick the indices of the top-7 scores within the set (f(x. as the predictions of the image x. Then the top-r error is bounded by / log(1 r)).
In other words, Corollary guarantees that trained CLIP model can achieve small top-r error, where r is an integer usually selected as or 3 in real-data experiments.
Remark 5.2. The result in Corollary |.1 can be generalized For example, we can deal with the case where the distribution of the prompts Dylve and the image distribution Dx are shifted. As long the x" distance between the shifted distributions is bounded, we can provide top-r error guarantee (see Appendix for a detailed discussion).
out-of-distribution zero-shot transfer.
Next, we will introduce a specific problem to illustrate how CLIP can learn transferable features with distinguishable margins, which is hard to achieve by simple square loss.
Definition 5.3 (A Case Study). Let shared feature E IR be random variable uniformly drawn from the set V (Vi..... VK} where ||v& 1, max (V&, v2) = 1 - y. Let & E R R Ks be unique random features satisfying |1 ||51l2 S R and are mutually independent given z. The image-text pair is generated as
z
x=
G
= G [
H
C
=
H H2S.
where G E 12 Ka) is the image dictionary with full rank (K1 K2). H ( is the text dictionary with full rank (Ki + Kz).
For the distribution in Definition 5.3 locked image-text tuning is enough to learn transferrable features (Zhai et al., 2022). In particular, we choose the score function as fw (g(x), h(y)) where
the embeddings are g(x) = Wx, = y. Next, we verify Assumptions 4. for the specified distribution.
Lemma 5.4 (Completeness). There exist score function fx(x.y) = (W*x,y) with W E Rddxd.
satisfying
| 1
For (x,y,z) Dxxyxz, variance (yy Varx| y))] Exxxx)
0.
Let x y Dyla, D Dy'lz' where z j z'. With probability we have that yx (x, - 7 and yx) S f* Y
Then the standard gradient descent the empirical loss to learn the score function f, i.e.,
((+ w( (w(ffTT))
The following theorem gives convergence guarantees for CLIP and provides the upper bound of its zero-shot transfer error.
Theorem 5.5. For sufficiently large n, set the learning rate 1 = GG] ))) gradient descent find W within 4|| (()) - /l/:/(re (nc) iterations such that LD= (f.r) s (ffT..]]. LDe(f (f= .T + where f y) In addition, the top-r zero-shot transfer error is bounded by
2/log(1 + r), where c' (CB + 2)- c 11 (2xxp((-71)) and CB = Ö(K/B).
5.1 SQUARE LOss FAILS ZERO-So SHOT LEARNING
Another conceivable method is to use the square loss to align the embeddings of x, y. Here, we investigate why such simple loss can not successfully learn transferrable representations and reveal the significance of contrastive loss in multi-modal learning. In particular, we use E| [[]gg(x) - y to learn the embedding g. By Lee et al. (0021), we know that the embedding g(x) indeed preserves the information of the shared feature Z and can be used to predict the label i (the index of in the
dictionary) using linear probing with additional Ō(K) examples {(k,)), x Dx }. Given the success of g as a representation for the downstream classification problem, a natural question arises: Can the learned embedding be used for the zero-shot transfer task, using only K prompts yı, k [K] where y& ~ Dyiva? ?
Surprisingly, the answer is negative. We find that even if we can train with population risk and get the Bayesian optimal predictor, the learned representation g is not suitable for the zero-shot transfer.

---

Published as a conference paper at ICLR 2024
To make fair comparison, we also consider the data distribution introduced in Definition 5.3 and present the following results.
Theorem 5.6. The Bayesian optimal representation g g(x) H
[E((2]]
Since E[(|z] lies in the unique feature space, the accuracy of zero-shot learning can be largely determined by the unique features Ç. i.e., the quality of the prompt. In detail, given set of prompts {ya}. we evaluate the similarity between representations g(x) and h(yu) = Ye under different similarity scores, including (1) inner product similarity: yz) (g(x), yu): (2) cosine similarity: ys) = ([xx)/llg(x]]la) y&/ /lllylll))) and (3) L2 similarity: (-1) llg(x) h(yx)]l2. The
following corollary formally states the negative result.
Corollary 5.7. For the distribution in Definition 5.3 with H
margin 7 < 1/3, text unique
feature Ç R drawn from {21,e2} with probability 1 1/3, 2/3 respectively. Then, the zero-shot top-1 error is at least 1/(3)) regardless of the three similarity scores.
Remark 5.8. By Theorem 5.5 we can achieve arbitrarily small top-1 error by CLIP as long and 7 are sufficiently small. However, for the representation learned from the square loss, the top-1 error at least a constant even if we can achieve the Beyasian optimal predictor.
LEARN BETTER REPRESENTATION VIA REGULARIZATION
In Corollary 5.I we know that CLIP can achieve small error for zero-shot transfer tasks. In this section, we investigate how large the margin can be achieved between different features z's. Under the same condition of Corollary 5.1 we present the following corollary.
Corollary 6.1. Suppose the result of Theorem 4.2 holds for the learned similarity function f. We calculate the similarity score f(x,ya vv) for all i [K] Then with probability at least 1 4c', the top-1 result gives the correct answer with a margin 7.
Here, the margin depends on the temperature parameter т. Note that we only achieve the margin with instead of y guaranteed the Assumption 4.1 Therefore, CLIP needs to choose 7 < y to ensure good performance, indicating a theoretical gap for the learned margin. To further investigate this gap, we consider the simple case study in Definition 5.3 and have the following negative result.
Theorem 6.2. Under the same condition as Theorem 5.5 there exists a special case with initialization ( such that when we train the model with polynomial iterations T' = poly(n e, d1, d2). ( with probability at least 0.99, the top-1 result can only give the correct answer with margin ((T).
Such phenomenon also exists in real data: the margin will decrease when temperature decreases (see Figure 3}. The reason is that softmax function I.(a) = log(E, exp(a:)) convex but not strongly convex and has an apponential-decyiin tail. Once the score function f with the features and h achieves the margin of order ((()), the gradient will exponentially decrease. Therefore, the weight will not be updated effectively. To obtain larger margin, it is natural to add the following regularization to maximize the score the positive pairs and minimize the score of the negative pairs.
R(S) S-
Σ (x-Y)ES-
f(x,y'
1
Σ f(x,y'), (x,Y")E*
(6.1)
where S., the of positive pairs that have the same shared feature z'. and S_ is the set of the negative pairs that have different shared feature z o z. However, the set S_ is very hard to determine since different image-text pairs in the batch can possibly have the same shared features, as we demonstrated in Figure On the other hand, the set of S4 can be simply chosen the training data set Therefore, we propose to use only one direction in 6. as the regularization, i.e.,
R(F) = S
Σ f(x,)). (x,y)E
In particular, when g and h are normalized representations with unit L2 norm and inner product similarity y(x = (g(x), h(y)), our regularization can be viewed as the Lo2 distance between the embeddings since
Rs( = 2|5
[g( + [|b())]l 2(g(x), h(y)))())))]
(xx))S
7

---

Published as a conference paper at ICLR 2024
llg(x) h 1.
2|5
(x.y)eS
Similarly, for a sampled batch S", the regularized loss is defined as Lg (f. r. )) = (f, + (f), where À > is a small regularization parameter. The following theorem shows that the regularization will provably improve the margin.
Theorem 6.3, Under the same condition as Theorem 6.2 with sufficiently small and appropriately chosen X, within polynomial iterations I' poly(n e, d1, d2). we can find a score function f with large margin. In partieular, with a probability of at least 0.99, the top-1 result gives the correct label with margin ((r).
Recall in Theorem 6.2 where the vanilla model achieves margin of Ö(T). the regularization term provably improves the margin to ((7). Lastly, our regularization term shares similar concept as SimSiam (Chen & He 202Tp, which only considers the positive pairs in the single modality setting.
7 EXPERIMENTS
In this section, present experiment results real datasets to verify theoretical findings. Accordingly, we examine our new CLIP-like training objective and showcase its improvement in performance diverse zero-shot transfer and linear probing tasks.
Datasets. For performance evaluation, we primarily foeus Conceptual Captions 3M (CC3M) (Sharma et al. 2018) as the pretraining dataset, in alignment with prior literature (Li et al. 2022 Goel al. 2022). Additionally, use MSCOCO (Chen et al. 2015) in order to conduct lightweight real data experiments to validate our theoretical findings.
Architectures. We consider the same setting for experiments on all baseline CLIP-objectives. Following the original CLIP paper, employ ResNet (He et al.) 2016) as the image encoder and the Transformer architecture (Vaswani et al. 2017) as the text encoder. We utilize pre-trained weights for both encoders to achieve faster convergence. These include the pre-trained ResNet-50 from the PyTorch Image Models library (Wightman 2019) and pre-trained DistilBERT from the Huggingface Transformers library (Wolf et al. 2020). We note that, the setting training from pre-traines weights is also considered in several previous literature (Zhai et al.) 2022 Alayrac al., (022). Lastly, our experiments can feasibly ran on a single GeForce RTX 2080 GPU. Detailed hyperparameters and additional experiments are presented in Appendix C
7.1 EFFECT OF TEMPER
ON MARGIN
In support of our theoretical discussions in Corollary 6.1 and Theorem 6.2 that find the positive correlation between the margin and the temperature parameter, we conduct real data experiments to confirm the impact of temperature on the margin. In Figure we examine the margin distri- bution of CLIP models trained at varying temperatures. Specifically, the margin is evaluated by the difference he- tween a diagonal value and an off-diagonal value within batch: f(x,, yy) fx yi) and f(,,, y:) fxx, yj) (see Appendix A for details). We collect the results of untrained and trained CLIP models on all batches within the MSCOCO training dataset with batch size 64.
random
temp=0.01
temp=0.07
Count 17.0P)
-0.5
0.0
0.
1.0
Margin
Figure 3: The distribution of the margins with regard to CLIP models trained at differ- ent temperature values. Margin is computed within each batch of the data.
As depicted in Figure 3 CLIP model with random ini- tialization at the projection layers has margins normally distributed near zero, whereas trained models exhibit pos- itive margins, signifying successful training. Furthermore, we consider CLIP models trained at fixed temperature values of 0.07 (the default starting value for the original CLIP) and 0.01 (the clipping value). As observed in the figure, the margin distribution shifts to the left temperature 7 decreases, suggesting that extremely small leads to small margins, aligning with the results in Corollary 6.I
7.2 ZERO-SHOT TRANSFER
To confirm Theorem 6.3 we investigate the advantages of incorporating our regularization term during training by evaluating zero-shot transfer accuracy and linear probing on various datasets. We consider the following training objectives when adding our regularization: (1) the original CLIP (Radford et al.) 2021), and (2) CyCLIP (Goel al. 2022) with cross-modal and in-modal consistency regularizations,
8

---

Published as a conference paper at ICLR 2024
adopting the same hyperparameters for the regularizations outlined in Goel al. (2022). All models are trained on CC3M using the same model architecture, batch size, and optimizer settings. Further experimental details provided in Appendix [
In Table [ present the zero-shot test accuracy of CLIP models trained with the original CLIP objective and the CyCLIP objective. Firstly, we demonstrate the model's performance when training solely on the regularization term (L2) and compare that of the CLIP objective. In alignment with our Corollary 5.7, we can observe on real data that training exclusively on the L2 objective leads large error and even random guessing
the zero-shot datasets. Combining with our theoretical analysis, we show that naive square loss fails to learn transferable representations. In the context of multi-modal learning, contrastive loss is important. Moreover, confirming our result from Theorem 6..3 incorporating the regularization term into the contrastive objective effectively enhances performance across the majority of zero-shot transfer tasks. It improves over the baseline on out of 6 datasets by good margin. The best performance achieved by adding regularization to the CLIP objective outperforms its original objective by 3.62% on CIFARIO 10 and by 2.06% on average of all datasets.
In Table 2 we report the results of linear probing, where logistic regression classifiers are fitted to the embeddings learned by the image encoders of our compared models. This table offers an assessment of the visual representation learning for each training objective. Similarly supporting Corollary 5.7 training on the regularization term only results in learning bad representations that yield unsatisfactory performances linear probing. Moreover, in alignment with Theorem 66.3 observe that adding the regularization term consistently improves CLIP's performance across various datasets by an average of 1.54%.
Table Zero-shot top-1 accuracy (%). Notably, adding the regularization term successfully improves the baselines on out of the 6 datasets.
CIFAR1O
CIFAR100
STL10
Food101
ImageNetV2
DTD
Average
Reg
10.04
1.05
9.95
1.08
0.11
2.07
3.47
CLIP
63.85
31.17
90.35
8.39
20.24
21.22
39.20
CyCLIP
60.71
28.87
89.98
9.72
19.66
20.21
38.19
CLIP+Reg
67.47
33.33
92.64
12.14
22.36
19.63
41.26
Table 2: Linear probing accuracy (%). All logistic regression models are trained till convergence. Adding regularization term to CLIP provides decent improvements across all datasets. On CyCLIP, also makes improvements on the majority datasets.
CIFARIO
CIFAR100
STL10
Food101
DTD
Flowers
OxfordPets
Average
Reg
14.09
2.17
17.86
1.73
3.40
2.18
4.12
6.51
CLIP
87.30
66.03
93.26
62.8
56.70
70.24
72.91
72.75
CyCLIP 86.31
63.93
93.69
61.57
56.86
70.56
70.46
71.91
CLIP+Reg 88.49
66.16
94.98
63.39
$7.66
72.21
77.13
74.29
CONCLUSION
In this paper, we rigorously investigated the theoretical underpinnings of transferable representation learning CLIP, addressing the challenges associated with feature domain alignment and shared feature sparsity. We provided insights through detailed examination of specific cases and corroborated our theory with empirical evidence. Lastly, we proposed a regularization term grounded in our theoretical findings to enhance CLIP's performance in various downstream tasks, including zero-shot transfer and linear probing. Combining rigorous theoretical analysis with empirical validation, we contribute to the advancement of understanding in multi-modal contrastive learning.
Limitations and future work. We emphasize that our primary contribution lies in providing theoretical insights into transferable representation learning in CLIP, which assumes a one-to-one mapping between image-text pairs. Interesting future works include extending the analysis to more modalities and exploring other multimodal training algorithms. Another limitation of our work the limited computational resources, where we used relatively smaller training data than the large-scale web data used by CLIP and are also restricted to smaller training batch sizes as compared to industry standards.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank the anonymous reviewers and area chair for their helpful comments. ZC, YD and QG are supported in part by the National Science Foundation CAREER Award 1906169, IIS-2008981, CHE-2247426 and the Sloan Research Fellowship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Peter L Bartlett and Shahar Mendelson. Rademacher and Gaussian complexitie: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, and Masashi Sugiyama. Large-marghn contrastive learning with distance polarization regularizer. In International Conference on Machine Learning, pp. 1673-1683, PMLR, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprini arXiv:1504.00335.. 2015.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 123.222 JMLR Workshop and Conference Proceedings, 2011.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
11162-1117.. 2021.
Kenji Fukumizu, Francis R Bach, and Michael Jordan. Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5(Jan): 73-99, 2004.
Kenji Fukumizu, Francis R Bach, and Michael Jordan. Kernel dimension reduction in regression. 2009.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramid- clip: Hicrarchical feature alignment for vision-language model pretraining. Advances in Neural Information Processing Systems, 35:35959-35970, 2022.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-6718 2022.
Lluis Gomez, Yash Patel, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4230 4239, 2017.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self superviidd deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944-10955 10956, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Andrej
Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 3128 1128-3137, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps:
Provable self-supervised learning. arXiv preprint arXiv:2008.00 01004, 2020.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable sule-supe learning. Advances in Neural Information Processing Systems, 34: 309-323, 2021.
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE international conference on computer vision, pp. 4257-4255, 2015.
Yangguang Li, Feng Liang. Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2022. URL
https:://pperrevll
net. I
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis- mhiil-Ph Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. arXiv preprint arXiv:2306.05268 2023.
Jovana Mitrovic, Brian Mc Williamms, Jacob Walker, Lars Buesing, and Charles Blundell. Representa- tion learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922, 2020.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXVI, pp. 529-544. Springer, 2022.
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pp. 4348 4380. PMLR, 2023.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 689-696 2011.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Imagee Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pp. 3498-3505 IEEE, 2012.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2/11.1000 2021.

---

Published as a conference paper at ICLR 2024
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference computer vision, pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. PMLR, 2021.
Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400 PMLR, 2019.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Cuen-Yu Lee, Kate Saenko, and Tomas Pfister. Prefix conditioning unifies language and label supervision. arXiv preprint arXiv:2206.01125, 2022.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pp. 153-170. Springer, 2020.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- ence on Machine Learning, pp. 5628-5637. PMLR, 2019.
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy, Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202.14037, 2022.
M. Moein Shariatnia. Simple CLIP, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578, 2020.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Journal of Machine Learning Research, 22(281):1-3 2021a.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Journal of Machine Learning Research, 22(281): 1--3,, 2021b.
Hio-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying self-supervised learning: An information-theorrttiall framework. arXiv preprint arXiv:2006.05576 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
12

---

Published as a conference paper at ICLR 2024
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9939. PMLR, 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pp. 11112-11122. PMLR,
2021.
Ross Wightman. Pytorch image models.
http:s///gtthub..com/v....g
pytorch- www.aaaaaooooo.ooo models 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pp. 68-85. Springer. 2022.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning image-text-label space. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19163-19173, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine- fine-grained interactive language-iman pre-training. In International Conference Learning Representations, 2022.
Amir Zadeh, Paul Pu Liang, and Louis-Philipps Morency. Foundations of multimodal co-learning. Information Fusion, 64: 11888193,, 2020.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133 2022.
Tong Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2(Mar):527-550 2002.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
A DISCUSSION ON THE MARGIN IN CLIP
"Margin" plays important role in unimodal contrastive learning (Wagg & Liu 2021). which measures the desired similarity difference between positive and negative pairs in the learned feature space: f(x, x fxx x This metric ensures that the similarity of positive pair representations exceeds a specific threshold, while preserving a greater distance for the negative pairs. In practice, large margin encourages the model to learn meaningful and discriminative data representations, thereby achieving better results in the downstream task (Chen et al. 2021).
In exploring the CLIP model, we focus on the concept of margin from a multi-modal perspective.
For two independent tuple (x, y, 2) ~ Dxxxxx and (x', y', 2 ~ Dxxyxx, we formally introduce measure as follows
a = fz (axn)))-(xx)))
f(x,y' 5T) +P x z', ((xxx))f(xxx))) f(x.))
(A.1)
where y denotes the margin, and at is failure probability of failing to achieve this margin. We note that when z z', x, Y, x', y' will form positive pairs, thus excluded in equation (A.I). Unfortunately, we can access Dxxx in real applications but have limited knowledge of the latent variable z. This limitation complicates the identification of all positive pairs within batch of data.
A.1 MARGIN AND RESUAL-SEMANIOS ALIGNMENT
When g and h are normalized representations with unit L2 norm and we use inner product similarity f(x) = (g(x), h(y)). The formula fxx y) f(x. y") can be expressed as
f(x,y) - y(x,y') = 2 |g(x) h(y) h())
- - 22 |lg[x) h(y h(y))
ll((x) h(y')l 213
[g(()) h h ].
(A.2)
Nogative-pail ppir Distance
Positive ive ppai Distance
where the second equality uses the property of unit L2 norm. By (A.)), we can see that a larger margin value implies that the embeddings g and h of the positive pairs remain in closer proximity, while the embeddings of negative pairs are far away from each other. This is a crucial aspect of contrastive learning, especially when considering the CLIP model.
In unimodal contrastive learning, y = x typically follows the same distribution of x, and h is chosen to be identical to g. Consequently, the embedding difference g(x) h(y) will generally exhibits a zero mean. In this scenario, the variance of the embedding, rather than its mean, becomes the dominant term for positive-pair distance in (A.2). However, this is not the case for the CLIP model since x, y belong to different modalities, and thus h is no longer chosen to be identical to g. Moreover, identifying negative pairs in a batch for image-text data is challenging. To empirically mitigate the issue, Yang et al. 2022) proposed UniCL for multi-moda contrastive learning. Unlike vanilla CLIP, UniCL additionally consider image-label data and group these data with identical classes, which facilitates negative pair identification within the dataset. However, this strategy necessitates additional group information about the dataset, being either class label concept. Our paper aims to thcoretically tackle the identification problem by integrating this grouping mismatch into our analysis. We recognize the significance of empirically addressing this issue like Yang et al. (2022), but it goes beyond the scope of current work.
A larger margin of f indicates an improved visual-semantic alignment. Thus, we favor function f that achieves a larger margin 7 with smaller a.y, Under Assumption 4.1 we define the (a, 3, 7) completeness, ensuring the existence of such function. To find a function with a larger margin more effectively, we introduce new regularizer in Section [6) specifically tailored for the CLIP model. This regularization approach does not require identifying negative pairs and is particularly suitable for CLIP, as it only penalizes the positive-pair distance [lg(x) hly)lž
Chen et al. (202T) proposes novel large-margin contrastive learning (LMCL) method in unimodal contrastive learning, regularizing both positive and negative pair distances. In our study, we choose to regularize only the positive pair distance, acknowledging the unique characteristics of the CLIP model: different embedding functions g, h for images and texts and the difficulty in identifying negative pairs. We also conducted an ablation study for only regularizing the off-diagonal term in the batch. We find that off off-diagonal pair regularization yields marginal improvements in downstream zero-shot tasks and lacks stability compared to the regularizer proposed in Section (detailed in Section
14

---

Published as a conference paper at ICLR 2024
A.2 ESTIMATION OF THE MARGIN
In this subsection, will discuss how to verify the Assumption 4.1 and measure the quality of the learned function with margin. We introduce an approximate measure a, as follows.
ã, IP ((xx)) <T) ( +x(/((x)))) f(x)))
f(x,y)
(A.3)
ā differs from the at since we didn't 't extinguish different classes in the probability. Therefore we can easily calculate a, without observe z. In practice, (A.3) can be evaluated by the difference between diagonal value and an off-diagonal value within a batch: f() f() and
f(x,, y:) fx,, Yj) (as illustrated in Figure 3).
Moreover, we have the following upper and low bounds, which show that a, can approximate an.
Theorem A.1. Let 2 0, then have that
i 04 Ciy
Pk kc(K]
Σ
where pa is the probability of the classes in Assumption exact equality for y = 0.
3.1
Besides, the second inequality becomes
Proof.
āj ā-
+ + -(xx))
+ + -( In)
=
J(x,)) - f(x, <>) + p - (5)))>s) 5n)
Approximate Error
The Approximate Error has naive lower bound of O, and we can upper bound as follows
P(2 ((x,)) ((x, ((x,'') m7 =P -P([(x))) f(x,y') S 7/1 z') PP(z z²)
1 P f(x,)) f(x,y') s 0/z =1z=z'') z') IP(z z')
= 1/2 Σ pi.
kE[k]
the the inequality due to fact that 7 and the last equality is because y' and y symmetric give = z'. Finally, the inequality is an exact equality for = O.
By Theorem A.I
Ctyy and ã. are close to each other if max Pk is small, since
Σ RE[K]
Σ
Ph
Ph
max Pk
Pk
max Pk-
RE(K]
KE(K]
kE[K]
RE(K]
RE[A]
Relation with the Figure 6 ā. has strong relationship with Figure 6 where have plot the and f(x,y) - f(x',)) The figure can be viewed as the figure
distribution of of the probability density function, and ā. can be viewed as the cumulative probability function, which is the integral of probability mass smaller than From Figure learned with regularization has consistently smaller ã. for all 7
6] we can deduce that the CLIP
B DISCUSSION ON THE TRAINABLE TEMPERATURE PARAMETER T This section considers the setting where the temperature is also trainable with the following loss.
Laa (f,)) E log
Σ cxp ()
f(1 ((x.))))))))
LE[B
+ E
log Σ
exp ([
tE(B
15

---

Published as a conference paper at ICLR 2024
Suppose is clipped to be within the range [Tiia Tmax] it natural to assume that obtain function f with temperature 7 E [minn Tmax] such that
Lps (f.F)) z
min rEl7min,7mas
LDa ,T) +e
(B.1)
LDD (
(L,
min rE((mia: Fmua)
(f=(((;7))
(B.2)
-
(B.3)
where ē c- (f min, f LDS (ft. S Since č is smaller than
can get smaller é' in Theorem 4.2 and thus get smaller top-r error in zero-shot transfer task by Corollary 5.1 This observation implies that the representation (f, 7) found by trainable temperature can be better than the representation (f', found with fixed temperature
C ADDITIONAL EXPERIMENT RESULTS
We consider the same model architecture CLIP (Radford 50 et al. 2016) as the image encoder and transformer (Vaswani et al. 2017) architecture as the text encoder. Specifically, we use pre-trained weights for the encoders for faster convergence in training. We follow the code framework in Shariatnia (2021) and use pre-trained ResNet-50 from the PyTorch Image Models library [Wightman] 2019) and pre-trained DistilBERT from the Huggingface Transformers library (Wolf et al., (220).. We further have linear projection layers on both image and text encoders, the same as in CLIP, and consider the embedding dimension to be 512. As we are training small-scale data with pre-trained encoders, follow SShariatnia (2021) and use AdamW optimizer with learning rate lc-4 on the image encoder, le-5 on the text encoder, and 1e-3 on the projection layers, with weight decay coefficient 1e-3. Our code is provided anonymously Github"] C.1 IMAGE-TEXT TEXT RETRIEVAL
2021) and consider ResNet-
We additionally consider the image-to- to-ts-tttxetttttt and rext-to-image retrieval downstream tasks in the zero-shot setting. Following the setting outlined by Goel et al. (2022), we use Flickr30K (Plummer al. 2015) and MSCOCO (Chen al. 2015) datasets, which are well-established benchmarks for image-text retrieval tasks. We similarly focus on the test data from the Karpathy (Karpathy & Fei-Feil 2015) split, with Flickr30K comprising 1k test instances and MSCOCO containing 5k. Consistent with the findings Goel et al. (2022). we observe that text retrieval for given image tends to be less challenging than image retrieval for given caption. This is due the nature both datasets, where each image is associated with 5 captions. Our results, as detailed in Table and Table align with this trend. Notably, while CyCLIP does not consistently outperform CLIP, adding our regularization term consistently enhances the performance of both the CLIP and CyCLIP.
Table 3: Zero image-to-texs and text-to-image retrieval results on Flickr30K test set for CLIP with different regularization techniques (CyCLIP, our regularization, both).
Text R@
Text R@5
Text R@
Image R@1
Image R
Image R
Average
CL.IP 87.36
93.0
95.18
26.88
54.18
66.22
70.47
CLIP+Reg
87.42
93.42
95.82
29.94
58.00
69.82
72.40
CyCLIP
87.34
93.12
95.04
29.00
56.50
67.62
71.44
CyCLIP+Reg
87.20
93.20
95.56
29.14
56.94
68.64
71.78
Table 4: Zero-shot image-to-ter and eext-to-image retrieval results MSCOCO test set for CLIP with different regularization techniques (CyCLIP, our regularization, both).
Text R@
Text R@5
Text 10
Image R@1
Image R@5
Image R
Average
CI.IP 81.19
83.21
84,42
4.73
11.66
15.93
46,86
CLIP+Reg
81.25
83,31
84.49
4.98
12.14
16.66
47.14
CyCLIP
81.06
82.92
84.28
4.70
11.66
46.86 15.93
CyCLIP+Reg
81.31
83.28
84.65
5.27
12.17
16.70
47.23
16

---

Published as a conference paper at ICLR 2024
C.2 DISCUSSION ON THE "NEGATIVE" PAIRS
As previously discussed in Figure I and Section 6 the use of unlabeled image-text data in CLIP pre-training may lead to batches containing off diggonal pairs that are not genuinely negative. In contrast, in the unimodal setting (Chen et al. 2021), accurately identifying truly negative pairs is more straightforward due to the availability of class labels. However, treating all off-diagonal pairs as negatives in the CLIP framework may not be ideal. We investigate taking off-diagonal pairs within batch as "negative" pairs and sum them into a regularization term. Again, during the training, we consider sample batch of image-captions pairs 5" the negative pairs is thus
y:) S. The regularization term for {xx, yi)
R()==A Σ Σ f(x;, RES' $#4
where À O is the regularization parameter. In experiments, let X 0.1/(B² B) and all the other settings remain the same as our previous experiments. In Table our results show that while positive pair regularization markedly improves performance, off-diagonal pair regularization yields only marginal enhancements on some datasets and no improvement on others. This unstable performance may attributed to the presence positive pairs among the off-diagonal elements in the unlabeled image-text data.
Table Zero-shot top-1 accuracy (%) with regularization on positive image-text pairs and "negative" pairs.
CIFARIO
CIFAR100
STL10
Food101
ImageNetV2
DTD
Average
CLIP
63.85
31.17
90.35
8.39
20.24
21.22
39.20
CLIP+Pos
67.47
33.33
92.64
12.14
22.36
19.63
41.26
CLIP+Neg
64.36
31.01
91.25
9.59
20.17
20.74
39.52
C.3 INVESTIGATION INTO THE IMAGE-CAPTION DATA
In Figure 4 we focus on the MSCOCO image-caption dataset, specifically examining the existence of objects present in images but omitted in their corresponding captions. We found that significant portion of the data pairs contain least one such object missing from the caption. In Figure
10
Count (10*)
6-
2.5
7.5
10.0 12.5
17.5
Number of objects not in caption
Distribution of image-caption pairs in MSCOCO, where we count the number of object that appeared in the image but was absent from the captions.
Figure 4:
we present random selection of the image-caption pairs in CC3M dataset. These examples are illustrative of the whole dataset, although we cannot provide an exhaustive representation of the numerous examples within the dataset.
C.4 EFFECT OF TEMPERATURE ON MARGIN
Setup. For lightweight exploration in section 7.1 use the training dataset from MSCOCO (Chen et al. 2015) Image Captioning Task as the data for languuaee contrastive pre- pre-training Specifi- cally, the dataset contains 82, 783 images where each image coupled with 5 captions. We consider each amage-caption pair as data example in pre- pre-training and therefore arrive at 413, 915 pre-training data pairs. We further randomly split the data to keep 20% of the data validation set and stops
17

---

Published as a conference paper at ICLR 2024
"A very typical station." Missing: people, building, ect.
"A flight was traveling when animal got free on tuesday night." Missing: cars, road, ect.
"Venture funded company ealled
"Dog kilt Missing: grass, tree, ect.
these sugar-glazed slow-smoked ribs."
Missing: plate, table, ect.
Figure 5: Examples of the image-text pairs from CC3M. We identify few missing visual objects in the captions.
training as the contrastive loss on validation data no longer decreases avoid overfitting the small dataset.
Margin. Given training data batch, the margin consider as the difference between diagonal value and off-diagonal value: f(x,, y.) y.) and f(x,, yz) f fx; We consider CLIP models trained at fixed temperature 7 = 0.07 and 0.01. We note that 0.07 is the default value for to start training in CLIP and 0.01 is the clamping value (equivalently as the maximum logit scale of 4.60 (.602.2) Figure 3 we collected the margins from all batches of size 64 in the MSCOCO training data, where the data is randomly shuffled.
Additional Experiments. Here, additionally compare the margin distribution of CLIP trained at temperature 7 = 0.01, without or with our regularization term. We could observe that the margin distribution shifts to the right with the regularization term, which alleviates the negative influence of an extremely small temperature value.
+rog
Count (105)
Margin
Figure 6: The distribution of the margins with regard to CLIP models trained 0.01 with or withour regularization. Margin is computed within each batch the data.
C.5 ZERO-SHOT TRANSFER AND LINEAR PROBING
Setup. In the evaluation of zero-shot transfer and linear probing, we use CC3M (Sharma al. 2018 as the pre-training dataset, which contains around 3, 318, 332 image-caption pairs gathered from the web. While some URLs are broken so that we cannot download the images, we eventually reached pre-training dataset of 2, 786, 288 data pairs. When training CLIP models, we use the default coefficients of CyCLIP regularization terms of A1 = 0.25 and X = 0.25. For our regularization coefficient of À 0.1. As in CLIP, set the temperature from 0.07, equivalently having maximum logit scale at 2.6593. Lastly, we use a training batch size 32 and trained for epochs in the results reported in section 7.2
term,
18

---

Published as a conference paper at ICLR 2024
Table Summary of datasets used zero-shot transfer and linear prohing.
Dataset
Classes
Class Description
CIFARIO
10
Categories of animals and vehicles
CIFARI00
100
Categories of objects including animals, foods, vehicles and people
STLI0
10
Categories of animals and vehicles Categories of foods/dishes
Food101
101
ImageNetV2
1000
Categories of objects including animals, foods, vehicles and people
DTD
47
Categories of textures
Flowers 102
102
Categories of flower species Categories of cats and dogs
Oxford-IIIT Pet
37
Evaluations. As similar in previous works (Radford et
2021 Yao et al. 2022 Mu al. 2022
Goel et al. 2022). we consider the following image classification tasks for zero-shot transfer and linear probing: CIFAR 10100 (Krizhevsky) 2009). STL (Coates et al., 201T), Food101 (Bossard
et al. 2014). ImageNetV2 (Recht al. 2019), DTD (Describable Cexxtures/Cimpar et al. (2014)),
Flowers (Nilsback & Zisserman 2008) and Oxford- Pet (Parkhi et al. 2012). The dataset statisties reported in Table 6 For zero-shot transfer,
the same prompt engineering and ensembling as the original CL.IP and report the top-1 accuracy. For linear probing, as the same in CLIP, we train logistic regression classifier on the image embeddings generated by the image encoder of pre-trained CLIP models on the training data from the considered datasets. The classifiers are all trained to convergence and report the test accuracy on each of the test dataset of the tasks. We note that, due to the limitation of the training data CC3M, the zero-shot test accuracy of all CLIP-objectives on Flowers 102 and Oxford-III Pet are near random guesses. Therefore, we omit these datasets for zero-shot transfer.
Additional Experiments. We additionally report the zero-shot transfer results of the original CLIP objective and adding our regularziation term, on a different visual encoder architecture of TinyVIT (Wu al. 2022) with pre-trained weights from Huggingface.
Table 7: Zero-shot top-1 accuracy (%). Notably, adding the regularization term successfully improves the baselines on 5 of the 6 datasets.
CIFARIO
CIFARI00
STL10
Food101
ImageNetV2
DTD
Average
CLIP CLIP+Reg
52.02
15.57
81.89
7.92
16.91
11.80
31.02
53,30
19.67
83.76
7.99
16.06
11.53
32.05
D PROOF OF RESULTS IN SECTION 3
Proof of Theorem
3.3
We first prove that Lsg (f,T) is upper bounded by 4M log B/T.
Lg(f,r)=
1
Σ
log Σ exp ([f(x), (f f(xx))))//))
B
JES
log
Σ JES
exp ([I(xis)s)) -(x;
B
IES'
S
1
Σ log Σ exp (2M/T))
Σ log
E
exp (2M/T)
'E
jES
SES'
JES
4M log B/r.
(D.1)
where the inequality is by the fact the lfl S M. On the other hand, we have that
(f.
Σ log
Σ exp
jES
log
Σ
exp ([f(x;))5) -(x:))))))
B
SE
JES'
2
log
exp ([f(x,)))) - f(x.))))/T))
VES
19

---

Published as a conference paper at ICLR 2024
O.
where the inequality is because Exp function is greater than O. Therefore we have proved that I.g (J.r) E (0, 4M lT] For all fi, f2 E F and any batch S' with size B, we have that
(i,(f1,) -) Ls (f2.
Σ İES'
log
Σ jES
exp ([fi(x) (aoo))((((() (x (1(
log
Σ JES'
exp (11(x))))))-1xxxx..)))
f2(x:
B
$ES!
+
log
Σ exp
- ( /T)
B
iES!
JES
log
Σ exp (Ifa(xx.)))
iES'
jES
log
Σ
exp ([fi(x))
-f (ixx::)))
B
VES'
JES'
log
Σ exp (
(14 -21151 -2lloo//t))
JES
Σ exp ([fi - (((x
+ B
log
iES!
JES
log
Σ jES
exp - 211 - falloc1/7)
= =4]l/1 - /2/lo0/T.
(J1,)) ( S 411l - S2lloO/T, which yields < 4 - f2ll /T. Taking the expectation gives that |LD (f2, T) Li Tf) < fl f2l x/T By the definition of the covering set, the function class F can be covered by subsets BI...., Bx. that is F = B₁ U... U BR. where K = W(F ((Frre/16) and B1,...By Bx are the balls of the radius c/16 centered fi,..., fk. Then we have that
Similarly, we can get another direction Lg (12,7)
to L (f2.T) Lo (fi.
PS-Da
sup LEF
Laa (J,7) [s([,T) 2
PSND
sup (f,5) - -s(f,r)) 2
JEB,
kc(K]
PSNDa
TuDn T) - -T.s(Ja; 22/2) 2012/
KE(K]
Σ PSNDR K)(K]
- (1/m) Σ
Ls, T) 2))/2(12
[E[n]
nE²T
< 2K exp
8.M log B
nE²T
2N( (6) exp
(D.2)
8M log B
the first inequality is by union bound, the second is by triangle inequality, and the third is by Hoeffding's 's inequality and (D.I). Finally, plugging the condition n >
(8r-le-* 22 log B) log (2N(F (F.MFFMM///))))) into (D.2 we have that
PANDA sup (f, Li( r) 2 56. JEF
which completes the proof.
20

---

Published as a conference paper at ICLR 2024
E PROOF OF RESULTS IN SECTION 4
Lemma E.1. For by 2 0,j [m]] we have that
log
+
Σ
by
log(1 + bz).
je[m]
JE/m
Proof. Notice that
I JEE +b2)) 2
1
Σ by-
JE[J]
Taking the logarithm over both sides completes the proof.
Lemma E.2. Suppose that ai..... are i..d random variable sample lies in [-R, R] where R 2 1, with mean μ == [[a1] and variance o² [a( B[a,]) Then have that
Ellog Σ exp(()) 2 log(m) + + 4mR
m
1=1
Proof. Let ā = |-mn a:) m
1
log
exp(a:)) log(m)
a + log
exp(o - a)
m
m
1
log(m)
+
+
log
1
a2
a|²)
RmR
1=1
i=1
m
2 log(m)
+
04
+
aa
0)2
m
4mR
1=1
1=1
where the first inequality is by exp(t) 2 1 + (3R
VL |- R, It). the second inequality is
due to log(1 +) 31/4, [0, 1/3).
n
Lemma E.3. Suppose f is the function that satisfies Assumption 4.1 then we have that
6MBB/M + 6MMB// 2B eBxpp(n2/1 LDo (f" 2E log Σ 1(z 1(z)=2)) ) tE
Proof. Let the event & be the case that either i) = 1 and [fx (f (x1, S or ii) 2: F 21 and (x( - ff")=((xxx (x1, S -7 We also denote the complementary set of & to E
By Assumption 4.1 we have that
P z: 21) 8/1p²
P(E,, 2, + 21) Sa.
the first inequality by Chebyshev's inequality, and the second by margin assumption. Therefore, we have that (E))) < a + p/.². Next, let us decompose LD (f* 7) into three parts,
Lad () = =
log Σ
1 (z + 21) ((E)) exp ([1 - (x1,
Σ
((8)) exp
(E[B]
Σ
1 I(z 21) 1(8)) exp ([f* - (xx.
[E[B]
+ E
log
Σ [E[
I ( + 21) ((8) exp ((x.x)))/1)
21

---

Published as a conference paper at ICLR 2024
(1)
(I1*
(E
+
Σ
[(z z1) 1(E)) exp (x) - (xxx)))))/+
[B]
< 2E
log
1 B exp (-//T) +
Σ
({E)) exp (2M/7) + Σ
1(z = z1) exp (p/T))
122
12
( (- (////) ]
S 2
log
1 Bexp
+
Σ
2
log 1 1 (1)) exp (2M/+))
122
12
+ 2
log
+ Σ
1(z, = 21) exp (0/T))
(E.1)
122
13
where the first inequality is by Assumption will bound J, I2, Is separately.
4.1 the second inequality is due to Lemma E.1 Next,
S B Bexp(-7/7),
(E.2)
where the inequality is due to the fact that log(1 + x) S x.
3M 1 (1+ exp (2M/+)) < (PE)) - = (a + 3/p²) 3M
I E ((E)) log
(E.3)
where the first equality is due to log (1 + (8 exp (2M/T)) = 0) when 1 = 0, the first
incquality is due to log (e+x (2M/T)) < 3M/TT. The last inequality is due to P(E)) - a + 3/p².
Is s E log
exp (p/T) +
' 1 1(1 21) cxp (p/T)
122
1(2 = 21)
= p/T + E log Σ
(E.4)
where the inequality is because 1 < exp(p/T).
Plugging E.2) E.3 and (E.4 into [E.1] gives that,
Topa (an((",))) r) < 2B expl (xp(->/T)>>>)) + 6MBo/T (t00://+MMBB//TT)) 2p/7
+ +2 log
Σ
1(z, = 21)
tE[B]
< 2E
log Σ
1(z, ((2x=21)) 1(2,=
21)
+ +GM/B//744//NNB//++++ exp( exp(-7/T),
[E
where the second inequality is by choosing p V6MBB.
Proof of Theorem
4.2 First by Lemma E.3 we have that
L (f,T) Ip"( (f*, + 2E
log
Σ 1(2, = =1)) +e [E[B]
(E.5)
where = 3 + 2B exp(-n/T). Notice that
Lpe(f. (f.T) = E log Σ
exp ([f(x))))) -(x.))))-(xxx))). v1))
*E[D]
22

---

Published as a conference paper at ICLR 2024
log Σ (E
exp
(E.6)
Next, we prove the bullets in Theorem First and Second Bullet in Theorem 4.2 Denote the event & as the case that for all / 2 1, ze # 21. which is the event that CLIP favored.
2
one by one.
We first lower bound I1.
1 = E log Σ [B]
(z: + 2i) exp x1) f(x,,))))
(z = 21) exp (1f(x,,))) - -(x)))))))
+
LE[B]
E log
Σ
(z + 21) exp -
[E|B]
Σ
1 = 21) exp ([f(x,,))) -(x1)x))///
BE[B]
1(z * exp F(x.)))))((xx))///)))))) + +1)]
> E (E)) log
Σ (E|B)
+ E ((8)) log
Σ
1(z =2) ([xx
tE(D
E ((8) log
Σ
1(z * 21) exp ([f(x)) ( F(x,,))) +
[E[B]
+ E log
Σ
1lz = 21) exp -(x,,)))))))
LE[B]
1(z * 21) exp ([f(x, v1) F(x, +1)]
2 E 1(8) log
Σ
[E|B]
+ E log
Σ
1(z = exp exp (E
1/T)
LE[B]
E 1(8) log
Σ
1(z * z1)
exp exp ([f(x). -(x)) + +1) 1)]
[E|B]
+ E log (l(1E [B] 21 21 }11
(E.7)
where the first inequality is because when & holds Enero 1(z 21) ([f(xe,))) ((1)))))/77 1 when E holds ΣiE[ay [(2 * 21) exp (1)x 2 0. the last second equality is because when & holds ΣeE(E) 1(2,- exp ([fxx; y1) )) the second inequality is because LogSumExp function is convex, and the last equality is due to [[[](x,, - f(xi, 2r, 21] = when z, = 21. Similarly, we can prove
/T)
1(z + 21) exp ([((x))))) f(x, +11 +1)]
Ig 2 E ((8) log
Σ [E|8)
+ E log (1( { [B 2 21 }1}
(E.8)
23

---

Published as a conference paper at ICLR 2024
Notice that when event & holds, # holds for all t 2. Therefore, plugging the (E.7 and (E.8) into (E6 gives,
E 1(8) log
Σ 122
- exp - (xx.))))/+t
+ +1) s (E.9) exp ([[xx),))) - ((x1 +
+ +1) se'.
E
((E) log
(B.10)
122
(E.11)
Let us compute the probability of & given 21. Let 21 without loss of generality, we have that
P(E v1) (1 P1) B-1
Therefore P(E z = v1) is always positive and is greater than 1/2 as long as B S L/P1-
Next, consider the following situation. Given 2 V1. we generate sequence zi,... 2/, with length
L = [og(2K)/ [ 1) min p 1 1). such that each z ........ z generated from D Dz The probability that the sequence includes Væ is
1 - (1 p+/(1 p=)) 2 1 - (1- p 2 - exp(- 2 1 - exp( min pk).
Therefore the probability that the sequence can cover all the other K classes is at least
1 K exp(-L p&) 2 1/2.
Then we look deeper into
E log Σ exp
([f(x),)1) - F(x1 + +1) =1. + V1.
zK # v1
122
We introduce L.(BB 1) copies x(" with I [L/(B- 11]] for t 2, then have that
L/(BB (//(B-1) E log Σ exp -
/r) + [(1,
21 v1, # Vi,-..
2x # v1
E
Σ log Σ exp ([f(x (1) y1) ,1))
f(xI,N1) /r) +1)
21 =V,, 22 ..... 2x + vi
122
E log ΣΣ exp
([F(x/) (1) .1) -(x))))
/T) +1 21 vI, (1)
..... zx X Av
!1)
exp ([f(x.,) - f(x" x))",)))/)) ) vv
2 E
log
(E.12)
KE[K]
where the first inequality is by Lemma E.I the second inequality is by the fact that the Exp function is greater than 0, and the Xx. x" in the last line are the ones that defined in Theorem 4.2 Plugging (E.12) into (E.9) and applying total expectation completes the proof for the second bullet. The proof for the first bullet is the same.
Third Bullet in Theorem 4.2; By the third equality in E.7 we have that
4 2 E
log
Σ [E[B]
1(z, =z) exp ([f(x. v1) - -f F(xx,
E
log
1(z = =1) exp (f(x,,
21. 21....,
Elf(
1 [B]
2: = -213 -
2: = =2z))/ n1] +E 2 E log (1f [B]
Varx (f (f(x)
111 21
(E.13)
24

---

Published as a conference paper at ICLR 2024
where the incquality is by Lemma E.2 Next we will We analyze the distribution of
[B] Z =
21 Without loss of generality, fix 21 = V1. We know that the probability that
f E [B] 2t =
21 2 2 is
1 (z2 + 21) ..... P + 21) 2 1 (1- min px) B-) 2 min(0.25 * min pa (B - 1), 0.25),
the last inequality holds since the strictly increasing function F((s) 11110 (1 " is and have derivative lower bounded by 0.25 when S 1/ min pk. Therefore we can further lower bound (E.1)) follows,
log (I(+ E
[2122 - =11)))
min (0.25 min pk (B 1),0.25)
Varx ((()))1))
IE
8M2
Similarly, we can prove that
12 log (I(+ E [B] ze zı }1} min (0.25 min pk (B - 1),0.25)
Vary (((x),11))
S.2
Plugging the bound of I1, 12 into [E6) completes the proof for the third bullet of Theorem 4.2
F PROOF OF THE RESULTS IN SECTION 5
Proof of Corollary 5.7 For (x Dxx., {yk D k E [K]], let y
Eve(K 1(2
vR)yk. Denote be the event that the top-r choice gives the wrong prediction. Then we have that,
2 E log Σ exp
-(xx
F(x,x')](+))
KE[K]
E ((8) log
Σ KE[K]
exp exp -
2 E 1(8) log(( +r)
P(E)) log(1 + +r),
where the first inequality is by the first bullet of Theorem 4.2 the second inequality is due to the fact that log
ExE[K] exp ([f(x, yK) vx) - x(xx)*))]
0, the last inequality is due to
log(1 + r) since there are at least r + I number of fxx, are greater than f(x,y*) if the prediction is wrong. Therefore, we have that P(E)
f(x, log EuE(K] exp f(xx f(x))))))))
// log(1 + +) which completes the proof.
Discussion for out-of-distribution zero shot learning. The result in Corollary 5. can be generalized to diito---iittibttionn zero-shot transfer learning. For example, we can deal with the case where the distribution of the prompts Dylv, and the image distribution Dx are shifted. particular, let us consider the case that the distribution of the prompts is shifted to D' D'vvv and the image distribution Dx is shifted to D'_ Then the original joint cumulative distribution function function P(x, Y1.1.... .))) shifted Q(x, z, y1....,y'' ).). Suppose Q is absolutely continuous with respect to P. and the Pearson x² distance is bounded
2 dQ ((8-1) dP SC
SC
Then we have that
/ log Σ exp (I) f(x dQ kE[A]
25

---

Published as a conference paper at ICLR 2024
log
Σ kE[R]
exp - )))))))/+)) ((xx)))))) (dp)
dQ dP
dp
/ log Σ kE[K]
exp ([f(x, Fox, F(x))) dP. v
dQ dP
2
dP
1JE',
where the first inequality is by Cauchy Schwartz inequality and the last equality is due to Then we can follow a similar analysis in the dQ ar (1- d - - dP d (
proof of Corollary 5.1 and have that top-r test error is smaller than v + 1)e' log(1 + r). There- fore, if the x2 distance between the shifted distributions is bounded. we can still provide top-r emrr guarantee. is worth noting the bound for cut-of-distribution zero-shot learning looser. If we want to do more general zero shot analysis, we may need to add more data structure in Assumption 4.1
Proof of Lemma 5.4 We can construct W
H( H) PP G)- where P
I
0
( Ka) is the projection matrix
0 with rank K1.
is easy to verify that H W.G = P. Therefore we have that
(W x, y') = (z, z²).
Then applying ||vill2 (VA, v/ S 7, Wk k' completes the proof.
Lemma F.1. [[VLsff.r)) lF S L where L 2r G
H
2
+ 1).
Proof. First, we have that
Vw (Wx,)) y)ll = ||xy T F1 - |
G
H (R² (2 +))
2 + 1) since LogSumExp function Therefore we have that VLs(fw, T) 2T H is an I-Lipschitz function.
Proof of Theorem
5.5
By the gradient update rule, have that
[W(0] w 12 - || w
(V w - - (w())) 13
2 2nIssWW() 2nL .T) - 12
(EII)
Take the telescope sum F. from to we have that
Σ L.s( (s(()),
| W -W(0--1FF
s (s( -) nT?2
T
2n/
s <is(W -) c/4+
= ['" e/2,
where the second inequality is by v) S e/(41?) and T = 4|| -W* / Therefore, there
exist t' < T - 1 such that Ls(w((), 7) < Ls(W",77 + e/2. Let T to be the first time that
Ls(W()) (T) r) < Ls(WW,)) e/2. Again take telescope sum of F.I) from 0 to T we have that
W 7) 2 7-1 ( w < TT (W
Σ
(w)
( W 1
< -ff + 0.5mf (0) -
() - 1/
26

---

Published as a conference paper at ICLR 2024
where the second inequality is due to the definition of T, the last inequality is due to -0.5nTe 0. Therefore, within 4|| ( we can find W w such that (s( (WT)) - Ls( (W", ,T) + c/2 and
W 11 s 2 1 ||
where the inequality is by triangle inequality. Therefore, for any x,y
f(x (W"x x, + (W- W'x,)
11+ W - W xX IF
$1+ W W H R2 +1)
3 W - w | H + +1).
Therefore the function f is bonded by .M 1+ - w || G | (R² 1). Moreover, the function f must belong the class J
2| W ++ W 2 Since
the linear function class F has finite covering the set [FF (Bartlett & Mendelson 2002 Zhang
2002). by Theorem 3.3 we know that when n 2 (8r -2 2M log 2/ probability at least
with
we have that
[Zs(P,t) - LD (f. e/4
[Zs(f LD (f" .T) s c/4.
Thus, we can conclude that
(P (f T.s(f.T) Ls( ,) + + La (f,7))
+ +Lsss T) Lpa (f*, S c/2 </4 c/4
=6.
where the first incquality is by the triangle inequality, the second inequality is by the bounded gap between empirical and population loss.
Proof of Theorem
5.6
E
[g(x)
y12 2 = E |g((x)
Elylz]
+ E[y|z]
y112
E
[l((x)
E
+
|
y
x12
where the second equality is due to x 1 y/z and E over both sides over z gives that
E Ely|22
y
0. Then taking total expectation
(x) y y1 = EE [[((x) Ely 2 2]|
+ E
Ily
Ily Ely [2]]
Obviously, E [[]g(x)) |1] achieves global minima when
g(x) = E[y|zi
z E(Ci2]
This function is also achievable. We can construct function (z H
and projection
E(çiz]
function gi (x) = z that is linear. Then we can define g g2 gi.
Proof of Corollary 5.7 Since is independent with z, have that
g(x)
E
= 1/3
e 0
+ 2/3
e
0
27

---

Published as a conference paper at ICLR 2024
Besides, we have that
2
y' = H
5"
5
6
Inner product similarity. We have that f(x, y') (z, zz) 1/3 1/3+1/3 1/3 1(5' e2). Since margin
< 1/3. There exist j. k such that (vy. v&) > 2/3. Then for z v,, we will sample K prompt
V
[v
YK- When y;
and Yk
e2
we have that
0.
0.
[(x, v,) = 4/3 < (x) + 2/3 f(x, ya),
which leads to the wrong top-1 prediction. The key insight behind this consequence is that f(x, y') = (z, x') 1/3 1/3 1(4"= 02) is greatly influenced by the unique feature Ç. A similar case also [
Vv
exists for z = Vv: with y, =
The probability that the above event occurs is at e2 and y: = ei
0.
0.
least 1/KK1/3 2/3 = 4/(9K) 2 1.K3)). Therefore, the test error is at least 1/(3K).
Cosine similarity. Notice that [lg(x)]l2 =v + 1/9 + 4/9 v14/3, and ||y||2 1, therefore the cosine similarity proportional to inner product similarity with factor vI4/3. Thus, the test error still at least 1/(3K).
T2 similarity. We have that f(x,y') = -1
z' 8/9 + 2/3 1('' = =e2). Since margin 2/3.
y < 1/3. There exist j, le such that [[v, V
Then for z = v,, we will sample K prompt
[
[v
y1.... YK- When y, = el
and Yı =
e2
we have that
0.
0.
f(x, = 8/9 < - Ily, vo + 2/3 = yx,,
which leads to the wrong top-1 prediction. The key insight behind this consequence is that f fxx = 02) is greatly influenced by the unique feature A similar case V
-||z 8/9 2/3 1(5'
V&
also exists for z = Vv with y;
e2
and yı
e 0.
The probability that the above event occurs
0..
is at least 2/ 1/3 2/3 = 4/(9K) 2 1/K3. Therefore, the test error is at least 1/(3K).
G PROOF OF RESULTS IN SECTION
6
Proof of Corollary 6. For (x,z) Dxxm= yyk Dylve,k E [K]}, let y" EvE(K] 1(z VR)yk. Denote & to be the event that the top-1 choice gives the wrong prediction or the margin smaller than Then we have that,
2 E log
Σ KE[K]
exp -x)
> E
1(8) log
Σ
exp (Ifix, - -(xx)))))))
kE[K]
2 E 1(E) log(1 + exp(-
= P(() log(1 + )
where the first inequality by the first bullet of Theorem 4.2 the second inequality due to
the fact that log ExE[K] exp ([f(x, ((xx,*))/T)]
0, the last inequality is due to
log EMETK] exp ([F(x, - f(x, f(x,x")\/7) 2 log(1 + e since there exists at least one similarity score f(x. yx) greater than f(x,y*) - with y& + y. Therefore, we have that PFE) < e'/ log(1 +1-1) 5 4e' which completes the proof.
28

---

Published as a conference paper at ICLR 2024
Consider the simplest setting where & and ( are all zero vectors, and we can access to the population loss and its gradient (notice that we are constructing the negative example). We will show that even under this ideal setting, the learned score function with corresponding representations may not achieve a margin greater than Ö(7). Notice that
Proof of Theorem 6.2
((f = Ww
log
Σ (E(
exp
(1f(x))))) -(xx)
+ V W E
log
Σ
- f(x))
exp
[E|B]
E
V log
Σ
exp (10
tE(
+
V log
Σ
exp -x - -(xx) /T)
[E|B]
exp ([I()), (
E
Σ, exp ((xxx)))) (y: - x1)x
[E|
exp exp((/xxxx))-/11xxxl) ((1, y1 (x3: -x1)7
+ E
Σ. exp -
[E
-
(z, # 21) 21)exx exp (1) V1)/TT
Σ, exp (1 (x), - ((x (ve (xx-
E
(z + 21) exp
Σ. exp (ff(x,))) - f(x, y yi x
+ E
[E
where the last inequality is by x; achieve a margin greater than log [16]] H (N² + 1)a 1 r. we have that the gradient
x1 and y: = Y1 when = Z1. Therefore suppose function f can
V EWEDDL(f.T))
(z + 21) exp (1
< 2|| +1)
Σ, exp (x) - ((x,,)))))))
[E[B]
(2, F exp ((xx
22] +1) E
Σ, exp (1)x
(E(B)
<2| || + +1) E 1(z, # 21)
exp (fsp
[B
+ H +1) .E
1(ee + 211 exp
< 0.255 | H 1/ (R² +
(G.1)
is very small. Now suppose the SGD trajectory start at (()) 2 2oo [ H
(R
Br 'T) - WT Obviously the score function with weight (()) achieve a margin
2 log
|6]] G | (1 + Br 1 v/T
m/T)= F. Suppose there exists a time S 7' such that (W y)
can achieve margin larger than log 16 G HI + B) >//) or can achieve margin
larger than log [ H + B Then there must exist a first time L << l' such
29

---

Published as a conference paper at ICLR 2024
that the margin at time t lies outsize the range between log [99]] (R + Br 'nT)=
and 3 Blog
( + BT B))) 5. By definition t (margin gap), we know that there exist y, such that (W x y) (W y) > т. On the other hand, we have that
[6]||] G B
(W x.y) (W x. .> = | w [ll||xy'l 1|
< 2 (R² + 1]] W W(
< | 2 (R² +1) T/T 0..55 G H ( + +1
< 0.5T,
contradiction! Therefore, such doesn't 't exist. The score function learned by SGD within T ( + +1)
iterations can't achieve margin greater than 3 log [ H
Theorem G.1 (Formal statement of Theorem 6.3 Under the same condition as Theorem 5.5 with 0. (This problem setting includes the special case considered in Theorem 6.2 Let S Xy² min m/imin// (32/03200] 2 and S 7/ 27/log(?2 min (64/0600 HJ within polynomial iterations, we can find score function / with large margin. In particular, with probability of least 0.99, the top-1 result gives the correct label with margin of at least 0.5%
Proof. For simplicity, consider the case that we can access the population loss and its gradient, i.e., n + 00. The regularized loss then becomes,
L new Loe (f.r) + XEllg(()) h(y) ||3)
Since the new loss is still convex and strongly convex. By applying the same technique in the proof of the Theorem 5.5 within polynomial iterations, we can find Inew (f,T,A) s L new (f .T, A) Besides,
1(z: = 21)] + + (eeep-pp/1
new (T)) = LDa (f", r) s 2E
log Σ
[
where the first equality is by plugging in W H(H H) -D g(() = Wx,h
y, the inequality is by Lemma E.3 Thus we have that
h(y) < 2E log Σ 1(z = =i) +E' LE(B)
Ap=(sr + xBElgg
where
C' c c 2B 2B exp( (-ppp-))))) By and
we know that LDd (f,T) 2
2E log Exe(oy l(z 21)
Therefore, we can conclude that
[x(]((x) h(y) <É/> < x² min phinpe/(1100/ H 2),
where the last inequality is by choose < 2r² min pk (3200/ H| and <
7/ log/(22 min (6/(000
Then by Chebyshev's inequality, for any z, with probability
1 - 0.01 we have [lg(x) - h S 1100 Pf B]]g((x) h N {|])) Then for any y' that has the different shared feature from y (i.e., z' # z) have that
(g(x), h(y'))(())))))))))))) - (g(x), h(y))
< (h(y), h(y')) (b(y), byy)) + llg(x) - h(y)ll2 (Ilb(y')ll + (2b())
< 5-72
< 2/2
where the first inequality is by triangle inequality, the second inequality is by [lg(x) - -h(y)l2 S 7/(4 [|]]2) and ||b())]l2 = [h(y)]la < |H| since
0.
30


Reviews:
[{"review_id": "rJpoz4lihX", "reviewer": "Reviewer_P1Cv", "summary": "This paper investigates transferable representation learning underlying CLIP and demonstrates how features from different modalities can be aligned. Then a new CLIP-type method is proposed, the effectiveness of the proposed method is proved through experiments on multiple benchmark datasets.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow. \n- This paper theoretically examines the transferable representation learning in CLIP. The theory seems sound.\n- This paper proposes an easy regularization technique for CLIP that can effectively improve its zero-shot performance.", "weaknesses": "My major concerns lie in the empirical studies.\n- The current pre-training experiments are all based on the CC3M, which is much smaller than the full 400M dataset used by the CLIP. It is unclear whether the proposed regularization technique holds when extended to a larger dataset. It is recommended to conduct experiments on datasets with different sizes.\n- In Table 1, why incorporating the regularization term into the contrastive objective is harmful to DTD? \n- It seems that the results of CyCLIP in Table 1 and Table 2 are inconsistent with the original CyCLIP paper.\n- Can the proposed regularization technology work in CyCLIP?\n- Is the method equally effective for other downstream tasks such as retrieval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iegnthFBIe", "reviewer": "Reviewer_7qr5", "summary": "This paper focuses on providing theoretical support for the CLIP training and its zero-shot transferability. The main claim is that the contrastive learning objective in CLIP may not cover all the positive pairs, e.g., some features in an image may not be present in its corresponding captions. In section 3, the authors show that the empirical loss converges to the true loss when the number of batches is large enough. In section 4, they show that the learned similarity score f_hat between negative pairs is smaller than the score between positive pairs given that there exists a score function f* such that this relation holds. Based on such assumption, in section 5, they conclude that a trained CLIP model can achieve small top-r error and this generalizes to different distributions as the distribution shift is bounded. Based on the prior assumption and derivations, they have three claims: 1) Margin depends on the temperature tau, 2) We should only regularize positive pairs instead of both positive and negative pairs, 3) With sufficiently small tau, we can find a f_hat with large margin. They test these claims with experiments on CC3M.", "questions": "My questions are listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides theoretical bounds on CLIP training and its zero-shot transferability.", "weaknesses": "1. Contrastive learning has been widely studied in the community with several variants of NCE loss, with different ways to regularize positive and negative pairs to improve the margins (e.g., [a]). The behavior of temperature in contrastive learning was also studied (e.g., [b]), and so was regularization (e.g., [c]). It is not surprising that adding the regularization of the distance between positive pairs can improve the performance. Also, how does the proposed solution compare to those methods?  \n\n2. The authors propose only to regularize the distance between positive pairs, but there is no ablation comparison to the variants that regularize both or negative pairs only.\n\n3. The introduction states that the contrastive learning objective in CLIP may not cover all the positive pairs, which makes sense. However, \n it is unclear how the proposed solution addresses this issue.\n\n4. The experiments are conducted on a relatively small dataset compared to CLIP. The training behavior and the generalization ability of the representation may be different.  \n\n5. The derivations make sense but they are under several assumptions.\n\n[a] Unified Contrastive Learning in Image-Text-Label Space, CVPR'22  \n[b] Understanding the Behaviour of Contrastive Loss, CVPR'21  \n[c] Large-Margin Contrastive Learning with Distance Polarization Regularizer, ICML’21", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pwVO1xjP6U", "reviewer": "Reviewer_CWhP", "summary": "This paper offers theoretical analysis of the underlying principles of CLIP, shedding light on why CLIP exhibits robust transferability. Additionally, the paper introduces a novel regularization technique designed to enhance the performance of CLIP.", "questions": "See `Weakness' above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is well-written and novel. This paper offers a robust analysis, including mathematical proofs, of CLIP. These contributions greatly contribute to our understanding of CLIP.", "weaknesses": "1. The paper's primary focus appears to be on CLIP's zero-shot transferability. While this is undoubtedly a significant aspect, it's worth considering that CLIP's robust zero-shot performance results from a strong semantic space based on extensive vision-semantic data. Therefore, an exploration of the visual-semantic alignment aspect could be an intriguing avenue for further investigation.\n\n2. In introduction section, the author cites \"blue sky\" and \"white cloud\" as examples of unique features. However, these instances might be seen as special cases.  As CLIP is based on a large amount of vision-semantic data, it's possible that the missing elements could appear in various other captions. Therefore, I question the significance of this problem. To address this concern, the author may need to conduct overall statistics on the data. Furthermore, the term 'unique features' could benefit from a more precise definition or explanation.\n\n3.  Some notations and definitions in the paper can be challenging to follow. For instance, the terms 'one-to-one mapping' or 'one-to-one matching' could benefit from clearer explanations for readers.\n\n4. Expanding the range of experiments to include various downstream tasks, rather than solely focusing on zero-shot and Linear probing, would provide a more comprehensive assessment of the paper's proposed methods and their practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZm0tdcxd2", "reviewer": "Reviewer_nFDE", "summary": "This paper theoretically examines transferable representation learning of CLIP. The analysis reveals that with a near-optimal network trained on the data, features from different modalities align, allowing for zero-shot learning when appropriate prompts are used. The paper also demonstrates that contrastive learning with sparse features can lead to unexpected positive pairs, emphasizing the need for careful consideration. Building on these general theoretical findings, the authors provide deeper insights into specific cases, illustrating how multi-modal learning aligns different features and how CLIP's learned features outperform those obtained through naive square loss. To validate their theoretical predictions, the authors conduct experiments on real data. Additionally, inspired by their theoretical findings, they propose a novel regularization technique for CLIP, effectively improving zero-shot performance across various tasks, as confirmed by empirical results.", "questions": "Can one simply replace the CLIP component in all CLIP-related work with CLIP+Reg to achieve a better performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and flows smoothly, making it relatively easy for readers to understand. \n2. This article focuses on what's behind the explosive effectiveness of CLIP, and the paper attempts to delve into the principles underlying CLIP, demonstrating a certain level of originality and innovation.", "weaknesses": "1. More relevant experimental results are expected, such as results from a wider range of downstream tasks.\n2. The article exclusively analyzes and experiments with CLIP, without thoroughly exploring the applicability of this new methods to other relevant contrastive learning approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)