On the Comparison between Multi-modal and Single-modal Contrastive Learning
Authors: Wei Huang
Andi Han
Yongqiang Chen
Yuan Cao
zhiqiang xu
Taiji Suzuki
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal contrastive learning
single-modal contrastive learning
optimization
learning theory
Primary area: other

Abstract: Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.

On the Comparison between Multi-modal and Single-modal Contrastive Learning
Wei Huang"1
Andi Han" RIKEN AIP
Yongqiang Chen
RIKEN AIP
The Chinese University of Hong Kong yqchen@cse. cuhk. edu. hk
huang, vr@riken. jp
andi han@rikkn.))) jP
Yuan Cao
Zhiqiang Xu MBZUAI
The University of Hong Kong yuancaoChku.
zhiqiang.o www.mmmuu..a..a...
Taiji Suzuki
University of Tokyo & RIKEN AIP taiji@mist. i. w-tokyo. ac. jp
Abstract
Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoret- ical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, introduce feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is per- formed on ReLU network trained with the InfoMax objective function. Through a trajectory-baseet optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which the signal-to noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-mode learning can achieve better feature learning, leading to improvements in performance downstream tasks compared to single-modal learning. Our analysis provides unified framework that characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-word real-world datasets further consolidate our theoretical findings.
Introduction
Large-scale pre-trained models have achieved unprecedented success, including GPT series 66 41). LLaMa [53], among many others. CLIP 42
typical example, uses multi-modal contrastive learning framework to learn from a massive scale of image-caption data. The multi-modal contrastive learning in CLIP has shown significant capabilities learn high-quality representations, which are ready to be adapted to a wide range of downstream tasks, forming the backbone of generative models
"Equal Contribution.
t "orresponniigg Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

like DALL-E2 433, prompt learning [6] as well as general purpose modtl agents 62 351 Given the huge success of models like CLIP that have stellar zero-shot and few-shot capabilities on wide range of out-of-distribution (OOD) benchmarks, they have been widely recognized as foundation models (FMs). More similar examples are given by ALIGN 28 Florence [59], IIIP
33. Flamingo 1
Despite the unprecedented success achieved by multi-modal contrastive learning, the fundamental mechanism that leads to greater performance, especially compared to single-modal contrastive learning still under-explored. Recently, several seminal works provided theoretical explanations for either single-modal [ 5 44 24, 7 50 49 [0] or multi-moda contrastive learning [38 37 44 [2]
For example, [56] studied how single-min modll contrastive learning learns the feature representations for neural networks by analyzing its feature learning process. As for multi- contrastive learning, provided explanations for why multi-modal contrastive learning demonstrates zero-shot transferabilityy and robustness to distribution shifts, than supervised learning. which offer valuable insights. Although both lines of the existing works provide valid theoretical insights under the respective settings, rare work has compared the optimization and generalization of the two types of contrastive learning under a unified framework. This motivates us to establish a systematic feature learning analysis for both single-modal and multi-modal contrastive learning.
12 58
In particular, we consider data generation model that contains two modalities of data, which are generated from signal and noise features. The signal feature correlates in different modalities, while there is no correlation between noise features among modalities. We then study the optimization of single-modal and multi-modal contrastive learning under gradient descent training. By studying the trajectories of signal learning and noise memorization, we establish the convergence conditions and further characterize the generalization ability in the downstream tasks. The results show that, through the cooperation between modalities, multi-modal contrastive learning can achieve better generalization in the downstream task. In contrast, without the help of the second modality, single- modal contrastive learning concentrates on learning noise from the data, and thus generalizes poorly on the downstream tasks. The main contributions of this work are summarized as follows:
This work establishes the first systemutic comparative optimizution analysis for single-modal and modti contrastive learning under gradient descent training in non-convex settings. We show that both single-mode and multi-modal can achieve near-zero training eπror under InfoMax contrastive loss after polynomial number of iterations, by overcoming the non-conve. difficulty.
By trajectory-basee analysis of the signal learning and noise memorization of the ReLU network from the data, we successfully characterize the difference in generalization between single-modal and multi-modal contrastive learning. The distinct SNRs of different modalities lead to a divergence in the generalization of downstream tasks for the two contrastive learning frameworks.
Our theory suggests that the advantage of multi-modal over single-modal contrastive learning comes from the high quality of the second modality and the cooperation between the two modalities through contrastive learning. This divergence is ultimately reflected in the difference in feature learning and the final gap in downstream task generalization. Experimental results on both synthetic and real-world datasets confirm our theoretical findings and understanding.
2 Related Work
Theoretical Understanding of Single-modal Contrastive Learning. The seminal work [4) started theoretical research on single-modal contrastive learning. They assumed that different positive samples are independently drawn from the same latent class, making a connection to supervised learning. (55) identified two key properties related to the contrastive loss: aligmment and uniformity. Alongside, (32) illustrated that predicting auxiliary prediction tasks helps in learning representations effective for downstream prediction tasks, and [52) provided theoretical analysis of contrastive learning in the multi-view setting. Besides, [31] proposed a theoretical framework to understand
contrastive self-supervised learning from an optimization perspective. [2] proposed loss that
performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective on neural net representations. [46) pointed out the importance of inductive biases of the function class and training algorithm in understanding contrastive learning. The most related work to us the work by [56]. Similar to them, this work studies ReLU networks and considers the signal-nois data model. However, we do not require the adjustable bias term in
2

---

the activation function, which plays a critical role in [56]. Furthermore, this work adopts unified framework compare with multi-modal contrastive learning, which is out of scope in [56].
Understanding of Multi-modal Contrastive Learning. As the multi-modal contrastive learning approaches such CLIP received great success, recent works have been proposing explanations from empirical perspective. (36] empirically showed that high train-test similarity is insufficient to explain CLIP's OOD performance. [60] illustrated that CLIP behaves similarly to Bags-of-words in language-basee baaed image retrieval, i.e., the order of words in the input sentence does not largely affect CLIP to find the corresponding image. Besides, [6] demonstrated that training data diversity and the ability to leverage the diversity as supervised learning is the key to the effective robustness of CLIP. Theoretically, [13 proved that multi-modee contrastive learning can block-identify latent factors shared between modalities by the generative data model. 44 analyzed the training dynamics of a simple multi-modal contrastive learning model and show that contrastive pairs are important for the model to efficiently balance the learned representations. Furthermore, (38] showed that each step of loss minimization by gradient descent can be seen as performing SVD on a contrastive cross-covariance matrix. Similar to us, [25] tried to answer why multi-modal learning is better than single model learning. However, they did not consider contrastive learning and thus cannot explain the success of multi-modal contrastive multi-modal learning.
Data Quality Matters for Multi-modal Contrastive Learning. Aligned with our theoretical results, there is a lot of empirical evidence showing that improving the alignment quality with more descriptive captions improves multi-modal contrastive learning. [66] show that the training distribution mostly determines the generalizability of CLIP. Furthermore, 47 40 18 17 find filtering poorly aligned image-caption samples used for training leads to further improvements. Besides, [55 39 15] demonstrate that improving the descriptiveness of the captions could further boost the performance of CLIP. Besides, (34] demonstrated that the caused by a combination of model initialization and contrastive learning optimization. However, their results do not take neural network architecture into consideration, and do not provide an analysis of test errors either.
3 Problem Setting
Notation. We use bold-faced letters for vectors and matrices otherwise representing scalar. We use 112 to denote the Euclidean norm of vector or the spectral norm of a matrix, while denoting ||- as the Frobenius norm of matrix. For a neural network, we denote a--) the activation function and we adopt ReLU activation where a (xx) = max(0, æ} in this work. To simplify, we denote [n] (1.2....,
Data Model. In this work, we consider the following data model, which consists of signal and noise. In the first modality, example (x, y) ~ D is generated as follows:
x= xx (2) I = [yy1 1 STTT, y ~ uni(((-
where 2d is the input feature and y {-1, is the corresponding label generated from Rademacher distribution. In particular, (1) = yı E Rd is the task-relevant signal vector, and x(2) E N(0,o21) E Rd is the task-irrelevant noise vector. Intuitively, if a network learns primarily from signal, it can effectively generalize to unseen data and vice versa. Similar data models have been adopted in recent thcoretical works on supervised learning 2 26 and self-supervised learning [56] 50 30).
Similarly for the second modality, a sample (x, v) ~ D is generated as
X (D) ST = [y T, . JT. y unif((-1,1)),
where the input feature X 2d and the label y shared with the first modality. Besides, the signal given vector zı E u and noise follows & ~ NN(0, (2) ) E Rdd The linear data models for multi-modal learning have also been studied in previous work [4]. To simplify the analysis, we set d = d, og og. However, we highlight that extensions to deal with unmatched dimension and noise level is possible.
3

---

3.1 Single-modal Contrastive Learning
We use a single-layer neural network h IR2 R with ReLU activation as our encoder, where m is the number of neurons, which represents the embedding dimension. More precisely,
h(x) = (x). [x)] E IR", where (x) = h, (x("))) h (x(2)),
(3)
here we let h, (x(()) o ((w,,x(0)) for [m]] i [2]. and (.) is the ReLU activation function. We adopt a Gaussian to initialize the weights w, NNO, og)), where dy severs as the strength.
Given pair of positive data samples, the contrastive loss function is based on the similarity measure defined as the inner product between the representation of two samples x, x' E TR2d:
Simm (x.x)
1 Σ )sg(h, (x((,,(())))) +
Σ hy (x (x(())
(4)
where the sg(-) is the stop-gradient operation, which is inspired by recent empirical works and theoretical work studying contrastive learning [56]. Here define positive sample
19 10
x
(x) T(2)TT [yr= E +ET]T, E o
In particular, we consider the form of augmentation where the signal stays invariant while the noise vector corrupted with added independent noise. Similar setup has been considered in (57] We
consider the contrastive loss presented as follows:
Sima (x. Rt)/
log( (x, Ene eSmm, (x,x)/
(6)
where 7 is the temperature parameter, n is the number of training samples, and M is the number of negative pairs. this work, to efficiently optimize the loss to near zero, we require negative sample pairs do not share the same label, i.e., Dj F in (6) Note that this setting is aligned with supervised contrastive learning 29 277.
We use gradient descent to optimize the contrastive learning loss, which leads to the gradient update:
w (((+1)
w(()
V w. L(
7
(x. yaa
TITT
n
M
7
E:
4g
10 (x ya
nMT
nIT
#
1
22) E
mMTT
where we denote (( (x) = o ((w (t) x)) 7) as the learning rate, and we define the loss derivatives as
Simm xx. 8,)/T
Simu (x,,,) x,//r
(8)
(x), E/#t (x.,,(x.xx)/==
Simp 8,1/7 + Enee Simma x,)//
Intuitively, when the similarity between positive pair high, and the similarity between negative time is low, we can see e((t) = 1 and ((t)) ~ 0, for E [n] and j [M]] Therefore. the gradient descent in Eq. (7) is close zero, indicating the near convergence result. Furthermore, from Eq. (7). we observe that the evolution direction of weight is composed of signal vector and noise vectors E; for i [n]. This observation plays critical role in our following theoretical analysis.
3.2 Multi-modal Contrastive Learning
We use two neural networks h Rd + Rm and g: Fdd R encode two input modality x and respectively. Both neural networks use ReLU activation function. More precisely,
hxx) (x), Tx)) RT where hx(x) hx(x + h, (x(x2))

---

g(x) 101 (x)..... Im (x)
E R" where (x(x)) = (x + gr (x(1))
we let (x(())) ((w,, x ) and gr (x()) o (w.. x))) Here o(-) is the ReLU activation
function, w, E R.d and w, E Rd for E [m] are the weights in two networks. Given the embedding, the similarity function of the two modalities is defined as
Sima.h (x, x) = m Er= h,
1
sg (x (x(1)) + E hr
Simg, h (X = -/ Σー=
Js h (xx + Σ (x,h,
The two similarity functions defined above modality-centered centered with stop-gradiie operation applied. The objective function of contrastive multi-modal learning can be expressed as
e Simn, (x, xx)/T
L =
log(
Siauh. &.)T + Enfi gSitm..(xx.,;))
(i,xxu(R;xx)/7
log( Simg h (2. + Ejzr &&.
(9)
71
Same to the single-mose mngle-modal learning whose objective function is governed by Eq. (6) the objective function for multi-modal contrastive learning adopt one positive pair and M negative pairs. Besides, require the negative pairs do not share the same label. To optimize the objective function (9) for multi-modal learning, gradient descent is applied to train two encoders simultaneously. The gradient descent rule for the first modal network is governed by the following expression.
w (++1)
1/ w. (WW())
w(()
(x (
TUTTTT
M
7
E.
7
9r
(x
nint
TUTT
jti
1
M
1]
(2) ).
(10)
TUTTET
1
Here with slight abuse of notation, we use e(0) g(()) modalities. Compared to signal-modal learning, the main difference for the multi-modal learning that the corresponding embedding is from another modality. The gradient update for the second modality can be derived similarly, which we omit here for clarity.
i,j represent the loss derivatives for both
3.3 Downstream Task Evaluation
To evaluate the diitt---istiibution generalization of mongle-modal and multi-modie contrastive learning for downstream task, we consider
distribution Dusst where a sample Xtest = jy T < "J Dreet is generated as follows. The test signal v satisfies (v, (u) O(I|m 1/22-112) and the test noise follows Ç ~ (O, I2 and follows Rademacher distribution. After the training is complete, we introduce linear head on top of the learned embedding h(xtesa) for adapting to test distribution, ie.. f(xtest) (w, h ((xtes))): Specifically, we consider the task of classification and define the population 0-1 test error as LD.... Xteat Draee yff (xtes)) 0].
4
Main Results
In this section, we introduce our key theoretical findings that elucidate the optimization and general- ization result for both single-modal and multi-modal contrastive learning through the feature learning analysis. We use a trajectory-baan based analysis for the iterations induced by gradient descent, following post-training analysis for the performance on the downstream test set. Below we provide the main assumption and main theorems.
-1 Assumption 4.1. Let SNR |1//|1// (ag Vd). Assume (1) d A(max(n² no 0g o0 (1 ( 2d_1)). (3) oo I Ō((max(o -1 (4) m, 2(1).
(2) n S mminn(m] (mmmmm/11113/ | .nmo
5

---

a, S a constant.
(6) n SNR* (1). (7) C. lell = 1/1l. where Cp. 2 2.66
(I) We adopt a high dimensional setting to ensure enough over parametttttri..ninnnn. (2.3) The learning rate and the strength of initialization are chosen to make sure the that gradient descent can effectively minimize the contrastive loss. (4) The choice of hidden size m and number of training sample n to provide adequate concentration. (5) The strength of augmentation is set to keep the similarity between two positive samples. (6) The relation between number of sample and SNR is to distinguish the feature learning process between single-modal and multi-modal contrastive learning. (7) To differentiate single-modal and multi-modal contrastive learning, we introduce a constant Cy. which enables the cooperation between the two modalities in multi-modal contrastive learning.
Theorem 4.2 (Single-Modal Contrastive Learning). Under the single-modal learning setup, suppose -2ad--2d---
2 Assumption 4.1 holds. Then after T D(nn'mnos 2 2d + 1) 1a-nmmmmmm "mno -2 dd) the with proba-
bility at least I 1/d, it holds that (1) Training error L(T") S and (2) Test ervor at down-stream task D (T = (1).
Theorem 4.2 states that despite the small training error achieved by single-modal contrastive learning, the test error large in the downstream task.
Theorem 4.3 MMutii-Modl Contrastive Learning). Under the single-modal learning setup, suppose Assumption 4.1 holds. Then after T = m + -n----nmm "mno dd) the with proba- bility at least 1/d, it holds that (1) Training error L(T") S and (2) Test error at down-stream
task D (T = o(1).
Theorem 4.3 demonstrates that trained multi-modal contrastive learning can achieve both small training error and downstream test error. Compared to Theorem 4.2 Theorem 4.3 shows that the generalization of multi-modal contrastive learning in downstream tasks is better than single-modal contrastive learning. The reason behind this difference is that the two modalities can cooperate with each other; the higher quality in one modality can boost the feature learning in the target modality, helping to generalize the downstream task. On the contrary. augmentation often maintains the same SNR as the original data, so single-moda modal learning hardly benefits from the augmentation and can only memorize the noise from the data, which is not applicable to downstream tasks.
5
Proof Roadmap
5.1 Proof Sketch for Single Modal Contrastive Learning
The proof is constructed by optimization analysis followed by a generlization analysis in the downstream task. Through the application of the gradient descent rule outlined in Eq. 7 we observe that the gradient descent iterate w is a linear combination of its random initialization w! (0) the signal vector μi and the noise vectors in the training data &, for i E [n]. Consequently, for E [m], the decomposition of weight vector iteration can be expressed:
(+)
w (0) + 7/()
1 14 +
14
| E
(11)
where 30 and p serve as coefficients and represent signal learning and noise memorization respectively. Based on the the gradient descent update 77) the iteration of 7 and p7 are given:
(8)
(t) Lemma 5.1 -Single-modal Contrastive Learning). The coefficients "/ Prá in decomposition 11 satisfy the following equations:
M
7 ((+1) = =7(
[1
(x(
Σ
1 (x} (2 1)llllll
(12)
TITT
1=1
M
Pr. Pr.st NUTT
(I+1) (1) 7
(1
ny (x)
hr
22) (x)
(x, 12
(13)
j+
where the initialization T0) .Pr. = 0.
(0)

---

Lemma 5.1
tells how the coefficients evolve under gradient descent update. In the following, we
introduce a two-stage dynamics to characterize the whole training process based on Eq 12 and Eq
First Stage: Exponential growth. During the first stage, we show before 7/00 or Pr.i grow to (1), the embedding (3) is close to zero, suggesting the similarity is bounded by I Sim, x') I C2 for some constant C2 > 1. The loss derivatives defined in (8) can thus be bounded within some constant range.
(t)
Signal learning. According to the update for signal learning in 12). we see the prop- agation can be simplified based on the hard-negative sampling strategy, i.e. the negative pairs do not share the same labels. This suggests the negative term is always zero as 5.8 a((w,) (()))))'((w? yap)) 0. The resulting update of 7f reduces to (1) ?
Ei=1 ΣM
((+1) " anr Ei o (((w! y.P))) (.:4)) Examining the prop- agation of ) we can divide the dynamics into two groups depending on the sign of weight initialization (w: pt). Let & fr: (w!" . > 0} and u - fr: (w! 1)) < 0}. Then for r E 4 we can show 70th 2 0 increases exponentially and thus the sign of inner produce stays
invariant with 4 o for all 2 On the other hand, for (0), we can show 7/ S 0 and decreases exponentially with _ = _
(0) for all t 0.
Noise memorization. Compared to signal learning, the behaviour of noise memorization requires more detailed analysis. This is mainly because the negative pairs can not be eliminated simply based on label difference, as the noise patch E, is generated independent of label y;- In addition, the added noise e; by augmentation can also contribute to the noise dynamics. We first show when the noise level o, much smaller compared to ag. the dynamics of noise memorization is largely remains unaffected.
By the sign of (w!!) E we partition the samples into two sets, i.e., I {i (w! E.) 0}.
and Z((1 {i: (w) (1) E,) < 0). We can verify for i E I the value of Pr stays at zero based on the update (L3) with induction argument. For samples i I with positive initialization, we analyze the noise memorization based on the joint dynamics of samples with the same label. In particular, we define total noise memorization of positive and negative samples respectively as
(t)
Bid A Ein. (p + (w/")
(0) 1 (EI! and r.- - Ein:=-i(P! (p/1 + (w, =,)) The
update of p in (13) then implies the dynamics of B,O) and Bl0 as follows
B((+1) B
( nopd nmT
(B.0 r.
1 B(D)) B B no (B
1 B()); nmr r. 2
where the coefficient of 1/2 appears as a result of the randomness of the sign of initialization. This result suggests, individual Pr,k:u==1 cannot grow too slow compared to the Pr. Following
(t) similar induction argument, we are able to show Pragge= has an exponential growth lower bound. On the other hand, for samples with y, = -1 but with different neuron, we can use the same strategy to
show exponential growth lower bound for some neurons that satisfy the initialization conditions.
Lemma 5.2. Under the Assumption 4. let Ti log (20/ vd)) /og (1 0.96 ya (0)
nmr we have
(T)) ((1) for all i [n]. ((1 (n) for all r [m] and t < Ti and max, Pr.i
Second stage: convergence and scale difference. At the end of first stage, the noise grows to a constant order while signal learning remains negligible. As a result, the loss derivatives are no longer bounded within some constant range. In the second stage, we aim to show the loss is able to converge to an arbitrarily small value c. Despite the unsupervised learning setup. we are still able to show loss convergence thanks to the hard negative samples. Let (W.,, (W, x:) Sim (x;, X,) be the similarity to the argumentation and (W, xx) = sim(x;,x)) x,) for j = 1,..., M be the similarity between the negative pairs. Then we can show there exists some W such that (VFo 2 log(22Mec while FV (FF,WW'T (w(), x)), W < (2M/c) for all j = 1, .... M. Then we can bound (VLs(W(()) W(() W Z D L (W()) (() c/2. This as a result allows to show a monotonic decrease in the loss function as L(W < 1 (( W [ W (1 + which guarantees convergence by telescoping over the inequality. Upon the convergence, we can also show
xx), W*) 2
7

---

the scale difference obtained at the end of the first stage is maintained, i.e., 71 = Ō01/7) while
max, pe.i 2(1) for all E [n]]. This suggests the non-linearly separability for the resulting embeddings and thus the downstream test error is non-vanishing. The formal convergence result is
established in Lemma C.18 Combined with the generalization error demonstrated in Appendix C.3
this completes the proof of Theorem 4.2
5.2 Proof Sketch for Multi-Modal Learning
Similar to single-modal contrastive learning, we decompose the decomposition of weight vector iteration for the network in the second modality and subsequently provide a two-stage analysis.
( (0) +
12 E + Pr.
(14)
where 76t) and P serve as coefficients for the weight decomposition in the text modal.
Lemma 5.3 (Multi-Modal), The coefficients 7:0 .Pr.s 74")
(6) . in decomposition satisfy
M
(1+1) = =/
(8)
7
Σ
2y M4
(15)
nMT
t=1
M
Pri = Pr.s
(++1)
1
19 CENDEIE
"
4
(16)
TUTT
TUTU
M
7((+1)
(1)
1
Σ Mo
(17)
nMT
1=1
Pri Pr.s
10+1)
((t)
17
( (E))))
"
Σ (E)
(18)
TITT
nUT
where the initialization satisfy 7/ .r.a. Pr.
(0) -0) (0)
0
First Stage: Exponential growth The first stage of multi-modal learning shares similar characteristics as single-modal learning.
Signal learning. For signal learning, we analyze the trajectories for both ?f and 7(*)) To this end, we partition the neurons depending on their initialization status. Apart from 4 and (0) defined in the single-modal learning, we additionally define ui! & fr (wr (2) > 03. 7 & fr (w/") ()) < 0} for the other modality. Then for 21 n 740),
can show 7/ 2
( 2 0 and are increasing. For neurons u n decreasing. Furthermore, the sign of inner product stays invariant, i.e., 21 nū = 44 nūt
i7 we can show 7/ < 0. 5t < 0 and are
and (t) T()() Sn (ū For neurons with only one of the modalities activated initially, i.e.,
r E t0) Tū or r (o) nū we can show there exists some time t' 2 0 such that the neurons either positively or negatively activated with u nū and 4 nt7
This shows synchronization of the signal learning patterns of two modalities. The pre-synchronization phase reduces the speed of learning and thus we can only focus on neurons with the same sign for the initialization in order to decide the lower and upper bound.
Noise memorization. For noise memorization, we partition the samples into two sets for both
modalities according to the initialization, namely, I {i (w!") .E)) 0). z {i
(w.") ( E.) < 0), and similarly for the second modality 7 ( Because the noise memorization
are correlated in the two modalities, we separately analyze the samples as follows.
(1) For i E _ n 7:00, we can show Pr. = 0,
x(
-z and p = 0, I,,- = 7,00
())
(2) For i E (
n (0)
7(0) we can show (1) I(t) ( and Pr. < 0.
(3) For i E I (0)
I,, n 7 (0) we can show P
(1)
0, [,(1) - and Pr. < 0.
(1)
8

---

Training Lass
That Accuracy
Signal Learning
Noise Memorization
Single Nodal
1.0
Mult Medal
0.8
0.8-
0.6
0.7
0.4
0.6-
0.5
0.0
01
50
1000 Epoch
150
200
50
100
150
200
0
50
300
150
200
0
50
100
150
200
Epoch
Epoch
Epoch
Figure 1: Training loss, test accuracy, signal learning and noise memorization of single-modal and multi-modal contrastive learning.
(4) For E I,, n without loss of generality, we consider upper bounding noise memorization
for the first modality and y; 1. To this end, we first define the individual and joint noise
memorization for the first modality vp Fu Pe.i (w, -.i), and (()) +. - =
A (e)
(0)
(w.")..) . (E not) B - Eég (pr. + (w: (0) -.)) (E nł Similar definitions
(0)
exist for the other modality. The joint dynamics of noise memorization can be first upper bounded by the other modality ( 10IC, (B
M+1 rtt, B
-.). Then we can upper bound
the individual noise memorization by
V (()) 7, S (1+ 1.06nood WUTT
y (0) (7
We show the combined dynamics of (0 and pr exhibits exponential growth while the magnitude of their difference shrinks exponentially. The results are summarized follows
Lemma 5.4. Under the Assumption 4.1 let Ti = log (20/(au) (00((ooll4()))) log (1 + 0.48C, vllalla
mr ), we have Pr.3 = O(1/v/) for all E [m], i [n). and s t S Tj and max, 77 (Tr) (11).
(Ti)
Second Stage: Convergence and scale difference.
The second stage presents similar patterns compared to single-modal learning. Thanks the correlation between the two modality during gradient descent training, the two neural network converge at the same time, minimizing the training loss. Besides, The scale difference at the end of the first stage carried over throughout the second stage until convergence. Therefore, it allows to show monotonic decrease in the loss function
T(W(()) W(O) < ||W(() 11 W W | [ W [[(++)))
WI + 2c, which guarantees convergence by telescoping over the inequality. At the same time, until convergence, we can show the scale difference obtained the end of the first stage is maintained,
namely max,,; p Õ(1/ (z) and max,- (+) = (2).). This suggests the signal learning dominates the noise memorization and thus the resulting embeddings are linearly separable, which guarantees small test error for downstream tasks. The formal convergence result is established in Lemma (.155
Combined with the generalization error demonstrated in Appendix D.3, this completes the proof of Theorem
Experiments
Synthetic experiments We conduct synthetic experiments to verify the theoretical results obtained in the previous sections. We generate samples following the theoretical setups, where we set the data dimension d = 2000, number of training samples n 100, number of test samples Niees 200, and the hidden size of all encoders as m 50. We adopt gradient descent with a learning rate of 0.01 as the optimizer to train the model by 200 epochs. In the single-modal setting, the l set to be [5, 0, .... 0) and the g AO I) for the in-distribution data, and the augmentation vector ~ (V 0.01 1). For the multi-modal setting, jı [0, 15, O,... 0JT In addition, for the OOD test data Xrest = - JT,
set [2. 0..., Oj and Ç NN I). We perform logistic regression based on the learned features h hxx...at) ) and apply the learned classifier head to evaluate OOD generalization error terms of prediction accuracy.
Results. In Figure I we see the training loss of both single- and multi-modal learning converges rapidly. At the same time, OOD test accuracy of multi-mode learning converges to nearly 1.0 while

---

that of single-modal learning stagnates around 0.5. This is primarily because under the setup where the other modality çe has a higher SNR, signal learning of f is lifted. This can be verified from the third plot of Figurel] [ where the signal learning of multi-modal framework is significantly higher than single-modal. Further, can be observed that single-moda contrastive learning exhibits more severe noise memorization, which suppresses signal learning. In contrast, multi-modal contrastive learning exhibits less severe noise memorization which would further encourage signal learning. These phenomena again support and align with our theoretical results.
Real-world experiments We now extend the comparison of single-modal and multi-modal learning to realistic image data, ColoredMNIST B [5]. which is a typical benchmark studying the generaliza- tion capability under distribution shifts. The ColSTedMNIST dataset is a variation of the standard MNIST dataset, where each digit is assigned a specific color based on its label. The two modalities are image, and text that describes the images. The task 10-class classification that recognizes the number of the colored MNIST images. Specifically, we have 10 colors to color 10 digits, and introduce spurious correlations via label noises following the literature:
For the training set, 10% of labels will be clipped to random class. For images with class 'O' (or 'I'), they will be colored red (or green) with a probability of 77.5%, and another random color with a probability of 22.5%. The coloring scheme introduces a spurious correlation.
For the test set. 10% of labels will be clipped to random class. For images with class "0" (or 'I'), they will be colored as green (or red) with a probability of 77.5%, and as another random color with probability of 22.5%. The coloring scheme be considered as reversing the training spurious correlations. Therefore, the evaluation on test set can reflect to what extent the model learns to use the spurious features, i.e., colors, classify images.
We implement the multi-modal learning following the practice in [54]. where we consider an ideal language encoder that successfully encodes the caption of the images into one- hot labels of colors and digits. For single-modal learning, we follow the implementa- tion of the SimCLR 9 to construct a set of augmentations to learn the representations.
Results. Under the distribution shift, we verify that multi-modal learning archives an -uttof-distribution test accuracy of 13%, which outperforms that of single- modal learning 12.68%. As result, we can claim that the effective SNR of invari- ant features (the shape of the digit) will be degraded under the impact of the injected color. Therefore, the performance of single-modal may be suboptimal as it cannot effectively utilize the information of the digit's 's shape. On the other hand, multi-modal demonstrates a better capacity for handling this scenario.
Table I: Performance comparison for single and multi-
modal contrastive learning.
Model
Train Accuracy
Test Accuracy
Single-modal Multi-modal
88.43%
12.68%
87.77%
82.13%
7 Conclusions
In this work, we have established a comprehensive comparison of the optimization differences during the pre-training stage and the generalization gap between single-modal and multi-moda contrastive learning for downstream tasks. With the cooperation between modalities, multi-modal contrastive learning can achieve better feature learning and generalization on downstream tasks compared to single-modal learning. On the other hand, data augmentation alone can hardly improve data quality and thus cannot boost the performance of single-modal contrastive learning. Together, these results quantitatively demonstrate the superiority of multi-modal learning over single-modal learning and emphasize the importance of data quality in multi-modal contrastive learning.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their insightful comments to improve the paper. Wei Huang is supported by JSPS KAKENHI Grant Number 24K20848. Yuan Cao is supported by NSFC 12301657 and HK RGC-ECS 27308624. Taiji Suzuki partially supported by JSPS KAKENHI (24K02905) and JST CREST (JPMJCR2015).
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 15:23716-23722 2022.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020.
[3] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 028999. 2019.
[4] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 1ww2.09229992 2019.
[5] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learn- ing recover global and local spectral embedding methods. Advances Neural Information Processing Systems, 35:26671 1166771-2665,.6 2022.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[7] Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssl interplay: Augmentations, inductive bias, and generalization. In International Conference on Machine Learning, pages 3252-3298. PMLR, 2023.
[8] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237- 25250, 2022.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 15750-15758, 2021.
[11] Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Under- standing and improving feature learning for out-of-distribun generalization. In Advances in Neural Information Processing Systems, 2023.
[12] Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable represen- tation learning and zero-shot transfer in clip. arXiv preprint arXhv:2310.00927, 2023.
[13] Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. arXiv preprint arXiv:2303.0916. 2023.
[14] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self- supervised learning by characterizing idealized representations. Advances in Neural Information Processing Systems, 35: 19111119999999 2022.
[15] Lijie Fan, Dilip Krishnan, Phillip Isola. Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In Advances in Neural Information Processing Systems, 2023.
[16] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In International Conference on Machine Learning, 2022.
[17] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.

---

[18] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh. Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Veneu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-eevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[19] Sean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- self-supervised learning, Advances in neural information processing systems, 33:21271-2122 2020.
[20] Jeff Z. HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. arXiv preprint arXiv:22 14699, 2022.
[21] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
[22] Wei Huang. Yuan Cao, Haonan Wang, Xin Cao, and Taiji Suzuki. Graph neural networks provably benefit from structural information: A feature learning perspective. arXiv preprint ftfi:2300.139226 2023.
[23] Wei Huang. Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding convergence and gener- alization in federated learning through feature learning theory. In The Twelfth International Conference Learning Representations, 2023.
[24] Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.0074 2021.
[25] Yu Huang. Chenzhuang Du Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems. 34: 14:40994-10956, 2021.
[26] Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, pages 9965-10040. PMLR, 2022.
[27] Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. Journal of Muchine Learning Research, 24(330):1-78, 2023.
[28] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[29] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[30] Yiwen Kou, Zixiang Chen, Yuan Cao, and Quanquan Gu. How does semi-supervised learing with labule-labelers work? case study. In International Conference on Learning Representa- tions, 2023.
[31] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting for two-layer relu networks. arXiv preprint urXiv:2303.0412 2023.
[32] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable supervipe learning. Advances in Neural Information Processing Systems, 14:309-323, 2021.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- lanouala understanding and generation. In International Conference Machine Learning, pages 12888-12900. PMLR, 2022.
12

---

[34]
Victor Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modie contrastive representation learning. Advances in Neural Information Processing Systems, 55:17612-17623 2022.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
[36]
Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does clip's generalization performance mainly stem from high train-test similarity? arXiv preprini 100.0520 2023.
[37]
Yifei Ming and Yixuan Li. Understanding retrieval-augmented task adaptation for vision- language models. arXiv preprint 2eti::4000001668,, 2024.
[38] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statisticy, pages 4348-4380, PMLR, 2023.
[39] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[40] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP, In Advances in Neural Information Processing Systems, 2022.
[41] OpenAl. Gpt-4 technical report, 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[44] Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint arXiv:2304.03711 2023.
[45] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is caption worth thousand images? a study on representation learning. In International Conference on Learning Representations, 2023.
[46] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pages 19250-19286. PMLR, 2022.
[47] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[48] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pages 31852-3188 876 PMLR, 2023.
[49] Yuandong Tian. Understanding deep contrastive learning via coordinate-wis optimization. Advances in Neural Information Processing Systems, 15:19111-19522, 2022.
[50] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.
[51] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00572 2020.
[52] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMIR, 2021.
13

---

[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977,, 2023.
[54] Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. Do clip models always generalize better than imagenet models? Advances in Neural Information Processing Systems, 2024.
[55] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pages 9929. PMLR, 2020.
[56] Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self- ceff-supeeiied contrastive learning. In International Conference on Machine Learning, pages 11112-11122
PMLR, 2021.
[57] Yihao Xue, Siddharth Joshi, Eric Gan, Pin- Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. In International Conference on Machine Learning, pages 38938-38970 PMLR, 2023.
[58] Yihao Xue, Siddharth Joshi, Dang Nguyen, and Baharan Mirzasoleiman, Understanding the robustness of multi-modal contrastive learning distribution shift. arXiv preprint arXiv:2310.09971, 2023.
[59] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint 0rXiv:2111.1143 1432, 2021.
[60] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-languag, models behave like bags-of-words, and what to do about it? arXiv e-prints, pages arXiv-2210, 2022.
[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for langunel models. International Journal of Computer Vision, 13009))2337,2348, 2022.
[62] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing rision-languag understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[63] Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. In International Conference Learning Representations, 2023.
14

---

Appendix
Contents
Limitations and broader impact
16
Preliminary Lemmas
16
Single-modal Contrastive Learning: Proof of Theorem 4.2
17
C.1 First Stage
18
C.I.1
Dynamics of Signal Learning: Upper Bound
20
C.1.2 Dynamics of Noise Memorization: Lower Bound
23
L.3 Noise Memorization: Proof of Lemmal5.2
28
Second Stage
29
Downstream Task Performance
36
D Multi-Modal Contrastive Learning: Proof of Theorem 4.3
36
D.I First Stage
36
D.I.1 Dynamics of Signal Learning: Lower Bound
37
D.1.2 Dynamics of Noise Memorization: Upper Bound
40
D.1.3 Learnigg: Proof of Lemma
5.4
46
D.2
Second Stage
47
D.3 Downstream Task Performance
51
Additional Experimental Details
51
15

---

A Limitations and broader impact
While our theoretical analysis novel in terms optimization and generalization, the data model can be further modified to be more practical. Our theoretical analysis may be further used for empirical and theoretical studies of contrastive learning. especially multi-modal contrastive learning. However, we do not foresee direct social impact from our theory.
B Preliminary Lemmas
Before the proof, we introduce lemmas that are useful in proving our main theorem.
Lemma B.1. Let (O, o²). Then P[[æ] s c) = 2erf (vzo) s 2/
2ca exp(- 0²
Proof. The probability density function for æ given by
x²
f(x) V2TO
exp
20
Then know that
x²
P( [c)
exp
dr.
2 m
20
By the definition of erf function
2 222)dr
arf(c)
expl-2"
VT
and variable substitution yields
/20 V2TO L0 exp (- x2
1
erf
dx.
2
Therefore, we first conclude P([x| S c) 2erf
(v20)
Next, by the inequality (x) s v1 - exp(x 1xx2 / /T),
we finally obtain
P([a $ c) s 2 1-cp cxp (- 2c
0
Lemma B.I introduces an anti- coneenncnnttttoon result. later sections, this lemma will be used to show that with relatively large initialization for the weight vector, some initial properties hold.
Lemma B.2. Under condition that d
400m
log(6n (5) -
log(1- 52 (1mm]]) and à N
Gode
400m
doog V v log(1 - 82/ -)2((4m2)) then with probability at least
log(6n/8)
5, we can show for all r E [m].
8 8log(6n/8)
(w(!) (0) 1441 2 100 SNR
d
log (en/))n))
(W (0) x 2 100 SNR
d n.
Proof of Lemma
B.2 By Lemma
B.1 because (w.") (0)
.p) ~ NNO. 11114111)) we can show
20a
P(I(W()) pi) c)
exp
(- 13
16

---

Let = 100 V 8 log(66//)) n 100mm|111220 d- 1ogrg'd''Vooooooooo V8log(6n/0) and plug it into the RHS of the above inequality, which becomes:
160000 log( (6n/5)n2
RHS 2 exp
P
Then we can verify that when satisfies that d 2
(6) 8 ((mm")) holds that RHS S
- log[1
(0) 5/m. This suggests for single neuron [m]] we have P([(w; 14) S 5/m. Applying union bound, we can show the desired result.
Similarly, with the same procedure, we prove the result for the other modality.
8 log (og(6n//) n2.
(w (0)
pi) 100 SNR
d
Lemma B.3 (3II). Let Si = {i E [n] y 11 and S-1 =
[n] y -1}. Then with n
probability at least 8,
n n log log (4/)), + V 1og(4/0))
[s 1-11
Lemma B.3 states that when the label is randomly sampled, the number of positive samples and negative samples is close 2. adequately.
Lemma B.4 (18). Suppose that d > ((log(mn/0))
((log(L/))). Then with probability at least 8, it satisfies that for all r E [m], i E [n].
kw (0)
14) v 202g(mmmm// ( |e/l2
[w (0) {.) < 2
((w.. e /
V
for all [n]
ooll a 11ll111222 max (w(
14) S V log((8/06)o 1112
ré(m]
dodg v dd4 <
max (w (0)
{ < 2 /ogg(8mn//)ooo
rE/m]
Lemma B.5 8
Suppose that > and d = ((oo((6n/))) Then with probability 1 5-5,
d2 < |6 < d0
(K,, < 20 / log((622)))
(() s ||||| 2og((6n/
o,d/2 s ||e, - 3/2
(e,, E s 20.00 Vd.0gVolog(mm (6n log(6n2///
(G, 11) < 2oog
for all i [n].
C Single-modal Contrastive Learning: Proof of Theorem 4.2
In this section, we provide the proof for Theorem 4.2 which states main results of single modal learning. The training dynamics are based on the coefficient iterations presented Lemma 5.1 Below, we provide a proof for this lemma:
17

---

Proof of Lemma 5.7 Recall that the weight decomposition is expressed as
(0)
w (0) + 1
2 E; +
We plug it into the gradient descent update as described by Equation 10 yields
w(++1) (0) + 7r
(f+1)
p1
+
Pr.
(E+1)
"s
( + ( 1111 2 p
(x yau
nmn
M
+ 1
(10)
7
h (x)
(x
jts
h
(x E,
jti
By comparing the coefficients in front of μ and &, both sides of the equation, we can obtain
7 (1+1)
=x(1)
| (0) (x,
nITT
Pr. PL0 nmT
(t+1)
[1
20 1/ hr (x)
1
(x (2)
which completes the proof.
According to the behavior of the defined loss derivative X we split the entire training dynamics into two phases. In the first stage, the loss derivative remains close to its initial value as the similarity is small from initialization. Later, as the similarity grows to constant value, the loss derivative is no longer close the initial value, and the dynamics transition to the second stage. In this stage, the similarity increases logarithmically, and the empirical loss converges.
C.1
First Stage
In the first stage, the derivative the loss is close to its initial value because the similarity is small. Below, we provide useful lemma for establishing such result.
Lemma C.I. Suppose that "ir (t) 0(1) and pi? 0(1) for all r [im] and i E [n]. Under
Assumption 4.1 then for any 8 > 0, with probability at least 8 8
log (og(6n2/0)
(w(()) !. (.E.)) -Pr.l < //
(0)
d
8 logg 6n/8)
(w(()) !0) 14) - - s SNR
d n
for all r E [m], e [n].
Proof of Lemma C.1
From the signal-noise decomposition of w (t)
we derive
(w w. (.)
(22 (a)
| (c, 1
P
log(Gn2214 /0)
| a5 / logg(6n60 4
dl
18

---

(e)
log (og((n2// (0) 5V
d
Equation (a) results from the weight decomposition (see Equation In the first stage, used the upper bounds for ( and |p."), and applied Lemma B.5 inequality (b). Finally, inequality (c)
follows from the condition nSNRR (1))
Further,
2 log 6n/6)
(w!1)
w(())
(0)
(4) - 7/ 10 =
(E,, n1) s 2n SNR
d
where we have used Lemma B.5
Now, we proceed to the lemma concerning the derivative of the loss as follows:
Lemma C.2. If max (1+²) .Prii = =11) and under Assumption 4.1 there exists a constant Ce >
such that
1 Ce(1+M) M)
S 50 (00) S
C₁
1
200 45
Ce
1 1+ M'M
C((M
+
1 14 M
for all i E [n].
Proof of Lemma C.2
From the update of w we have
log log(6n2/0))
((w.), E. S kw (0) E.) + 2 + 5/
7
d
(b)
log 1og(6n2/0
< 2 (lllo(88mm/00ggg Vd pr, + 5v
d
O(1),
where (a) is by Lemma C.1 and (b) is by Lemma B.4 Finally, in inequality (c) we have used the condition that og S
2 3log(Smmm/8jogv and d n log(6n2/65) according to Assumption 4. and
max(-+ (0) p.0)) = O(1). At the same time,
8 llog (6n/d) Hi + ( + SNRV
kw (4)
[w (0)
d
8 Blog (6n /0)
v 2 (log(Sm/o)onhl) SNR
O(1),
where (a) is by Lemma C.1
(b) is Lemma B.4 and (c) is by 00 $
2 >ogg(8m (00088mm))ll and >
SNR (og(6n2///) 28) according to Assumption 4.1 and max (-t Pr..) 0(1). Besides,
(1)
(w/() ..),= ((w <s)
(2) M (m, e:) +
2 (Eir e:)
(a)
5 (( E + |11l lo /2 log(6n/0) + 4V log( 15)
g.05.11
(
8 2 (ogg vd 1111 a, llg((6n/0) + AV
log (6g2//2
0.05
3 0(1),
where (a) follows from Lemma B.5
(b) from Lemma B.4 and (c) from the conditions do S |A| d > n² log(6n2 /8), and a. < dg.
2 22/lggBBmn//6 06)0 '8' <
/ (6n(/66)))
19

---

Next, we calculate the upper bound of the similarity measure. First, we examine the negative pair. For any i.j E [n], we have
Simp x,) (h(x; 11)
sg (x(x) ()) + - (h(x,)) (g(((())))) ((((xx))))
m
o( 5.)) .E,)) +
o((w!)) (aH))))(((W)) -1s44)
77
S max (w w.1)) (WW. إ,)1 0(1).
Similarly, for positive pair.
(xin(x;;88) 82)
Σ ((w)) (ww)) ,5 6s)) + 1
m Σ ((w ( yite) ( (o(())) ,3n)))
m
max (w!2) ya (w.)) w (. E + E O(1).
According the above result, we can say that S (Sim)(xxx)) I Ce, where Ce is positive constant. Then we can provide the upper bound for f and fijj
Sim, (x:1//
Ce
Simp, (x. &:)//
Σ M
(Simm(x,x))//+
1 M
Simm &211
(()
4
Simup 8x)7
Ejjt eSimm
C₂(1 M)
Simu(x;x;)//
(1)
1.j
(x;
Σjit (x.,5))
Ce(M +1)
(
(xi)/(x;x))/7
C₁
Sim, (x;:)/
Eggt (xx,)(x;xx)/
M +1
This completes the proof.
C.1.1 Dynamics of Signal Learning: Upper Bound
In the first stage, the growth rate of signal learning is exponential. We establish an upper bound for the growth of signal learning.
Then we consider the growth of signal learning coefficient (0 Depending the initialization, define _ = (r E [m] (w!") (t) p)) > 0} and ( (r E [m] (w!") (t)
p2) < 0).
Lemma C.3. Under the condition d > 400n.
n [loggll1=/(aa 84/(4m" and Assumption 4.1 for all 1 2 O,
log (Gn/8)
auag
we have 5hth = _ 4t
o and 7
> 0 is an increasing sequence for all r E u and
(t) 7r < 0 and is decreasing sequence for all r E _
Proof of Lemma C We prove the claims by induction. To better understand the dynamics, we first
derive the propagation for signal learning from the first step. For (") i.c. (w!") 11) > can see
(1) = (0)
(0)
)a(( (aa))o'' m.
nUTT
1=1
M
(0) 1g
o (w, (0)
,3,u))o' ((w w
i=1 j
[0)
(1 ( (o (,. va ((w (w (a
TUTT
20

---

M
e ij
(( (0)
(iiu) )o
((w, 3114) )4 1
VITI
viy=1 :uy) =-1
4
Σ
(1
W 00)
M 0.
:yy=1
where last equality is by (w, (0)
14 > 0, 7t = 0, and the fact that negative samples satisfy yj + yr-
Thus, we verify that the sign of 7r follow its initialization and 7f O.
Next, we show the propagation of inner product at = 1, for r E ut we have
8 log(6n//))
(w-+), p)) (w!)) pa) + -SNRV
d
(.99 (1) 0,
where inequality (a) is by Lemma C.1 the second inequality (b) is by Lemma B.2 and the last by 07 (1) O. Hence we verify u. _
Now suppose at iteration t, the claims are satisfied, namely (w! .e) > 0 and 7: (t-
2 0 for
r E t Then following similar argument, for r u
[1+1)
(8)
7
Σ (1
(w
2 7(1 2 O,
TUTT
i:y
where we use the induction condition that (w!") (1) > Further by Lemma C.1 and B.2
(w
((+1) .p) (.9
(0) (1) + (1+1) > 0,
where we use 7/ ((+1) 2 0. This completes the induction for r E ut
Similarly, for those neuron that satisfies (w.), 11) O, i.i. _0)), we have
7/11)
1)
Σ
(1
( ( hew.://..)))u)n )ailla (00) )o((w (0) ,
nmT
1=1
M
1
((w (U) 1)))))o )o')) ((w 10) ma
TMTT
( (w (() y. 1113 O.
nmT
where the last equality is by (w. 14) < 0, 3/ -1. the property of ReLU activation, and the fact that yú # ys in the negative pair term. Hence we see 7/ I O. Similarly, for the inner product at t = 1,
(0)
(w()) (1) 14) - (w,
(0) 14) + + + Σ Pri 1 14)
8 (6g/))/))
< (w. (0) 14) + + (1) + SNR
d
S 0.99 (w |4م + 7 (1)
< 0,
where the first inequity is by Lemma last inequality follows from (f
C.I the second inequality follows from Lemma B.2 and the
< 0.
21

---

Now suppose at iteration I, the claims are satisfied, namely (w! p4) < 0 and )イ 5x
I 0 for
r E (.) Then following similar argument, for r (t
7 THIT
Σ (1 11-6 (w) Ml S 7f 2
where we use the induction condition that (w! pe) < 0. Further
(w (w(++1) (e) (w( (L) + 7/ (1+1)
Eell ( (E;: 14)
(a) (w pi) + o (1+1) S
8 8log (log(6n/0)
(b)
d n
0,
where inequality (a) follows from Lemma C.1 inequality (b). This completes the induction for r E (")
7
((+1)
< 0 and Lemma B.2 in deriving
With Lemma C.3 hand, we are ready demonstrate the upper bound of the growth rate for signal learning.
Lemma C.4. With the same condition as in Lemma C.2
and Lemma
C.3 and n 2 2500 log(4)))
define Al (0) + (wr u) for r E 2 and A
(0)
(1)
(wr (1) for _t With
probability at least 8, we have
0.52m||
1 1+ A 1
A,
mT
Proof. Lemma C.3 suggests that for r E [m].
we can upper bound for bod
Without loss of
generality, we consider
(o) By Lemma
C.1 we can see
8 log (og(6n/8)
(w) p1) S 7/00 + (wi") 11) SNR < 11001(7 (1) + (w! HH),
d
(19)
log (6//) /8)
(w (1) 11) 2 )r + (w (0) 14) - SNRR
d
2 (1. (1) + (w ,H),,
(20)
where we use Lemma B.2 and x/ 2 O.
Then, the update equation for 7/ follows
(1+1) nuT
( (w 11
Then find that
A((+1)
A(9)
Σ (1
e! (w) 11)1111
Rige m1
(a)
||.111101140
Cr(M +1)
(6)
1
A
log(4/6)
1.014.))
mt
2n
(0)
(1
0.52m||
1+
(()
(21)
1/1
22

---

where the first inequality (a) is by (19) and Lemma C.2 The second inequality (b) is by Lemma B.3
and CaMM+T The last inequality (c) is by the condition 2 2500 log(4/6).
C.1.2 Dynamics Noise Memorization: Lower Bound
To establish the lower bound of noise memorization in the first stage, we require that the added noise level o. S / for sufficiently large, with C" 2 (()). We prove the following result that upper bound the scale of (wr (t) G.).
Lemma C.S. Under the same condition Lemma C.1 and d = (max((o 211ml12 ,no, og 1))
there exists sufficiently large constant Cg >> O such that
Kw E. 2 Cel(w()) G
for all i E [n].
Proof of Lemma C.5
According to the decomposition of w, we can show
(w [.,] kw (0) E + ,, || (n,, E.) +
Σ
| ll (Epp) ea)
$ (0 (0) Gill + ,t 1 1112 (u, Gill
Ip
P l1a KEz, Gil
(0)
log Dog(6n2/1
2 2 (oV/oo((mm))moogo Vd || a. 2ll.//log(6//// + 20.00"
d
$ </C (0) E.),
where the second inequality (a) follows from Lemma B.5 and the last inequality is by the following anti-concentration result.
Because (w/") (0) ) a Gaussian random variable with mean zero and variance do Vd, by Lemma
B.I we compute
2ca
(o (0) E.i Se) <
1 1-exp
exp
Ro
20a When 2 (0r(l===((42)))))eeeee we have P(((w,") (0) &:)l S c) s 8/n /n and by union bound, we have
with probability at least 1- 6, it holds kw,") E.) > c. Here we let (ogg//lo(8mm))//6 v d+
|| o. /e.loo.2llol/n///./ 40.05" log( /51 n) 2 (l( Ee)) Then we can show the desired
result with the condition (2mmxx oo 2 || _no -
Define that Ir, {i (wl)) E.) 0} and _ (i (w E.) < 0}. To show the result regarding I(0) and I_!),, prepare the following onticconcentration result:
300 log(Gn2 /6)
Lemma C.6. Under the condition Z
then with probability at least
log(1-8 /n magg??
satisfies
(w (0) ..)I > 150 1100/ooo(t0 (og(Gn2/8)/4
Proof of Lemma C.6 Here want to show (ww (0) & 150 150 1og((622/)) & e with high probability. To see this, because (w (0)
E.) is Gaussian random variable with mean zero and
23

---

variance og by Lemma B.1
we compute
20a
(([(w.)) E
S c) s 1
1 1 exp
Rojaoo
Thus when d
Toy( 10060166700 64/4m2 A we have PP P(((w) (0) .E:I - c) S 8/n and by union bound,
1on (6)
we have with probability at least 6, holds (w,) (0) E > 150 /lgg (logg66n//7) /6)/d.
We then show that neurons with negative inner products with the noise at initialization would stay negative and the corresponding p stays zero.
Lemma C.7. Under the same condition Lemina C.1 and Lemina C.6 for all O, we have
I _ and Pr.a 0 for all E I
Proof of Lemma
C.7 The proof by induction. We first consider E I At 1,
7 (1
( )(( &. + (e)))' (0) (,))1
Prçi
Pr..
TITTUT
((w (0) -E,) Jo ((w +)) 0,
TUTUTT
where we have used the condition that (w: (0) E. < 0 and the property of ReLU activation.
Next we consider
(w(') = ((w(") + + Mlllll Σ
Pri P 2
E=1
= (w(()) {,) + x (p,, E.) 1 2 + Pr.s + Σ
Pr.e I1 ( E.)
8
E + / log (nn 15)
(b) < 0,
d
where inequality (a) is by Lemma C.1 and p = 0, and inequality (b) is by Lemma C.6
Suppose iteration I, the claim is satisfied, i.e., (w.?,,, ,.)) 0.
(t) P O, for all E ( Then
(t+1) Pr.b
" (1 ((00 (0)))((()) (1) & + ) ((w 1.),
Pr. +
(1) 13 (,)))) ((w9))(( .E 0,
TUTT
Next we consider the update of inner product as
(w((+1), E.) (ww())
(1+1)
2 Σ
pri 11::112 2E)
(e+1)
7p
= (ww")) EE)) + 2 (1+1)
(1,, E.) 1| + pr.
(++1)
+
Prpp I1& 11222 ( Ei)
(++1)
< (w.)).)) EE)) + x + (r. 1llllll
+
lp,.
11 111x 31, E.)
(++
log (6n /0)
< (w
(0) (E)) + /
n
O,
where the last inequality is by anti-concentration analysis shown in Lemma C.6 This completes the induction for E (1)
24

---

Before formally stating the main lemma on the lower bound of noise memorization, we prepare several lemmas that will be useful.
Lemma C.8. Suppose that ó > O. Then with probability 1 for all E [m], we have:
Σ Figel1
1
4 - SV2 log(4/5).
n
Proof.
The proof is by Hoeffding's inequality, for arbitrary t 0, we have that
2t
P
Σ
L.EI,++
- E
Σ
LIeIn
St S 2ex
(-m
By the randomness of initialization and Rademacher distribution, we have:
n
E
IeInt
Setting = /n/2logg log/2)g(44/5) and taking a union bound over r E [m=], we conclude with high probability least 1 5, it holds:
NEI,,
S V
n
log(4/8).
4
To establish the lower bound for noise memorization we define
(())
Σ (0 + (w("), (w()),.)) (EZ!))
(Ct) A Σ (p + (w(()))))) 1e
r,+
d-p=-1
Lemma C.9. Suppose 5 > O, the with probability at least - 8, we have
B dgo d /log(1/8)/2) B S ooog soogvdn(11 + \lo(1//6/22)
Proof. By Bernstein's incquality, for arbitrary > 0, we have
P( oooe ooag Vnd > t) < exp( t²
2n /0/0
Setting t oode ogoe /ldlog////// we further have
1 < ano + Vlo (1)
Similarly, the same result holds for B
8Ca log(1/8)
Lemma C.10. For arbitrary constant C > 0, under the condition n probability at least - 8, it satisfies
1og(1/(1(6//me)) then with
CB 0))
CB
kw -E))
(( (0)
n
Proof of Lemma C.10 Here we want to show kw, (0) (.))
with high probability. To see this, because (w/") { is random variable with positive mean and variance dd Besides, by Lemma
have
B
00) r
doog / /n + + log(1/0)/22
25

---

By Lemma B.1 we recall
8t2
P(Kw E.H. S _
1 - exp
FofoRd
8Ca log(1/5) (0/1/(1-00/))3) we have
Thus when n 2
Conog Vnn( + v log(1/0)/22
P(((w (0) =
S 8/m
n
and by union bound, we have with probability least - 6, holds ((w ,5,1
B
and
((,, (0) E. B
Besides, we define
(C) Pr. (w! E.), with y; 1, (1) (0)
(0)
T - p (w, E.), with y: = -1, 2
With all the results (lemmas) and definitions outlined above hand, we are ready to state the lemmas that provide the lower bound for noise memorization follows.
Lemma C.11. Under the same condition as Theorem then with probability at least 5,
0.96no
(1 +
yu (0) r.
(22)
2,
TUTT 101C;
((() P,
M +1 B(d)
>
(23)
Prayy =1 1 (e 2 0.
(1)
(24)
Proof. The proof is by induction. We can check
0.96no
Pronge LieT.4 = 0, w 2 (1 +
(0) r.a
nun
Besides, by Lemma C.10 and M y = o(n 1/2),, it holds that
C₁ B
T (0)
(w (0) 101
M ++1
Assuming that the results hold at t, we continue the proof by calculating the propagation of 23 and B respectively. First, we can see by Lemma C.1 and by induction, for y = and E Ir.t.
log (6n² (6n²/0) n
(t) EEiIl - GV E, + 6:) 2 (w E.) + 2. - ((
(0)
2 (1 (1- (1)) (0) (.) + + 0.
We can show that
B((+1)
(1)
B
+ 7
Σ
(1 (1- ( (0
> ((t)
bif
(w.)),,)
1 1iEz, DiEEn
RMT
Jfi
C(MM (1)++))- ((w."), ,.) + p.0 + ((w(()), ee + 6y
log(6n2/8)
< ((t) r,-
+ ?
Σ
n))
MMT
+1)+1)
Eil
=-1
1l LiEI.- Ci(M + 1) ((w!0) (0),))))))))) + Pr.
1
(0)
6V
log (g((622/6 18) (
m) o2 (dlog(6n2//0 (0) Lt..
344
26

---

( + n
Cc(M + +1 - 1.01
E
1III+ Ce(M + 1)
1
0.99
;,- (otd + /d /llg((622/))
RTm
BG(M+1]
i===11
1og(6n2/6) d
1E Ce(M + 1)
(B
Σ 6V
(o]d + oī /Vloog(6n2/)) m)
jt
( + 7
1.01
0.99²
((t)
1.0350ld
RUTT
2C
r,
$ 1 + "
1.05B
0.5B +++
ofd
1.05mo.d
(1
((0)
where the first inequality is by Lemma
C.2 Lemma C.1 and Lemma B.5 Furthermore, the second
inequality is by Lemma C.5 i..., kw (1)
e - 1/Cg (w, (.)1 (for Cg > 200), (0.01( (0) 3 2
6/log (oog(6n2//50 (ond/,) the definition of B and ( and the induction ( 2 B _ 2 _
7,+
The third inequality is by M 2 100C₂ - 1, M E 21 + o(n (12)) d > 10000 log(6n2 /8) and
Lemma
C.8
The last inequality is by ((0) r,+ 2 O.
As result, we conclude that
1.05mo 2d
B < (1
YBB
(0)
r.-
num
Finally, the induction step for ( r. can be calculated as follows:
(1+-1)
(p())
" NITT
(1 (00) (w!) E
( +G) - 1 (2 Eg))
1 115.1/
r,
V (1)
" mMF
(M+1) Ce ((w!()) .E,) +
(.)) +
(t) - ((w!!) e;)l - 6V
1og(6n2/00) m)
(M 11)
d
LjTT..1
((w))),))) M+1 ((w/2)) .5,) +
Ct
(1) P.g + +6V
log /0)
11)
(o og d dlog(6n2//00 LieT..1 Li)
1 (!)
n
(M+1) 1) Ci
M +1 0.99
Ce
1.01
(0,d og v /dlog66n2//0
nmT
n.
M +1
1.-
"
0.992 (()) - 1.01
C B(t)
0.990
mITT
M + 1
(1)
0.96T (1)
where the first inequality is by Lemma Lemma and Lemma B.5 Furthermore, the second
incquality is by Lemma C.5 i..., kw
(t)
e.)l < 1/Ce (wr") 0) &.) (for Ce > 200), (0.01(w!"), & 2
Gloo (6n2 fo) the definition of (())
+,+ and B(1) The third inequality is by M 2 100C, 1.
The last inequality is by induction
23
Then we check induction induction
23 through following inequalities:
1.05mo
(1() -. < (1 +
YBB("0 -
nMr 0.96no {" yu 7.
(1) V r.
2 (1
nmr
Together it confirms that v 000
101C₂ (00
M+1
27

---

Finally, we check the sign of Pr. for i: = and E I
(9)
no d 2nmT
Prii
Tp (1) (w/ (0)
2 (1 +
2nmr ni
ypp (1)
(w) .)) >
V
(1)
> 0.
This completes the induction proof.
C.1.3 Noise Memorization: Proof of Lemma|5.2
Before proving Lemma 5.2 we require lower bound for the initialization. Define that 4 {rr (w, (0) -.,) 0} for = 1, and _ = {r: (w, ...)) <
0} for y/ --1.
Lemma C.12. Suppose that 8 and m 2 ((1).
Then with probability at least we have
2 0.2000 d,
Σ
+ (0) 2 0.2000g Vd.
rEUR (
rE
Proof of Lemma C.12 Consider y/ = Note that (w, E.) NN -1 We define that
the event A (r E [m]. (w,"), <.) 0}. Then we can compute that (w: (0) (AA) becomes half-normal distribution with the expectation
(0)
V2ool Elll
[w (0) (A))
VT
We then apply the sub-Gaussian concentration inequality that with probability least ô
Σ rEc
v (0)
mou
E.
S mm22.
2
m0
Then we have
1 2 0.40ollE.11a 2 oog.Vd.
rEM
where we have used Lemma B.5
Similarly, we can show that for y = 1, the same result holds.
Proof of Lemma 5.2 From the upper bound on (1). we take the maximum over 4 which
gives
max A < (1 0.52
max A
mT
< 1 + 0.52 M 11lpe 12
ooll log(8/88))
Under the SNR condition SNR² 1.8, max,,; (6) rg and max,, A at the end of first stage.
there exists scale difference between
At the same time, for noise memorization, from the lower bound established Lemma C.I wc have that
0.96no2d miT
Σ
(0 ru 2 (1
Σ
(0)
r.
rE (0)
rEZ
2 (1 0.96no2d 2d
y 100.2000g Vd.
mmT
where the second inequality is due to Lemma C.12
28

---

Let
Ti log (20/(o00g vd)/ log + 0.96 VI7TET
Then we have 1 E re)
(t) r, reach 3 within Ti iterations by 22 Similarly, we can also show that
1
re (0) d ( reach 3 within 7 iterations.
On the other hand, we compute the scale of max, A
max ((11) & (1 + 0.52
الا Ti Jull 2allll//2log(mBm////
MT
( + 0.52 Yl 1
=exp
log ( (110g/ (olldd))olllllll 2 21gg(88/)))
log(1 + 0.96 1o
S exp (0..5 n SNR 2 + O nll log(0 22lgl//lm
I exp (00.5 n SNR2 + 0.01) 100000000000000000000/0000//0000
S exp (1g388/0/oorrr000)0oo)l////2/220
20 logg
(V (VV/lo(mm/0//n)
= ÖOnnn/1/
where we choose 7) sufficiently small for the third inequality. The last inequality by the SNR condition. Because we can choose 2 Cloo(m/6) for sufficiently large constant C, max, A(T o(1).
C.2 Second Stage
Proposition C.13. Let T be the maximum admissible iteration and let log(3MT*), Then can show
h S a, |p:3 S a.
Proof of Proposition C.13 We need to show Pr S a. We prove the claim by induction. It is clear
(4)
when t = 0, p.! = 0 Suppose for all 0 < < T 1, we have P S a. We aim to show the
claim holds for T.
By the update of Pr..
(8)
((+1) Pr.
S Prt
7 (1 nmt
((w e;)))
Here without loss of generality, we consider the r, i pairs such that i I ie., (w.". .g))) 0 with
y: Let try be the last time such that (4) S 0.5a. Then we have
(1w)
(( )o (a((W E e3))1
Pr.
Pr.s
nmT
Σ
l (00) )((( E, Ei)) 12.11
(25)
WIT
S
The second term can bounded as
7
(1
) E, + nmr ( (w:")) (D)_E)) (EE)) + Pr. )
(br
29

---

3no
(0.3a + 0.5a)
2nae
x 0.25a,
(26)
where the first inequality is by 1 - e((e.) I 1 and modified Lemma C.1 with Pz O(a). The
second inequality is by (0) ..) - 0.2a and S 5 umz
24 ofd
For notation convenience, we let
(x(W,,) Simm (x,, Rt)/T
1
1
Σ (((w../))ag(o((wwses. +c6)))
F,W x.) = Simm (xi, 82)/T
m
m
mt Σ
((w,, ((w.,.E)))gg(((((:s.)) for j 1, M
mIT
Next, we show the bound for Fo (W())) x:) and F2 (WO) x;) for j = 1,.,. M as
Fo(W(), x.)
o(( (( +.)).
Further, we have for j 1, ... M
F (1) xx) Σ ((w ((W)) E.)).
Then bound the difference between Fo (WO) xx) and F, (W()) x;)
1 Σ o((w ((( (o)) ,,)))
Fo x:) - (W(( xi)
(( (())(((W E (
a ((w (o()) (1) 5. + +(+)) - [((w!9,E))
mr
0.5a
(w(0) (.) Pr. 2
(1)
3
(w()) (0) (.,)) -199)/T
0.5al
(w(0) E.) + 2P
(0)
(w(
10) ,,)))T
2
W
0.5a 0.10/T
M a.
(27)
where the second inequality is by a(ww!) (1) -() 2 (w, (0) 5.) (() + + 2 /0 2 0.5a for E T.02
Further or (w!") (() &. + + 2 1 (wr
(0)
{.) + Pr. o((w)) (9) (x,))))))))))))) $ (w! .E,)) + Pr.j" The fourth
(t)
(t)
inequality is by induction and (w, E.) 3 (w, (g)) 2 -0.15a. The last inequality is by the
(0)
condition that a 2 207.
Further, we bound the loss derivative as
A + j= exp(F, x.) - Fo((((),
$1 + M exp(-aa)
T
$1 T+
30

---

(28)
T + 1
where the second inequality is by Finally, we bound
3nd ad
Σ
11 (1 E7
+
2a.
Σ (1-67
nuT
2mmT
SIST
G.STET
3n0 dd
T
a
nmr
T +1
< 0.25c
(29)
where the first inequality is by Lemma B.5 and (((,)) 5. + e,)) - (w (E + + - 2a by
induction. The last inequality is by the condition 1) S a2a"
Combining (26) and (29 with 25 gives
Prái S 0.5a 0.25a + 0.25a =
which completes the induction.
Lemma C.14. Under Assumption for S11 we have
[VLs(W()) S (maxx ( (W(().
Proof of Lemma C.14
First, we can write the gradient of Vw. Iu (W) as
M
(W)
V L(W) Σ
V.. FWW
OF, (W.x).
5=0
where
(W)
Fo.....)
-+ Fo + Ene F.
OF,
(L)(W)
((Wx))
(W. + E (..)) for j = 1,... M
0F,
By the derivation of the gradient.
(W)()))))
1
m
L. (W()))))
Ww (w())))))
O OF, x:)
V (W,,
ΣΣΣ
(W)
|| x.))l)))
(30)
OF,(W,
where the first inequality by triangle inequality and second inequality uses E, S (E, a,) for a, 2 0 and the last inequality is by triangle incquality.
31

---

Now we upper bound
|| Ww, w.
(W, xx)) as
1 ||
|V Fo (W, x,)
yi ( w..)) to (ww..... (,) G))
1
mr
+o E O) (max(lllll2
mr
where the first inequality is by triangle inequality and the second inequality is by Jensen's inequality. Similarly, we can obtain for j I...,,M .M
VV F3 W. x S mt
(o((wr, U514)) + max
For clarity let
2r. (:p4) +o c,)
žrj = o(w, )) + o((w,: E,)), for = -I,..,, M
Substituting the above results into 30
gives
aL W
2 (23) )0(( max (ax((ll((.
vLs W 11
OF, x:)
i=1
M
aL W)
M
2
knd) 0 (1al1
OF, W x,)
m ==
1=0
where the second inequality is by Σ, a.b; E a:) b:) for a, b. 0.
Next we can verify that
M
a (W)
x.)
M (W.W..))))
OF, (W, x2)
(.) e (W.x)))
FoW - (W..)...)
x.)
=2(1 Fo((W.))) ΣØ= (W.))
M
Fo(..x.))
< -6log( cool. EJ eF)).x) = (L),
(31)
where the last inequality is by - r < - 3lg((z) for x E 10, 1]. Furthermore, we can show
e x.))
2(1
kn = 0(1).
Fo(W.x:) + En (W.)))
which leads to
M
OL, (W(")
2 > 0 max (11
m M
(W(s)
7T
OF, (W(), xx)
r=1j=0
M
(F,(W!), 2)) 1) 2
(W())
1) O max (11e113. oZ
0 max
|
-/c (d) Σ L(W())
< 0 max
| 11 oad Ls (W()),
where the third incquality is by
31 and Cauchy-Schwart Schwartz inequality.
32

---

We define
w, = w + 2r log (2M//e)
E
Recall in the first stage
w(( w + + |ell 2 r Σ
(Ti Pr.
15:112" -E:
and we have
max, 176 (/22 for all 0 S S1S.1 Ti.
max,, PT.s 2 2.
(21)
Lemma C.15. Under Assumption 4.1 we have W W S 2P 1/11))
Proof of Lemma C.15
We have
[W(T)) W & (w())
W |le ||
-W
(2:)
+ (vmm) max
Prá
(Tt)
ll + 0(m //2 1 1/2)
|ell2
E
I (mm og 1-----11 1/2).
Lemma C.16. Under Assumption 4.1 we have for all [Ti,T"] T*)
(W()) x;), W*) > 2og(2MM
FV x.), W S log(
Proof of Lemma C.16
Based on the definition of W and F, (W(t), x.), we can derive for j = 0,
Fo (W()) (c)
x:). W
m
w. Pol (0 xx), w
((ww (i (w.)) w11)
mt Σ d' ((w (() +e)))
mI
o d((( (aa1))) v0)1)) (w.") vipl) 2og Σ E 11))
((w!)) (.wl"))))))owwwv ++)) ((w("), ..)) log(2M/c) + 27 (2MM2 > &>..E))11
MT
((w.), (ww)) & + 2T log(2M/o)
Σ a((w 2104))
1 m
m
a((w (000)) 11-)
a((w E, + (
1T
1
a ((w (t), E, + 22) 2/
2r
MT
33

---

where the inequality is by Lemma B.5
Next, we bound 1,, 12, 13, 4. 1s separately. For 41, we take maximum over r, which results in
m 1
((w 1 E E:))
o (w:0) (0) E.) + 2 2.
m/1
Thus we obtain
11 2 4log(2M/44)
In addition, we can show by upper bound on (4) and (0
(w(), E, + <s) S (w/)) (0) [.) 1p:3 S Ö(1)
(w! 11) S 2 !w) p1) + + S Ö(1).
This implies
12 S ((0olll1l2) 13 I log(2MMee0
d- 14 S (( Is S 0(mmd-1/2)
Based on the conditions on do. d, we can show
(VFo x:), W 2 4log(2M/e) - 1l2 - 13 14 Is 2 2 (2M Now we prove for the claim for FJ (W((), W*) for j 1,... M as follows.
(VF, (W(9) xx), W
1
d ((w))
mam + Σ o' ((w!) (w)) (()))((())))))).)))))))))))))))) (ww),,,))(."...)
mT ==1
o' ((w (9) (()) (.))) (w)))
11T
13 + 5 (2M
where the second equality is by y1 + yj-
Lemma C.17. Under Assumption
4.1 we have
|W(
w P w((++)
W" n/s (W(1))
Proof of Lemma C.17 First, we verify that for =0
(VF(( x.), W(1))
Σ v Fo Fo(
m
((W,, (ww.. w(())
MT
(o(( w!!)
mr
((w,, (ww...)))sgll((w....e.
1/T
Fo ( xx).
(32)
34

---

Similarly, we can show for j 1,... M, it satisfies that
(VE(W()) x:); -P;(((,,
(33)
By the update W(O, we have
IW - W 13 W W 13
L (0), w - (W
2n
"
M (L.(W(9)
(VE((W x:), w -"'
2n
Σ
(L.(W(O)
(5(W0) x,) (VF((W)) xx), x,), w.) (
0F, (WO), x;)
2n
(W(()
(Fo(Wl),x) - 2log(2M/c)) + + (L.(W(O) (F(WO),))) 1og(2MM())
M
Σ
(WO) x:)
(F((W xx)
1=1
j=1
1
2n
2log(2MMcc)
(L((W() log( 22og((MM/)) + Melog(2M/e) - (
n
20
(L.(W(9) - log(1 ()) - -
2 nLs(wW() - ne.
The first inequality is by Lemma C.16 The second inequality is due to the convexity of negative log-Softmax function. The last inequality is by Lemma C.14 (and the conditions on 17) and log(1 + x) S x for 2 O.
where the third equality is by (32) and 33
Lemma C.18. Under Assumption 4.1 let Ti w
= Ti Ti dd 2n
Then we have max, 1/(
Õ(1 for all Ti ST In addition, we have
|W W 17(1 Ti 11
Ls (w)))
+
for all Ti S. Thus there exists an iterate w for [T with training loss smaller than 2e.
Proof of Lemma C.18
By Lemma C.17 for [Ti,7],
[ w
[++1)) w 2 nLs(w(()
for all 8 < L. Summing over the inequality and dividing both sides by / T + yields
Lg (w(()))) < [W(T)) W + ne(t - T1 +1)
|W(T)) 11
ny(t - + + < 2e,
nyt Ti +11
==
where the last inequality is by the definition 7f for all Ti < 5 T. In addition, we have
Σ (s( < 2T) - w Ō(1 'mndddd 'o,
(34)
"
Next we prove the claim that max, (0 < 33, where B = mmxx 77 (Ti) = Ō(1V/)) for all
T < < T. Without loss of generality, we only consider ( First it is evident that at t Ti
35

---

we have max, 7/t) = B S 33. Next suppose there exists T E [Ti,T] such that max, (0)
S 3,3 for
all t [Ti, T 11. Then we let eit = max, 1 and we have by the update of y,4)
((++1) S +
7
max (w f0 1)) 111B
3
nMT
ty-=1
$ <000)
( 2 2 2log(8m/5)ooll)) )(.)) ||
TUTT
S 0
" Ls (s(())))
3
2log(8m/600oll14) + )1
2
where the third inequality is by 31
Now summing over i = =TT, T 1, we have
ブ-1
S (
11
Lss(sss (W())) 2 /2 210g(8m/8)ooolllll + 110011
Ls
mr
(=
Ý-1
1
S (T) + O
1/3
Σ (w(W
M&
(=T
S ( + ( 1d 1d= 2)8
< (TL) + +(SSNRR)
S ( + 23 33
where the second incquality is by induction and the third incquality is by (34). The last inequality is by the condition of SNR.
C.3
Downstream Task Performance
Recall that after the pre-training stage on the training data at time T, the signal learning and noise memorization satisfy
max A(T) =2)=000 =(1/vm)
max = 511 for i E [n].
Then, on the downstream task, the corresponding embedding can be calculated as follows:
hr hr (x = a ((w!") x ( = Ö(1/Ván),
h (x(2)) = ((w
(T)
( (2))))
= ((1/va).
Then, it is straightforward to check that the embedding of a finite size of samples during the finc-tuning stage is not linearly separable. Thus, the downstream task performance follows L..... (T*) (1).
D
Multi-Modal Contrastive Learning: Proof of Theorem 4.3
D.1
First Stage
Similar to the single-modal case, in the first stage, the loss derivative is close its initial value for both modalities.
(1) (0) Lemma D.I. maxax(1 (Tr) .Pr.0 P 0(1), there exists a constant Ce > 1 such that
700
1
S =((00 <
C2
Cx(1 M)
1 1 M
Ce
<
(()
<
CaMM +
1 M
for all i E [n].
36

---

Proof of Lemma D.1 The proof follows from Lemma C.2
Lemma D.2. Suppose that "tr
(t) -t (1) and P Pa.
( for all r E [m]] and e [n].
Under Assumption 4.1 then for any 5 > O, with probability at least 1 8
log( 6n 2 /6)
kw E.) P.. S 5V
log (6n/5)
kw (()) 14) 1 S SNRV
n,
log (og((n2//) (5)
Kw)(0 - E.) p b/
d
8log (6n/5)
Kw() (( [2] 7 SNR
for all r E [m]) E [n]]
D.1.1 Dynamics of Signal Learning: Lower Bound
We first analyze the dynamics of signal learning for both two modalities. Similar in the single-
modal learning, we partition the neurons, depending on the initialization, i.e., (4()) {r E [m]
(w/
(1) ( 0) and (1) p1) > 0} and (n {r [m] (w!" 1a) 0), 07. {r E [m] (w/))
(4)
7 fr E [m]] (w! p4) < 0}.
Lemma D.3. Under Assumption and the same condition as Lemma D.2 for all t > 0, we have
(1) 4 nūt u4 - and 7/
2 0, 70(0 2 0 and are increasing.
(2) n nū = _ i and 'r
(t)
< 0, 7 <0 0 and are decreasing.
(3) For ( nū orr E _0) (0) nū there exists time t' such that E u nū(
or E (t) nū!!)
Proof of Lemma
D.3
We first analyze the neurons r Eu nit By the update of 70
(1+1) = 700
+ nma
Σ (1
)((!)) . vi) ((w!)) Ba)))a 1
i=1
)o ((w w,)))ya
nmTT
izi
Σ (1
2 (0
(35)
::y==
where the second equality is by yi Vý and the derivative of ReLU activation. Similarly for the other modality,
((+1) = 7(0 +
200 )o( s1))) ((W B;) 1712
nimer
M
C ((w (1)
((w!") 12
THTET
im أ
-/0
(1 2
)
(w 1 5
(36)
nmT
1:
37

---

Hence we see at / 0, the claim is satisfied as 7/ 2 Tt" for E u n rut Then we prove
the claim by induction. Suppose at iteration / the claims are satisfied, i.e. 7 2 up-ro (t-1) 2 0 for E 4 and _. Then we can see from 355
(++1) 2 0.
Similarly, we show from (66 that (++))
(1) / 0.
Then for the inner product, we have
(0)
8 log (6n/0) d n
(w
p 77
+ (w("), (2) SNR (1+1)
(w. 11) > 0,
where inequality (a) is by Lemma D2 and inequality (b) is by Lemma
B. Similarly, the same result and the induction is complete.
holds for the other modality. Together, it shows r E 4 (((+1)) ((
Similarly, we can use the same strategy to prove claim (2) for r E _ (
Next, we analyze the case where r E u nū7
n
(1+1))= = 10 + num
(1
((
(o(( uu))) ((w!) 3ep4))) 1
M
o
Jo ((w!") 1
nmT
2m
M
(1)
<,,
(37)
VATEET
jtz
where the second equality follows from ys + yj. Similarly,
((++))
//
70
+
Ja((W (n ((W (1: 1pl1
nmm
12
M
(ol(((W (t) ,yu4))o (WW(" u
nmT
i=1
i
M
70
19 (o(( (9)
700
(38)
1:21:
Then it is clear that the change of "r
(() does not align with the sign of its initialization. Therefore there exists some time 1' 2 such that either 4 nūt or E u²th) (t) We prove
this claim by contradiction. Suppose for all L 2 0, r E u n 7 then by (3) and 38
see "T (t+1) (t)
7i < 0 and 7(2+1 2 7/ > 0. Because (w! (2) 14) S 1.01 (w,
and
(1) (
0.99 (0)
p)) + 77" this raises a contradiction as either (w (4) S or
IV
0.
With Lemma D.3 at hand, we are ready to demonstrate the lower bound of the growth rate for signal learning.
Lemma D.4. With the same condition as in Lemma C.2 and Lemma
C.3 and n 2 2500 log(4/8), define A = =/ (1) + (w. (0) 1) for E L and A -1 (t) (w/ pb) for E ()) Similarly,
we define A, = (()) + (w))) (0) for E ( and A,0) -((() (w, Tt) for r E 7 Then
with probability at least 1 5, have
A 2 0.48m p [3 2 (A((0) + A(0)) /C,)
mT
38

---

1 2 (1 0.48m| pa [ 1 (C,A,t)) + A - 1.
Mr
Proof. Without loss of generality, we consider r E u4 nūt Then by the update of y/ we have from 35 and 36
(0) (0)
(1+1) = 7
"
(1
Jo 1))1
(39)
#yy=1
((++1)
7/-
Σ
((() )0 ((w) 1
(40)
TITL
To achieve the lower bound, we define
A - /
(w, (0)
p))
A = 7/0 + (w: A
By 39 we have the lower bound of update equation,
A
A
71
Σ
(1
Jo (( 1111
TUTT
>
A
1
O(vn))
C₁ 7 |
nMT
1 M
0.48m/ p
A(
mT
The inequality by Lemma B.3 arrive at
Lemma D.1
where we recall C2
C2 M41
Similarly, by 40 we
0.48n 11 A
A 2
A
A + A
m7
mr
Then by combining the above two inequalities, we conclude that
CpA(t+)) A((+1) C,A(1) + A 0.484/1a1B (C,,A,0) + C3A
mr
2 (1 0.48m1lu1B,, (C + A(1)).
mn
Thus, we see the joint dynamics of (t) and 7/ exhibits exponential growth, i.c.
CHA + 7 > 1 + 0.48m 1m113 ) (A4(0) + A(O)).
(41)
To characterize the dynamics of 7 individually, we next track the dynamics of C,A subtracting (40) from (39), which gives
A by
C,A((t2) T(+12)_ A, 7
Σ -1-200) (C,o(()) M)) - (wo((w
TITT
Cullllll1
Σa- (CruAft)- - -A(1).
TITTT
f1y=11
Then from the the propagation, notice that the gap C,A(()) - A exhibits exponential decay regardless of the sign. Thus, we have
GLA
A(t)) (1
(1-20))
(C, -x0)
(42)
k===1
Therefore, combining
41 and (42) yields:
0.48m p 13 / ) -.0
C,A,0) + A N (1 +
mT
39

---

C,A,,
A 2 (1
Σ
11- (1- (0) (C,, - )
Then it concludes that
4 1 0.487 1BC (AA + /C,) + +11 7 )0) (4 A (c,)
TITT m:y=1
2 1 0.48n|lu|B2c (A. + ATO /C,) - -14,0) -A,0) /Cub
mT
1 2 (1
0.487|1a1BBc (C,A,)) + + -00--11 (1 2|1اe12
(1) (C,4,))) -A
mi
TeTn =1==1
0.48m|lulBC, (C + A - -CCAA - A!O11
2 1 +
mT
where we use the fact that (1- - 7|lea112
nymt Σ E (1 111))² S 1. Next, we derive an upper bound on
( T /Cpl.
AC,,A) (0) - A₂
as follows.
14 / /Cull S | I/ S 22 2og/log(mm///) /ell2 S 1
where we recall that Calllal/2 = ||0||2 and the second inequality is by the condition on dp S 0.5 18g(8m/8))--1/) 1/2 |11|
0(11)ll)") Similarly, we can show (C,.A, 1 S 1 and thus
we obtain
0.48n) || p 1BC ? "AAT0) + /C²) 1
A 2 ( +
(43)
MT
A > (1 0.48m|1 C 1 (C,,AA0)) + A!o0) -1
(44)
mT
D.1.2 Dynamics of Noise Memorization: Upper Bound
(()) In order to characterize the growth of Pr.e Pr... we partition the samples according to its sign of inner product.
- = {i E [m] : (w)) ()) > 0}, - = (i E [<] ())))))) > 0},
I = (i e [n] (w (1) (.) < <). 7 = (i in] (v < <0.
We further define
(0)
(p + (w.", (0 (1) sez( n7
+
ky==1 =1
(00
(0 + (w,") (E)) (EZ! n7
(4)
B(0)
(p + (w (0) E.) iez!!) ni
h=-=1
((0)
(pr + (w (0) E sz ni
On the other hand, we define
(())
+ (W (0) (.) 11 (
B(0)
Pr + !w!") ...))) 11 ( mz!
fy==
D0
(p + (w: (0,2))11 se nب!
(t)
-===-1
40

---

D
Σ cy==-1
Prt (w!))
(E)) (e (e70 oz
Before we state the main result, we provide some useful lemmas.
Lemma D.5. Suppose 8 > 0, the with probability at least 6, we have
B $ go00 V dd (log(1/))/)),
B _
2 goo0 Vdn(1 + \lo(11/8)2))
2
D r,t+ 2
S goog dn(1 v /og(L/))/2),
D g Vdn(1+ Vlog(1/6)/)
1
2
Proof. By Bernstein's inequality, for arbitrary > 0, we have
t
P( +, g Vnd > t) s exp(
2n 40
Setting / = 1000 log (ondlogg())))))) (1/6)/2, we further have
3 good Vono /log(1/8)/2)
1
Similarly, the same result holds for B
300 / 2n log (6n2/6) /6)
Lemma D.6. Under the condition d
an v log(l/(1 (6/mm))))) then with probability least 1 5,
satisfies
log 6n? 6n2/2)
log(6n2 /5) n +
110)
150
n:
(0)
150g
d
Proof of Lemma
D.6
/1ottt//6 1o)(66"/0)) n² with high probability. To ++ is random variable with positive mean and variance ojo 4, by Lemma
Here we want to show (
+
150
this, because B
B.I
compute
8t
P(D r.t. <t) < exp
Tojo
Thus when m0TE 1og(1/(1 (11-0/111 we have PB ,,+,- S 150
300 (og(Gn
lug(6n2 /8)
n22
I 8/m and by
union bound, we have with probability at least 8, it holds (0)))
+ 150
/ogg(6n n²
d
llog(188) Lemma D.7. Under the condition n log(1/11(6/mm)))) then with probability at least 8, satisfies
D(0) -.
Kw (0)
E.
r.t.t
(w (0)
E.)
n
Proof of Lemma D.7 Here we want show (ww") E.)l
with high probability. To see this, {,) is random variable with positive mean and variance of Besides, by Lemma
because (w()) (0)
C.9 we have
B (0) + <
dogeVdn(1 /lo((15)/2
n
n
By Lemma B.1 we compute
812
P(((W:0) (0) -E)1 S
exp (-
To30?d
41

---

Thus when n 2 log (07///1 we have
8log(1/6)
(n + v Mlg(1/6)/22
P(Kw!)) (0) 6 S
S 5/m
n
and by union bound, we have with probability at least 1 - 6, it holds ((w, (0) E.i.
Similarly, we can conclude that (w,
E.)
Lemma D.8. Under Assumption
with probability at least 6, we have
((4) +
S (1 +
1.05n,
)
B -,,,- S B (0)
p
nmT
B $ (1 +
1.05m
(0)
B - < B
niT
((0) B S 511+
1.057,
(tt) - S B
nmn
( 7 S (1
1.05m
B (0)
- B (0)
niT
Proof of Lemma D.8
According to the iteration equations for Pr. and P7.. we have
(t) (t)
(((+1) [1()
+
1]
E.)
E
49 (W/ EE) jein+
1 nzm, 112:11
nmT
ドーリー=1
Ca(M +1 Cc(M++)
1og(6n2/8)
<
(1)
+
IET nix.
((W()) E.) + + +6V (0)
n2)
nmT
d
(1) Ce(M +1M ((w()) + Pr.j - Gy log(6n2/0))
Σ
1) jEIn (o + og V /dlog(6n2/8)
d
1:0==1
CM(M + -1 (1)
(og(6n2/8)
< ((() BBiti++
+ + Σ
iEI Innt
6V
d n))
ntm
Ce(M +1)
Σ LiE :w=1
1
(B(())
-.000
Σ 66
1og(662/0)
ni) (ofd ol
Ce(M + +1)
iti
0.992
< ((()
+
numT
(1)
r,
(1)
40,
(B + +D(00
1.0350ed
(()) +,
(0) S. r. (B + (())
1 (
o?d,
where the first inequality is by Lemma C.2 Lemma C. and Lemma B.5 Furthermore, the second
inequality is by Lemma C.8 and the definition of D ,, and 0 -,- The third inequality is by
Lemma D.6
( ++ 150 log(Gn n², d > 10000 and n 2 1280000 log(4/5), The d last inequality is by choosing C2 = 1.01.
Next, we establish the following inequality
-, B
(((+1) (.0)
Σ
I 1 1|5.112
TITTT
ty=ll
-ーツ==1
log(6n2 2/) Ca(M +1) (w.)0),)) + + - 6V dl
< [(()) 1,
Σ -:
1) jeīn+
THITT
=1=11
(0 of 1 /log(66n2/0)
42

---

1 LiEIn. Ce(M + +1)
(1)
B
B
Σ 1#1
66
log(6n2 /8)
m)
m
(o²d o dllog( (2ogg66)))))
< ( -,t.- TUTTT
7
0.992
(B()) r + B(0) 0.990
4Cr
BC 7
0. - + B ----)) ord,
where the first inequality is by Lemma Lemma and Lemma B.5
Furthermore, the second
inequality is by Lemma C.8 and the definition of B B, and B -,,-.- The third inequality is by
Lemma and +,t.+ 150
log(Gn /8) n d > 10000 log(6n² (5), and n 2 1280000 log(4/6)
Similarly, we have,
B $ ((4)
7
1.05B 0.25(B r,ti
((9) rt.
oēd.
((++1) y--,- ((t) "
(.24(B(() +. + B.0) -/ ofd
nmT
For the second modality, with the same derivative, we could obtain the following inequalities:
((+1))
(+)
1.05
((+1) h+.+ 0.25 B
dd
+,+, +
TUTTT
1 +,++ nmm
((+1)
<
(())
0.24(BL0 P- + - o2d,
B < (())
+ TITTT
1.05
(411) r- 0.25(B (1) [(1)
Bitut
r,t,t Br.t.-) R.d.
+ < (+))
0.24(B,, + B/00 - o2d. {
1ITT
which completes the proof.
We define
(() (t)
Pr.i + (w))) Ed) i I
(() & Pr. + (w))) Ed) 7(0)
Lemma D.9. Under Assumption 4.1 with probability at least 1 for all 0 < 1 1 Ti
(1) for T n 7 we have Pr. 0, I ( and Pr.i I,,- -
(1)
(2) for I n [ we have p = 0, [1(1]
= (()) and Pr.i <
(t))
(3) for E I n 7 we have Pr. = 0, 7(1) 7
(11)
-,- and Pr.i < 0.
(4) for I n (0)
we have
1.06no2d
(t) < (1 +
)(1)(() 7,1 + (
(45)
MMT
( 101C; (B(1) (t)
M +1 -. +
(46)
Proof of Lemma
D.9
We partition the dynamics of P and Pr: into one of the four cases according
(9)
=!))
to its initialization (1) i I (0)
7 7( (2) [0 n n (3) i E (()) 7 (4) I n 7(
43

---

(1) In the case of E - n T,,-0 it holds that
Pr 0, - 1,0;
Pru 0, I(! -
The proof is by the induction method. is clear when 1 = O, Prçi = 0 and (0))
0. Therefore, the
induction argument holds at the initial step.
Suppose at iteration c, we have Pr. and P O. Then we have
(w/)) E.) - 2
W
(0)
[.) + Pr,i
(0)
<1) < 0,
2
(w(),E)) E.) S (w (0)
E.) + +P (w/
(
<0.
(1+1) Thus by the update of Pr.i we have
(t+1)
(1))
1
0 (00 (o(() (.))) ((7 (W.))))))
Pr
Pr.s
nmT
M
i ((w( (() E O.
j
Here we have used (w, [.) < Similarly,
(6)
(1+1)
(1)
7
(1 (10) ( (E))))))))))((( ((w 13
Prá
Pr.
nmt
M
(10 2, ( (,)) (.)" E 0.
(2) In the case of E I
(0)
We use induction. It clear when t O, we have p 0.
Suppose iteration t, it holds that p Then by the propagation of (4) we have
Prá
(t+1)
Pr.
77
(1
((()) Jo (w!!) (o)) (
29 (((!)) (,))''(( 12 o.
nuTe
3
On the other hand,
(1+1) (t) 1
(1 (o (())) 1
Pr. Pr. nmT
M
(()
((w ((W!) E... 1.
TUTTUT
[(0)
17
Σ
(WW( {,))
1
Pr.
JEIN+
Pro
where the first equation by ((ww)) {.) 2 0.
(3) In the case of I n 7(0) we use induction to prove such claim. It is clear when f = 0. (0)
Pr. 0. Suppose at iteration I, we have Pro = 0. Then we have
-
2 ( (0) (.) + -/1
(w (0)
(.E 0.
2
44

---

Thus by the update of Pr.i
((1+1);
p(+11)
Prs
= (1
+
1 (1 (oW) ( E
nmT
M
(00
2. ((((W))))))))(((())))))))))) 1 O.
VUTTUT
jfi
On the other hand,
(1+1)
(1 1 )o((
Prá
Pri
TUT
11
((o ()))) I..I.
TUTT
Σ
19 (w!! (w(",,,))))
11E:12
Pr
LJEI++
Pro
where the first equation is by (((w,
(1) E.)) 2 O.
(4) Finally, in the case of _0 n
7 we have
yp (1++1)
(t)
TD (1
T
(1)
VUTTET
1.05no d
(1)
mMT
r,
Similarly, for the other modality.
I (1+1)
17 (1
((()
niT
1.05n07 d
T
(1)
r
nMT
r,a"
Together, we can achieve that
1.05mpo2
T S (1
10 r. + T (
(0)
(47)
nim
On the other hand size, we calculate the upper bound of
(0 0.
T6 as follows
(1+1)
(++1)
(t)
(2)
" 71 (1 4/00 )o (w)! E (l))) ((w.2, 1.))
mmT
n (1 e: (o)))((())) (1) (.)) (W))) 21)
"
o ((w())
7
(1) o (w (t) E2)) E
WHTT
A
0.96no²d
1oonooe Ct
1
(v((1) ((
aimer M 1 +
(B.))
+BO
2. 2d
< (1
nmF
where the first inequality is by Lemma D.2 and Lemma D.I the second inequality is by induction 46 Therefore we conclude that
0.95mo²d
T ((()
(1)) r. ri < (1
)' (v((0) r.
(0) - ((0))
(48)
nmm
45

---

Combining 47 and 48 yields
1.05no
0.93mo
W (1
(1 (0) (0) (1
1( r. (100)
(0)
nint
1.06no
S (1 +
(00 ni
(0)
+ T
D.1.3
Signal Learning: Proof of Lemma 5.4
Before proving Lemma
5.4
we require the following lower bound for the initialization. Recall the
definition that A! 7F
+ (w/") (0)
.1)) for r E u and A. - (w/") (0) (4) for E _
Similarly, we have A
(() (w, (0) (1) for E u and A -7(( (w,)))
(0) ( for E _h)
Lemma D.10. Suppose ã > 0 and m 2 (()). Then with probability at least 6, we have
Σ
(A
1 /C,) 0.200ollall2
rEuL (ū)
Σ
(A,0) + 1 /C,) 2 0.2001114112
m
rEU 99 (7
Proof of Lemma
D.10 We first note that (w; (0)
μb) NO ا and (w, ..)
(0)
(0, o 1 We define the event A = fr E [m] (w, (0) .4) > 0, (w, A > 0}. Then
(0)
we can compute
(w 11) (AA)
(/C,)((A
Elww
+ (W IAA)
= EW
p 0)] + 1 _ (0) ( > 0]
Gollu collma112
22T
where the independence of neurons in two modalities. Let S == Em (w,"), P (AA)
(0)
(w, ((A), Then we apply the sub-Gaussiaa concentration inequality that with probability least 1 8/2,
(ww 1²) + ,T/C,,,C))) moo 14ll (mm-11/2
Σ rEA
V2T
Then suppose m = πτ(1), we have
1
Σ ((w (w (/C,,) 2 0.200l|1441: rEA
m
Similarly, we can show the same for the event where (w!") (0) union bound completes the proof.
14) 0, (w!") (0)
( < 0 and taking the
Proof of Lemma 5.4 From the upper bound on noise memorization 45 we take the maximum over i, which gives
max (t) 1
< [ + 1.06
max y (0)
TUTTT
r.a
< 1.06
2 Iog({mm/88jo0 Vo.
nmr
46

---

At the same time, for signal learning, from the lower bound on signal learning in Lemma D.4 we have for the first modality that
0.48n|1ua ? (4, + /Cx) -
A (1
mr
Taking summation over the E Eu
nūt we obtain
1
Σ
A
mT 11
0.48n|1 1C
Σ
(4 /C,) -1
m
rEu
1 0.48n|| |C
0. 0.20oll -1
mT
where the second inequality is due to Lemma D.10
Under the SNR condition n SNR* 2 1.7 and C₁ > 2.66, we can see there exists a scale difference between max,,i r. and 1. m -rauSh (i A. at the end of first stage. Let
Ti = log (20/ (20/(oollx1l2)) log (1 + 0.48C,
Then we have Ereunna A reach 3 within T₃ iterations. Using similar analysis, we can show at the same time -reuhha (0) 1 Ereugnaz Alo 1 EreuSnat A also reach
3.
(T) On the other hand, we compute the scale of max,, r, as
max (Ti) r2
S (1 + 1.06
2 lo//o/mmm/))000/
WTTT
goed
log(1 + 1.06
log (20/(ooll44)))) (0ggs)///o/ Va
cxp
log(1 + 0.48C, /4
all
n )(2) log (o0//o/ll
exp ((2.11/CC,,n SNRR) + O(( nmn
exp ((2.11/(C,,) (C,)((CCnSSR)))) + 0.01) log (B0(coll
I exp (0.5 (20/(aoll)) /4
V 24 24 Vo
/1/21//2
Ö(n 1/2).
where we choose sufficiently small for the second inequality. In third inequality, we have applied the condition that nSNR² = O(1) and op I T The last inequality is by the SNR condition.
Because choose > C'log(m/5) for sufficiently large constant C, max,, (7h)
o(1). O
D.2 Second Stage
We first show similar result as in Lemma C. for both two modalities.
Lemma D.11. Under conditions, for 0 < < T*, we have
(VLs(WW()) || < (mmxx | [
VVLs(WW) || < Ommax L
Proof of Lemma D.17 The proof follows from that of Lemma C.14 14 and hence is omitted for clarity.
D
47

---

For notation convenience, we let
Fo xx) = Simm, (xi, xT)/T
mTT ((w..Bu
m
1
Σ
m ((w,, (.w.sloo))ll(........ E.)))
mt
(W, xx) Simm. (xi, 8,)/T
Σ ((w...)))sgl((((...... for 1,... M
Fo(W, x.) = gi i (Xe, x:)/T
sg(o sglo(((wnB4u)))) mt
1
Σ sg(o (v(o((w..)))))o((...
mr
(W xx) = Simg, h (R,, x//TT m
Σ m
(w))(( E.)), for 1,.., M
mT
is worth mentioning that F,(,, terms of the derivatives.
Fj (W x;) in terms of numerical values. They differ in
We further denote
Simn x Sx.1
Ls( (W)
La (W)
Σ log
(x, + > 15 Zize (x,
6 Simm, .g 8.1/
eoo(W.x.
where (W) = - -log
= llog (.. + EE (W..))
Simmn g (x. 3,1/7 + A
SSmm ((xx)/(
Ls(W) 71 i=1
Σ log
Sitng, (2.x E M
jzzi Simg in (2.12))/T/
Simg. h x//T
(o.(W.
where I,(W) = - -log
+ Ejet (im = log
Simg (
(Fo(W, + + e (.).)
L(W, V) = Ls()sWWW))))) + Ls(W)
Here T(W, W) is the combined loss function for two modalities.
Let 0, = r E (0) i.e. (w: .14) and 0, = -1 if e 24 i.e. (w, 11) < Similarly,
let 0, = 1 if E u740) and 0,
1 if -1
( Then we define
w w (0) 2T log(2MMe)) 0,
4
(0) 27 log log(2.M/e) 0,
Lemma D.12. Under Assumption 4.1 we have [T)) - S Õ(m 12 Iml)= and (T)))
W < (/(m//211)))
Proof of Lemma D.12 The proof follows exactly the same as the proof in single-modal case. include it here for completeness. Without loss of generality, we focus on the case for (T)),
By the scale difference at Ti, we have
(W())
W | < (w))) ((()) (0)
|| +
W (0)
W
(Ti)
((())
0(m log log([/e))]l
|
48

---

s Omilla O(nmo)) + O(m log(1/e)lllllz))
(m /2 |m112')
where the first inequality is by triangle inequality and the second inequality is by decomposition of (Ti)
(fi) and Lemma B.5 The last W(I) and W The third inequality by the bound on 7r and Pr. inequality is by condition on O0-
Lemma D.13. Under Assumption
4.1 we have for E [T
(VFo xi), W 2 2 log(2M/cc
[VF] (W x:), W I log(2M/e) (e), for 1,... M
FV ) W 2 llg(2MM
FVF xx)) W log(2MM/c) for 1.... M
Proof of Lemma D.13 The proof follows similarly from Lemma C.16 and here we only show the result for the first modality. Based on the definition of W and Fj( (W()), x,), we can derive for j = 0,
Poo x:), W
V Fo (6() x;), w
m
m
d' ((w (.) + mn
d' d(ww ( [ (W)) E.)
mt
(((
(w:)) vnp) + 27 2ogg/
((w!")
(w("),,, E.) 2r0,, log((M/))) 91144) |1112
MT
m
m
o ((w ( ( Σ a (( 1), (
MT
mt
o ((( 201/ Σ (w (o0o0 r=1
F=1
First, bound Ig ((oollu11)) Is < log(2M//0((c]l1ll))) Ia Ō(ogog by the global bound on ((w,) E.)), (( (.)) = O(1)
Further, we lower bound / follows. Without loss of generality, suppose yı then we have
1 Σ
h
a( y. log(2M/e)) 2 4lg((2M/e)
WT
rEU
where the last inequality is by Lemma 5.4 and the monotonicity of 7(())
Then obtain
(VF((W()) x;), W*) 2 4 4lgg(2M/c) - 11------- 1 42 (og(2M/c)
The proof for F3 (W()) W follows the same argument as in Lemma C.16
Lemma D.14. Under Assumption 4./ we have for all E [Ti,
IW( w + (1 W w
w 11
| - ( W(0) 27
49

---

Proof of Lemma D.14 First, we see that I(w W = Ls(WW)) + (s((W() is decompos-
able in terms of W and W(t), This suggests that L(W W L (W(9) and
V WW ) = vLs W()).. Then following similar analysis as in Lemma C.17 show
we can first
(W x,), W(O) - F.( x.),, ,x.), for j = .,,, M,
(49)
(F, x.), W (W( x,), for j 0, ..., M.
(50)
Then by the gradient descent update
I - 1 [| w 1
W W*) 7 V L.S (W
2n
M
OL (W())
(VE)(W x.), W -
- - -
DF
2m
M
(L((W()
(WO) x;) (F)(W1))))) x.) (VE) xx). x.). w -
- -
11
20
M OF (Fo(Wlo,,x xx) 2l0g(2MM/e) Σ (W(()
OL, (W
OF (W()), x,) (F)(WO) .x)) 1og((MM())
2n
2
(L) + log( (2loR(2M/e) + McloR(2M/e)) 7 (
20
(L(((()) - log(1 + -) - (
n
2 nLs( (W()) -
where the third equality is by (49). The first inequality is by Lemma D.13 The second inequality is due to the convexity of negative log Softmax function. The last inequality is by Lemma D.1I (and the conditions on 77) and log(1 z) S x for x 2 O.
Similarly, we can show the same for the other modality as
(8)
[ W 2 (bbs(W()) - r/e
Combining the two results completes the proof.
4.1 let T = Ti + I W.B woll Ti
Lemma D.15. Under Assumption
Ömmn ---n== Then we have max,, | < oooe Vd and max,; plg I ooog for all
te T. In addition, we have for all Ti ST
1
W W1 + || 1 2e. 11(1 MU--T+1 T1+1)
L(W()) w(t)
1-
Therefore, we can find iterate (W()) W()) for [Ti, T] with training loss smaller than 3c.
Proof of Lemma D.15
By Lemma
D.14 for / E [Ti, T). we have for any 8 S
[[((')
W
(W
w [W(+1) w || [[++1) W' 2 ([(W w(o)) - 22c.
Summing the inequality yields
| w 1| + ((T) W 2ne(t T + 1)
L(W(s), w(o)
50

---

Dividing both sides by t -- -- + and setting = T gives
1
I( W(o) | w W
1|
2r 3c.
- 1
7 -
==
D.3
Downstream Task Performance
Recall that after the pre-training stage the training data time T,, the signal learning and noise memorization satisfy
max AT7) =[((1,
max (T)) Ö(L/Vn) for i [n]]
(T)
Then, the downstream task, the corresponding embedding can be calculated as follows:
h (x = o(( x 5(1/Va),
h (x(2) = a((w (2)))) O(1/Van).
Then, it straightforward to check that the embedding of a finite size of samples during the fine-tuning
stage is linearly separable. Thus, the downstream task performance follows Li (I*) = o(1).
E
Additional Experimental Details
We implement our methods using PyTorch. For the software and hardware configurations, we ensure consistent environments for each dataset. We run all the experiments on Linux servers with NVIDIA V100 graphics cards and CUDA 11.2, completing them within one hour.
51

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the primary contributions of the paper, including the theoretical advancements in optimization and generalization analysis of multi-modal contrastive learning and single-modal contrative learning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of this work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
52

---

Answer: [Yes]
Justification: We have provided all assumptions in Assumption 4.1 in the main paper. The
proof sketch and complete proof are provided in Section S and Appendices C and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the complete configuration in Section 6 uploaded the code in the supplementary material.
We have also
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
53

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have uploaded the code with instructions in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccpppppccceeeeeeeeee..c.ett httaa//gggiiee//eeuubbiisissssss /Cc////oddiiisssseeeeeee ssiiiiiiscyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
We have provided all the training and test details in Section 6 and uploaded code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We plot 1 1-sigma error bar for results shown in Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
54

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided sufficient information about computer resources in Appendix
E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://eurrpp::: wwcppblicc/bhicsuuuuee
Answer: [Yes]
Justification: The research does not involve any human subjects, personal data, or interactions that would raise ethical concerns about consent, privacy, or respect for persons. In conclusion, the research aligns with the ethical principles outlined in the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
55

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
56

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the asshts?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
57


Reviews:
[{"review_id": "0Uan1Fqbmk", "reviewer": "Reviewer_bcfY", "summary": "This paper provides a theoretical analysis comparing single-modal and multi-modal contrastive learning. The authors develop a unified framework to analyze the optimization dynamics and generalization capabilities of both approaches. Key findings include:\n\n- Both single-modal and multi-modal contrastive learning can achieve small training error.\n- Multi-modal contrastive learning generalizes better to downstream tasks compared to single-modal learning.\n- The advantage of multi-modal learning comes from cooperation between modalities and higher quality data in the second modality.\n\nThe analysis is based on a two-stage optimization process and uses a signal-noise data generation model. Theoretical results are supported by synthetic experiments.", "questions": "Line 205: “On the contrary, augmentation often maintains the same SNR as the original data, so single-modal learning hardly benefits from the augmentation and can only memorize the noise from the data.” This claim significantly contradicts empirical experiments such as SimCLR and MoCo. How would you justify this?", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Develops a unified framework to analyze both single-modal and multi-modal contrastive learning, allowing for direct comparisons.\n\n- Provides detailed theoretical analysis, including convergence guarantees and generalization bounds.\n\n- Identifies key factors (signal-to-noise ratio, cooperation between modalities) that explain the superior performance of multi-modal learning.\n\n- Supports theoretical findings with synthetic experiments that align well with the analysis.", "weaknesses": "- The paper could benefit from more intuitive explanations of the key theoretical results to make them more accessible to a broader audience. For example, can you provide intuition for why the cooperation between modalities leads to better signal learning in the multi-modal case? How might this insight be leveraged to improve single-modal contrastive learning?\n\n- The paper lacks discussion on how the insights could be applied to improve existing contrastive learning methods or guide the development of new approaches. Do your theoretical insights suggest any practical strategies for improving multi-modal contrastive learning, such as ways to select or preprocess data to increase the effective signal-to-noise ratio?\n\n- The theoretical setup and assumption of the linear data generation model are somewhat simple and restricted. Do you expect the main insights to hold for more complex data distributions?\n\n- The experimental validation is limited to synthetic data. Including experiments on real-world datasets, even if simplified, would strengthen the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their constructive feedback and insightful comments on our paper. We appreciate the opportunity to address the points raised and clarify our contributions. Below, we provide detailed responses to the main comments and questions.\n\n---\n\n**W1**: Can you provide intuition for why the cooperation between modalities leads to better signal learning in the multi-modal case? How might this insight be leveraged to improve single-modal contrastive learning?\n\n**R1**: Both the single-modal and multi-modal contrastive learning aim to maximize the similarity between positive pairs while minimizing the similarity between the negative ones. The difference is single-modal learning forms positive pairs by data augmentation while multi-modal learning forms positive pairs by the correspondence between the two modalities. Usually, the data augmentation does not change the signal-to-noise ratio (SNR), while the two modalities can have different SNR. Due to the use of contrastive loss, the learning of signals in one modality highly depends on the signal learning in the other modality. Thus, when the other modality has a higher SNR, the learning of the first modality is lifted by aligning with the second modality. This cooperative learning leads to better overall signal learning compared to single-modal contrastive learning.\n\nTo improve single-modal contrastive learning, one could potentially simulate the effect of multi-modal learning by augmenting the single modality with additional synthetic views that enhance the signal-to-noise ratio, thereby improving the overall learning process.\n\n---\n\n**W2**: Do your theoretical insights suggest any practical strategies for improving multi-modal contrastive learning?\n\n**R2**:  Please refer to the global rebuttal\n\n---\n\n**W3**: Do you expect the main insights to hold for more complex data distributions?\n\n**R3**: While our current analysis is based on a linear data generation model, we believe that the main insights regarding the benefits of modality cooperation and the importance of signal-to-noise ratio (SNR) are likely to extend to more complex data distributions. The core principles of multi-modal learning, such as leveraging complementary information from different modalities to enhance signal learning, are not inherently tied to the simplicity of the data model.\n\nTo ensure broader applicability, future work will focus on validating these insights with non-linear and more realistic data models. This will involve experimenting with a variety of data distributions and network architectures to confirm that the advantages of multi-modal cooperation and improved SNR hold true in more practical and complex scenarios.\n\n---\n\n**W4**: The experimental validation is limited to synthetic data. \n\n**R4**: We have extended the comparison to realistic image data, ColoredMNIST [1,2], which is a typical benchmark studying the generalization capability under distribution shifts. The task is a 10-class classification that recognizes the number of the colored MNIST images. The two modalities are image and text that describes the images. We implement the multi-modal learning following the practice of [2], where we consider an ideal language encoder that successfully encodes the caption of the images into one-hot labels of colors and digits. For single-modal learning, we follow the implementation of the SimCLR paper to construct a set of augmentations to learn the representations. Then, under a mild and realistic distribution shift, we verify that CLIP archives an out-of-distribution test accuracy of 82.13\\%, which outperforms that of SimCLR 12.68\\%.\n\n| Model | Train Accuracy | Test Accuracy |\n|-------|----------------|---------------|\n| SimCLR | 88.43% | 12.68% |\n| CLIP   | 87.77% | 82.13% |\n\n\n---\n\n**Q**: Line 205: “On the contrary, augmentation often maintains the same SNR as the original data, so single-modal learning hardly benefits from the augmentation and can only memorize the noise from the data.” This claim significantly contradicts empirical experiments such as SimCLR and MoCo. How would you justify this?\n\n**A**: Our argument does not contradict the success of single-modal contrastive learning methods such as SimCLR and MoCo. The statement is made in a comparative context with multi-modal contrastive learning. In our analysis, when comparing single-modal to multi-modal contrastive learning, the augmentation in the single-modal case often does not improve the signal-to-noise ratio (SNR) as effectively as having a second, complementary modality would. Therefore, in scenarios where the SNR is not significantly enhanced by augmentation, single-modal learning may perform worse than multi-modal learning.\n\nHaving said that, this does not imply that single-modal methods like SimCLR and MoCo fail in general. Empirical evidence shows that these methods are quite effective, particularly when the number of training samples is large and the SNR is sufficiently high. Our theoretical framework can indeed provide guarantees for the success of single-modal contrastive learning under such conditions.\n\n\n---\n\n**References**\n\n\n[1] Invariant Risk Minimization arXiv 2019.\n\n[2]  Do CLIPs Always Generalize Better than ImageNet Models? arXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpYif767CX", "reviewer": "Reviewer_7UFF", "summary": "This paper presents a theoretical framework to understand the differences between multi-modal and single-modal contrastive learning approaches. It emphasizes the impact of signal-to-noise ratio (SNR) on the generalizability of these learning methods in downstream tasks. The authors argue that multi-modal learning, through the cooperation of modalities, can achieve better feature learning and performance in downstream tasks compared to single-modal learning.", "questions": "The theoretical analysis is predicated on rather strong assumptions. If the volume of data and the complexity of the models are significantly increased, would conclusions still be applicable?", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper explore an interesting problem. As to why multi-modal contrastive learning might be more effective than its single-modal, this paper provides a feature learning theory framework for analyzing differences between multi-modal and single-modal contrastive learning which is valuable for the machine learning community.\n- This paper delves into the nuanced dynamics of multi-modal and single-modal contrastive learning by employing a data generation model that incorporates signal and noise, thereby enabling an in-depth study of the optimization trajectories under gradient descent.\n- The authors notably identify the signal-to-noise ratio (SNR) as a pivotal determinant of the generalizability across learning methods, and compellingly demonstrates that the orchestrated interplay between modalities in multi-modal contrastive learning fosters superior generalization in downstream tasks, underscoring the importance of harmonizing diverse data streams for effective feature learning.", "weaknesses": "- **Strong assumptions made in the theoretical analysis** The analysis relies on some strict assumptions, such as specific signal-to-noise ratios and network initialization conditions, which might not hold in real-world applications. \n- **Some influential previous works are not introduced** such as UMT/UME[1], QMF[2], MMParato[3], and ReconBoost[4].\n\n[1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.\n\n[2] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023\n\n[3] MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024\n\n[4] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their constructive feedback and thoughtful comments on our paper. We appreciate the opportunity to address the points raised and clarify our contributions. Below, we provide detailed responses to the main comments and questions.\n\n---\n\n**W1**: Strong assumptions made in the theoretical analysis The analysis relies on some strict assumptions, such as specific signal-to-noise ratios and network initialization conditions, which might not hold in real-world applications.\n\n**R1**: The data model adopted in this work aligns with related works in the field. Specifically, we use small weight initialization from a Gaussian distribution, ensuring that the network with gradient descent can effectively perform feature learning, as demonstrated in [a,b]. If large initialization were used, the training might fall into the lazy training regime or neural tangent kernel (NTK) regime, which has different learning dynamics and generalization properties [c,d]. The specific SNR values were chosen to create a scenario where we can make a meaningful comparison between single-modal and multi-modal contrastive learning. Similar strategies have been adopted in prior works [e,f].\n\n---\n\n**W2**: Some influential previous works are not introduced such as UMT/UME[1], QMF[2], MMParato[3], and ReconBoost[4].\n\n**R2**: We thank the reviewer for pointing out these valuable references. Indeed, the mentioned works provide significant theoretical and empirical insights into multimodal learning:\n\n[1] \"On Uni-Modal Feature Learning in Supervised Multi-Modal Learning\" (UMT/UME) explores feature learning in multimodal contexts, although its setting and claims differ from ours.\n\n[2] \"Provable Dynamic Fusion for Low-Quality Multimodal Data\" (QMF) focuses on a popular multimodal fusion framework from a generalization perspective.\n\n[3] \"MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance\" identifies previously ignored gradient conflicts between multimodal and unimodal learning objectives through optimization analysis.\n\n[4] \"ReconBoost: Boosting Can Achieve Modality Reconcilement\" explores a novel multimodal alternating learning paradigm that aims to reconcile the exploitation of unimodal features with the exploration of cross-modal interactions.\n\nThese studies are relevant to our work, and we will cite them and add a discussion in our revised version to provide a more comprehensive overview of the related literature and to position our contributions in the broader context of multimodal learning research.\n\n---\n\n**Q**: The theoretical analysis is predicated on rather strong assumptions. If the volume of data and the complexity of the models are significantly increased, would conclusions still be applicable?\n\n**A**: Our theoretical analysis indeed can cover scenarios with large data sizes (when sample size $n$ and dimension $d$ increases). This ensures that our conclusions about training dynamics and generalization hold even with an increase in the volume of data. \n\nWe believe that the main insights regarding the benefits of modality cooperation and the importance of signal-to-noise ratio (SNR) are likely to extend to more complex data distributions (such as non-linearly separable) as well as more complex model architecture (such as transformers). The core principles of multi-modal learning, such as leveraging complementary information from different modalities to enhance signal learning, are not inherently tied to the simplicity of the data model. We acknowledge that more complex scenarios require further investigation and we consider this an interesting direction for future work.\n\n---\n\n**References**\n\n[a] Cao, Yuan, et al. \"Benign overfitting in two-layer convolutional neural networks.\" Advances in neural information processing systems 35 (2022): 25237-25250.\n\n[b] Suzuki, Taiji, et al. \"Feature learning via mean-field langevin dynamics: classifying sparse parities and beyond.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[c] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural tangent kernel: Convergence and generalization in neural networks.\" Advances in neural information processing systems 31 (2018).\n\n[d] Chizat, Lenaic, Edouard Oyallon, and Francis Bach. \"On lazy training in differentiable programming.\" Advances in neural information processing systems 32 (2019).\n\n[e] Wen, Zixin, and Yuanzhi Li. \"Toward understanding the feature learning process of self-supervised contrastive learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[f] Zou, Difan, et al. \"Understanding the generalization of adam in learning neural networks with proper regularization.\" arXiv preprint arXiv:2108.11371 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jNUglQza0a", "reviewer": "Reviewer_nN5z", "summary": "In this paper, the authors performed theoretical analysis between single-modal and multi-modal contrastive learning. The authors proved theorems that indicates single-modal contrastive learning tend to perform worse on test datasets after converging on the training set, while multi-modal contrastive learning tend to perform better on the test set. They also conducted small-scale simulated experiments that confirmed their theoretical results.", "questions": "1. Please address the problems in the weakness section.\n\n2. Does your theorems provide any new insight into future research contrastive learning? Any actionable suggestions?", "limitations": "Discussion is adequate.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper studies the important topic of contrastive learning, and put effort into creating a theoretical understanding of the difference between single-modal and multi-modal contrastive learning. They also conducted small-scale simulated experiments that confirmed their theoretical results.", "weaknesses": "There are many problems with the proof:\n\n1. The starting point of the proof is Assumption 4.1. However, Assumption 4.1 contains many issues that puts the soundness and correctness of the entire proof as well as the usefulness of the results into question:\n\n(a) Incorrect use of big-O-like notations. Big-O notations are formally defined only as sets of functions with variables, but in Assumption 4.1, the authors used big-O-like notation to bound constants, which doesn't make sense.\n\n(b) Assumption (1) is extremely unrealistic. It basically requires that $d>n^2$, meaning that the input data's dimension is larger than the square of the number of data points we have in the dataset. In any real world scenario, your data's input dimension will not be anywhere close to the square of the size of your dataset. For example, OpenAI CLIP is trained with $n=400M$ data instances, and the instance dimension is in the order of $d<2\\times 10^5$, so in this case, $d < \\frac{1}{10^{11}}n^2$.\n\n(c) Undefined variable $\\sigma_0$. Technically $n$ and $\\eta$ is also never formally defined in text, but we can guess that they mean the size of the dataset and learning rate respectively.\n\n(d) It is also unclear how assumption (6) distinguishes between the single-modal and multi-modal features, as the assumption only applies to the first modality's SNR. \n\n2. Theorem 4.2 and 4.3, the main results of the paper, are not well-defined. $\\epsilon$ is supposed to be a vector according to Eq(5), but the main result of Theorem 4.2 says that the train loss (which is scalar) is less than $\\epsilon$.\n\n3. Confusing definition of the loss derivatives in Eq (8). Are these definitions for $l_i^{(t)}$ or for $l'^{(t)}_{i}$? These expressions does not look like derivatives.\n\n4. Lemma 5.1 is not proven anywhere, but is then directly used in proving Lemma C.3 and Lemma C.7. \n\n5. The overall proof is organized poorly. It's very difficult to understand what each Lemma is trying to achieve and how are the Lemmas connected to each other, especially between the main text and the Appendix. For example, it is very hard to find where Lemma 5.2 (in the main paper) is proved in the Appendix. Perhaps the authors should add references in the main paper to the appendix (for example, state after Lemma 5.2 \"proof in Appendix X.X\").\n\n6. The proof roadmaps in Section 5 does not lead to conclusions in Theorems mentioned in Section 4. For example, the roadmap in Section 5.1 does not explicitly tell us how we end up with Theorem 4.2.\n\n7. In proof of Lemma B.2, on Line 560, it is unclear what \"RHS\" is referring to. Is it the RHS of the last equation on the previous page? Or is it the one immediately before it on line 560? If it is the latter case, you shouldn't be allowed to \"set\" it to $\\delta\\/m$ since $\\delta$ already appears in the expression before. \n\nIn addition to problems in the proof, it is also unclear how the \"noise memorization\" and \"signal learning\" evaluation numbers are defined in the experiment in Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed and constructive feedback on our paper. Below, we provide detailed responses to the main comments\n\n---\n\n**W1(a)** Incorrect use of big-O-like notations. \n\n**R1(a)** We appreciate the reviewer's attention to the details of our notation. We have followed standard usage of big-O notation, such as in [1,2,3,4]. Big-O notation is indeed a mathematical tool used to describe the asymptotic behavior of functions. While it is most commonly used to describe functions in terms of variables, it can also be used to describe constant bounds. For instance, the notation $O(1)$ is widely accepted and used to indicate a constant upper bound regardless of the size of the input.\n\n---\n\n**W1(b)** $d>n^2$ is unrealistic.  \n\n**R1(b)** We appreciate the reviewer's concern regarding the assumption $d> n^2$. Our work seeks to provide a theoretical understanding of the difference between multi-modal and single-modal contrastive learning in the over-parameterization regime. The high dimensional setting we adopted ensures sufficient over-parameterization, as the number of trainable parameters is $d \\times m$. This setting is common in theoretical studies to simplify analysis and highlight key phenomena. Similar high-dimensional settings have been adopted in relevant prior works [3,4,5,6].\n\n---\n\n**W1(c)** Undefined variable σ_0, n, η.\n\n**R1(c)** Thank you for bringing this to our attention. $\\sigma_0$ represents the initialization strength of the weights of the neural network, where $\\mathbf{w}^{(0)}_r \\sim \\mathcal{N}(0, \\sigma^2_0 I)$. $n$ is the number of training samples and $\\eta$ is the learning rate. We will ensure that these variables are explicitly defined and clearly explained in the revised version of the manuscript. \n\n---\n\n**W1(d)**  Unclear how assumption (6) distinguishes between the single-modal and multi-modal features.\n\n**R1(d)** Assumption (6) specifically applies to the first modality's SNR. Assumption (7), on the other hand, specifies the relationship between the signal strengths of the two modalities, given that we assume the noise scale is the same across both modalities (as stated on Line 123). Together, assumptions (6) and (7) establish the requirements on the SNR for the second modality, thereby distinguishing the roles of single-modal and multi-modal features.\n\n---\n\n**W2** ϵ is a vector according to Eq(5), but is scalar in Theorem 4.2.\n\n**R2** In our paper, we use bold-faced letters to denote vectors and matrices, while non-bold letters represent scalars (line 106). Therefore, the scalar $\\epsilon$ and the vector $\\boldsymbol{\\epsilon}$ are distinct variables. We will ensure to clarify this notation in the revised version of the manuscript to avoid any confusion.\n\n---\n\n**W3**  Confusing definition of the loss derivatives in Eq (8). \n\n**R3** We use $\\ell'^{(t)}_i$ to denote the 'effective' loss derivative, which is a component of the gradient of $L$ with respect to $L$ with respect to $\\mathbf{w}_r^{(t)}$. The complete form of gradient is given in Eq. (10) in the main paper. We will ensure to clarify this notation in the revised manuscript to avoid any confusion.\n\n---\n\n**W4** Lemma 5.1 is not proven.\n\n**R4** Lemma 5.1 is derived directly from the weight decomposition (Eq 11) and gradient descent training (Eq 7). Similar results can be found in the literature [3,7]. We will include the proof of Lemma 5.1 in the appendix of the revised manuscript to ensure completeness and clarity. Thank you for pointing this out.\n\n---\n\n**W5**: Hard to find where Lemma 5.2 is proved in the Appendix.  \n\n**R5**: Thank you for the comments. We will make sure to restructure the proof to improve the clarity and add explanations for each lemma. For the proof of Lemma 5.2, it can be found in Appendix C.1.3. Apart from the contents of the appendix which we have already provided, we will add references to Lemma 5.2. \n\n---\n\n**W6**: The roadmap in Section 5.1 does not explicitly tell us how we end up with Theorem 4.2.\n\n**R6**: We would like to highlight that proof roadmaps are to provide *major* steps and lemmas (rather than all of them) that lead to a final conclusion. And we leave the remaining part in the appendix given the limited space in the main paper. In our revised version, we will add more explanations by including the remaining proof sketches that lead to Theorems in Section 4.\n\nHere we make an explanation using single-modal learning (Section 5.1) as an example. After we have shown convergence of the dynamics in the second stage, we can show the maximum signal learning is on the order of $\\tilde O(1/\\sqrt{n})$, while maximum noise memorization is on the constant order (Lemma C.17). This means for downstream tasks, the embedding of a test data would be dominated by the noise, overshadowing the signal. Thus the embedding is not linearly separable, which gives a constant test loss. \n\n---\n\n**W7**:  Line 560, it is unclear what \"RHS\" is referring to.  \n\n**R7**: the RHS denotes RHS of the inequality\n\n$ \\mathbb{P} \\big( |\\langle \\mathbf{w}^{(0)}_{r} , \\boldsymbol{\\mu} \\rangle| \\leq c \\big) \\leq 2 \\sqrt{1 - \\exp\\big(  - \\frac{2 c^2}{\\sigma_0^2 \\\\| \\boldsymbol{\\mu} \\\\|_2^2 \\pi} \\big)}$\n\nThe $\\delta$ here is defined in line 558 of Lemma B.2. In the proof of Lemma B.2, it is the first time to use $\\delta$, thus it is valid. Similarly, the $\\delta$s in Lemma B.3, B.4, B.5 are all defined within the scope of the respective lemma.\n\n---\n\n**W8**:It is also unclear how the \"noise memorization\" and \"signal learning\" are defined in the experiment in Figure 1.\n\n**R8**: The signal learning and noise memorization refer to the coefficients $\\gamma_r^{(t)}$ and $\\rho_{r,i}^{(t)}$, which we have already explicitly defined in Line 214. In the experiments, we track the coefficients along the optimization path to plot the results.\n\n---\n\n**Q**: Does your theorems provide any new insight into future research contrastive learning? \n\n**A**: Please refer to the global rebuttal", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and ACs

Thank you all for your time and constructive feedback! We are truly encouraged to see many positive comments on our work, such as the the *unified framework for an interesting problem* (Reviewer bcfY, Reviewer 7UFF, Reviewer nN5z), *thorough theoretical analysis* (Reviewer bcfY, Reviewer 7UFF), *identification of key factors* (Reviewer bcfY, Reviewer 7UFF), and *supportive simulations* (Reviewer bcfY, Reviewer nN5z).

Your insights have greatly helped us strengthen our manuscript. Here, we provide a response to a common question raised by Reviewer bcfY (W2) and Reviewer nN5z (Q):


**Q**: Do your theoretical insights suggest any practical strategies for improving multi-modal contrastive learning?/ Does your theorems provide any new insight into future research contrastive learning? Any actionable suggestions?

**A**: Our theoretical results highlight that increasing the effective signal-to-noise ratio (SNR) across modalities is crucial for improving multi-modal contrastive learning. This leads to two practical strategies for developing new approaches:

- **Selecting or Generating High-Quality Data Pairs** Ensuring that the signal is strong and well-aligned across modalities can significantly improve the performance. For instance, improving the quality of aligned image-caption samples by filtering out poorly aligned pairs used for training can enhance the overall learning process [1,2,3]

- **Improving SNR for the Text Modality** Enhancing the descriptiveness of the captions can boost the SNR for the text modality, leading to better performance in multi-modal contrastive learning [4,5,6]

Should the reviewers have any further suggestions or wish to discuss any points in more detail, we would be more than delighted to continue our productive exchange. Once again, we deeply appreciate the reviewers' time and valuable comments.  

Best regards,

Authors of Submission 555


---

**References**

[1] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.

[2] Nguyen, Thao, et al. "Quality not quantity: On the interaction between dataset design and robustness of clip." Advances in Neural Information Processing Systems 35 (2022): 21455-21469.

[3] Gadre, S.Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J. and Orgad, E., 2024. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36.

[4] Santurkar, Shibani, et al. "Is a caption worth a thousand images? a study on representation learning." The Eleventh International Conference on Learning Representations. 2023.

[5] Nguyen, Thao, et al. "Improving multimodal datasets with image captioning." Advances in Neural Information Processing Systems 36 (2024).

[6] Fan, L., Krishnan, D., Isola, P., Katabi, D. and Tian, Y., 2024. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36.

Meta-review: The authors develop a unified framework to analyze both single-modal and multi-modal contrastive learning. They explore why multi-modal contrastive learning might be more effective than its single-modal counter part. The reviewers noted some concerns about the clarity of the proofs and the underlying assumptions (which are found to be too strong), but the rebuttal resolved the main questions and concerns. The authors are urged to incorporate all suggestions and additional clarifications into the paper or the appendix of the paper. Despite the concerns, the theoretical framework can be valuable for the machine learning community.
Decision: Accept (poster)