Title: On the Comparison between Multi-modal and Single-modal Contrastive Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-modal contrastive learning

single-modal contrastive learning

optimization

learning theory
Abstract: Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.

On the Comparison between Multi-modal and Single-modal Contrastive Learning

Abstract

Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.

1 Introduction

Large-scale pre-trained models have achieved unprecedented success, including GPT series [6, 41], LLaMa [53], among many others. CLIP [42] as a typical example, uses a multi-modal contrastive learning framework to learn from a massive scale of image-caption data. The multi-modal contrastive learning in CLIP has shown significant capabilities to learn high-quality representations, which are ready to be adapted to a wide range of downstream tasks, forming the backbone of generative models

like DALL-E2 [43], prompt learning [61] as well as general purpose multi-modal agents [62, 35]. Given the huge success of models like CLIP that have stellar zero-shot and few-shot capabilities on a wide range of out-of-distribution (OOD) benchmarks, they have been widely recognized as foundation models (FMs). More similar examples are given by ALIGN [28], Florence [59], BLIP [33], Flamingo [1].

Despite the unprecedented success achieved by multi-modal contrastive learning, the fundamental mechanism that leads to greater performance, especially compared to single-modal contrastive learning is still under-explored. Recently, several seminal works provided theoretical explanations for either single-modal [4, 5, 14, 48, 24, 7, 50, 49, 20] or multi-modal contrastive learning [38, 37, 44, 12]. For example, [56] studied how single-modal contrastive learning learns the feature representations for neural networks by analyzing its feature learning process. As for multi-modal contrastive learning, [12, 58] provided explanations for why multi-modal contrastive learning demonstrates zero-shot transferability, and robustness to distribution shifts, than supervised learning, which offer valuable insights. Although both lines of the existing works provide valid theoretical insights under the respective settings, rare work has compared the optimization and generalization of the two types of contrastive learning under a unified framework. This motivates us to establish a systematic feature learning analysis for both single-modal and multi-modal contrastive learning.

In particular, we consider a data generation model that contains two modalities of data, which are generated from signal and noise features. The signal feature correlates in different modalities, while there is no correlation between noise features among modalities. We then study the optimization of single-modal and multi-modal contrastive learning under gradient descent training. By studying the trajectories of signal learning and noise memorization, we establish the convergence conditions and further characterize the generalization ability in the downstream tasks. The results show that, through the cooperation between modalities, multi-modal contrastive learning can achieve better generalization in the downstream task. In contrast, without the help of the second modality, singlemodal contrastive learning concentrates on learning noise from the data, and thus generalizes poorly on the downstream tasks. The main contributions of this work are summarized as follows:

This work establishes the first systematic comparative optimization analysis for single-modal and multi-modal contrastive learning under gradient descent training in non-convex settings. We show that both single-modal and multi-modal can achieve near-zero training error under InfoMax contrastive loss after polynomial number of iterations, by overcoming the non-convex difficulty.

By a trajectory-based analysis of the signal learning and noise memorization of the ReLU network from the data, we successfully characterize the difference in generalization between single-modal and multi-modal contrastive learning. The distinct SNRs of different modalities lead to a divergence in the generalization of downstream tasks for the two contrastive learning frameworks.

Our theory suggests that the advantage of multi-modal over single-modal contrastive learning comes from the high quality of the second modality and the cooperation between the two modalities through contrastive learning. This divergence is ultimately reflected in the difference in feature learning and the final gap in downstream task generalization. Experimental results on both synthetic and real-world datasets confirm our theoretical findings and understanding.

2 Related Work

Theoretical Understanding of Single-modal Contrastive Learning. The seminal work [4] started theoretical research on single-modal contrastive learning. They assumed that different positive samples are independently drawn from the same latent class, making a connection to supervised learning. [55] identified two key properties related to the contrastive loss: alignment and uniformity. Alongside, [32] illustrated that predicting auxiliary prediction tasks helps in learning representations effective for downstream prediction tasks, and [52] provided a theoretical analysis of contrastive learning in the multi-view setting. Besides, [51] proposed a theoretical framework to understand contrastive self-supervised learning from an optimization perspective. [21] proposed a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective on neural net representations. [46] pointed out the importance of inductive biases of the function class and training algorithm in understanding contrastive learning. The most related work to us is the work by [56]. Similar to them, this work studies ReLU networks and considers the signal-noise data model. However, we do not require the adjustable bias term in

the activation function, which plays a critical role in [56]. Furthermore, this work adopts a unified framework to compare with multi-modal contrastive learning, which is out of scope in [56].

Understanding of Multi-modal Contrastive Learning. As the multi-modal contrastive learning approaches such as CLIP received great success, recent works have been proposing explanations from empirical perspective. [36] empirically showed that high train-test similarity is insufficient to explain CLIP's OOD performance. [60] illustrated that CLIP behaves similarly to Bags-of-words in language-based image retrieval, i.e., the order of words in the input sentence does not largely affect CLIP to find the corresponding image. Besides, [16] demonstrated that training data diversity and the ability to leverage the diversity as supervised learning is the key to the effective robustness of CLIP. Theoretically, [13] proved that multi-modal contrastive learning can block-identify latent factors shared between modalities by the a generative data model. [44] analyzed the training dynamics of a simple multi-modal contrastive learning model and show that contrastive pairs are important for the model to efficiently balance the learned representations. Furthermore, [38] showed that each step of loss minimization by gradient descent can be seen as performing SVD on a contrastive cross-covariance matrix. Similar to us, [25] tried to answer why multi-modal learning is better than single model learning. However, they did not consider contrastive learning and thus cannot explain the success of multi-modal contrastive multi-modal learning.

Data Quality Matters for Multi-modal Contrastive Learning. Aligned with our theoretical results, there is a lot of empirical evidence showing that improving the alignment quality with more descriptive captions improves multi-modal contrastive learning. [16] show that the training distribution mostly determines the generalizability of CLIP. Furthermore, [47, 40, 18, 17] find filtering poorly aligned image-caption samples used for training leads to further improvements. Besides, [45, 39, 15] demonstrate that improving the descriptiveness of the captions could further boost the performance of CLIP. Besides, [34] demonstrated that the caused by a combination of model initialization and contrastive learning optimization. However, their results do not take neural network architecture into consideration, and do not provide an analysis of test errors either.

3 Problem Setting

Notation. We use bold-faced letters for vectors and matrices otherwise representing scalar. We use ∥ · ∥ 2 to denote the Euclidean norm of a vector or the spectral norm of a matrix, while denoting ∥ · ∥ F as the Frobenius norm of a matrix. For a neural network, we denote σ ( · ) as the activation function and we adopt ReLU activation where σ ( x ) = max { 0 , x } in this work. To simplify, we denote [ n ] = { 1 , 2 , . . . , n } .

Data Model. In this work, we consider the following data model, which consists of signal and noise. In the first modality, example ( x , y ) ∼ D is generated as follows:

where x ∈ R 2 d is the input feature and y ∈ {-1 , 1 } is the corresponding label generated from Rademacher distribution. In particular, x (1) = y µ ∈ R d is the task-relevant signal vector, and x (2) = ξ ∼ N ( 0 , σ 2 ξ I ) ∈ R d is the task-irrelevant noise vector. Intuitively, if a network learns primarily from signal, it can effectively generalize to unseen data and vice versa. Similar data models have been adopted in recent theoretical works on supervised learning [2, 26, 8, 23, 31, 63, 22, 11] and self-supervised learning [56, 50, 30].

Similarly for the second modality, a sample ( ˜ x , y ) ∼ ˜ D is generated as

where the input feature ˜ x ∈ R 2 ˜ d and the label y is shared with the first modality. Besides, the signal is a given vector ˜ µ ∈ R ˜ d , and noise follows ˜ ξ ∼ N ( 0 , σ 2 ˜ ξ I ) ∈ R ˜ d . The linear data models for multi-modal learning have also been studied in previous work [44]. To simplify the analysis, we set d = ˜ d , σ ξ = σ ˜ ξ . However, we highlight that extensions to deal with unmatched dimension and noise level is possible.

3.1 Single-modal Contrastive Learning

We use a single-layer neural network h : R 2 d → R m with ReLU activation as our encoder, where m is the number of neurons, which represents the embedding dimension. More precisely,

here we let h r ( x ( i ) ) = σ ( ⟨ w r , x ( i ) ⟩ ) for r ∈ [ m ] , i ∈ [2] , and σ ( · ) is the ReLU activation function. We adopt a Gaussian to initialize the weights w (0) r ∼ N ( 0 , σ 2 0 I ) , where σ 0 severs as the strength.

Given a pair of positive data samples, the contrastive loss function is based on the similarity measure defined as the inner product between the representation of two samples x , x ′ ∈ R 2 d :

where the sg( · ) is the stop-gradient operation, which is inspired by recent empirical works [19, 10] and theoretical work studying contrastive learning [56]. Here we define positive sample as

In particular, we consider the form of augmentation where the signal stays invariant while the noise vector is corrupted with added independent noise. Similar setup has been considered in [57]. We consider the contrastive loss presented as follows:

̸

̸

where τ is the temperature parameter, n is the number of training samples, and M is the number of negative pairs. In this work, to efficiently optimize the loss to near zero, we require negative sample pairs do not share the same label, i.e., y j = y i in (6). Note that this setting is aligned with supervised contrastive learning [29, 27].

We use gradient descent to optimize the contrastive learning loss, which leads to the gradient update:

̸

̸

where we denote ¯ h ( t ) r ( x ) = σ ( ⟨ w ( t ) r , x ⟩ ) , η as the learning rate, and we define the loss derivatives as

̸

̸

Intuitively, when the similarity between positive pair is high, and the similarity between negative time is low, we can see ℓ ′ ( t ) i ≈ 1 and ℓ ′ ( t ) i,j ≈ 0 , for i ∈ [ n ] and j ∈ [ M ] . Therefore, the gradient descent in Eq. (7) is close to zero, indicating the near convergence result. Furthermore, from Eq. (7), we observe that the evolution direction of weight is composed of signal vector µ and noise vectors ξ i for i ∈ [ n ] . This observation plays a critical role in our following theoretical analysis.

3.2 Multi-modal Contrastive Learning

We use two neural networks h : R d → R m and g : R ˜ d → R m to encode two input modality x and ˜ x respectively. Both neural networks use ReLU activation function. More precisely,

we let h r ( x ( i ) ) = σ ( ⟨ w r , x ( i ) ⟩ ) and g r ( ˜ x ( i ) ) = σ ( ⟨ ˜ w r , ˜ x ( i ) ⟩ ) . Here σ ( · ) is the ReLU activation function, w r ∈ R d and ˜ w r ∈ R ˜ d for r ∈ [ m ] are the weights in two networks. Given the embedding, the similarity function of the two modalities is defined as

The two similarity functions defined above are modality-centered with stop-gradient operation applied. The objective function of contrastive multi-modal learning can be expressed as

̸

̸

Same to the single-modal learning whose objective function is governed by Eq. (6), the objective function for multi-modal contrastive learning adopt one positive pair and M negative pairs. Besides, we require the negative pairs do not share the same label. To optimize the objective function (9) for multi-modal learning, gradient descent is applied to train two encoders simultaneously. The gradient descent rule for the first modal network is governed by the following expression.

̸

̸

Here with a slight abuse of notation, we use ℓ ′ ( t ) i , ℓ ′ ( t ) i,j to represent the loss derivatives for both modalities. Compared to signal-modal learning, the main difference for the multi-modal learning is that the corresponding embedding is from another modality. The gradient update for the second modality can be derived similarly, which we omit here for clarity.

3.3 Downstream Task Evaluation

To evaluate the out-of-distribution generalization of single-modal and multi-modal contrastive learning for downstream task, we consider a test distribution D test , where a sample x test = [ y · ν ⊤ , ζ ⊤ ] ⊤ ∼ D test is generated as follows. The test signal ν satisfies ⟨ ν , µ ⟩ = O ( ∥ µ ∥ 2 2 d -1 / 2 ) and the test noise follows ζ ∼ N ( 0 , σ 2 ξ I ) and y follows Rademacher distribution. After the training is complete, we introduce a linear head on top of the learned embedding h ( x test ) for adapting to test distribution, i.e., f ( x test ) = ⟨ w , h ( x test ) ⟩ . Specifically, we consider the task of classification and define the population 0-1 test error as L D test = P x test ∼D test [ yf ( x test ) < 0 ] .

4 Main Results

In this section, we introduce our key theoretical findings that elucidate the optimization and generalization result for both single-modal and multi-modal contrastive learning through the feature learning analysis. We use a trajectory-based analysis for the iterations induced by gradient descent, following a post-training analysis for the performance on the downstream test set. Below we provide the main assumption and main theorems.

(5) σ ϵ ≤ min { ˜ Θ( ∥ µ ∥ 2 ) , σ ξ / ˜ Ω(1) } . (6) n · SNR 2 = Θ(1) . (7) C µ ∥ µ ∥ 2 = ∥ ˜ µ ∥ 2 , where C µ ≥ 2 . 66 is a constant.

(1) We adopt a high dimensional setting to ensure enough over-parameterization. (2,3) The learning rate and the strength of initialization are chosen to make sure the that gradient descent can effectively minimize the contrastive loss. (4) The choice of hidden size m and number of training sample n is to provide adequate concentration. (5) The strength of augmentation is set to keep the similarity between two positive samples. (6) The relation between number of sample and SNR is to distinguish the feature learning process between single-modal and multi-modal contrastive learning. (7) To differentiate single-modal and multi-modal contrastive learning, we introduce a constant C µ , which enables the cooperation between the two modalities in multi-modal contrastive learning.

Theorem 4.2 (Single-Modal Contrastive Learning) . Under the single-modal learning setup, suppose Assumption 4.1 holds. Then after T ∗ = ˜ Θ( η -1 mnσ -2 ξ d -1 + η -1 mnσ -2 ξ d -1 ϵ -1 ) , the with probability at least 1 -1 /d , it holds that (1) Training error L ( T ∗ ) ≤ ϵ and (2) Test error at down-stream task L D test ( T ∗ ) = Θ(1) .

Theorem 4.2 states that despite the small training error achieved by single-modal contrastive learning, the test error is large in the downstream task.

Theorem 4.3 (Multi-Modal Contrastive Learning) . Under the single-modal learning setup, suppose Assumption 4.1 holds. Then after T ∗ = ˜ Θ( η -1 mnσ -2 ξ d -1 + η -1 mnσ -2 ξ d -1 ϵ -1 ) , the with probability at least 1 -1 /d , it holds that (1) Training error L ( T ∗ ) ≤ ϵ and (2) Test error at down-stream task L D test ( T ∗ ) = o (1) .

Theorem 4.3 demonstrates that trained multi-modal contrastive learning can achieve both small training error and downstream test error. Compared to Theorem 4.2, Theorem 4.3 shows that the generalization of multi-modal contrastive learning in downstream tasks is better than single-modal contrastive learning. The reason behind this difference is that the two modalities can cooperate with each other; the higher quality in one modality can boost the feature learning in the target modality, helping to generalize to the downstream task. On the contrary, augmentation often maintains the same SNR as the original data, so single-modal learning hardly benefits from the augmentation and can only memorize the noise from the data, which is not applicable to downstream tasks.

5 Proof Roadmap

5.1 Proof Sketch for Single Modal Contrastive Learning

The proof is constructed by a optimization analysis followed by a generlization analysis in the downstream task. Through the application of the gradient descent rule outlined in Eq. (7), we observe that the gradient descent iterate w ( t ) r is a linear combination of its random initialization w (0) r , the signal vector µ and the noise vectors in the training data ξ i for i ∈ [ n ] . Consequently, for r ∈ [ m ] , the decomposition of weight vector iteration can be expressed:

where γ ( t ) r and ρ ( t ) r,i serve as coefficients and represent signal learning and noise memorization respectively. Based on the the gradient descent update (7), the iteration of γ ( t ) r and ρ ( t ) r,i are given:

Lemma 5.1 (Single-modal Contrastive Learning) . The coefficients γ ( t ) r ρ ( t ) r,i in decomposition (11) satisfy the following equations:

̸

̸

where the initialization γ (0) r , ρ (0) r,i = 0 .

Lemma 5.1 tells how the coefficients evolve under gradient descent update. In the following, we introduce a two-stage dynamics to characterize the whole training process based on Eq 12 and Eq 13.

First Stage: Exponential growth. During the first stage, we show before γ ( t ) r or ρ ( t ) r,i grow to Θ(1) , the embedding (3) is close to zero, suggesting the similarity is bounded by 1 ≤ Sim h ( x , x ′ ) ≤ C ℓ for some constant C ℓ > 1 . The loss derivatives defined in (8) can thus be bounded within some constant range.

̸

Signal learning. According to the update for signal learning in (12), we see the propagation can be simplified based on the hard-negative sampling strategy, i.e., the negative pairs do not share the same labels. This suggests the negative term is always zero as ∑ n i =1 ∑ M j : y j = y i σ ( ⟨ w ( t ) r , y j µ ⟩ ) σ ′ ( ⟨ w ( t ) r , y i µ ⟩ ) = 0 . The resulting update of γ ( t ) r reduces to γ ( t +1) r = γ ( t ) r + η nmτ ∑ n i =1 (1 -ℓ ′ ( t ) i ) σ ( ⟨ w ( t ) r , y i µ ⟩ ) σ ′ ( ⟨ w ( t ) r , y i µ ⟩ ) y i ∥ µ ∥ 2 2 . Examining the propagation of γ ( t ) r , we can divide the dynamics into two groups depending on the sign of weight initialization ⟨ w (0) r , µ ⟩ . Let U ( t ) + ≜ { r : ⟨ w ( t ) r , µ ⟩ > 0 } and U ( t ) -≜ { r : ⟨ w ( t ) r , µ ⟩ < 0 } . Then for r ∈ U (0) + , we can show γ ( t ) r ≥ 0 increases exponentially and thus the sign of inner produce stays invariant with U ( t ) + = U (0) + for all t ≥ 0 . On the other hand, for r ∈ U (0) -, we can show γ ( t ) r ≤ 0 and decreases exponentially with U ( t ) -= U (0) -for all t ≥ 0 .

Noise memorization. Compared to signal learning, the behaviour of noise memorization requires more detailed analysis. This is mainly because the negative pairs can not be eliminated simply based on label difference, as the noise patch ξ i is generated independent of label y i . In addition, the added noise ϵ i by augmentation can also contribute to the noise dynamics. We first show when the noise level σ ϵ is much smaller compared to σ ξ , the dynamics of noise memorization is largely remains unaffected. By the sign of ⟨ w ( t ) r , ξ i ⟩ , we partition the samples into two sets, i.e., I ( t ) r, + = { i : ⟨ w ( t ) r , ξ i ⟩ > 0 } , and I ( t ) r, -= { i : ⟨ w ( t ) r , ξ i ⟩ < 0 } . We can verify for i ∈ I (0) r, -, the value of ρ ( t ) r,i stays at zero based on the update (13) with an induction argument. For samples i ∈ I (0) r, + with positive initialization, we analyze the noise memorization based on the joint dynamics of samples with the same label. In particular, we define total noise memorization of positive and negative samples respectively as B ( t ) r, + ≜ ∑ i : y i =1 ( ρ ( t ) r,i + ⟨ w (0) r , ξ i ⟩ ) 1 i ∈I ( t ) r, + and B ( t ) r, -≜ ∑ i : y i = -1 ( ρ ( t ) r,i + ⟨ w (0) r , ξ i ⟩ ) 1 ( t ) i ∈I r, + . The update of ρ ( t ) r,i in (13) then implies the dynamics of B ( t ) r, + and B ( t ) r, -as follows

where the coefficient of 1 / 2 appears as a result of the randomness of the sign of initialization. This result suggests, individual ρ ( t ) r,i : y i =1 cannot grow too slow compared to the ρ ( t ) r,i : y i = -1 . Following a similar induction argument, we are able to show ρ ( t ) r,i : y i =1 has an exponential growth lower bound. On the other hand, for samples with y i = -1 but with different neuron, we can use the same strategy to show an exponential growth lower bound for some neurons that satisfy the initialization conditions.

Lemma 5.2. Under the Assumption 4.1, let T 1 = log ( 20 / ( σ 0 σ ξ √ d ) ) / log ( 1+0 . 96 ησ 2 ξ d nmτ ) , we have γ ( t ) r = ˜ O (1 / √ n ) for all r ∈ [ m ] and 0 ≤ t ≤ T 1 and max r ρ ( T 1 ) r,i = Ω(1) for all i ∈ [ n ] .

Second stage: convergence and scale difference. At the end of first stage, the noise grows to a constant order while signal learning remains negligible. As a result, the loss derivatives are no longer bounded within some constant range. In the second stage, we aim to show the loss is able to converge to an arbitrarily small value ϵ . Despite the unsupervised learning setup, we are still able to show loss convergence thanks to the hard negative samples. Let F 0 ( W , x i ) = Sim( x i , ̂ x i ) be the similarity to the argumentation and F j ( W , x i ) = Sim( x i , x j ) for j = 1 , ..., M be the similarity between the negative pairs. Then we can show there exists some W ∗ such that ⟨∇ F 0 ( W ( t ) , x i ) , W ∗ ⟩ ≥ 2 log(2 M/ϵ ) while ⟨∇ F j ( W ( t ) , x i ) , W ∗ ⟩ ≤ log(2 M/ϵ ) for all j = 1 , ..., M . Then we can bound ⟨∇ L S ( W ( t ) ) , W ( t ) -W ∗ ⟩ ≥ 1 n ∑ n i =1 L i ( W ( t ) ) -ϵ/ 2 . This as a result allows to show a monotonic decrease in the loss function as L ( W ( t ) ) ≤ 1 η ( ∥ W ( t ) -W ∗ ∥ 2 F -∥ W ( t +1) -W ∗ ∥ 2 F ) + ϵ which guarantees convergence by telescoping over the inequality. Upon the convergence, we can also show

the scale difference obtained at the end of the first stage is maintained, i.e., γ ( t ) r = ˜ O (1 / √ n ) while max r ρ ( T 1 ) r,i = Ω(1) for all i ∈ [ n ] . This suggests the non-linearly separability for the resulting embeddings and thus the downstream test error is non-vanishing. The formal convergence result is established in Lemma C.18. Combined with the generalization error demonstrated in Appendix C.3, this completes the proof of Theorem 4.2.

5.2 Proof Sketch for Multi-Modal Learning

Similar to single-modal contrastive learning, we decompose the decomposition of weight vector iteration for the network in the second modality and subsequently provide a two-stage analysis.

where ˜ γ ( t ) r and ˜ ρ ( t ) r,i serve as coefficients for the weight decomposition in the text modal.

Lemma 5.3 (Multi-Modal) . The coefficients γ ( t ) r , ρ ( t ) r,i , ˜ γ ( t ) r , ˜ ρ ( t ) r,i in decomposition (11) satisfy

where the initialization satisfy γ (0) r , ρ (0) r,i , ˜ γ (0) r , ˜ ρ (0) r,i = 0

First Stage: Exponential growth The first stage of multi-modal learning shares similar characteristics as single-modal learning.

Signal learning. For signal learning, we analyze the trajectories for both γ ( t ) r and ˜ γ ( t ) r . To this end, we partition the neurons depending on their initialization status. Apart from U ( t ) + and U ( t ) -defined in the single-modal learning, we additionally define ˜ U ( t ) + ≜ { r : ⟨ ˜ w ( t ) r , ˜ µ ⟩ > 0 } , ˜ U ( t ) -≜ { r : ⟨ ˜ w ( t ) r , ˜ µ ⟩ < 0 } for the other modality. Then for r ∈ U (0) + ∩ ˜ U (0) + , we can show γ ( t ) r ≥ 0 , ϕ ( t ) r ≥ 0 and are increasing. For neurons r ∈ U ( t ) -∩ ˜ U ( t ) -, we can show γ ( t ) r ≤ 0 , ˜ γ ( t ) r ≤ 0 and are decreasing. Furthermore, the sign of inner product stays invariant, i.e., U ( t ) + ∩ ˜ U ( t ) + = U (0) + ∩ ˜ U (0) + and U ( t ) -∩ ˜ U ( t ) -= U (0) -∩ ˜ U (0) -. For neurons with only one of the modalities activated initially, i.e., r ∈ U (0) + ∩ ˜ U (0) -or r ∈ U (0) -∩ ˜ U (0) + , we can show there exists some time t ′ ≥ 0 such that the neurons are either positively or negatively activated with r ∈ U ( t ′ ) + ∩ ˜ U ( t ′ ) + and r ∈ U ( t ′ ) -∩ ˜ U ( t ′ ) -.

This shows synchronization of the signal learning patterns of two modalities. The pre-synchronization phase reduces the speed of learning and thus we can only focus on neurons with the same sign for the initialization in order to decide the lower and upper bound.

Noise memorization. For noise memorization, we partition the samples into two sets for both modalities according to the initialization, namely, I ( t ) r, + = { i : ⟨ w ( t ) r , ξ i ⟩ > 0 } , I ( t ) r, -= { i : ⟨ w ( t ) r , ξ i ⟩ < 0 } , and similarly for the second modality ˜ I ( t ) r, + , ˜ I ( t ) r, -. Because the noise memorization are correlated in the two modalities, we separately analyze the samples as follows.

0

50

100

150

200

Epoch

0

2

4

6

8

Training Loss

Single-Modal

Multi-Modal

0

50

100

150

200

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

Test Accuracy

0

50

100

150

200

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Signal Learning

0

50

100

150

200

Epoch

0

1

2

3

For i ∈ I (0) r, + ∩ ˜ I (0) r, + , without loss of generality, we consider upper bounding noise memorization for the first modality and y i = 1 . To this end, we first define the individual and joint noise memorization for the first modality as Ψ ( t ) r,i ≜ ρ ( t ) r,i + ⟨ w (0) r , ξ i ⟩ , and B ( t ) r, + , + ≜ ∑ i : y i =1 ( ρ ( t ) r,i + ⟨ w (0) r , ξ i ⟩ ) 1 i ∈I ( t ) r, + ∩ ˜ I ( t ) r, + , B ( t ) r, + , -≜ ∑ i : y i =1 ( ρ ( t ) r,i + ⟨ w (0) r , ξ i ⟩ ) 1 i ∈I ( t ) r, + ∩ ˜ I ( t ) r, -. Similar definitions exist for the other modality. The joint dynamics of noise memorization can be first upper bounded by the other modality as ˜ Ψ ( t ) r,i ≥ 101 C ℓ M +1 ( B ( t ) r, + , + + B ( t ) r, + , -) . Then we can upper bound the individual noise memorization by

We show the combined dynamics of γ ( t ) r and ρ ( t ) r exhibits exponential growth while the magnitude of their difference shrinks exponentially. The results are summarized as follows

Lemma 5.4. Under the Assumption 4.1, let T 1 = log ( 20 / ( σ 0 ∥ µ ∥ 2 ) ) / log ( 1 + 0 . 48 C µ η ∥ µ ∥ 2 2 mτ ) , we have ρ ( T 1 ) r,i = ˜ O (1 / √ n ) for all r ∈ [ m ] , i ∈ [ n ] , and 0 ≤ t ≤ T 1 and max r γ ( T 1 ) r = Ω(1) .

Second Stage: Convergence and scale difference. The second stage presents similar patterns compared to single-modal learning. Thanks to the correlation between the two modality during gradient descent training, the two neural network converge at the same time, minimizing the training loss. Besides, The scale difference at the end of the first stage is carried over throughout the second stage until convergence. Therefore, it allows to show a monotonic decrease in the loss function as L ( W ( t ) , ˜ W ( t ) ) ≤ ∥ W ( t ) -W ∗ ∥ 2 F + ∥ ˜ W ( t ) -˜ W ∗ ∥ 2 F - ∥ W ( t +1) -W ∗ ∥ 2 F - ∥ ˜ W ( t +1) -˜ W ∗ ∥ 2 F +2 ϵ , which guarantees convergence by telescoping over the inequality. At the same time, until convergence, we can show the scale difference obtained at the end of the first stage is maintained, namely max r,i ρ ( t ) r,i = ˜ O (1 / √ n ) and max r γ ( t ) r = Ω(1) . This suggests the signal learning dominates the noise memorization and thus the resulting embeddings are linearly separable, which guarantees a small test error for downstream tasks. The formal convergence result is established in Lemma D.15. Combined with the generalization error demonstrated in Appendix D.3, this completes the proof of Theorem 4.3.

6 Experiments

Synthetic experiments We conduct synthetic experiments to verify the theoretical results obtained in the previous sections. We generate samples following the theoretical setups, where we set the data dimension d = 2000 , number of training samples n = 100 , number of test samples n test = 200 , and the hidden size of all encoders as m = 50 . We adopt gradient descent with a learning rate of 0 . 01 as the optimizer to train the model by 200 epochs. In the single-modal setting, the µ is set to be [5 , 0 , ..., 0] T and the ξ ∼ N ( 0 , I ) for the in-distribution data, and the augmentation vector ϵ ∼ N ( 0 , 0 . 01 ∗ I ) . For the multi-modal setting, ˜ µ = [0 , 15 , 0 , ..., 0] T . In addition, for the OOD test data x test = [ ν ⊤ , ζ ⊤ ] ⊤ , we set ν = [2 , 0 , ..., 0] and ζ ∼ N ( 0 , I ) . We perform logistic regression based on the learned features h ( x test ) and apply the learned classifier head to evaluate OOD generalization error in terms of prediction accuracy.

Results. In Figure 1, we see the training loss of both single-modal and multi-modal learning converges rapidly. At the same time, OOD test accuracy of multi-modal learning converges to nearly 1.0 while

that of single-modal learning stagnates around 0.5. This is primarily because under the setup where the other modality ˜ µ has a higher SNR, signal learning of µ is lifted. This can be verified from the third plot of Figure 1, where the signal learning of multi-modal framework is significantly higher than single-modal. Further, it can be observed that single-modal contrastive learning exhibits more severe noise memorization, which suppresses signal learning. In contrast, multi-modal contrastive learning exhibits less severe noise memorization which would further encourage signal learning. These phenomena again support and align with our theoretical results.

Real-world experiments We now extend the comparison of single-modal and multi-modal learning to realistic image data, ColoredMNIST [3, 54], which is a typical benchmark studying the generalization capability under distribution shifts. The ColoredMNIST dataset is a variation of the standard MNIST dataset, where each digit is assigned a specific color based on its label. The two modalities are image, and text that describes the images. The task is a 10-class classification that recognizes the number of the colored MNIST images. Specifically, we have 10 colors to color 10 digits, and introduce spurious correlations via label noises following the literature:

For the training set, 10% of labels will be clipped to a random class. For images with class '0' (or '1'), they will be colored as red (or green) with a probability of 77.5%, and as another random color with a probability of 22.5%. The coloring scheme introduces a spurious correlation.

For the test set, 10% of labels will be clipped to a random class. For images with class '0' (or '1'), they will be colored as green (or red) with a probability of 77.5%, and as another random color with a probability of 22.5%. The coloring scheme can be considered as reversing the training spurious correlations. Therefore, the evaluation on test set can reflect to what extent the model learns to use the spurious features, i.e., colors, to classify images.

We implement the multi-modal learning following the practice in [54], where we consider an ideal language encoder that successfully encodes the caption of the images into onehot labels of colors and digits. For single-modal learning, we follow the implementation of the SimCLR [9] to construct a set of augmentations to learn the representations.

Results. Under the distribution shift, we verify that multi-modal learning archives an out-of-distribution test accuracy of 82.13%, which outperforms that of singlemodal learning 12.68%. As a result, we can claim that the effective SNR of invariant features (the shape of the digit) will be degraded under the impact of the injected

color. Therefore, the performance of single-modal may be suboptimal as it cannot effectively utilize the information of the digit's shape. On the other hand, multi-modal demonstrates a better capacity for handling this scenario.

7 Conclusions

In this work, we have established a comprehensive comparison of the optimization differences during the pre-training stage and the generalization gap between single-modal and multi-modal contrastive learning for downstream tasks. With the cooperation between modalities, multi-modal contrastive learning can achieve better feature learning and generalization on downstream tasks compared to single-modal learning. On the other hand, data augmentation alone can hardly improve data quality and thus cannot boost the performance of single-modal contrastive learning. Together, these results quantitatively demonstrate the superiority of multi-modal learning over single-modal learning and emphasize the importance of data quality in multi-modal contrastive learning.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their insightful comments to improve the paper. Wei Huang is supported by JSPS KAKENHI Grant Number 24K20848. Yuan Cao is supported by NSFC 12301657 and HK RGC-ECS 27308624. Taiji Suzuki is partially supported by JSPS KAKENHI (24K02905) and JST CREST (JPMJCR2015).

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 , 2019.

Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neural Information Processing Systems , 35:26671-26685, 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.

Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssl interplay: Augmentations, inductive bias, and generalization. In International Conference on Machine Learning , pages 3252-3298. PMLR, 2023.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:2523725250, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Understanding and improving feature learning for out-of-distribution generalization. In Advances in Neural Information Processing Systems , 2023.

Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable representation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.00927 , 2023.

Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. arXiv preprint arXiv:2303.09166 , 2023.

Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving selfsupervised learning by characterizing idealized representations. Advances in Neural Information Processing Systems , 35:11279-11296, 2022.

Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In Advances in Neural Information Processing Systems , 2023.

Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In International Conference on Machine Learning , 2022.

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models , 2023.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Jeff Z HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. arXiv preprint arXiv:2211.14699 , 2022.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Wei Huang, Yuan Cao, Haonan Wang, Xin Cao, and Taiji Suzuki. Graph neural networks provably benefit from structural information: A feature learning perspective. arXiv preprint arXiv:2306.13926 , 2023.

Wei Huang, Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding convergence and generalization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations , 2023.

Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.00743 , 2021.

Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems , 34:10944-10956, 2021.

Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning , pages 9965-10040. PMLR, 2022.

Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research , 24(330):1-78, 2023.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

Yiwen Kou, Zixiang Chen, Yuan Cao, and Quanquan Gu. How does semi-supervised learing with pseudo-labelers work? a case study. In International Conference on Learning Representations , 2023.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting for two-layer relu networks. arXiv preprint arXiv:2303.04145 , 2023.

Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems , 34:309-323, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612-17625, 2022.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS , 2023.

Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does clip's generalization performance mainly stem from high train-test similarity? arXiv preprint arXiv:2310.09562 , 2023.

Yifei Ming and Yixuan Li. Understanding retrieval-augmented task adaptation for visionlanguage models. arXiv preprint arXiv:2405.01468 , 2024.

Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics , pages 4348-4380. PMLR, 2023.

Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of CLIP. In Advances in Neural Information Processing Systems , 2022.

OpenAI. Gpt-4 technical report, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint arXiv:2304.03717 , 2023.

Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth a thousand images? a study on representation learning. In International Conference on Learning Representations , 2023.

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning , pages 19250-19286. PMLR, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning , pages 31852-31876. PMLR, 2023.

Yuandong Tian. Understanding deep contrastive learning via coordinate-wise optimization. Advances in Neural Information Processing Systems , 35:19511-19522, 2022.

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning , pages 10268-10278. PMLR, 2021.

Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578 , 2020.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory , pages 1179-1206. PMLR, 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. Do clip models always generalize better than imagenet models? Advances in Neural Information Processing Systems , 2024.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning , pages 11112-11122. PMLR, 2021.

Yihao Xue, Siddharth Joshi, Eric Gan, Pin-Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. In International Conference on Machine Learning , pages 38938-38970. PMLR, 2023.

Yihao Xue, Siddharth Joshi, Dang Nguyen, and Baharan Mirzasoleiman. Understanding the robustness of multi-modal contrastive learning to distribution shift. arXiv preprint arXiv:2310.04971 , 2023.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv e-prints , pages arXiv-2210, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. In International Conference on Learning Representations , 2023.

Contents

Appendix

A Limitations and broader impact

While our theoretical analysis is novel in terms of optimization and generalization, the data model can be further modified to be more practical. Our theoretical analysis may be further used for empirical and theoretical studies of contrastive learning, especially multi-modal contrastive learning. However, we do not foresee a direct social impact from our theory.

B Preliminary Lemmas

Before the proof, we introduce lemmas that are useful in proving our main theorem.

Proof. The probability density function for x is given by

By the definition of erf function and variable substitution yields

Therefore, we first conclude P ( | x | ≤ c ) = 2erf ( c √ 2 σ ) .

Next, by the inequality erf( x ) ≤ √ 1 -exp( -4 x 2 /π ) , we finally obtain

Lemma B.1 introduces an anti-concentration result. In later sections, this lemma will be used to show that with a relatively large initialization for the weight vector, some initial properties hold.

Proof of Lemma B.2. By Lemma B.1, because ⟨ w (0) r , µ ⟩ ∼ N (0 , σ 2 0 ∥ µ ∥ 2 2 ) , we can show

Then we know that

Let c = 100 · SNR √ 8 log(6 n/δ ) d n = 100 n ∥ µ ∥ 2 σ -1 ξ d -1 √ 8 log(6 n/δ ) and plug it into the RHS of the above inequality, which becomes:

Then we can verify that when d satisfies that d ≥ 400 n σ 0 σ ξ √ log(6 n/δ ) -π log(1 -δ 2 / (4 m 2 )) , it holds that RHS ≤ δ/m . This suggests for a single neuron r ∈ [ m ] , we have P ( |⟨ w (0) r , µ ⟩| ≤ c ) ≤ δ/m . Applying union bound, we can show the desired result.

Similarly, with the same procedure, we can prove the result for the other modality.

Lemma B.3 ([31]) . Let S 1 = { i ∈ [ n ] : y i = 1 } and S -1 = { i ∈ [ n ] : y i = -1 } . Then with probability at least 1 -δ ,

Lemma B.3 states that when the label is randomly sampled, the number of positive samples and negative samples is close to n 2 , adequately.

Lemma B.4 ([8]) . Suppose that d ≥ Ω(log( mn/δ )) , m = Ω(log(1 /δ )) . Then with probability at least 1 -δ , it satisfies that for all r ∈ [ m ] , i ∈ [ n ] ,

and for all i ∈ [ n ]

Lemma B.5 ([8]) . Suppose that δ > 0 and d = Ω(log(6 n/δ ))) . Then with probability 1 -δ ,

for all i, i ′ ∈ [ n ] .

C Single-modal Contrastive Learning: Proof of Theorem 4.2

In this section, we provide the proof for Theorem 4.2, which states main results of single modal learning. The training dynamics are based on the coefficient iterations presented in Lemma 5.1. Below, we provide a proof for this lemma:

Proof of Lemma 5.1. Recall that the weight decomposition is expressed as

We plug it into the gradient descent update as described by Equation 10 yields

̸

̸

By comparing the coefficients in front of µ and ξ i on both sides of the equation, we can obtain

̸

which completes the proof.

According to the behavior of the defined loss derivative (8), we split the entire training dynamics into two phases. In the first stage, the loss derivative remains close to its initial value as the similarity is small from initialization. Later, as the similarity grows to a constant value, the loss derivative is no longer close to the initial value, and the dynamics transition to the second stage. In this stage, the similarity increases logarithmically, and the empirical loss converges.

C.1 First Stage

In the first stage, the derivative of the loss is close to its initial value because the similarity is small. Below, we provide a useful lemma for establishing such a result.

Lemma C.1. Suppose that γ ( t ) r = O (1) and ρ ( t ) r,i = O (1) for all r ∈ [ m ] and i ∈ [ n ] . Under Assumption 4.1, then for any δ > 0 , with probability at least 1 -δ

for all r ∈ [ m ] , i ∈ [ n ] .

Proof of Lemma C.1. From the signal-noise decomposition of w ( t ) r , we derive

̸

Equation (a) results from the weight decomposition (see Equation 11). In the first stage, we used the upper bounds for | γ ( t ) r | and | ρ ( t ) r | , and applied Lemma B.5 in inequality (b). Finally, inequality (c) follows from the condition n SNR 2 = Θ(1) .

Further,

where we have used Lemma B.5.

Now, we proceed to the lemma concerning the derivative of the loss as follows:

Lemma C.2. If max { γ ( t ) r , ρ ( t ) r,i } = O (1) and under Assumption 4.1, there exists a constant C ℓ > 1 such that

for all i ∈ [ n ] .

Proof of Lemma C.2. From the update of w ( t ) r , we have

where (a) is by Lemma C.1, and (b) is by Lemma B.4. Finally, in inequality (c) we have used the condition that σ 0 ≤ 1 2 √ log(8 mn/δ ) σ ξ √ d and d > n 2 log(6 n 2 /δ ) according to Assumption 4.1, and max { γ ( t ) r , ρ ( t ) } = O (1) . At the same time, r,i

where (a) is by Lemma C.1, (b) is Lemma B.4, and (c) is by σ 0 ≤ 1 2 √ log(8 m/δ ) ∥ µ ∥ 2 and d > SNR 2 n 2 log(6 n 2 /δ ) according to Assumption 4.1, and max { γ ( t ) r , ρ ( t ) r,i } = O (1) . Besides,

where (a) follows from Lemma B.5, (b) from Lemma B.4, and (c) from the conditions σ 0 ≤ 1 2 √ log(8 mn/δ ) σ ϵ √ d , σ ϵ ≤ ∥ µ ∥ 2 √ 2 log(6 n/δ ) , d > n 2 log(6 n 2 /δ ) , and σ ϵ < σ ξ .

Next, we calculate the upper bound of the similarity measure. First, we examine the negative pair. For any i, j ∈ [ n ] , we have

Similarly, for positive pair,

According to the above result, we can say that 1 ≤ e Sim h ( x , x ′ ) ≤ C ℓ , where C ℓ is a positive constant. Then we can provide the upper bound for ℓ ′ i and ℓ ′ i,j

̸

This completes the proof.

C.1.1 Dynamics of Signal Learning: Upper Bound

In the first stage, the growth rate of signal learning is exponential. We establish an upper bound for the growth of signal learning.

Then we consider the growth of signal learning coefficient γ ( t ) r . Depending on the initialization, we define U ( t ) + = { r ∈ [ m ] : ⟨ w ( t ) r , µ ⟩ > 0 } and U ( t ) -= { r ∈ [ m ] : ⟨ w ( t ) r , µ ⟩ < 0 } .

Lemma C.3. Under the condition d ≥ 400 n σ 0 σ ξ √ log(6 n/δ ) -π log(1 -δ 2 / (4 m 2 )) and Assumption 4.1, for all t ≥ 0 , we have U ( t ) + = U (0) + , U ( t ) -= U (0) -and γ ( t ) r > 0 is an increasing sequence for all r ∈ U (0) + and γ ( t ) r ≤ 0 and is a decreasing sequence for all r ∈ U (0) -.

Proof of Lemma C.3. We prove the claims by induction. To better understand the dynamics, we first derive the propagation for signal learning from the first step. For r ∈ U (0) + , i.e., ⟨ w (0) r , µ ⟩ > 0 , we can see

̸

̸

̸

̸

̸

where last equality is by ⟨ w (0) r , µ ⟩ > 0 , γ (0) r = 0 , and the fact that negative samples satisfy y j = y i . Thus, we verify that the sign of γ (1) r follow its initialization and γ (1) r > 0 .

Next, we show the propagation of inner product at t = 1 , for r ∈ U (0) + , we have

where inequality (a) is by Lemma C.1, the second inequality (b) is by Lemma B.2, and the last by γ (1) r > 0 . Hence we verify U (1) + = U (0) + .

Now suppose at iteration t , the claims are satisfied, namely ⟨ w ( t ) r , µ ⟩ > 0 and γ ( t ) r ≥ γ ( t -1) r ≥ 0 for r ∈ U (0) + . Then following similar argument, for r ∈ U (0) +

where we use the induction condition that ⟨ w ( t ) r , µ ⟩ > 0 . Further by Lemma C.1 and B.2

where we use γ ( t +1) r ≥ 0 . This completes the induction for r ∈ U (0) + .

Similarly, for those neuron r that satisfies ⟨ w 0 r , µ ⟩ < 0 , i.e., r ∈ U (0) -, we have

̸

̸

where the last equality is by ⟨ w (0) r , µ ⟩ < 0 , y i = -1 , the property of ReLU activation, and the fact that y j = y i in the negative pair term. Hence we see γ (1) r ≤ γ (0) r = 0 . Similarly, for the inner product at t = 1 ,

where the first inequity is by Lemma C.1, the second inequality follows from Lemma B.2, and the last inequality follows from γ (1) r ≤ 0 .

Now suppose at iteration t , the claims are satisfied, namely ⟨ w ( t ) r , µ ⟩ < 0 and γ ( t ) r ≤ γ ( t -1) r ≤ 0 for r ∈ U (0) + . Then following similar argument, for r ∈ U (0) -

where we use the induction condition that ⟨ w ( t ) r , µ ⟩ < 0 . Further

where inequality (a) follows from Lemma C.1; we use γ ( t +1) r ≤ 0 and Lemma B.2 in deriving inequality (b). This completes the induction for r ∈ U (0) -.

With Lemma C.3 at hand, we are ready to demonstrate the upper bound of the growth rate for signal learning.

Lemma C.4. With the same condition as in Lemma C.2 and Lemma C.3 and n ≥ 2500 log(4 /δ ) , define A ( t ) r = γ ( t ) r + ⟨ w (0) r , µ ⟩ for r ∈ U (0) + ; and A ( t ) r = -γ ( t ) r - ⟨ w (0) r , µ ⟩ for r ∈ U (0) -. With probability at least 1 -δ , we have

Proof. Lemma C.3 suggests that for r ∈ [ m ] , we can upper bound for | γ ( t ) r | . Without loss of generality, we consider r ∈ U (0) + . By Lemma C.1, we can see

where we use Lemma B.2 and γ ( t ) r ≥ 0 .

Then, the update equation for γ ( t ) r follows

Then we find that

where the first inequality (a) is by (19) and Lemma C.2. The second inequality (b) is by Lemma B.3 and 1 -1 C ℓ ( M +1) < 1 . The last inequality (c) is by the condition n ≥ 2500 log(4 /δ ) .

C.1.2 Dynamics of Noise Memorization: Lower Bound

To establish the lower bound of noise memorization in the first stage, we require that the added noise level σ ϵ ≤ σ ξ /C ′ for sufficiently large, with C ′ ≥ ˜ Ω(1) . We prove the following result that upper bound the scale of ⟨ w ( t ) r , ϵ i ⟩ .

Lemma C.5. Under the same condition as Lemma C.1 and d = ˜ Ω(max { σ -2 0 ∥ µ ∥ -2 2 , nσ -1 0 σ -1 ξ } ) , there exists a sufficiently large constant C ξ > 0 such that

for all i ∈ [ n ] .

Proof of Lemma C.5. According to the decomposition of w ( t ) r , we can show

where the second inequality (a) follows from Lemma B.5 and the last inequality is by the following anti-concentration result.

Because ⟨ w (0) r , ξ i ⟩ is a Gaussian random variable with mean zero and variance σ 0 σ ξ √ d , by Lemma B.1, we compute

When d ≥ 2 c 2 -log(1 -δ 2 / (4 n 2 )) πσ 2 0 σ 2 ξ , we have P ( |⟨ w (0) r , ξ i ⟩| ≤ c ) ≤ δ/n and by union bound, we have with probability at least 1 -δ , it holds |⟨ w (0) r , ξ i ⟩| > c . Here we let c = C ( 2 √ log(8 mn ) /δσ 0 σ ϵ √ d + ∥ µ ∥ -1 2 σ ϵ √ 2 log(6 n/δ ) + 4 σ ϵ σ -1 ξ √ log(6 n 2 /δ ) d n ) ≥ C |⟨ w ( t ) r , ϵ i ⟩| . Then we can show the desired result with the condition d = ˜ Ω(max { σ -2 0 ∥ µ ∥ -2 2 , nσ -1 0 σ -1 ξ } ) .

Define that I ( t ) r, + = { i : ⟨ w ( t ) r , ξ i ⟩ > 0 } and I ( t ) r, -= { i : ⟨ w ( t ) r , ξ i ⟩ < 0 } . To show the result regarding I ( t ) r, + and I ( t ) r, -, we prepare the following anti-concentration result:

Lemma C.6. Under the condition d ≥ √ 300 log(6 n 2 /δ ) -log(1 -δ 2 / 4 n 2 ) πσ 2 0 σ 2 ξ , then with probability at least 1 -δ , it satisfies

Proof of Lemma C.6. Here we want to show |⟨ w (0) r , ξ i ⟩| ≥ 150 √ log(6 n 2 /δ ) /dn ≜ c with high probability. To see this, because ⟨ w (0) r , ξ i ⟩ is a Gaussian random variable with mean zero and

variance σ 2 0 σ 2 ξ d , by Lemma B.1, we compute

Thus when d ≥ √ 300 log(6 n 2 /δ ) -log(1 -δ 2 / 4 n 2 ) πσ 2 0 σ 2 ξ , we have P ( |⟨ w (0) r , ξ i ⟩| ≤ c ) ≤ δ/n and by union bound, we have with probability at least 1 -δ , it holds |⟨ w (0) r , ξ i ⟩| > 150 √ log(6 n 2 /δ ) /d .

We then show that neurons with negative inner products with the noise at initialization would stay negative and the corresponding ρ stays zero.

Lemma C.7. Under the same condition as Lemma C.1 and Lemma C.6, for all t > 0 , we have I ( t ) r, -= I (0) r, -and ρ ( t ) r,i = 0 for all i ∈ I (0) r, -.

Proof of Lemma C.7. The proof is by induction. We first consider i ∈ I (0) r, -. At t = 1 ,

̸

where we have used the condition that ⟨ w (0) r , ξ i ⟩ < 0 and the property of ReLU activation. Next we consider

̸

where inequality (a) is by Lemma C.1 and ρ (1) r,i = 0 , and inequality (b) is by Lemma C.6.

Suppose at iteration t , the claim is satisfied, i.e., ⟨ w ( t ) r , ξ i ⟩ < 0 , ρ ( t ) r,i = 0 , for all i ∈ I (0) r, -. Then

̸

Next we consider the update of inner product as

̸

where the last inequality is by a anti-concentration analysis shown in Lemma C.6. This completes the induction for i ∈ I ( t ) r, -.

̸

Before formally stating the main lemma on the lower bound of noise memorization, we prepare several lemmas that will be useful.

Lemma C.8. Suppose that δ > 0 . Then with probability 1 -δ , for all r ∈ [ m ] , we have:

Proof. The proof is by Hoeffding's inequality, for arbitrary t > 0 , we have that

By the randomness of initialization and Rademacher distribution, we have:

Setting t = √ n/ 2 log(4 /δ ) and taking a union bound over r ∈ [ m ] , we conclude with high probability at least 1 -δ , it holds:

To establish the lower bound for noise memorization we define

Lemma C.9. Suppose δ > 0 , the with probability at least 1 -δ , we have

Proof. By Bernstein's inequality, for arbitrary t > 0 , we have

Setting t = σ 0 σ ξ √ nd log(1 /δ ) / 2 , we further have

Similarly, the same result holds for B (0) r, -

.

Lemma C.10. For arbitrary constant C > 0 , under the condition n ≥ 8 C 2 log(1 /δ ) log(1 / (1 -( δ/m ) 2 )) , then with probability at least 1 -δ , it satisfies

Proof of Lemma C.10. Here we want to show |⟨ w (0) r , ξ i ⟩| > B (0) r, + n with high probability. To see this, because ⟨ w (0) r , ξ i ⟩ is a random variable with positive mean and variance σ 2 0 σ 2 ξ d . Besides, by Lemma C.9, we have

Thus when n ≥ 8 C 2 log(1 /δ ) log(1 / (1 -( δ/m ) 2 )) , we have

and by union bound, we have with probability at least 1 -δ , it holds |⟨ w (0) r , ξ i ⟩| > B (0) r, + n and |⟨ w (0) r , ξ i ⟩| > B (0) r, -n .

Besides, we define

With all the results (lemmas) and definitions outlined above at hand, we are ready to state the lemmas that provide the lower bound for noise memorization as follows.

Lemma C.11. Under the same condition as Theorem 4.2, then with probability at least 1 -δ ,

Proof. The proof is by induction. We can check

Besides, by Lemma C.10 and M ∈ n 2 (1 ± o ( n -1 / 2 )) , it holds that

Assuming that the results hold at t , we continue the proof by calculating the propagation of B ( t ) r, + and B ( t ) r, -respectively. First, we can see by Lemma C.1 and by induction, for y i = 1 and i ∈ I r, + ,

We can show that

̸

̸

̸

where the first inequality is by Lemma C.2, Lemma C.1 and Lemma B.5. Furthermore, the second inequality is by Lemma C.5, i.e., |⟨ w ( t ) r , ϵ i ⟩| ≤ 1 /C ξ |⟨ w (0) r , ξ i ⟩| (for C ξ > 200 ), | 0 . 01 ⟨ w (0) r , ξ i ⟩| ≥ 6 √ log(6 n 2 /δ ) d -1 n , the definition of B ( t ) r, + and B ( t ) r, -and the induction B ( t ) r, + ≥ B (0) r, + , B ( t ) r, -≥ B (0) r, -. The third inequality is by M ≥ 100 C ℓ -1 , M ∈ n 2 (1 ± o ( n -1 / 2 )) , d > 10000 log(6 n 2 /δ ) and Lemma C.8. The last inequality is by B ( t ) r, + ≥ 0 .

As a result, we conclude that

̸

Finally, the induction step for Ψ ( t ) r,i can be calculated as follows:

̸

where the first inequality is by Lemma C.2, Lemma C.1 and Lemma B.5. Furthermore, the second inequality is by Lemma C.5, i.e., |⟨ w ( t ) r , ϵ i ⟩| ≤ 1 /C ξ |⟨ w (0) r , ξ i ⟩| (for C ξ > 200 ), | 0 . 01 ⟨ w (0) r , ξ i ⟩| ≥ 6 √ log(6 n 2 /δ ) d -1 n , the definition of B ( t ) r, + and B ( t ) r, -. The third inequality is by M ≥ 100 C ℓ -1 . The last inequality is by induction (23).

Then we check induction induction (23) through following inequalities:

Together it confirms that Ψ ( t ) r,i ≥ 101 C ℓ M +1 B ( t ) r, -.

Finally, we check the sign of ρ ( t ) r,i for i : y i = 1 and i ∈ I ( t ) r, + :

This completes the induction proof.

C.1.3 Noise Memorization: Proof of Lemma 5.2

Before proving Lemma 5.2, we require a lower bound for the initialization. Define that U (0) + = { r : ⟨ w (0) r , ξ i ⟩ > 0 } for y i = 1 , and U (0) -= { r : ⟨ w (0) r , ξ i ⟩ < 0 } for y i = -1 .

Lemma C.12. Suppose that δ > 0 and m ≥ ˜ Ω(1) . Then with probability at least 1 -δ , we have

Proof of Lemma C.12. Consider y i = 1 . Note that ⟨ w (0) r , ξ i ⟩ ∼ N (0 , σ 2 0 ∥ ξ i ∥ 2 2 ) . We define that the event A = { r ∈ [ m ] , ⟨ w (0) r , ξ i ⟩ > 0 } . Then we can compute that ⟨ w (0) r , ξ i ⟩ 1 ( A ) becomes a half-normal distribution with the expectation

We then apply the sub-Gaussian concentration inequality that with probability at least 1 -δ

where we have used Lemma B.5. Similarly, we can show that for y i = -1 , the same result holds.

Proof of Lemma 5.2. From the upper bound on (21), we take the maximum over r ∈ U (0) + , which gives

Under the SNR condition n · SNR 2 ≤ 1 . 8 , we can see there exists a scale difference between max r,i Ψ ( t ) r,i and max r,i A ( t ) r at the end of first stage.

At the same time, for noise memorization, from the lower bound established in Lemma C.11, we have that

where the second inequality is due to Lemma C.12.

Then we have

Let

Then we have 1 m ∑ r ∈U (0) + Ψ ( t ) r,i reach 3 within T 1 iterations by (22). Similarly, we can also show that 1 m ∑ r ∈U (0) -Φ ( t ) r,i reach 3 within T 1 iterations.

On the other hand, we compute the scale of max r A ( T 1 ) r as

where we choose η sufficiently small for the third inequality. The last inequality is by the SNR condition. Because we can choose n ≥ C log( m/δ ) for sufficiently large constant C , max r A ( T 1 ) r = o (1) .

C.2 Second Stage

Proposition C.13. Let T ∗ be the maximum admissible iteration and let α = log(3 MT ∗ ) . Then we can show

Proof of Proposition C.13. We need to show ρ ( t ) r,i ≤ α . We prove the claim by induction. It is clear when t = 0 , ρ ( t ) r,i = 0 ≤ α . Suppose for all 0 ≤ t ≤ ˜ T -1 , we have ρ ( t ) r,i ≤ α . We aim to show the claim holds for ˜ T .

By the update of ρ ( t ) r,i ,

Here without loss of generality, we consider the r, i pairs such that i ∈ I (0) r, + , i.e., ⟨ w (0) r , ξ i ⟩ > 0 with y i = 1 . Let t r,i be the last time t such that ρ ( t ) r,i ≤ 0 . 5 α . Then we have

The second term can be bounded as

where the first inequality is by 1 -ℓ ′ ( t r ) i ≤ 1 and modified Lemma C.1 with ρ ( t ) r,i = O ( α ) . The second inequality is by ⟨ w (0) r , ξ i ⟩ ≤ 0 . 2 α and η ≤ 5 24 nmτ σ 2 ξ d .

For notation convenience, we let

Next, we show the bound for F 0 ( W ( t ) , x i ) and F j ( W ( t ) , x i ) for j = 1 , ..., M as

Further, we have for j = 1 , ..., M

Then we can bound the difference between F 0 ( W ( t ) , x i ) and F j ( W ( t ) , x i ) as

where the second inequality is by σ ( ⟨ w ( t ) r , ξ i ⟩ ) ≥ 3 4 ⟨ w (0) r , ξ i ⟩ + ρ ( t ) r,i ≥ ρ ( t ) r,i ≥ 0 . 5 α for i ∈ I (0) r, + . Further σ ( ⟨ w ( t ) r , ξ i + ϵ i ⟩ ) ≥ 1 2 ⟨ w (0) r , ξ i ⟩ + ρ ( t ) r,i , σ ( ⟨ w ( t ) r , ξ j ⟩ ) ≤ 3 2 ⟨ w (0) r , ξ j ⟩ + ρ ( t ) r,j . The fourth inequality is by induction and 1 2 ⟨ w (0) r , ξ i ⟩ -3 2 ⟨ w (0) r , ξ j ⟩ ≥ -0 . 15 α . The last inequality is by the condition that α ≥ 20 τ .

Further, we bound the loss derivative as

where the second inequality is by (27).

Finally, we bound

where the first inequality is by Lemma B.5 and σ ( ⟨ w ( t ) r , ξ i + ϵ i ⟩ ) ≤ 3 2 ⟨ w (0) r , ξ i ⟩ + ρ ( t ) r,i ≤ 2 α by induction. The last inequality is by the condition η ≤ 1 12 nmτ σ 2 ξ d .

Combining (26) and (29) with (25) gives

which completes the induction.

Lemma C.14. Under Assumption 4.1, for 0 ≤ t ≤ T ∗ , we have

Proof of Lemma C.14. First, we can write the gradient of ∇ w r L i ( W ) as

where

By the derivation of the gradient,

where the first inequality is by triangle inequality and second inequality uses ∑ i a 2 i ≤ ( ∑ i a i ) 2 for a i ≥ 0 and the last inequality is by triangle inequality.

where the first inequality is by triangle inequality and the second inequality is by Jensen's inequality. Similarly, we can obtain for j = 1 , ..., M

For clarity let

Substituting the above results into (30) gives

where the second inequality is by ∑ i a i b i ≤ ( ∑ i a i )( ∑ i b i ) for a i , b i ≥ 0 . Next we can verify that

where the last inequality is by 1 -x ≤ -3 log( x ) for x ∈ [0 , 1] . Furthermore, we can show

which leads to

where the third inequality is by (31) and Cauchy-Schwartz inequality.

We define

Recall in the first stage

and we have

Lemma C.15. Under Assumption 4.1, we have ∥ W ( T 1 ) -W ∗ ∥ F ≤ ˜ O ( m 1 / 2 n 1 / 2 σ -1 ξ d -1 / 2 )

Lemma C.16. Under Assumption 4.1, we have for all t ∈ [ T 1 , T ∗ ]

Proof of Lemma C.16. Based on the definition of W ∗ and F j ( W ( t ) , x i ) , we can derive for j = 0 ,

̸

where the inequality is by Lemma B.5.

Next, we bound I 1 , I 2 , I 3 , I 4 , I 5 separately. For I 1 , we take maximum over r , which results in

Thus we obtain

In addition, we can show by upper bound on ρ ( t ) r,i and γ ( t ) r ,

This implies

I 2 ≤ ˜ O ( σ 0 ∥ µ ∥ 2 ) , I 3 ≤ log(2 M/ϵ ) ˜ O ( nm ∥ µ ∥ 2 σ -1 ξ d -1 ) , I 4 ≤ ˜ O ( σ 0 σ ξ d ) , I 5 ≤ ˜ O ( nmd -1 / 2 ) . Based on the conditions on σ 0 , d , we can show

Now we prove for the claim for F j ( W ( t ) , W ∗ ) for j = 1 , ..., M as follows.

̸

where the second equality is by y i = y j .

Lemma C.17. Under Assumption 4.1, we have

Proof of Lemma C.17. First, we verify that for j = 0 ,

Similarly, we can show for j = 1 , ..., M , it satisfies that

By the update of W ( t ) , we have

where the third equality is by (32) and (33). The first inequality is by Lemma C.16. The second inequality is due to the convexity of negative log-Softmax function. The last inequality is by Lemma C.14 (and the conditions on η ) and log(1 + x ) ≤ x for x ≥ 0 .

LemmaC.18. Under Assumption 4.1, let T = T 1 + ⌊ ∥ W ( T 1 ) -W ∗ ∥ 2 F ηϵ ⌋ = T 1 + ˜ O ( mnσ -2 ξ d -1 η -1 ϵ -1 ) . Then we have max r | γ ( t ) r | ≤ ˜ O (1 / √ n ) for all T 1 ≤ t ≤ T . In addition, we have

for all T 1 ≤ t ≤ T . Thus there exists an iterate W ( s ) for s ∈ [ T 1 , T ] with training loss smaller than 2 ϵ .

Proof of Lemma C.18. By Lemma C.17, for t ∈ [ T 1 , T ] ,

for all s ≤ t . Summing over the inequality and dividing both sides by t -T 1 +1 yields

where the last inequality is by the definition of T , for all T 1 ≤ t ≤ T . In addition, we have

Next we prove the claim that max r | γ ( t ) r | ≤ 3 β, where β = | max r γ ( T 1 ) r | = ˜ O (1 / √ n ) for all T 1 ≤ t ≤ T . Without loss of generality, we only consider r ∈ U (0) + . First it is evident that at t = T 1 ,

we have max r γ ( t ) r = β ≤ 3 β . Next suppose there exists ˜ T ∈ [ T 1 , T ] such that max r γ ( t ) r ≤ 3 β for all t ∈ [ T 1 , ˜ T -1] . Then we let ϕ ( t ) = max r γ ( t ) r and we have by the update of γ ( t ) r ,

where the third inequality is by (31).

Now summing over t = T 1 , ..., ˜ T -1 , we have

where the second inequality is by induction and the third inequality is by (34). The last inequality is by the condition of SNR.

C.3 Downstream Task Performance

Recall that after the pre-training stage on the training data at time T , the signal learning and noise memorization satisfy

Then, on the downstream task, the corresponding embedding can be calculated as follows:

Then, it is straightforward to check that the embedding of a finite size of samples during the fine-tuning stage is not linearly separable. Thus, the downstream task performance follows L D test ( T ∗ ) = Θ(1) .

D Multi-Modal Contrastive Learning: Proof of Theorem 4.3

D.1 First Stage

Similar to the single-modal case, in the first stage, the loss derivative is close to its initial value for both modalities.

Lemma D.1. If max { γ ( t ) r , ρ ( t ) r,i , ˜ γ ( t ) r , ˜ ρ ( t ) r,i } = O (1) , there exists a constant C ℓ > 1 such that

for all i ∈ [ n ] .

Lemma D.2. Suppose that γ ( t ) r , ˜ γ ( t ) r = O (1) and ρ ( t ) r,i , ˜ ρ ( t ) r,i = O (1) for all r ∈ [ m ] and i ∈ [ n ] . Under Assumption 4.1, then for any δ > 0 , with probability at least 1 -δ

for all r ∈ [ m ] , i ∈ [ n ] .

D.1.1 Dynamics of Signal Learning: Lower Bound

We first analyze the dynamics of signal learning for both two modalities. Similar as in the singlemodal learning, we partition the neurons, depending on the initialization, i.e., U ( t ) + = { r ∈ [ m ] : ⟨ w ( t ) r , µ ⟩ > 0 } and U ( t ) -= { r ∈ [ m ] : ⟨ w ( t ) r , µ ⟩ < 0 } , ˜ U ( t ) + = { r ∈ [ m ] : ⟨ ˜ w ( t ) r , ˜ µ ⟩ > 0 } and ˜ U ( t ) -= { r ∈ [ m ] : ⟨ ˜ w ( t ) r , ˜ µ ⟩ < 0 } .

Lemma D.3. Under Assumption 4.1 and the same condition as Lemma D.2, for all t > 0 , we have

Proof of Lemma D.3. We first analyze the neurons r ∈ U ( t ) + ∩ ˜ U ( t ) + . By the update of γ ( t ) r

̸

where the second equality is by y i = y j and the derivative of ReLU activation. Similarly for the other modality,

̸

Hence we see at t = 0 , the claim is satisfied as γ (1) r ≥ γ (0) r for r ∈ U (0) + ∩ ˜ U (0) + . Then we prove the claim by induction. Suppose at iteration t the claims are satisfied, i.e., γ ( t ) r ≥ γ ( t -1) r ≥ 0 for r ∈ U (0) + and r ∈ U ( t ) + . Then we can see from (35)

Similarly, we can show from (36) that ˜ γ ( t +1) r ≥ ˜ γ ( t ) r ≥ 0 .

Then for the inner product, we have

where inequality (a) is by Lemma D.2, and inequality (b) is by Lemma B.2. Similarly, the same result holds for the other modality. Together, it shows r ∈ U ( t +1) + ∩ ˜ U ( t +1) + and the induction is complete.

Similarly, we can use the same strategy to prove claim (2) for r ∈ U (0) -∩ ˜ U (0) -.

Next, we analyze the case where r ∈ U ( t ) + ∩ ˜ U ( t ) -.

̸

̸

̸

where the second equality follows from y i = y j . Similarly,

̸

Then it is clear that the change of γ ( t ) r , ˜ γ ( t ) r does not align with the sign of its initialization. Therefore there exists some time t ′ ≥ 0 such that either r ∈ U ( t ′ ) + ∩ ˜ U ( t ′ ) + or r ∈ U ( t ′ ) -∩ ˜ U ( t ′ ) -. We prove this claim by contradiction. Suppose for all t ≥ 0 , r ∈ U ( t ) + ∩ ˜ U ( t ) -, then by (37) and (38), we see γ ( t +1) r ≤ γ ( t ) r ≤ 0 and ˜ γ ( t +1 r ≥ ˜ γ ( t ) r ≥ 0 . Because ⟨ w ( t ) r , µ ⟩ ≤ 1 . 01 ⟨ w (0) r , µ ⟩ + γ ( t ) r and ⟨ ˜ w ( t ) r , ˜ µ ⟩ ≥ 0 . 99 ⟨ ˜ w (0) r , ˜ µ ⟩ + ˜ γ ( t ) r , this raises a contradiction as either ⟨ w ( t ) r , µ ⟩ ≤ 0 or ⟨ ˜ w ( t ) r , ˜ µ ⟩ ≥ 0 .

With Lemma D.3 at hand, we are ready to demonstrate the lower bound of the growth rate for signal learning.

Lemma D.4. With the same condition as in Lemma C.2 and Lemma C.3 and n ≥ 2500 log(4 /δ ) , define A ( t ) r = γ ( t ) r + ⟨ w (0) r , µ ⟩ for r ∈ U (0) + ; and A ( t ) r = -γ ( t ) r -⟨ w (0) r , µ ⟩ for r ∈ U (0) -. Similarly, we define ˜ A ( t ) r = ˜ γ ( t ) r + ⟨ ˜ w (0) r , ˜ µ ⟩ for r ∈ ˜ U (0) + and ˜ A ( t ) r = -˜ γ ( t ) r -⟨ ˜ w (0) r , ˜ µ ⟩ for r ∈ ˜ U (0) -. Then with probability at least 1 -δ , we have

Proof. Without loss of generality, we consider r ∈ U (0) + ∩ ˜ U (0) + . Then by the update of γ ( t ) r , ˜ γ ( t ) r , we have from (35) and (36)

To achieve the lower bound, we define

By (39), we have the lower bound of update equation,

The inequality is by Lemma B.3, Lemma D.1 where we recall c 2 = 1 -C ℓ M +1 . Similarly, by (40) we arrive at

Then by combining the above two inequalities, we conclude that

Thus, we see the joint dynamics of γ ( t ) r and ˜ γ ( t ) r exhibits exponential growth, i.e.,

To characterize the dynamics of γ ( t ) r individually, we next track the dynamics of C µ A ( t ) r -˜ A ( t ) r by subtracting (40) from (39), which gives

Then from the the propagation, we notice that the gap C µ A ( t ) r -˜ A ( t ) r exhibits exponential decay regardless of the sign. Thus, we have

Therefore, combining (41) and (42) yields:

Then it concludes that

where we use the fact that ( 1 -η ∥ µ ∥ 2 2 nmτ ∑ i : y i =1 (1 -ℓ ′ ( t ) i ) ) t ≤ 1 . Next, we derive an upper bound on | A (0) r -˜ A (0) r /C µ | , | C µ A (0) r -˜ A (0) r | as follows.

where we recall that C µ ∥ µ ∥ 2 = ∥ ˜ µ ∥ 2 and the second inequality is by the condition on σ 0 ≤ 0 . 5(2 log(8 m/δ )) -1 / 2 ∥ µ ∥ -1 2 = ˜ O ( ∥ µ ∥ -1 2 ) . Similarly, we can show | C µ A (0) r -˜ A (0) r | ≤ 1 and thus we obtain

D.1.2 Dynamics of Noise Memorization: Upper Bound

In order to characterize the growth of ρ ( t ) r,i , ˜ ρ ( t ) r,i , we partition the samples according to its sign of inner product.

We further define

On the other hand, we define

Before we state the main result, we provide some useful lemmas.

Lemma D.5. Suppose δ > 0 , the with probability at least 1 -δ , we have

Proof. By Bernstein's inequality, for arbitrary t > 0 , we have

Setting t = 1 2 σ 0 σ ξ √ nd log(1 /δ ) / 2 , we further have

Similarly, the same result holds for B (0)

r, -.

Lemma D.6. Under the condition d ≥ 300 √ 2 n 3 log(6 n 2 /δ ) σ 0 σ ξ √ π log(1 / (1 -( δ/m ) 2 )) , then with probability at least 1 -δ , it satisfies

Proof of Lemma D.6. Here we want to show ˜ B (0) r, + , + ≥ 150 √ log(6 n 2 /δ ) d n 2 with high probability. To see this, because ˜ B (0) r, + , + is a random variable with positive mean and variance σ 2 0 σ 2 ξ dn/ 4 , by Lemma B.1, we compute

Thus when d ≥ 300 √ 2 n 3 log(6 n 2 /δ ) σ 0 σ ξ √ π log(1 / (1 -( δ/m ) 2 )) , we have P ( ˜ B (0) r, + , + ≤ 150 √ log(6 n 2 /δ ) d n 2 ) ≤ δ/m and by union bound, we have with probability at least 1 -δ , it holds ˜ B (0) r, + , + > 150 √ log(6 n 2 /δ ) d n 2 .

Lemma D.7. Under the condition n ≥ 8 log(1 /δ ) log(1 / (1 -( δ/m ) 2 )) , then with probability at least 1 -δ , it satisfies

Proof of Lemma D.7. Here we want to show |⟨ w (0) r , ξ i ⟩| > B (0) r, + , + n with high probability. To see this, because ⟨ w (0) r , ξ i ⟩ is a random variable with positive mean and variance σ 2 0 σ 2 ξ d . Besides, by Lemma C.9, we have

Thus when n ≥ 8 log(1 /δ ) log(1 / (1 -( δ/m ) 2 )) , we have

and by union bound, we have with probability at least 1 -δ , it holds |⟨ w (0) r , ξ ⟩| > B (0) r, + , +

Similarly, we can conclude that |⟨ ˜ w (0) r , ˜ ξ i ⟩| > ˜ B (0) r, + , + n .

Lemma D.8. Under Assumption 4.1, with probability at least 1 -δ , we have

Proof of Lemma D.8. According to the iteration equations for ρ ( t ) r,i and ˜ ρ ( t ) r,i , we have

̸

where the first inequality is by Lemma C.2, Lemma C.1 and Lemma B.5. Furthermore, the second inequality is by Lemma C.8 and the definition of ˜ B ( t ) r, -, + and ˜ B ( t ) r, -, -. The third inequality is by Lemma D.6, ˜ B (0) r, + , + > 150 √ log(6 n 2 /δ ) d n 2 , d > 10000 log(6 n 2 /δ ) , and n ≥ 1280000 log(4 /δ ) . The last inequality is by choosing C ℓ = 1 . 01 .

Next, we establish the following inequality

i n .

Lemma D.9. Under Assumption 4.1, with probability at least 1 -δ , for all 0 ≤ t ≤ T 1 ,

Proof of Lemma D.9. We partition the dynamics of ρ ( t ) r,i and ˜ ρ ( t ) r,i into one of the four cases according to its initialization (1) i ∈ I (0) r, + ∩ ˜ I (0) r, -(2) i ∈ I (0) r, -∩ ˜ I (0) r, + , (3) i ∈ I (0) r, + ∩ ˜ I (0) r, + , (4) i ∈ I (0) r, -∩ ˜ I (0) r, -.

̸

where the first inequality is by Lemma C.2, Lemma C.1 and Lemma B.5. Furthermore, the second inequality is by Lemma C.8 and the definition of ˜ B ( t ) r, -, + and ˜ B ( t ) r, -, -. The third inequality is by Lemma D.6 and ˜ B (0) r, + , + > 150 √ log(6 n 2 /δ ) d n 2 , d > 10000 log(6 n 2 /δ ) , and n ≥ 1280000 log(4 /δ ) . Similarly, we have,

For the second modality, with the same derivative, we could obtain the following inequalities:

which completes the proof.

We define

(1) In the case of i ∈ I (0) r, -∩ ˜ I (0) r, -, it holds that

The proof is by the induction method. It is clear when t = 0 , ρ (0) r,i = 0 and ˜ ρ (0) r,i = 0 . Therefore, the induction argument holds at the initial step.

Suppose at iteration t , we have ρ ( t ) r,i = 0 and ˜ ρ ( t ) r,i = 0 . Then we have

Thus by the update of ˜ ρ ( t +1) r,i , we have

̸

Here we have used ⟨ ˜ w ( t ) r , ˜ ξ i ⟩ < 0 . Similarly,

(2) In the case of i ∈ I (0) r, -∩ ˜ I (0) r, + , We use induction. It is clear when t = 0 , we have ρ (0) r,i = 0 . Suppose at iteration t , it holds that ρ ( t ) = 0 Then by the propagation of ρ ( t ) , we have r,i r,i

̸

On the other hand,

̸

where the first equation is by σ ( ⟨ w ( t ) r , ξ i ⟩ ) ≥ 0 .

(3) In the case of i ∈ I (0) r, + ∩ ˜ I (0) r, -, we use induction to prove such claim. It is clear when t = 0 , ρ (0) r,i = 0 . Suppose at iteration t , we have ˜ ρ ( t ) r,i = 0 . Then we have

Thus by the update of ˜ ρ ( t +1) r,i :

On the other hand,

̸

where the first equation is by σ ( ⟨ ˜ w ( t ) r , ˜ ξ i ⟩ ) ≥ 0 .

(4) Finally, in the case of i ∈ I (0) r, + ∩ ˜ I (0) r, + , we have

Similarly, for the other modality,

Together, we can achieve that

On the other hand size, we calculate the upper bound of Ψ ( t ) r,i -˜ Ψ ( t ) r,i as follows

̸

̸

where the first inequality is by Lemma D.2 and Lemma D.1, the second inequality is by induction (46). Therefore we conclude that

̸

Combining (47) and (48) yields

D.1.3 Signal Learning: Proof of Lemma 5.4

Before proving Lemma 5.4, we require the following lower bound for the initialization. Recall the definition that A ( t ) r = γ ( t ) r + ⟨ w (0) r , µ ⟩ for r ∈ U (0) + ; and A ( t ) r = -γ ( t ) r -⟨ w (0) r , µ ⟩ for r ∈ U (0) -. Similarly, we have ˜ A ( t ) r = ˜ γ ( t ) r + ⟨ ˜ w (0) r , ˜ µ ⟩ for r ∈ ˜ U (0) + and ˜ A ( t ) r = -˜ γ ( t ) r -⟨ ˜ w (0) r , ˜ µ ⟩ for r ∈ ˜ U (0) -.

Lemma D.10. Suppose δ > 0 and m ≥ ˜ Ω(1) . Then with probability at least 1 -δ , we have

Proof of Lemma D.10. We first note that ⟨ w (0) r , µ ⟩ ∼ N (0 , σ 2 0 ∥ µ ∥ 2 2 ) and ⟨ ˜ w (0) r , ˜ µ ⟩ ∼ N (0 , σ 2 0 ∥ ˜ µ ∥ 2 2 ) . We define the event A = { r ∈ [ m ] : ⟨ w (0) r , µ ⟩ > 0 , ⟨ ˜ w (0) r , ˜ µ ⟩ > 0 } . Then we can compute

where we use the independence of neurons in two modalities. Let S := ∑ m r =1 ⟨ w (0) r , µ ⟩ 1 ( A ) + ⟨ ˜ w (0) r , ˜ µ /C µ ⟩ 1 ( A ) . Then we apply the sub-Gaussian concentration inequality that with probability at least 1 -δ/ 2 ,

Then suppose m = ˜ Ω(1) , we have

Similarly, we can show the same for the event where ⟨ w (0) r , µ ⟩ < 0 , ⟨ ˜ w (0) r , ˜ µ ⟩ < 0 and taking the union bound completes the proof.

Proof of Lemma 5.4. From the upper bound on noise memorization (45), we take the maximum over r, i , which gives

At the same time, for signal learning, from the lower bound on signal learning in Lemma D.4, we have for the first modality that

Taking a summation over the r ∈ U (0) + ∩ ˜ U (0) + , we obtain

where the second inequality is due to Lemma D.10.

Under the SNR condition n · SNR 2 ≥ 1 . 7 and C µ > 2 . 66 , we can see there exists a scale difference between max r,i Ψ ( t ) r,i and 1 m ∑ r ∈U (0) + ∩ ˜ U (0) + A ( t ) r at the end of first stage. Let

Then we have 1 m ∑ r ∈U (0) + ∩ ˜ U (0) + A ( t ) r reach 3 within T 1 iterations. Using similar analysis, we can show at the same time 1 m ∑ r ∈U (0) -∩ ˜ U (0) -A ( t ) r , 1 m ∑ r ∈U (0) + ∩ ˜ U (0) + ˜ A ( t ) r , 1 m ∑ r ∈U (0) -∩ ˜ U (0) -˜ A ( t ) r also reach 3 .

On the other hand, we compute the scale of max r,i Ψ ( T 1 ) r,i as

where we choose η sufficiently small for the second inequality. In third inequality, we have applied the condition that n SNR 2 = Θ(1) and σ 0 ≤ 1 ∥ µ ∥ 2 . The last inequality is by the SNR condition. Because we can choose n ≥ C log( m/δ ) for sufficiently large constant C , max r,i Ψ ( T 1 ) r,i = o (1) .

D.2 Second Stage

We first show a similar result as in Lemma C.14 for both two modalities.

Lemma D.11. Under conditions, for 0 ≤ t ≤ T ∗ , we have

Proof of Lemma D.11. The proof follows from that of Lemma C.14 and hence is omitted for clarity.

For notation convenience, we let

It is worth mentioning that F j ( W , x i ) = ˜ F j ( ˜ W , ˜ x i ) in terms of numerical values. They differ in terms of the derivatives.

We further denote

Here L ( W , ˜ W ) is the combined loss function for two modalities.

Let θ r = 1 if r ∈ U (0) + , i.e., ⟨ w (0) r , µ ⟩ > 0 and θ r = -1 if r ∈ U (0) -, i.e., ⟨ w (0) r , µ ⟩ < 0 . Similarly, we let ˜ θ r = 1 if r ∈ ˜ U (0) + and ˜ θ r = -1 if r ∈ ˜ U (0) -. Then we define

Lemma D.12. Under Assumption 4.1, we have ∥ W ( T 1 ) -W ∗ ∥ F ≤ ˜ O ( m 1 / 2 ∥ µ ∥ -1 2 ) and ∥ ˜ W ( T 1 ) -˜ W ∗ ∥ F ≤ ˜ O ( m 1 / 2 ∥ ˜ µ ∥ -1 2 )

Proof of Lemma D.12. The proof follows exactly the same as the proof in single-modal case. we include it here for completeness. Without loss of generality, we focus on the case for W ( T 1 ) .

By the scale difference at T 1 , we have

̸

̸

̸

̸

where the first inequality is by triangle inequality and the second inequality is by decomposition of W ( T 1 ) and W ∗ . The third inequality is by the bound on γ ( T 1 ) r and ρ ( T 1 ) r,i and Lemma B.5. The last inequality is by condition on σ 0 .

Lemma D.13. Under Assumption 4.1, we have for all t ∈ [ T 1 , T ∗ ] ,

Proof of Lemma D.13. The proof follows similarly from Lemma C.16 and here we only show the result for the first modality. Based on the definition of W ∗ and F j ( W ( t ) , x i ) , we can derive for j = 0 ,

First, we can bound I 2 ≤ ˜ O ( σ 0 ∥ µ ∥ 2 ) , I 3 ≤ log(2 M/ϵ ) ˜ O ( σ ξ ∥ µ ∥ -1 2 ) , I 4 ≤ ˜ O ( σ 0 σ ξ √ d ) by the global bound on σ ( ⟨ ˜ w ( t ) r , ˜ ξ i ⟩ ) , σ ( ⟨ ˜ w ( t ) r , y i ˜ µ ⟩ ) = ˜ O (1) .

Further, we lower bound I 1 as follows. Without loss of generality, we suppose y i = 1 , then we have

where the last inequality is by Lemma 5.4 and the monotonicity of ˜ γ ( t ) r .

Then we can obtain

The proof for F j ( W ( t ) , W ∗ ) follows the same argument as in Lemma C.16.

Lemma D.14. Under Assumption 4.1, we have for all t ∈ [ T 1 , T ∗ ] ,

Proof of Lemma D.14. First, we see that L ( W ( t ) , ˜ W ( t ) ) = L S ( W ( t ) ) + L S ( ˜ W ( t ) ) is decomposable in terms of W ( t ) and ˜ W ( t ) . This suggests that ∇ W L ( W ( t ) , ˜ W ( t ) ) = ∇ L S ( W ( t ) ) and ∇ ˜ W L ( W ( t ) , ˜ W ( t ) ) = ∇ ˜ L S ( ˜ W ( t ) ) . Then following similar analysis as in Lemma C.17, we can first show

Then by the gradient descent update

where the third equality is by (49). The first inequality is by Lemma D.13. The second inequality is due to the convexity of negative log-Softmax function. The last inequality is by Lemma D.11 (and the conditions on η ) and log(1 + x ) ≤ x for x ≥ 0 .

Similarly, we can show the same for the other modality as

Combining the two results completes the proof.

Lemma D.15. Under Assumption 4.1, let T = T 1 + ⌊ ∥ W ( T 1 ) -W ∗ ∥ 2 F + ∥ ˜ W ( T 1 ) -˜ W ∗ ∥ 2 F ηϵ ⌋ = T 1 + ˜ O ( mη -1 ϵ -1 ∥ µ ∥ -2 2 ) . Then we have max r,i | ρ ( t ) r,i | ≤ σ 0 σ ξ √ d and max r,i | ˜ ρ ( t ) r,i | ≤ σ 0 σ ξ √ d for all t ∈ [ T 1 , T ] . In addition, we have for all T 1 ≤ t ≤ T ,

Therefore, we can find an iterate ( W ( s ) , ˜ W ( s ) ) for s ∈ [ T 1 , T ] with training loss smaller than 3 ϵ .

Proof of Lemma D.15. By Lemma D.14, for t ∈ [ T 1 , T ] , we have for any s ≤ t

Summing the inequality yields

Dividing both sides by t -T 1 +1 and setting t = T gives

D.3 Downstream Task Performance

Recall that after the pre-training stage on the training data at time T , the signal learning and noise memorization satisfy

Then, on the downstream task, the corresponding embedding can be calculated as follows:

Then, it is straightforward to check that the embedding of a finite size of samples during the fine-tuning stage is linearly separable. Thus, the downstream task performance follows L D test ( T ∗ ) = o (1) .

E Additional Experimental Details

We implement our methods using PyTorch. For the software and hardware configurations, we ensure consistent environments for each dataset. We run all the experiments on Linux servers with NVIDIA V100 graphics cards and CUDA 11.2, completing them within one hour.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline the primary contributions of the paper, including the theoretical advancements in optimization and generalization analysis of multi-modal contrastive learning and single-modal contrative learning.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitation of this work in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided all assumptions in Assumption 4.1 in the main paper. The proof sketch and complete proof are provided in Section 5 and Appendices C and D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the complete configuration in Section 6. We have also uploaded the code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded the code with instructions in the supplementary material. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided all the training and test details in Section 6 and uploaded code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot 1-sigma error bar for results shown in Figure 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided sufficient information about computer resources in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research does not involve any human subjects, personal data, or interactions that would raise ethical concerns about consent, privacy, or respect for persons. In conclusion, the research aligns with the ethical principles outlined in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts in Section A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the asshts?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors performed theoretical analysis between single-modal and multi-modal contrastive learning. The authors proved theorems that indicates single-modal contrastive learning tend to perform worse on test datasets after converging on the training set, while multi-modal contrastive learning tend to perform better on the test set. They also conducted small-scale simulated experiments that confirmed their theoretical results.\n\nSTRENGTHS:\nThe paper studies the important topic of contrastive learning, and put effort into creating a theoretical understanding of the difference between single-modal and multi-modal contrastive learning. They also conducted small-scale simulated experiments that confirmed their theoretical results.\n\nWEAKNESSES:\nThere are many problems with the proof:\n\n1. The starting point of the proof is Assumption 4.1. However, Assumption 4.1 contains many issues that puts the soundness and correctness of the entire proof as well as the usefulness of the results into question:\n\n(a) Incorrect use of big-O-like notations. Big-O notations are formally defined only as sets of functions with variables, but in Assumption 4.1, the authors used big-O-like notation to bound constants, which doesn't make sense.\n\n(b) Assumption (1) is extremely unrealistic. It basically requires that $d>n^2$, meaning that the input data's dimension is larger than the square of the number of data points we have in the dataset. In any real world scenario, your data's input dimension will not be anywhere close to the square of the size of your dataset. For example, OpenAI CLIP is trained with $n=400M$ data instances, and the instance dimension is in the order of $d<2\\times 10^5$, so in this case, $d < \\frac{1}{10^{11}}n^2$.\n\n(c) Undefined variable $\\sigma_0$. Technically $n$ and $\\eta$ is also never formally defined in text, but we can guess that they mean the size of the dataset and learning rate respectively.\n\n(d) It is also unclear how assumption (6) distinguishes between the single-modal and multi-modal features, as the assumption only applies to the first modality's SNR. \n\n2. Theorem 4.2 and 4.3, the main results of the paper, are not well-defined. $\\epsilon$ is supposed to be a vector according to Eq(5), but the main result of Theorem 4.2 says that the train loss (which is scalar) is less than $\\epsilon$.\n\n3. Confusing definition of the loss derivatives in Eq (8). Are these definitions for $l_i^{(t)}$ or for $l'^{(t)}_{i}$? These expressions does not look like derivatives.\n\n4. Lemma 5.1 is not proven anywhere, but is then directly used in proving Lemma C.3 and Lemma C.7. \n\n5. The overall proof is organized poorly. It's very difficult to understand what each Lemma is trying to achieve and how are the Lemmas connected to each other, especially between the main text and the Appendix. For example, it is very hard to find where Lemma 5.2 (in the main paper) is proved in the Appendix. Perhaps the authors should add references in the main paper to the appendix (for example, state after Lemma 5.2 \"proof in Appendix X.X\").\n\n6. The proof roadmaps in Section 5 does not lead to conclusions in Theorems mentioned in Section 4. For example, the roadmap in Section 5.1 does not explicitly tell us how we end up with Theorem 4.2.\n\n7. In proof of Lemma B.2, on Line 560, it is unclear what \"RHS\" is referring to. Is it the RHS of the last equation on the previous page? Or is it the one immediately before it on line 560? If it is the latter case, you shouldn't be allowed to \"set\" it to $\\delta\\/m$ since $\\delta$ already appears in the expression before. \n\nIn addition to problems in the proof, it is also unclear how the \"noise memorization\" and \"signal learning\" evaluation numbers are defined in the experiment in Figure 1.\n\nQUESTIONS:\n1. Please address the problems in the weakness section.\n\n2. Does your theorems provide any new insight into future research contrastive learning? Any actionable suggestions?\n\nLIMITATIONS:\nDiscussion is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical framework to understand the differences between multi-modal and single-modal contrastive learning approaches. It emphasizes the impact of signal-to-noise ratio (SNR) on the generalizability of these learning methods in downstream tasks. The authors argue that multi-modal learning, through the cooperation of modalities, can achieve better feature learning and performance in downstream tasks compared to single-modal learning.\n\nSTRENGTHS:\n- This paper explore an interesting problem. As to why multi-modal contrastive learning might be more effective than its single-modal, this paper provides a feature learning theory framework for analyzing differences between multi-modal and single-modal contrastive learning which is valuable for the machine learning community.\n- This paper delves into the nuanced dynamics of multi-modal and single-modal contrastive learning by employing a data generation model that incorporates signal and noise, thereby enabling an in-depth study of the optimization trajectories under gradient descent.\n- The authors notably identify the signal-to-noise ratio (SNR) as a pivotal determinant of the generalizability across learning methods, and compellingly demonstrates that the orchestrated interplay between modalities in multi-modal contrastive learning fosters superior generalization in downstream tasks, underscoring the importance of harmonizing diverse data streams for effective feature learning.\n\nWEAKNESSES:\n- **Strong assumptions made in the theoretical analysis** The analysis relies on some strict assumptions, such as specific signal-to-noise ratios and network initialization conditions, which might not hold in real-world applications. \n- **Some influential previous works are not introduced** such as UMT/UME[1], QMF[2], MMParato[3], and ReconBoost[4].\n\n[1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.\n\n[2] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023\n\n[3] MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024\n\n[4] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.\n\nQUESTIONS:\nThe theoretical analysis is predicated on rather strong assumptions. If the volume of data and the complexity of the models are significantly increased, would conclusions still be applicable?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis comparing single-modal and multi-modal contrastive learning. The authors develop a unified framework to analyze the optimization dynamics and generalization capabilities of both approaches. Key findings include:\n\n- Both single-modal and multi-modal contrastive learning can achieve small training error.\n- Multi-modal contrastive learning generalizes better to downstream tasks compared to single-modal learning.\n- The advantage of multi-modal learning comes from cooperation between modalities and higher quality data in the second modality.\n\nThe analysis is based on a two-stage optimization process and uses a signal-noise data generation model. Theoretical results are supported by synthetic experiments.\n\nSTRENGTHS:\n- Develops a unified framework to analyze both single-modal and multi-modal contrastive learning, allowing for direct comparisons.\n\n- Provides detailed theoretical analysis, including convergence guarantees and generalization bounds.\n\n- Identifies key factors (signal-to-noise ratio, cooperation between modalities) that explain the superior performance of multi-modal learning.\n\n- Supports theoretical findings with synthetic experiments that align well with the analysis.\n\nWEAKNESSES:\n- The paper could benefit from more intuitive explanations of the key theoretical results to make them more accessible to a broader audience. For example, can you provide intuition for why the cooperation between modalities leads to better signal learning in the multi-modal case? How might this insight be leveraged to improve single-modal contrastive learning?\n\n- The paper lacks discussion on how the insights could be applied to improve existing contrastive learning methods or guide the development of new approaches. Do your theoretical insights suggest any practical strategies for improving multi-modal contrastive learning, such as ways to select or preprocess data to increase the effective signal-to-noise ratio?\n\n- The theoretical setup and assumption of the linear data generation model are somewhat simple and restricted. Do you expect the main insights to hold for more complex data distributions?\n\n- The experimental validation is limited to synthetic data. Including experiments on real-world datasets, even if simplified, would strengthen the paper's impact.\n\nQUESTIONS:\nLine 205: “On the contrary, augmentation often maintains the same SNR as the original data, so single-modal learning hardly benefits from the augmentation and can only memorize the noise from the data.” This claim significantly contradicts empirical experiments such as SimCLR and MoCo. How would you justify this?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}