Contrastive Learning with Data Misalignment: Feature Purity, Training Dynamics and Theoretical Generalization Guarantees
Authors: Jiawei Sun
Shuai Zhang
Hongkang Li
Meng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Contrastive Learning
Training Dynamics
Generalization Guarantees
Learning Theory
Primary area: theory

Abstract: Contrastive learning is a powerful framework for learning discriminative representations from image-text pairs. 
Despite its success, its theoretical foundations, especially when the image-text pair exhibits misalignment, remain underexplored. 
This paper provides the first theoretical analysis of contrastive learning under data misalignment, proving how the ground-truth modality-paired features are amplified while spurious features are suppressed through the training dynamics analysis. Specifically, we study two nonlinear encoders trained jointly with a contrastive loss and demonstrate that noisy (or misaligned) data pairs result in mixed representations and degrade the model's generalization ability. In contrast, recaptioning and filtering improve the data alignment, which in turn purifies the features learned by neurons and subsequently enhances generalization. Our analysis identifies feature purity as a key factor in the success of contrastive learning and offers insights into how data quality and training procedures impact representation learning and downstream generalization. Theoretical insights are  supported by experiments on standard benchmarks.

Contrastive Learning with Data Misalignment: Feature Purity, Training Dynamics and Theoretical Generalization Guarantees
Jiawei Sun Rensselaer Polytechnic Institute sunj [omj110rp..emm edu
Shuai Zhang
New Jersey Institute of Technology sz457Onjit.en edu
Hongkang Li University of Pennsylvania 1 lihkOseas. upenn. edu
Meng Wang
Rensselaer Polytechnic Institute wangm70rpi.. edu
Abstract
Contrastive learning is powerful framework for learning discriminative repre- sentations from image-text pairs. Despite its success, its theoretical foundations, especially when the image-text pair exhibits misalignment, remain underexplored. This paper provides the first theoretical analysis of contrastive learning under data misalignment, proving how the ground-trute truth modality-paiie features are amplified while spurious features are suppressed through the training dynamics analysis. Specifically, we study two nonlinear encoders trained jointly with a contrastive loss and demonstrate that noisy (or misaligned) data pairs result in mixed representa- tions and degrade the model's generalization ability. In contrast, recaptioning and filtering improve the data alignment, which turn purifies the features learned by neurons and subsequently enhances generalization. Our analysis identifies feature purity as key factor in the success of contrastive learning and offers insights into how data quality and training procedures impact representation learning and downstream generalization. Theoretical insights are supported by experiments on standard benchmarks.
Introduction
Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks such as Essoonlanguage understanding and generation. State-of-the-ee methods like CLIP [37] and [50) contrastive learning to train dual encoders large-scal scale image-text pairs scraped from the web, aligning embeddings by pulling paired samples closer in a shared space. These models excel in zero-shot scenarios, requiring task-specific fine-tuning.
However, web-sourced captions are often noisy misaligned, containing irrelevant or spurious details that hinder cross-modal alignment and reduce representation quality. For example, [34] cites an image a blue Mercedes- Benz parking lot paired with the caption: "2003 Mercedes-Bent C240 sedan, Leather, MUST BE SEEN - $6199." The price information in this caption is only superficially correlated with the image and does not contribute meaningfully to understanding the image context.
To mitigate this issue, many works [3 34 46
38 16, 45) adopt text generation methods during VLM training to produce high-quality synthetic captions more faithful to the corresponding images. Models like LaCLIP [33 and BLIP 25 show that such recaptioning improves both the quality and diversity training data, leading to significantly better performance. Further, (34] demonstrates that the cosine similarities between BLIP2 generated captions 24 and their paired images is higher than
3
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Comparison with existing theoretical works on contrastive learning.
Train
Zero-shot Gen.
Multi- modal
Joint
Work
Nonlinear
Recaption
Dyn.
Encoder
(Wen & Li, 2021) [51]
(Nakada et al., 2023) [3]
V
(Chen et al., 2024) (10]
v
(Lee et al., 2021) 21
(Zhang al., 2023a) 660
(Pareek et al., 2025)
35
This paper
that of raw captions. [22 analyzes conformity on MSCOCO and finds that it correlates with how common or rare an image- caption embedding is, reflecting its degree of alignment within the dataset.
Despite the impressive success of VLMs and the practical advancements driven by recaptioned texts, their theoretical foundations remain relatively underdeveloped. Several critical questions remain mostly open:
How do contrastively pre-trained VLMs align modalities, extract feature representations, and achieve zero-shot capabilities? How does text recaptioning on noisy image-text pairs provably enhance generalization performance?
Notably, even the theory of vanilla multimodal contrastive learning still incomplete, For instance, 115 60] extend spectral contrastive loss to multimodal settings, showing that the objective can be related to matrix factorization. 35 provides theoretical characterization of when data filtering improves multimodal contrastive learning, offering a complementary, data-centric perspective to objective-level analyses. Also, [T7 21 show that, under certain conditions, multimodal models outperform unimodal ones with better representations. However, these works assume an optimal solution to the non-convex problem without analyzing the training dynamics that lead to strong generalization. The zero-shot ability of VLMs also lacks full theoretical study. To the best of our knowledge, only [0] analyzes CLIP's zero-shot performance, showing it learns shared features while ignoring modality-speciife ones. Yet, their setup does not consider real-world issues like misalignment between image and text. Beyond standard contrastive learning, [33] proposes a modified loss using unpaired data to detect ground-truth pairs and improve results, but only for linear models. So far, no work has theoretically studied the effect of text recaptioning on VLMs.
Contributions: To the best of our knowledge, this is the first theoretical work explaining why text recaptioning improves zero-shot generalization VLMs, especially under image-text misalignment, where text may include spurious missing features. We analyze the training dynamics of stochastic gradient descent (SGD) multimodal contrastive learning and derive the generalization behavior of the learned model. Our analysis uses one-hidden-layee ReLU network, which remains the state-of-te -ort model in theoretical studies of contrastive [51] and supervised learning [1]. All findings validated empirically practical VLMs like CLIP. A comparison to prior theory works is shown in Table Key contributions include:
1. Theoretical training dynamics and generalization analysis of contrastive learning in nonlinear VLMs. We provide a theoretical analysis of jointly training two nonlinear encoders with contrastive loss. Prior works training dynamics in contrastive learning [51 10 33) either analyze a single encoder or are restricted to linear neural networks. In contrast, our analysis captures the joint learning behavior of both nonlinear encoders with ReLU activation functions.
Theoretical characterization of the impact misaligned image-text pairs on pre-training performance. We analyze a data model with modality misalignment, where some texts may contain features spuriously correlated with the image and others may omit relevant features. We show that spurious and missing features cause neurons to entangle true and irrelevant representations, which hinders the ability of the vasion-language model to disentangle semantic components, ultimately degrading generalization performance.

---

3. Theoretical justification of enhanced out-of-domain generalization through pre-training with text recaptioning. We first analyze the training dynamics of the text generation process and formally prove that the resulting text after recaptioning has reduced spurious correlation and enhanced semantic relevance with the corresponding images. When these filtered texts are used for contrastive pre-training, the resulting model exhibits improved feature purity and succeeds in out-of-domain zero-shot classification, whereas the model trained on raw data provably fails.
1.1
Related Works
Vision-Language Models: VLMs [58 48
26 are trained via contrastive learning on large web-sourced image-text pairs. Following CLIP, later models 30 (5)) aim to boost zero-shot performance. Data quality has become a key bottleneck, leading to recent filtering efforts [13 24
19
49
20
27
For example, LaCLIP [13]] uses LLM- generated caption rewrites as augmentation, and leverages synthetic captions to drop noisy pairs, enhancing feature quality and robustness.
BLIP
25
Theoretical Exploration on Contrastive Learning. Recent studies explore why contrastive learning yields effective representations. [47] identifies alignment and uniformity key properties of con- trastive loss. [15] shows that solving auxiliary prediction tasks improves contrastive representations. 44 highlights the role of inductive biases shaping learning dynamies. (29 proves that multimodal contrastive learning can recover shared latent factors under a generative model.
Generalization analyses of Neural Networks (NNs). Various approaches have been developed to analyze the generalization of feedforward NNs. The neural tangent kennel (NTK) approach shows that overparameterized networks can be approximated by kernel methods in the limiting case [18] 2). The model estimation approach assumes the existence of ground-truth one-hidden-layer model with desirable generalization and estimates the model parameters using the training data (63). The feature
learning approach analyzes how shallow NN learns important features during training and thus achieves desirable generalization
31 23 43
Problem Formulation and Algorithm
VLMs leverage large-scale web-based datasets containing paired visual and textual data to pre-train two separate encoders: image encoder and text encoder h, parameterized by weights W and V, respectively. Contrastive learning serves as the core framework, ensuring the learned embeddings of matching pairs are closer while separating mismatched pairs.
Specifically, let S be the indices of the image-text pairs, e.g.. (Ip, Up) with p S. (Ip, Up) referred to as positive pair, while (I Bn.) with p * is referred to as negative pair. We minimize the following spectral loss function:
(f(ap). + Σ h(gp)))
h(()))) NES
L(f,h)
Σ PES
Σ
27
2T
nEST())
where the hhper-parameee parameter 5 >> 0 referred as the temperature. The spectral contrastive loss L in (I) has been extensively utilized in recent theoretical works 42 (60. Although it differs from the commonly used SimCLR 9 in practice, the spectral contrastive loss closely resembles SimCLR
numerically, shown in 15
2.1 Training Framework
Let S S4, U S., include human-annotated high-quality image-text pairs with indices in S₁. and noisy web low-quality datasct with indices in S.. Due to the inherently noisy nature of web data, the learned embeddings from ( may be suboptimal. To mitigate this, many practical training methods [35] incorporate recaptioned text to improve the quality and diversity of image-text pairs. While specific implementations vary, most frameworks follow a similar four-stage approach:
(S1) Image-text contrastive pre-training (ITCP) on raw data: The image encoder f and text encoder h are trained using the image-tex text pairs {(zp. Vp) hpes by minimizing the contrastive loss as in 11. Let W and V denote the learned weights in and h. We then estimate the image and text
3

---

embeddings of (Ip, Up) by L,_ = (w( and E, = hv(3p). Due to the low-quality data in S when training the encoders, these estimations might not be accurate.
The high-quality data pairs in Sr, are used to finetune image- grounded text decoder G, which maps an image Ep to text through G(zp). Then, the learned G is applied to every image Zp in Sw to generate a synthetic caption Up G(Tp). Next, the estimated text emhedding of Up is computed as Zy, (v())) (G(x)) where represents the weights of h learned from Stage (S1).
(S2) Generating text captions:
(S3) Filtering: For every (Ip, My) in Sw compute the cosine similarity between the image embedding >'z, and the text embeddings of the original caption Ey,, and the synthetic caption &y.. respectively. If the pair (#e,, )y.) has higher similarity to each other than the pair (Zeg- 2/8)), (xp, Up) replaced with (xp, (p). Let S.,, denote the index set of the resulting data pairs. By filtering noisy captions in Sw with synthetic captions that better align with image embeddings, S., becomes a cleaner dataset.
The image encoder f and text encoder h are trained by minimizing the contrastive loss in T repeating the procedure from Stage (S1) with the only difference being that the original dataset S replaced by 5 S₁, U S... The resulting loss is denoted by L(f, h). Let W and V denote the resulting learned weights. fw and gy can produce improved embeddings compared with fw and 9v-
(S4) ITCP on filtered data:
We employ stochastic gradient descent (SGD) with step size 17 and batch size B, following standard practice. Despite the non-convexity of ① we present a detailed analysis of the resulting training dynamics and establish convergence guarantees in Section [ This stands in contrast to existing works
39 53
10
that assume the attainability of a global optimum.
2.2
Downstream Tasks
As demonstration of the performance of the learned model (/w my), we consider a downstream image classification task in a zero-shot setting. Unlike the regression and binary classification tasks evaluate the uni-modal contrastive learning in [51], we consider K KCclassificattonn problem for any constant K 2 2. Each class label is associated with a given text prompt yk. where k E [K] For any image x with its ground-truth label lg [K], the zero-shot predicted label by the pre-trained models (Ww 9V-) is computed as arg |a (f w (x). (M..).. This approach follows the typical setting
of zero-shot image classification using VLMs [10 25 The prediction is considered accurate
and only if arg maxke[K] (Sw (x), By (JN)) L.
3 Technical Assumptions and Setups
We introduce a set of assumptions that are either derived conceptually from the real data distribution or follow existing approaches in contrastive learning theory.
3.1 Backbone of the Encoders
We use a two-layer neural network with ReLU activation functions as the image and text encoders, respectively. Formally, we have
Definition 3.1. The image encoder fw Rdd - Tim and text encoder bv Rdd -> I is
f(x) (fi(z), ... [m)) ER", with se(x) ((wi, x)) b.) - o (- (Wi, xx) - b:), (2)
h(y) (h1(g), hm (y)) E R". with h₂(y) ((v), y) bi) o (- (vs, v) bb) (3) where a is ReLU function, and W [W1, u2, wm]'
[U1 ......, um] E mmdd
Because deep neural networks are highly nonlinear, amalyzing the training dynamics and resulting generalization performance of learned models remains challenging. As result, existing theoretical
studies are largely limited to one-hidden-layer neural networks 2 61 51 33 where the learning problem already nonconvex. In this paper, we extend this line of research to a more complex setting, where two such encoders are jointly trained for image and text modalities.

---

Recall that G' is employed in Stage (S2) to generate synthetic text captions. In practice, the core is to train the encoder-decoder model G and 3.2 Data Model for ITCP
Our data model in Assumption builds on the sparse coding framework, which has been widely used in both uni-modal contrastive learning for images [2] and multi-modal Image-text contrastive learning [TO]. This sparse coding model has been employed in theoretical analyses
7 14
because
it effectively models the practical NLP 5 6 36 11 and image data [55 52 54]
Assumption 3.1 (Sparse coding model for image-text pairs). Each image-text pair (Fpr p E S.
is generated Li.d. from the following spurse coding form:
Ip =p= =MZx + Expx
Up -H + Exp'
(4)
where Tp. Up E IR Zay: Bvp E RR, and d1 = poly We assume:
(a) Image dictionary: M = [M Mal E R xd is column-orthonomma.
(b) Text dictionary: [HI...., Ho] E gd is column-orthonormal
(c) Additive noise: Eay, Euy NO0, I.) with (1/di) S of S (O(Vlogd/dtto)
(d) Sparse latent vector: Zxp = (z (..,....,, with 2 E {0, =1}. where |=2,1 Cernoull(C../d)
Notably, we operate in a regime where the noise magnitude can dominate the signal: since (1/d1) < of S 0 (oTo d/ol++o/ | we have ||E||3 >> ((1) > ||Ma||a, indicating that the overall noise energy significantly exceeds that of the signal. Nevertheless, we will show that contrastive learning remains effective even under such high-noise conditions, due to the encoders' ability to extract denoised and purified features, as characterized in Theorem (4.4. An intuitive explanation for why feature recovery still possible lies in the different alignment properties of the signal and noise: for any active feature æj * O, the signal aligns well with its corresponding basis: [(Mz, M2)1 @(1), while the noise contribution remains small, (C, M,) s O(1/Vd).
We introduce Assumptions 3.2 and 3.3 to capture the characteristics of the dataset S S₁. U.S. Notably, the number of high-quality pairs in S₁. may be significantly fewer than that of low-quality pairs in S.. with |Sa| (dd) and |S.. poly(d)
w(d)).
Assumption 3.2 (High-quality image-text pairs). Every high-quality image-text pair (Fp:Bp) with p E Sp. satisfies zx, Zyp- iee. the image and text have the same latent vector.
Compared to high-quality pairs in Sh, low-quality pairs in S.e show modality misalignment due to spurious image-text text correlations and missing descriptions of key visual features.
Assumption 3.3 (Low-quality misaligned image-text pairs), There exists a constant C, E (w/ log d), 1/2) such that for every low-quality pair (Ip, Up) in Su and every image feature M2 (j E [d] in Tp, we have
Pr (2/ F0p
1 112,,111)
Pr Pr 0 -1) =
(5)
where the first term in (5) is the probability that a text feature Hy (j' + j) is spuriously correlated to the image feature M,. and the second term the probability that H, is missing in the text while the imagefeature feature M, exists.
Consider the blue Mercedes-Benz example from 34 Here, M, denotes the car's visual feature, while H. refers to unrelated price information spuriously correlated with Mj, illustrating the first term in (5). The correct text feature "Mercedes- relevant Teature, as captured by the second term in (5). We focus on a single spurious pair (i.j") for simplicity. Since our analysis depends on the total spurious feature probability (bounded by C.). the results extend to multiple spurious features as long as their total probability stays within C..
is H,; its absence reflects the omission of
3.3 Image-Gr Text Decoder in Stage (S2)
idea behind the widely adopted approaches (25 58 48
'The columns M, and H, are cottnn-otthooommal with each entry bounded by Ō(1/Vdi). ensuring small inner products with isotropic noise.
5

---

leverage the high-quality image-text pairs Sy to improve its performance. In this paper, we consider simplified form of G. given by:
G( = V V (WZp),
=
(6)
where a denotes the ReLU function. The parameters W and are learned by solving
Le Lo= = - V - vpl12.
min W.V
PES.
initialized at W and V, using SGD with step size 7) Although G in 66) is conceptual simplification. where o Wxp)) acts as the encoder and V as the decoder, it serves as a realistic abstraction to illustrate the underlying advantages of synthetic text caption generation.
3.4 Zero-Shot Generalization on Image Classification
We consider an out-of-domain (OOD) setting for testing images and text prompts as follows. Image: Each test image r can be approximated by sparse coding model with dictionary M',
M 2
Exx
2 llo = O(1), || = @(1),
(8)
where M' = MP1, and max,, (Pi),) dugه < O(1/Vd) The noise & matches the training distribution (Assumption 3.1 I.l d)) and dig denotes the Kronecker delta function.
Text: Each class E [K] has a prompt that has a sparse decomposition
yk H yn. + Ema.
1 llo = @(1) 1l llmax O().
x belongs to class then among all K binary vectors a 2/ is maximally aligned with z
"y*"
[1x2) 1/,, 112 > [[(()) 2 lla,
Ak' 1"*
(10)
This formulation reflects the intuition that x belongs to class 1 its sparse representation is most similar to the sparse representation of class k's text prompt.
4
Main Results
4.1 Intuition and Informal Insights
Before presenting our main results, first offer an intuitive explanation of the encoder-learnee success. To learn the latent representation from input pair (æ, y). well-trained image encoder f and text encoder g must ensure that each feature pair (M,, H2) captured by at least one neuron pair (w;, U;), without interference from spurious signals. We call this a purified feature, meaning the neuron pair encodes only one true feature with contamination. In this case, (w),x) 2 and (Vi, y) R zy. so f and recover the full latent space z. But in real data, where high-quality pairs in Sh rare and noisy pairs with misaligned image-text pairs in Sw, dominate, achieving this is difficult. See Appendix B. for proof sketches and we summarize main findings below:
(I) SGD provably solves the nonconvex training problems (l. The existing training dynamics and convergence analyses are limited to either single-modal contrastive learning (51] or linear networks Theorem 4.1 provides convergence analysis of SGD for solving the nonconvex ITCG problem when the network contains nonlinear activations for both modalities.
10 33
(II) Failure of learning due to spurious correlations. Theorem 4.2 provides negative result: if and g are directly trained on the raw data S, the model inevitably learns M, and M, together via some wi, and H2 and H,- together via some As result, the model fails to distinguish between these spuriously correlated features.
(III) Successful learning with recaptioning and filtering. Theorem demonstrates that recap- tioned texts significantly suppress spurious features and enhance relevant feature aligmment. Building on this, Theorem 4.4 states that training f and g on the recaptioned data Š enables the resulting en- coder pair to learn purified representations of M, and H2 accurately, as if trained solely on sufficient high-quality data. This highlights the advantage of leveraging the recaptioned data S...

---

(IV) Enhanced zero-shot image classification accuracy due to text recaptioning. The advantage of using synthetic text captions further validated in downstream tasks. As shown Theorem 4.5 for a zero-shot out-of-domain multi-class image classification task, ITCP trained using $ achieves high accuracy, whereas ITCP directly using S fails to generalize accurately.
4.2 Feature Purity Improvements in Converged Models via Recaptioned Data
We first characterize the training dynamics and convergence of solving ① using SGD in Stage (S1) and (S4) in Section 2.1 Let I, and ī,* denote the optimal values of the contrastive loss the raw dataset and the filtered dataset Š, respectively. Note that (W, V) and (W, V) are the converged
weights from contrastive training on S and Š Stage (S1) and (S4), respectively.
Theorem 4.1 (Convergence of ITCP). Suppose Assumptions 3.7 3.3 hold. Let the model complexity be m = d!.01, initialized at wh , (0) ~ ~N(0,o where of =
(zpoly(a)) (T.piia))) After
== O (d log d) iterations with batch size B (d(d) and 17 O(1), the returned weights achieve a loss that is sufficiently close to the optimal loss in Stage (SI) and (S4), respectively, i.e,
[f (T) - 11) o(1), (ilfw. -r)) |-) (().
(11)
Remark 4.1. Theorem 4.1 demonstrates that SGD iterations can converge to weights that achieve near optimal loss of respectively. This result is of independent interest, existing training dynamics and convergence analyses for contrastive loss are limited to linear networks. Here, we extend such analysis nonconvex optimization settings where the network contains nonlinear ReLU activations. Next, we characterize the feature purity of the learned models.
Theorem 4.2 (Unsuccessful learning ITCP on raw data S with low feature purity). For each newron pair (w;, 02) in (W, V), there exists spurious feature pair (j,j') E [d] such that
W;=- = o M, + a... My ris
U; =0:, H, + 0i."" H S;
(12)
where o o = + (1 113 || [107113)) and ||r:112 S <0((1 1 3 11) Moreover, for every spuriously correlated pair (') there exist least 21(1 neuron pairs (w,, 6.) that primarily learn the mixed feature pair (M,. My, Hyy)
Remark 4.2. Theorem 4.2 indicates that the model learned by ITCP on raw data achieves only limited feature purity. Specifically, a neuron pair (w;, 0.) learns a mixture of image and text features, respectively. M, and My are always mixed together, as are Hj and Hyr. As a result, the learned weights W and V fail to produce purified representations, making it difficult to distinguish between features j and j', which ultimately degrades downstream performance shown in (15).
Theorem 4.3 (Spurious feature suppression and relevant feature preservation by recaptioned texts). After T = Od d) steps of SGD, the decoder G in (6). finetuned by solving O converges
to weights (V, v) with expected loss Le S (1/dd) The recaptioned texts in Su are computed by
Up G(zzp). Then for any index j [d] such that |22,1 1, the decoder output satisfies:
Pr(2/_ 80 =1
Pr(z/ 0p
= | 1) s (1/d), Vj' * (13)
Remark 4.3. After captioning and filtering, the resulting text contains fewer spurious features and more aligned feature pairs than raw data. Compared with Assumption [33] the probability of spurious features can be reduced from a constant C₂ in S. to ((1/d) in S,.. while the probability of retaining all aligned feature pairs increases from C, in S,, to 1 - θ (1/d) in S.. The resulting dataset S = S₁ U S. has better-aligned image-text pairs, enabling higher feature purity in contrastive training. We next show how ITCP trained on Š improves feature purity.
Theorem 4.4 (Successful learning of ITCP on filtered data Š with high feature purity). For each neuron pair (wi, (2) in (W, V), there exists [d] such that (wi, 6x) primarily learns (M,, H,)
l = ā. M, + To.
To = ā. H, + 5,
(14)
where ā²,-- = =11 ((: and || S 1 S O (1I + 11.. Moreover, for every feature j E [d], there exist least 2(1) neuron pairs (1i, b.) that primarily learn purified feature pair (M,, H,).
7

---

Remark 4.4. Theorem 4.4 indicates that the model learned by ITCP on filtered data achieves purified representation. Specifically, neuron pair (wi ū;)) learns one single feature pair H), respectively. As result, W and V yield purified representations that effectively separate individual features, enabling improved downstream performance shown in (T6).
4.3 Performance Comparison on Downstream Tasks
We next compare the performance of the models (ww. (F) and (fw. (5) on the zero-shot image classification problem with out-of-domain data described in Sections Se. and
Theorem 4.5 (Zero-Shot Image Classification). For the OOD zero-shot K K-class image classification problem, the model (fw, 9v Iv) from ITCP using raw data has a constant failure probability:
Pr
arg max (fw (V())) =b1 ) -1- kE[K]
(15)
In contrast, the model (fw- 96) from ITCP using filtered caption succeeds with high probabilit:
Pr
arg max (fw (1), (ux)) =1z = =1-o(1) kE[K]
(16)
Remark 4.5. Theorem 4.5 first demonstrates that the zero-shot performance of (Sw [7 is unsatis-
factory, resulting from the low feature purity in (Jw: , as established in Theorem 4.2 Theorem 4.5 further shows that (fw.9z) achieves accurate classification. This success is attributed to high
feature purity in (fw. ). ). as described in Theorem 4.4 Note that Theorem 4.5 holds for image data with distribution shift from the training data.
Experiment
5.1 Simulated Experiment
Paribol Repissatttonn
50-
Pustoc Pattoudfepppee hep.ssttttttte
Faw Data Riteredd Doca
6.10
45
0.0:
Devmotream
0.08
60-
cation 0
6.8
0.06-
Ran
0,42
Sputios
Pobbsiitty Algnmert
(a)
(b)
(c)
(d)
Figure 1: Performance comparison of ITCP on raw data and filtered (recaptioned) data when the probability of spurious correlation C, changes. (a) Number of features that have purified representation in the model (b) Average magnitude of purified presentations (c) Zero-shot out-of- domain classification accuracy (d) Silhouette Score with cosine distance.
Experiment Setup. We first validate our results via simulated experiments, using the same framework from Section 2.1 We adopt more general spurious correlation model than Assumption 3.3 allowing each M, to be spuriously linked with multiple H, (j' * j). while keeping the total spurious 50, S.S.1 5000. SSa] 1000, and use
correlation probability C.. We set dj 2500, m = 80 neurons. Matrices M, H are drawn from standard Gaussians and orthonormalized via QR decomposition. Sparse codes 2, follows Bernoulli Noise variance of = 1/d. SGD runs with batch size 500 and step size 0.001. Downstream evaluation uses 5-way classification with test Za ~ Bernouuli(0.2)) class codes Zy, partition the d-dim space. Results are averaged over 20 trials. Models (W, V) and (W, V) are trained raw and filtered data, respectively.
Improved feature representation using filtered (recaptioned) data. say a weight w; learn purified representation of M, if its projection along M, achieves the largest magnitude and satisfies (w,, /,>l/|[ > 0.5. The same applies to [W, V). Figure I a) shows the number of features M, (out of d 50 total features) for which at least neuron in W (or W, respectively) Iearns purified

---

Table 2: of CLIP and LaCLIP on Accuracy (%) and Silhouette Score.
Food-101
CIFAR-10
Caltech-101
CIFAR-100
STL-10
Model
Acc
SS
Acc
SS
Acc
SS
Acc
SS
Acc
SS
Acc
SS
CC12M CLIP
50.8
0.034
64.9
0.113
77.4
0.225
38.5
0.005
64.1
0.069
91.0
0.195
CC12M LaCLiP
60.7
0.038
75.1
0.157
83.3
0.276
43.9
0.029
72. 4
0.070
95.
0.273
RedCaps CLIP
81.5
0.125
70.4
0.100
72.8
0.210
39.9
-0.002
82.7
0.091
92.8
0.226
RedCaps LaCLiP
85.0
0.175
74.8
0.107
76.4
0.233
40.7
0.011
78.2
0.074
91.4
0.275
LAION CLIP
85.5
0.116
93.0
0.181
91.2
0.258
71.7
0,078
90.1
0,122
97.3
0.223
LATON LaCLIP
86.5
0.148
93.5
0.215
92.4
0.306
73.9
0.108
90.9
0.152
98.4
0.260
representation. The results show that ITCP trained on filtered data learns purified representations for nearly all features, even at high spurious correlation levels (C, = 0.3). In contrast, ITCP on raw data degrades significantly, with purity dropping faster as C. increases. Moreover, Figure T[b) shows the average of the largest projection magnitudes among neurons that learn purified features. The magnitude from W (TTCP on filtered data) is consistently higher than that from W, indicating stronger purified representations. This aligns with Theorems
4.4andd Remark [.44
Improved zero-shot out-of-domain performace using filtered (recaptioned) data. Figure compares the classification accuracy of both models on zero-shot out-of-domain data. The model trained on filtered data consistently outperforms the one trained raw data, with the performance gap widening as spurious correlations in the raw data increase. We also adopt the widely used Silhouette Score (SS) with cosine distance [57 32 62 to evaluate feature embedding quality in different clusters, as shown in Figure d). A higher SS indicates better intra-class alignment and inter-class orthogonality, reflecting more purified representations. These results verify Theorem 4.5
Impact of feature purity. When C, reaches 0.35 in Figure even the filtered data fails to maintain full feature purification: the number of neurons learning disentangled representations of all d = 50 features drops significantly (Figure [ [T)),, and the SS (Figure (()) and classífication accuracy (Figure ()) both decline sharply. This highlights that feature purity-the extent to which each neuron aligns to a single semantic direction- is a key bottleneck in contrastive pretraining and downstream generalization. We provide extra results in Appendix A.I
5.2 Experiments on Practical Data and Models
LaCLIP improves generalization over CLIP via recaption. Tables 2 compare CLIP 337 and LaCLIP [13, which share the same architecture and datasets, except LaCLIP replaces part of the original captions with LLM-generated rewrites. "CC12M CLIP" denotes a CLIP model pretrained on raw CC12M M. while "CC12M LaCLIP" uses the same model and data but with LLM-rewritten captions. Other models are obtained similarly using RedCaps [2] and LAION 40 datasets. We evaluate their zero-shot classification accuracy and Silhouette Scores various downstream datasets. LaCLIP generally outperforms CLIP in both metrics, empirically validating that higher-quality captions improve zero-shot generalization. Additional ImageNet results are reported in Table 3 of Appendix A.2 Next, we study the feature purity using a CLIP model pretrained on CC3M 41 Both the image and text encoders are 12- 12-layer transformers that produce features in IR 708 which are subsequently projected into shared embedding space of R512 through final linear projection layers, as illustrated in [ Appendix A.2 The final linear projection layer has 5 512 neurons and is functionally aligned with V in our theoretical model. We now present two key findings from this setting:
Purified neurons enhance generalization. To investigate the effect of feature purity on general- ization, we prune the neurons in the final linear layer in different ways and evaluate the resulting zero-shot classification performance. Specifically, we rank the 512 neurons by their average pairwise absolute cosine similarity to all other neurons, from lowest to highest. The absolute cosine similarity of neurons Uj. By is computed as K15, l//l//le, I for all j.J' (1,2..... 512). A lower average indicates higher feature purity (i.e., more orthogonal representations), while a higher value suggests feature mixing. We evaluate three pruning strategies: (I) retaining figh-purity neurons, i.e., with lowest similarity, (2) retaining low-purity neurons, i.e., with highest similarity, and (3) retaining a random subset of neurons. The number of retained neurons is varied from 200 to 500. As shown in Figure 2 (a-c,c-g), downstream performance is the best when retaining high-purity neurons,

---

followed by random selection, with low low-purity neurons performing the worst. These results highlight the critical role of purified features in downstream generalization.
Data misalignment reduces feature purity. To study how image-text misalignment affects feature purity, randomly shuffling texts across image-text pairs in CC3M with probability C.n as illustrated in Figure of Appendix AA.2) thereby introducing a controlled probability of modality misalignment. We then use the shuffled dataset fine-tune the last linear projection layer only of the pretrained CLIP model, freezing other layers. We then compute the cosine similarities of all 512 neuron weight vectors V/ T6768 of the fine-tuned model. Figure 2 (d) reports the average absolute cosine similarity of all neuron pairs, while (h) presents a histogram of cosine similarity (Dz, / ||y One can see that as C. increases, the average absolute cosine similarity increases, and the neurons become less orthogonal to each other and tend to encode mixed representations, resulting in lower feature purity. This coincides with the decreases classification accuracy in downstream tasks, as shown in Table Appendix A.2
F00010, Arsrery
CAlle-ie Accoray
0.45
Buemnc (00, ait--00
9:12
0.10
0.450
0.30
0.425
0.35
0.00
High Purtyy
Bandom
0.400
High Purtly
Hit hurty
0.07
Puttyy
0.375
Purlly
Law Purity
0.08
são
300
300
500
0.0
02
0.4
0.8
03
Number Selectid Nourons
Number Sclected Ncurors
Selected houtons
Shufting ProbabilityCC Co
Food-101 Acc.
CIFAR-IO Acc.
(c) Caltech-101 Acc.
(d) Avg. Cos. Sim.
O15810 Sibe Sibeuerte Sooee
Caulcorong Sihouette Spot
0-3300
0.10
100010 Sthovett
0.042
0.1275
0.0
0.036
0.00
-0.005
0.014
0.1225
0.04
-0.053
6.3200
0.22
0.012
350
500
300
300
1.0
-55
0.0
05
1.0
(e) Food-101 SS
(1) CIFAR-10 SS
(g) Caltech- 101 SS
(h) Hist of Cos. Sim.
Figure 2: Left (a-c,e-g): Retaining high-purity neurons outperform random and low-purity neurons in downstream tasks. More datasets shown in Figure Right(d.h): When Cm increases, the neurons have higher cosine similarity and reduced feature purity.
6 Conclusion
This paper provides a theoretical analysis of contrastive learning with nonlinear networks, linking training dynamics to generalization. We identify feature purity as central to generalization and show that text recaptioning enhances purity and zero-shot performance. The theory is empirically validated on benchmarks. Future work includes extending to Transformer models and tasks like retrieval and visual question answering.
Acknowledgments
This work was supported by National Science Foundation (NSF) #2430223, Army Research Office (ARO) W91 1NF-25-1-0020, and the Rensselaer-IBe IBM Future of Computing Research Collaboration (htt:://arcc.pp.eedu)) The work of Shuai Zhang supported by NSF #2349879. We also thank all anonymous reviewers for their constructive comments.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances neural information processing systems, 35:23716-23728 2022.
10

---

[2] Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 977-988. IEEE, 2022.
[3] Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. Cont: Contrastive neural text generation. Advances in Neural Information Processing Systems, 35:2197-2210, 2022.
[4] Sanjeev Arora, Rong Ge, and Ankur Moitra. New algorithms for learning incoherent and overcomplete dictionaries, In Conference on Learning Theory, pages 779-806. PMLR, 2014.
[5] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent vari- able model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
[6] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
[7] Boaz Barak, Jonathan A Kelner, and David Steurer. Dictionary learning and tensor decom- position via the sum-of-square method. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 143-151, 2015.
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 3558-3568, 2021.
[9] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[10] Zixiang Chen, Yihe Deng. Yuanzhi Li, and Quanquan Gu. Understanding transferable rep- resentation learning and zero-shot transfer in clip. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 2024.
[11] Mingyang Deng, Lucas Tao, and Joe Benton. Measuring feature sparsity in language models. arXiv preprint arXiv:2310.0783/ 2023.
[12] Karan Desai and Justin Johnson. Redcaps: Web-curated image-text data for robust image captioning. In NeurIPS, 2021.
[13] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544-35575. 2023.
[14] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference on machine learning, pages 399 406, 2010.
[15] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
[16] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up languae pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980-17989, 2022.
[17] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances Neural Information Processing Systems, 34: 14:10094410999 10956, 2021.
[18] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

---

[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yum-Hsuar Sung. Zhen Li, and Tom Duerig. Scaling up visual and Aasonnlanguage representation learning with noisy text supervision. In International conference on machine learning, pages PMIR, 2021.
[20] Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, and Sangdoo Yun. Hype: Hyperbolic entailment filtering for underspecified images and texts. In European Conference on Computer Vision, pages 247-265. Springer, 2025.
[21] Jason D Lee. Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable supervieee learning. Advances in Neural Information Processing Systems, 34:309-323, 2021.
[22] Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of clip. arXiv preprini arXiv:2411. 14517, 2024.
[23] Hongkang Li, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, and Meng Wang. Understanding mamba in in-context learning with outliers: A theoretical generalization analysis. In High- dimensional Learning Dynamics 2025, 2025.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[26] Junnan Li. Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-999 9705, 2021.
[27] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[28] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[29] Xuechen Li, Tengyu Ma, and Sanjeev Arora. Identifiability results for multimodal contrastive learning. In International Conference on Learning Representations (ICLR), 2023.
[30] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2022.
[31] Yuanzhi Li and Yingyu Liang. Learning overparameterizee neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems, 31, 2018.
[32] Yujie Mo, Zhihe Lu, Runpeng Yu, Xiaofeng Zhu, and Xinchao Wang. Revisiting self-supervised heterogeneous graph learning from spectral clustering perspective. Advances in Neural Infor- mation Processing Systems, 1344133--31633 2024.
[33] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pages 334884380 PMLR, 2023.
[34] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Im- proving multimodal datasets with image captioning. Advances Neural Information Processing Systems, 36, 2024.
12

---

[35] Divyansh Pareek, Sewoong Oh, and Simon Shaolei Du Understanding the gain from data filtering multimodal contrastive learning. In The Thirty Annual Conference on Neural Information Processing Systems.
[36] Victor Prokhorov, Yingzhen Li, Ehsan Shareghi, and Nigel Collier. Learning sparse sentence encoding without supervision: An exploration of sparsity in variational autoencoders. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-2021) pages 34-46, Online, 2021. Association for Computational Linguistics.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[38] Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leverag- ing large language models for enriched fused image captions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5689-5700, 5700, 2024.
[39] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pages 19250-19286.
PMLR, 2022.
[40] Christoph Schuhmann, Robert Beaumont, Richard Vencu, et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021. arXiv preprint arXiv:2111.0111
[41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018
[42] Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International conference on machine learning, pages 19847-19878. PMLR, 2022.
[43] Jiawei Sun, Hongkang Li, and Meng Wang. Theoretical learning performance of graph neural networks: The impact of jumping connections and layer-wise sparsification. arXiv preprint arXiv:2507.05533, 2025.
[44] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578 2020.
[45] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-tex. transformer for vision and language. arXiv preprint arXiv: 14100, 2022.
[46] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sen tequeeneec learning framework. In International conference on machine learning, pages 33318-23340 PMLR, 2022.
[47] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 16070-16080, 2020.
[48] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign lan- guage: Beit pretraining for vision and Languae tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[49] Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, and Simon Shaolei Du. Cliploss and nose-based data selection methods for multimodal contrastive learning, arXiv preprint arXiv.2405.19547 2024.
13

---

[50] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai. Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint 0rXiv:2108.1000 2021.
[51] Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pages 11112-11122. PMLR, 2021.
[52] Pan Xiao, Peijie Qiu, Sung Min Ha, Abdalla Bani, Shuang Zhou, and Aristeidis Sotiras. Sc-vae: Sparse coding-based variational autoencoder with learned ista. Pattern Recognition, 161:111187, 2025.
[53] Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference on Machine Learning, pages 24851-2487 PMLR, 2022.
[54] Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conference on computer vision and pattern recognition, pages 1794-1801, IEEE, 2009.
[55] Meng Yang. Lei Zhang, Jian Yang, and David Zhang. Robust sparse coding for face recognition. In CVPR 201 1, pages 625-633 IEEE, 2011.
[56] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
[57] Chunlin Yu, Ye Shi, and Jingya Wang. Contextually affinitive neighborhood refinery for deep clustering. Advances in Neural Information Processing Systems, 36:5778-5790, 2023.
[58] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 27X5v:200..01917, 2022.
[59] Amir Zadeh, Paul Pu Liang, and Louis Louis-Philippe Morency. Foundations of multimodal learning. Information Fusion, 64:188-193, 193,8,9,9200 2020.
[60] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In International Conference on Machine Learning, pages 41677-41693. PMLR, 2023.
[61] S. Zhang, M. Wang. P.-Y. Chen, Liu, S. Lu, and M. Liu. Joint edge-model sparse learning is provably efficient for graph neural networks. In The Eleventh International Conference on Learning Representations, 2023.
[62] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre- treiiiaining for time series via time- frequency consistency. Advances in neural information processing systems, 35:3988 4003, 2022.
[63] Kai Zhong, Zhao Song, Prateek Jain, Peter L Bartlett, and Inderjit Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International conference on machine learning, pages 4140-4149. PMLR, 2017.
14

---

The overall structure of the appendix is as follows. Each appendix provides supplementary information that supports the main content of this document but not included in the main body maintain clarity and flow.
Appendix A: Extra Experiments Additional experiments including both synthetic simulations and CLIP/LaCLIP evaluations on omitted datasets.
A.1 Extra Simulated Experiment
Complements Section 5.1 with further analysis of neuron behavior trained simulated data.
-.2 Extra CLIPAL Experiment
Complements Section 5.2 by evaluating on datasets omitted due to space.
Appendix B: Preliminaries
Mathematical preliminaries and notation used throughout the paper. A proof sketch is also provided outline the key ideas behind the main results.
Appendix C: Technical Lemmas
Full statements and proofs of supporting lemmas used in the theoretical analysis.
Appendix D-J: Proofs and Theoretical Analysis
Appendix D-F: ITCP Raw Data (Phase I-III)
Theoretical proof of ITCP across three training phases on raw data.
Appendix G: Captioning
Theoretical proof of reception using high quality data.
Appendix H: Filtering
Theoretical proof of filtering noisy caption-text pairs.
AAppendix ITCP on Synthetic (Recaptioned) Data
Theoretical proof of training dynamics when using synthetic recaptions.
Appendix J: Downstream Task Evaluation
Theoretical implications for performance on downstream tasks.
Checklist
15

---

A Extra Experiment
All experiments were conducted on an internal compute cluster using 8 NVIDIA A5000 GPUs with 24 GB memory each, and each run completed within 50 GPU-hours. No large-scale pretraining or resource-intensive tuning was performed beyond the reported experiments.
A.1 Extra Simulated Experiment
This section extends the analysis in Section 5.1 by providing additional simulated experiments on neuron behavior under synthetic data training.
Neurons trained on filtered data exhibit a more concentrated distribution. Figure 3 visualizes the histograms of (e; H and Kū, ü for all [m]] and E [d]. The values of (o,, H / are more concentrated, typically around 0.05 and 0.7. In contrast, the values
for (0;, H | indicates that for every H,. certain neurons i V predominately learns Hj. In such cases, (W), H approaches 1, while Kū, H //lll,lll approaches 0 for j' * j. The concentrated values of 0.05 and 0.7 observed in Figure 3 are due to noise the data. contrast, feature alignment is less significant for V. leading less concentration of the corresponding values. Similar results are obtained for image encoder (w,, M, deferred to Figure
are less concentrated. This phenomenon is consistent with Theorem 4.4 which
0.12
Rew Data
Raw
0.10
Synthetic
0.0025
20.08
0.
0.0020
3.0015
0.00
0.02
0.0005
0.0
0.0
0.1
0.2
0.3
0.0000
0.3
0.4
0.5
0.7
Figure 3: Histogram of (e,, || for ITCP raw data and (,;, H,),/||o.l| for ITCP on filtered
data (split into two figures to highlight the significant differences in the value distributions).
Systhetc Dat3
0.001.25
0.00
0.00190
0.00
0. 0.00075
0.04
9. 0.00000
0.02
0.80025
0.00
Figure Histogram of (w;, MM,)///0 for ITCP on raw data and (xū M,) /w; for ITCP on filtered data (split into two figures to highlight the significant differences in the value distributions).
Enhanced class separation of downstream tasks by ITCP with recaptioned data. Figure visualizes the t-distributed stochastic neighbor embedding (t-SNE) of the feature embeddings gen- erated by the two models, computed as (I,) and fw (Ip) for each Tp, respectively, The t-SNE method projects the high-dimensional embeddings onto a two-dimensional map. One can see that embeddings from different groups are more distinctly separated in the model trained using ITCP on recaptioned data, indicating that this approach achieves better feature alignment.
A.2 Extra Experiment on CLIP and LaCLIP
To complement the results in Section 5.2 we report additional experiments on CLIP and LaCLIP using datasets omitted from the main text due to space constraints.
ImageNet Results. The LaCLIP variants consistently surpass their CLIP counterparts both Top-1 and Top-5 accuracy. Higher silhouette scores further indicate cleaner feature separation after recaptioning, line with our theoretical predictions.
CLIP architecture. Figure illustraaes the CLIP architecture used in our experiments. Both image and text inputs are independently encoded by 12-layer transformer backbones, each producing
16

---

(a) Raw (C, 0.3) (b) Recaption (C, 0.3)
(c) Raw (C, = 0.5) (d) Recaption (C, 0.5)
Figure 5: t-SNE visualization of text embedding with spurious correlation probability C..
Table 3: Comparison of CLIP and LaCLIP on ImageNet: Top-1 (%). Top-5 (%), and Silhouette Score.
Model
Top-1 (%)
Top-5 (%)
Silhouette
CC12M CLIP
35.04
62.10
-0.014639
CC12M LaCLIP
42.62
70.17
-0.008141
LAION-400M CLIP
58.34
84.73
-0.029893
LAION-400M LaCLIP
62.27
86.34
-0.056593
RedCaps CLIP
37.66
63.31
-0.022045
RedCaps LaCLIP
39.66
66.06
-0.012299
768-dimensional feature vector. These features are then projected into shared 51 512-dimensional embedding space through learned linear projection matrices W E 7668x112 and V TR 768x512 corresponding the image and text encoders in theorem, defined in Eq. (2). The resulting embeddings are aligned via a contrastive loss that maximizes similarity for matched image-text pairs while minimizing similarity for unmatched pairs. This architecture forms the foundation for our analyses on neuron selection and feature purity in the shared embedding space.
12-Layer Transformer Output Dim: Ro0
Lineer Projection
Image Embedding R012
Image
W R7001512
Imege-Text Contreefive Loes
12-Layer Transformer Output Dim: R700
Linear Projection V 723x-112
Text Embedding
Text Data
R512
Figure 6: Architecture of CLIP used in our experiments. Both image and text encoders are 12-layer transformers that output features in R. 768 which then projected into shared 1 embedding space via final linear projection layers W and V, corresponding to Eq. ② and Eq. (3) our theoretical analysis. Contrastive loss is computed between the resulting image and text embeddings.
Simulating Modality Misalignment via Caption Shuffling. Figure iillustrates how modality mis- alignment is introduced by randomly shuffling text captions across image-text pairs with probability Om. resulting in noisy supervision for contrastive learning.
Purified neuron selection enhances generalization. Figure presents additional experimental results on CIFAR-100. Pets, and STL-10. complementing the main results reported in Figure 2 Due to space constraints, we include only Food-101, CIFAR- 10, and Caltech-101 in the main text. All experiments follow the same protocol, evaluating zero-shot classification accuracy and Silhouette Score under different neuron selection strategies. These results consistently support our core finding: selecting high-purity neurons leads to improved downstream performance across diverse datasets.
Higher shuffling probability leads to reduced generalization and feature purity. Table additional experimental results on CLIP models finetuned with different levels of randomly shuffling probability C,,. simulate spurious correlation, showing that both accuracy and Silhouette Score consistently decrease as C'm increases.
17

---

Oriiial Algnee Fairs
Imagee
Image
Text
Ra rolon ahufling g cxptian with probabiitty Upp
Misalgnne
Image
Image
Image
Image
Image
Text S
Tixei
Twxt
Text
Trxt
Figure 7: Simulating Modality Misalignment via Caption Shuffling. Starting from original aligned image-text pairs, controlled probability Cm of misalignment is introduced by randomly shuffling the text captions. This results in noisy pairs that reflect varying levels of spurious correlations.
0.20
0.80
OPARLOO C Accurscy
0.18
PETS Accurety
0.78-
0.32
0.76-
0.18
0.31
STL10
0.74
0.14
Random Purity
Random High
0.72
Random
Purily
Low Purity
Low Purtyy
0.70
Purty
0.12
200
300
400
500
300
400
500
Selected Neurons
Number Neurons
Selected
C1FAR-100 Acc.
(b) Pets Acc.
(c) STL-10 Acc.
CIFARLOO Sihcuette Score
PETS Sihowette Score
0.0000
ST.10 Silhauctte score
0.114-
-0.0025-
0.112-
-0.0050
0.110
Random
-0.0075-
Random
0.108-
Random
High Purity
High Purity
Purity
-0.055
Low Purity
-0.0100
Low Purity
0.106-
Low Purity
300
Number of Selected Neurons
Number Selected Nourons
Mumber of Selected Neurons
(d) CIFAR-100 SS
(c) Pets SS
(f) STL-10 SS
Figure 8: Zero-shot classification accuracy (top) and Silhouette Score (bottom) under different neuron selection strategies for CIFAR-100, Pets, and STL-10 datasets.
B
Preliminaries
We first restate some important notations used in the Appendix, which are summarized in Table 5
B.1 Proof Scratch
Theorem is proven by integrating the convergence analyses Appendix F and Appendix Appendix Flestablishes convergence for ITCP raw data, while Appendix (eeetendss the convergence result to ITCP on synthetic data. Together, they verify that SGD with ReLU networks achieves near-optimal contrastive loss on both datasets.
18

---

Table 4: Accuracy (%) and Silhouette Score of CLIP models finetuned with varying C. on six
datasets.
Datasct
Cm 0
Cm
0.1
Cm
0.3
C.
0.5
C.
0.8
Acc
SS
Acc
SS
Acc
SS
Acc
SS
Acc
SS
Caltech101
59.7
0.160
48.2
0.124
47.9
0.121
43.6
0.117
44.5
0.115
CIFAR-10
57.9
0.030
50.7
0.012
49.5
0.013
46.5
0.013
44.1
0.011
CIFAR-100
26.4
-0.038
19.5
-0.042
17.8
-0.043
17.4
-0.044
16.2
-0.048
Food-101
12.9
- 0.073
10.9
0.052
10.9
- -0.06
11.1
-0.057
11.1
- -0.059
Pets
13.9
-0.005
13.3
0.006
13.2
- -0.099
13.4
00.011
12.6
-0.012
STL-10
86.3
0.164
79.8
0.103
79.2
0.102
78.8
0.100
78.3
0.097
Table 5: Summary of Notations
Notations
Annotation
M Bdixd xd. H E Rd: xd
M is the image dictionary matrix, H is the text dictionary matrix.
W mxxd V Rmxd
W is the weight of image encoder, V is the weight of text encoder.
Tp E (Rd,, Bp E Rd
Tp and Up represent an image and a text data, respectively.
Exy; Sup E R"
Ex, and zy, are the sparse signals of image and text, respectively. Fym the sparse signal for the text prompt yk-
zyy Zbpp
2 is the j-th coordinate of Zup; 2/1 is the j-th coordinate of zys-
L, LC
L is the loss for ITCP; La is the loss for Image-grounded Text De- coding.
S= SS S.w
S., is the noisy web low-quality dataset; S₁ is the human-annotated high-quality dataset.
$= Sh U.S.
S. replaces noisy captions in Sw, with synthetic captions.
Phase of ITCP with 66") =
Ti
Phase II of ITCP with 01 (1+11 (1 g)
T2
Phase III of ITCP with b(++1) b ((T))
T3s
Tc
Stage of training caption generators.
S
The set of well-initialized neurons (w,, 0z) on features (M,, Hy).
Theorem 4.2 is proven across Appendix D Appendix E and Appendix F. Specifically, Appendix D models Phase I training (t < Ti) and proves that neurons simultaneously align with true features and spuriously correlated features due to comparable gradient contributions, preventing pure feature separation. Appendix EE analyzes Phase II training (Ti < S T2) and shows that this spurious alignment continues to strengthen, as neurons with initial mixed alignment further amplify their entanglement during continued SGD updates. Appendix F establishes the convergence behavior during Phase III (T2 < t < Tz), showing that the network stabilizes into mixed solutions where each neuron represents combination of multiple features. These detailed stages collectively prove the failure of purified feature alignment as formalized in Thcorem
Theorem 4.3 is proven across Appendix Gnd Appendix H Specifically, Appendix G analyzes the captioning stage, where the decoder is fine-tuned on clean data to generate synthetic captions. proves that for neurons aligned with true features, the aligmment towards the true features grows exponentially while the alignment towards spurious features remains negligible. This ensures that the synthetic captions preserve relevant features and suppress spurious ones. Appendix H then formalizes the filtering process, demonstrating that after replacing noisy captions with synthetic ones, the resulting dataset satisfies much stronger feature purity conditions, with spurious correlations
19

---

suppressed to (1/d) and true features preserved with probability 1 - (1/d)) These results directly
support the purified feature learning described in Theorem 4.3
Theorem 4.4 is proven in Appendix [ which integrates the proofs of Phase I, Phase II, and Phase III for ITCP synthetic data. Specifically, Appendix [ first establishes in Phase I that purified training pairs allow neurons aligned with true features to grow exponentially without spurious interference. It then shows in Phase II that these alignments continue to strengthen while suppressing non-informative neurons, leading to clear feature separation. Finally, it proves in Phase III that the model converges, achieving a bounded final loss and dominant true feature alignment. Since the overall proof structure closely mirrors that of Theorem 4.2 (which was proven separately across Appendix [ Appendix and Appendix F). we consolidate all stages into a single appendix for brevity and clanty.
Theorem 4.5 is proven in Appendix J which analyzes the downstream zero-shot classification. Appendix shows that for ITCP raw data, spurious features cause a constant classification error, while for ITCP on synthetic data, true and spurious features become separable with high probability, leading an (1) error rate. This directly supports the main text conclusion on downstream generalization.
B.2 Feature Coupling and Expected Values in Sw
The following Assumption B.I corresponds to the more specific forms of Assumptions 3.2 and 3.3 discussed carlier.
Assumption B.1 (High and low quality pairs). The high-quality image-texi pairs in Sy have size |Sa| = (d))). The low-quality image-text pairs in S. have size [Swl = poly(d) > w(d)).
In Sn, for positive pair (Tip Bp we assume perfect alignment, meaning Zup "yy- Consequently, the following holds:
E
C2 d
E 2
2 - -
)
1 dc
3 }j
(17)
To model the misaligned features in low- low-quality pairs in Sw. where spurious misalignment occurs at a non-negligi level, we assume [d] can be divided into d/2 disjoint sets, each containing exactly two entries. Let (j.j') [d] denote one such set, referred to "spuriously correlated set." The following assumptions capture the nature of spurious and true alignments:
Pr(1= = 12 = 1) 0(1 2'
(18)
Pr(1=[]] = |= 1) Pr(1= = 12 1)
These assumptions imply that true alignment dominates, with 3p spurious alignment exists constant percentage level, making it non-negligible. The intuition behind this assumption is that each feature is paired with exactly one spuriously correlated feature j'. ensuring that j is not associated with any other feature j" * j'. This design simplifies the analysis while effectively capturing the key challenges posed by low-quality data.
- 11 - while
Then, for positive pair (Ipz Up) with in Sic. we have:
C2
E
2
22
Up
=
(19)
C2
E
9p
= θ
<
2d
where (J,j") is spuriously correlated set. For negative pairs (Ip, ny), where p + q. and p. 9 E S. we have:
1 7P
E
θ
Vj.j' [d].
(20)
34
In Sy, mismatched text and image pairs are prevalent compared to Sp. For a postive pair (zp: Up), we = 1
assume 2ogg d Ing(1/ro) < Pr((21
| = 1) < 1. To model this, assume that for each primary
20

---

feature j E [d], there exists exactly one spurious feature j' such that j and j' are uniquely coupled. This implies that j cannot be associated with any other feature j" * Mathematically, the coupling is defined as:
Pr(1= 3y
| = Pr(>, |||
1=,,-=1)=
(21)
For a positive pair (Ip, Mp) in Sw, the probabilities of spurious and aligned features are further constrained:
log(1/co) < - -1 1
< 2
(22)
2log d
The lower bound is established in Lemma C.8
and:
Pr( = |2 =) = - Pr( = - 11).
(23)
Under these assumptions, the expected values for the aligned and spurious features are calculated as follows:
For the aligned feature j. we have:
2 2x Jr
= Pr(1z} 8p = 1, 22, = 1)
= Pr[[2 |= =1
| = 1) Pr(
20 up
= 1)
(24)
C₂
Pr[[= - | = 1)
d
For the spurious feature j', we have:
j 2 2xp = Prr(lzz] = 1, 2 = 1)
E
= =Pr(1z// = 1 |24 = 1) Pr(1 1)
(25)
C.
= =Pr(|=]]|= = 1)
The total expected value across both aligned and spurious features satisfies:
C2
+ E
5' 2p
(26)
d
Here, j' denotes the spurious feature associated with j.
B.3 Gradient
The contrastive loss in vision-language models (VLM) is defined follows:
((f(0,,99)) Σ
(f (xx), (2b))
Σ
27
PES
(27)
(CF0 (1))))))
2T
where 7 > 0 is a temperature parameter.
We perform stochastic gradient descent (SGD) on this contrastive loss. Let f(() and (t) be the image encoder and text encoder networks at iteration respectively. The network parameters are updated as follows:
w (++1)
w." (1) 1f V ww ((f(t),)( 1,0),
(28)
2
(++1)
1i (t) L( (0),,
(29)
where 66") the bias term, is manually tuned during training and thus excluded from gradient updates.
21

---

The gradient of L(MO), 10(0) with respect to w/ at iteration is given by:
V L(f ,1')) 0 8p)a, 1 Kkmf0 -,) (00" w2)
(f (xn), (( (Bp)) Bp)
1 (cos0 and) 20000 mmi//2000
1 1:0500 2001
7
E9t
V..D. (xp), h (v,))
2 (wof") =0)/2660
1 ((00)) (nn) 226
(30)
Similarly, the empirical gradient of L(slo, h() with respect to of is:
V (001,,1000 = - (w: xp)Upp : K (200)/>)2200)) 1 "K0000 (000))0)22000
(2() (xp)))))) (1o?") an)8e 1 ((ol") (nn) 2000) "|K000 mo) 12000
7
T' Em
(xp),, (1n))) (mu -xplUn
+
(ao! (oot) =2)//0000 1 1/of0 Koftr (n) 22)
E
(31)
B.4 Alignment Updates
We analyze how each neuron i E [m]] aligns with the feature Mj during each iteration of SGD, The alignment can be described by the following update rule:
(w ((+1) M2) = (w) (1) M2) - Vw, L(Ftt, ). M,) (0) (w; (1) M₁) + ( H3) + n= (vf H2) + Err.
(32)
Similarly, for (o) (++1)
Hy). the update rule becomes:
(v) H₂) (u) H V L(f ) HJ)
(++1)
(33)
(0) H + (w M2) + nh (w) (1)
M2y) Err
Using Lemma C.6. we know that with high probability, E., en (aw), (spll I 0(2), so in Eq 30 the sum of second term and third term always less than the first term, until (T,), h (1) (1,)) = (d).
The updates for the components (wo(++)), M2). (of (o((+1) H,), (w((-"),, My and (0(+t)) Hyy) (where j' represents the spurious aligned feature corresponding to j) can be expressed concisely in matrix form as follows:
(tt
M,)
b
(w (t)
M,)
(ofet2) H
b
(u(0) H2)
+ Errr.
(34)
(w) (++1) My))
0 c
a
(w M2y
(1. (1t)) Hyy)
0
b
(o Hyy)
where the coefficients are defined as:
1, b= 2/ 1 10000 (z,))
12:00 110600 mn) 2200
21 20
1 ((m(")=2))220 22) (of
w. 20000
22

---

Therefore, we have
(wg (w;",,,) (0/ H2)
(a + b + c)
+
(a + bb-b- c)c
(<w:)) M2) (v:0) HH))
4
(35)
+ b e)c
(a b 0)
(wm My)) + (m) H,,)
4
and
(a b +c+ c)c
+ (a bcc) (Lu!) MMy) + (0 Hyy) (a + b e)¹ (wm) M2) + ( H2)
(uy (og",H,) (a +cb
(36)
4
This matrix representation highlights the interactions between the alignment of true and spurious features during SGD updates. The diagonal elements a dominate the contribution from existing alignments, while the off- off-diagonal terms b, c capture the mutual influence between paired features and spurious alignments. Note that if is very small, indicates that the spurious alignment (7") has minimal influence, allowing W; focus on learning purified features. Conversely, if e is large, the spurious alignment could significantly interfere with the learning process, hindering the purification of features. The error term Err, accounts for higher-order noise or unmodeled effects in the update process.
Assuming single spurious feature is simplification for presentation that was made for case of presentation in the proof and can be extended to a more general setting without altering the underlying insights. If each feature has 1 spurious correlates, and N; = j. in the last sentence of Theorem 4.2 contains j and other K-1 features. Our analysis relies the total spurious feature probability (bounded by Cs). the number of correlated features, so as long as the sum of all spurious feature probabílities is upper bounded by Cs, the core mechanism and insights of the theorem remain unchanged.
becomes 2K 2K matrix,
C Technical Lemmas
Definition C.1 (Neuron Characterization).. Let us define a few notations to characterize each neuron w(th) behavior. For every constant co (0, 1) and 7 (0, 0.1), by choosing c1 2 + 2(1 7)co and C2 = -ycu, we define:
1. Let j.aure S [m]] be those neurons i E [m] satisfying
2 Ei (m0)) .M,))² 2 (og MM T w (2)
( En (w (c) M2,))² (a -2) log
logd
MM T w
(1 1|2
Let Siip S [m]] be those neurons E [m] satisfying
(w(²) M2 [oge MM 1
Lemma C.1 (Geometry at initialization). We initialize the parameters by (0 ~ oN(,, where
of
Aspoty(a)) ). We have with probability 2 o(1/d²) over the random initialization, for all
j E [d]:
((0)
Ω (d3co) =2)(
E
J.aure
(0)
50 (cP300)
=: Ez
J
Proof. If is standard Gaussian, then for every t > 0,
1
(t)
2/12
Pr
Ig >
1 -212
(37)
/2T
12
+1
g0
V2T (t)
23

---

We initialize the parameters by 0
of ) where of 0 ( )
We have
ΣΣ (w (0) M.) N (0.
Therefore, for every E and j E d, we have
= Pr
1
Σ w 00 M₁}
2 (ai
+ σu
log
n
(38)
θ
log d
dlc)t02/22
θ
Vlogd
d d31-1/2)co
2 w M₁))
P2 Pr
2 (cl
22) n
log
1
(39)
log d
d o22
Vlogd
d. d(1-33/2)c0
Let s(0)
j.sure S [m] those neurons i [m]] satisfying
( EE (w: (0)
M2))²² 2 tert d
MM wi (0)
12
d
2 Er -
M M₁))2
(c) do
log
MM
(0) Wc
By concentration with respect to all m choices of [m.] we know with probability at least 1-o (#)
satisfies ( Ω (d
(0) Let
j.p S [m]] be those neurons i E [m]] satisfying
(w) 22 (or-
|MM
W.
(0)
By concentration with respect to all m choices of i [m.] we know with probability at least 1-o (++
it satisfies
((0) j.pot
O (d2roo).
More details of the proof can be found in Lemma B.2 2
Lemma C.2. With high probability 1 polytaj for every E [m]] the following holds:
Pr
(0)
M,) (w; (0)
M2
>
1 1
log d
2 1 - Ta
(40)
2m
d 2n
Lemma C.3. With high probability 1 poly(d) for every i E [m], the following holds:
|MM T W 112 + HH
(0)
1 2do3
[ 1- (V (-) 1 1 (111
1
1
(41)
Proof. Let X ~ 2 By standard properties of the chi-squared distribution, we know that with probability least 1 5,
X - nl < 22 log(1/8).
(42)
24

---

In our case, we consider |MM " 113+ HHH 12
X2.- Setting ò poly(a)) we have = 2d, and
thus, with high probability d)l(d))
the following holds:
|MM
+
2d S 2 ld
(43)
0
Rearranging and incorporating the scaling factor oi we get:
[ 1 ( ( (1:+)))) 1 v7 (
(0) 1 |MM T w, 2 + HH vi (0)
2do9
(44)
Lemma C.4 (Noise Projection Bound). For the spurious dense noise Ex, ~ ~N where the variance satisfies
+ oe 0 (2), the following holds with high probability D(dr);
1w:l1 11
kw,, E)12 SO
Vi [m].
(45)
d1000
Proof.
For all j E [di], by the properties of the Gaussian distribution, we have:
Pr
MM <)² 00 di
2 1 -a(d) -(d))
(46)
Now, consider the term 1(ws 2. We decompose it as:
Kws, 1)
Σ
Kw,, M,) (M,
E Kws, My), 1(M),
(47)
jE
JE
For the first term, since {M,, s 0 (guice) with high probability, have:
Kw,, M,)
Kw,, M,) (M, jE(d
Σ
Σ O
(48)
d dl
jE(d]
Similarly, for the second term:
Kus,, Kus,, M dl+co
Σ
o
(49)
jEL
JEM
Combining these, we have:
||M will
|M-M' M wil13
(1, 3 50
(50)
dl+cc
di++o
Since |MM w + M M
w
1|w,l12. we conclude:
[ di+es
2
((,,, E S
(51)
Thus, the lemma holds.
Lemma C.5 (Tail Bound for Matrix Product). Let Q E Raxa be symmetric matrix, and let w, v be independent zero-mean Gaussian random vectors with covariance matrix I,,, Define
Z:=
Qiywey..
(52)
Then, for any 6 > 0, the following kail bound holds:
82
Pr[[ 2 8) < 4exp
(53)
Q + 45 Q
25

---

Lemma C.6 (Bound Inner Product). Consider the inner product between the feature vectors at initialization:
m 11
(f()), (Wr. Vo) W1
(1,,9,) 1b ) m--
(54)
1=1 6,=1
Q ry T, with [1 O |Q|| (1) and oū 0 Zypody(at
Here, using Lemma C.5
Then, at initialization (t 0), the following holds:
(z), (w))) 2 2(1) v -oly
(55)
Lemma C.7 (Concentration bound for empirical loss and gradients). There exist N 2 poly(d) for some sufficiently large polynomial and all [wall2 O(d), E [m] it satisfies
Σ PE[N]
LCM (
(A): DED.. 10;;; (xp, mbmii
S <012)
(56)
Σ Vw, L,,0), (1p, Hp)) - DED, w., L(S()) ,100 (xp, vp)) [E[N]
s0
(57)
Proof. The proof can be done by trivial VC dimension or Rademacher complexity arguments similarly to Lemma A.2. [2].
Lemma C.8 (Misalignment Probability Bound). The probability of spurious alignment satisfies:
log
2xca
Pr(1=//1== 1 <
(58)
2 log d
Proof. By concentration over all m choices i E [m]]. we find that with probability at least 1 1-o (#)) the number of neurons satisfying:
(c1 + 4c2) o
Σ wwr, M,)
log
(59)
is o(1).
In addition, for all neurons, have:
max (wu)
(Ti) (luf M,,))
C1 - 302 log
| (Ti) 113 +
|10 (T))
(T))
112
<
(60)
2
2
Define:
A (Ti) (a + b - Ti
(wu (0) M2) + (o/0), H,) (w/)) M,)) - (11! H.))
(61)
Thus:
(w(T) (Ti)
2 = max (wu (Ti) M2²) (wfT)),,,)) - A (Ti)
2 Ca C2 log d
(T
(T1)
+
||0
(Ti)
(62)
d
2
We begin by expressing a + e and a b + as functions of P. === =Pr((a/,) = || = 1) and
Pg = Pr(12g]11 = |2 = 1), where Pj + Ps = 1:
a b ny + (Pi P2)C2 log logd
(63)
26

---

(PI + P2)C2 log
a a+b+c= An
(64)
Using Eq 62 Eq
35
and Eq 36 we derive:
(a + b c)2r:
1
c1 + 302
C1 02
S 202
(65)
(a + b + c)2r
2
2
Substituting back, we find:
log
2750
P1
< 2
(66)
log d
For example, setting co -0.1, 7 = 0.005, 100, and di 10000, we calculate:
1
1
4
Pr([
v,
1 1) 2
(67)
This concludes the proof by bounding Pr(1= =
| = 1) under the given conditions.
D ITCP on Raw Data
In this section we analyze Phase I of ITCP on Raw Data as the training iterations t S Ti, where
Ti = θ dlogd is the iteration when all
|| (T1) 11
|l + 1l 1 When s Ti.
7
we set b/ = O. For every neuron i E [m]. the weights w; and e; exhibit an increase in alignment along the direction of informative features M and H, while showing negligible increase in alignment along the direction of noise features M and H
Based on subsection B.2, we have Pr 1 | = ((1), so E [2] and
2 both
in θ (1)- In this case, w (++1), is jointly influenced by M, and My, with both features contributing comparably to the updates.
To simplify our analysis, we consider the worse case where Pr(1 up
=
355 Eq (36 and b(0) 8p
|2|
= 1) = Pr(1 =
1 1 = 1) 1/2 such that E
[z] E
[ C so using Eq
0.
we have
(we MM,) (a + b++b+c) (ww M2)
.M,) + (v. (0)
H2) (w: (0)
My) + (u) H,y))
(68)
4
(w M₂)
(a b
(mw (0) ,,,) + (v:") (0) H2) + (w, 00) M,,) + (v) Hyy)
(69)
4
This represents the worst-case scenario the contributions of the aligned feature E [ x]] and the which would result in (1n((+11,, M,)) (we(+t) My)). However, in this corst-case scenario, the equality of contributions prevents the network from prioritizing purified features, resulting in equal
spurious feature 2 zy are identical. Under real circumstances, we expect [
=
((+1)
magnitudes for (11. (f=1) .M,) and (w: (++1) MMp), thereby hindering effective feature separation.
We first provide lower bound for ||MM' T w we have:
(t) [|2 for iterations t < t1. From Eq 122 and Eq 66
(a + b) ((n! M2) + (o) H₂))
b+ ((v/ .M2) + (v Hy,) )}
[MM w 12)
1
(1 1+
1/222 d
21 |MM (0) B HH
(0) 1
8
(70)
27

---

[M (M-)
w 1133 1+ poly(d))
(71)
The detailed proof of Eq 711 can be found in Hypothesis C.4 of [51].
A similar result holds for HHH 0.
112 and
H [H
Eq 770p and Eq (71) shows that the image and text dictionary features M, H can grow exponentially, while the noisy features M H remain almost unchanged when t S Ti.
For Mj where [di] \ [d], using Eq
(11. obtain:
( d
1 di
1
(T)
11
+ 110
(Ti)
(wi M so
ISO
(72)
This result demonstrates that the noisy features MJ experience nearly increase during this phase, remaining insignificant in their contribution to the alignment of wz:
D.1 Lower Bound of Alignment for i Syre
This section provides a analysis of the alignment growth for neurons i E Sy.uee- Specifically, we demonstrate that for every E [d]. i E Sy the alignment (My, "M,,,/").. (2 and its spurious alignment
(M', y increase exponentially when t t Ti.
We now prove the lower bound of (w()T)
(Ti) M2 for i E Sy,sure
C2
27:
w (0)
M,) + (0) Hyy) + (w, M2) + (0 H,)
Kw (T))
(T))
22
+
11
d
4
I/0
C2
27:
(q + 02)
log
|MM w/ 10) 12 HH (
+ 1)
d
1|e
(c) c2) log d
MM
w.
(T1)
11
|H
0
(Tr)
2
(ci + 02)
log
(Th)
12
110
(Ti)
12
10
10)
d
Co
ycu) log
(Th)
||0.
(Th)
d
2
(73)
In O we use Definition C.1
In V we Eq
70p. In we use
(
1
(T)
2 w
(0)
11 +
1|
00
(0)
1
In we use C1 + C2
2[1 + yco)
Similarly, Kw, (Ti) 21 have the same lower bound.
D.2 Upper Bound of Alignment for i Sj
In this subsection, we analyze the alignment of neuron i Sy,pot with the feature My and provide an upper bound for (w 22 While neurons i & Sj still exhibit exponential growth in their alignment, their weaker initialization results in significantly smaller alignment compared to neurons in sune limiting their contribution learning the feature M,.
28

---

To establish the bound, we begin with the following expression:
C2
27:
[ M2 (v H2)) (w, MM (0, (0)
(0)
H2)
2
(1. M2)
11
4
C2
27
(ar logl
||M wi
(0) 11
HH o
+ 17
(c1 02) log d
MM
wi (T1) 11 +
HH
2
(74)
Here, D, Lemma C.1
which captures the reduced alignment for neurons outside Sj
Similar to the analysis for i Sj,sre the alignment strength for & Sjypot is weaker, as ci 02 is less than 211 + co 7co), leading to:
Kw ( M2) (1+c0 3yco) logd
(T)) |u
11 + 1|0 (7))
(75)
d
This inequality highlights the slower alignment for newrons outside pot distinguishing their behavior from neurons in Sy.are- Consequently, & Sj contributes less significantly the alignment of M,, reinforcing the importance of initial affinity for effective alignment.
D.3 Summary
At this stage (1 < Ti), we do not consider the worst-case scenario where the probability bounds for feature coupling satisfy
log(1/co)
< Pr(1= |= < Pr Pr((= |=
=
=) <1 <1
2
(as assumed in SubSection B.2). Thus, summarize the results when t T, as follows:
For E Su.srres the alignment strength satisfies:
kw 22) Kuc (Ti) M2) (1 cu ycp) logd
(Ti)
Iu (Th)
+
1
(T))
(76)
d
2
where j' represents the corresponding spurious alignment feature.
2. For i & Sy,pa the alignment strength satisfies:
Kwy (T)) M2) (1 + co 3yc0) log d
||
(Th) 112 +
||0
(T)
(77)
For M where j E [di] \ [d]. we have:
(
(Ti)
12 + 110
(Tr)
(w (++1) 2 <0
(78)
2
These results demonstrate that when < Ti, all features in M increase, but the alignment for i Sj.ure, including the corresponding spurious alignment, grows significantly larger due to favorable initialization. In contrast, noisy features M remain unchanged.
E
ITCP on Raw Data II
The Phase of ITCP on Raw Data defined as the training iterations Tj < S T2, where
T2 Ti θ (log 7
At the beginning of this phase, we set the bias threshold as:
6 (T)
+ co
2yco) log d
(i)
12
+
10
(79)
d
2
29

---

During training, the bias threshold is iteratively updated as:
b((+1) = -1 3) 1,90 2 (d)) w 12
(80)
until all neurons satisfy:
1 (T2)
(81)
In this phase, the dynamics of alignment vary depending on whether neuron belongs to Siuur or not:
For i & Sj.pa: The weights wy and show negligible alignment growth with both the informative features M,. H, and the noise features M H This is due to their weaker initialization, as shown in Phase I, and the effect of the indicator function when / 2 Ti which prevents them from being activated. As result, their capacity to learn meaningful alignments during this phase is significantly limited.
This
For E Sy.sur: The weights wi and v, exhibit continued alignment growth with the informative features M, Hy. Additionally, their alignment with the corresponding spurious features My. y also increases due their strong initialization, as shown in Phase I, and the effect of the indicator function when t 2 Ti, which ensures they are always activated.
By the end of this stage = T2), the weights wa, 2s will predominantly focus on the features M,. H, if E Sy.urr. while largely ignoring the features M,. H, if i & Sj.pot. as well as the noise features M H This separation lays the foundation for the Phase II of ITCP on Raw Data, where spurious alignments are expected further diminish due to the dominance of true feature alignments.
Similarly to the proof of t S T1 To simplify our analysis, we still consider the worse case where Pr(12/] 2 1) Pr(1
| 1) such that E 2 2f
2
E.1 Alignment for i SJ.sure
This section provides a analysis of the alignment growth for neurons i E S Specifically, we demonstrate that for every [d], i E Sj.uee, the alignment (M,, w ( and its spurious alignment
(M', ) increase exponentially when Ti < s T2-
For Sj.srrs using Lemma C.4 the following holds with high probability 1 e0(d)) when T < L & 72:
w dltta
(w0 (u("9.)) .5)
2 So
(82)
Therefore, with high probability e (R(d1), using Eq 76) and Eq (9) the indicator function satisfies the condition when = Tı:
(Ka/)>>)\/2 220) 2 11 or \20 1,
(83)
we can ensure that:
C2
2 Kio" (0)"),,))2200 12000 Kof0 My w,>\20
=
d
(84)
Using Eq (166) we know that (1 24 > (1 + 2) and using Eq
34 we have
(w
(++1)
M, > (1 + " "00000 = ((+1))
(85)
This implies that when t Ti, the alignment strength of informative features surpasses the updated bias threshold t(t) Consequently, the indicator functions become consistently activated Ti < t < T2 such that
12))20(00
(140) 20p) 12000 =
(86)
30

---

Using Eq (34). the weight dynamics for (wu ((u((+1)
(1+1) _M,) can be expressed as when 71 <1 S T2:
(wu M2)1 (1+ 17 :aaa (wo (0) M2) + (of Hy.) + (w)) (2)
.M.) + (u." H,)) ) (87)
4
Similarly, Kw, (TI), (T), My have the same result.
E.2 Alignment for i & Sy.pot
In this section, we analyze the alignment behavior for neurons i & Sopp Specifically, we demonstrate that for every j E [d]. if i & S3.eu. the alignment (Mj, w) exhibits negligible growth during the interval Ti <t T--
For & Sj.pot using Eq 156). Eq (79) and Eq 76). we have with high probability - 10---((66 -(d1),
similarly to the proof of E Sj. sure: the indicator function satisfies the condition when / - Ti;
Kam 20) 220) Krt -0 120 0,
(88)
We ensure that:
1 d2
E
2/ (mm")=))) )200/ 1 K1500 mn) ((1 |2000
S
(89)
Using Eq (116) we know that (1 ( (1 + 9) and using Eq
34
have
kw (1+)) M,حi < (1+ + +++2765 200000 6: ((+11)
(90)
This implies that when t > Ti, the alignment strength of informative features does not surpass the updated bias threshold "!". Consequently, the indicator functions become consistently not activated Ti < S T2 such that
Kalf (uul)>))\200) KKt" a> |200 = O,
(9))
Using Eq (34.
the weight dynamies for (uo(tt2),, M,) can be expressed as when Ti < S T2:
M,) S (1+ (2)) (w (T)(T)) M2) + (U)T)) H,y) + (T( My) (T) H,)) Kw: (++1)
(92)
Because (1+ T2 < 1+ + o (2). the growth in (w:"") .M,)l negligible. Consequently, we have:
1 + (2) Kw, (1)) M,y1²
(uufTa) M2)1² v
(93)
E.3 Summary
When T2- θ ddoggd
we know (1+ 1 To poly(d). Using Eq 76), we can ensure that when
7
all neurons satisfy the following condition:
(Ta)
12 2 ((d)]
w
(Ti llz.
(94)
we terminate the training process at T2 (doogd) "
This ensures that the alignment has sufficiently
progressed for effective learning.
Thus, using Eq
(93) and Eq (7Tp we have
(u(Ta) M2 Kw (T2),
M2)) 2 = ||!!! (Ta)
(T)) M2
Σ {w) (2) .M,))²
jE( EN;
JE(d:]M
2 ||w! |2)
(1 + of 1
+ 01
(T)) 112 Kw! (T)) .,,) - 1( (T))
(Ti) 1
> (T)
|| (T> - ol
(95)
31

---

Thus, at this stage (I₁ < <t T2), we do not consider the worst-case scenario where the probability bounds for feature coupling satisty
log(1/co) log d
< Pr(( = | = 1) < < Pr( 2 = - =1 <1
We summarize the results when Ti < 1 < 72 follows: 1. For i E Sysur the alignment strength satisfies:
Kw (1a) M2)1
(w (Ta)
M
1
1 (T2)
+
(96)
2
where j' represents the corresponding spurious alignment feature.
2. For i & Sj.pa. the alignment strength satisfies:
Ku ( M2) S 01G)
T2)
12
10
1
(T2)
(97)
2
For M.- where j [di]] \ [d]. have:
(w (wx((t)) M <0
(14) di
W
(Tb)
12 +
110
(Ta)
(98)
2
These results demonstrate that when Ti t T2, the alignment for Sy.sur. including the corresponding spurious alignment, grows significantly larger. In contrast, the alignment strength for i & Soptot and noisy features M remains unchanged. Similar results also hold for u;.
F ITCP on Raw Data III Convergence
In the previous section, we demonstrated that for t T2, the neurons (w, et) are sparsely activated and remain consistently activated for E Sj.uur. Building on this result, this section establishes the convergence of these neurons to sparse solutions, providing detailed analysis of their behavior during Phase of ITCP on Raw Data. The following theorem outlines the convergence guarantees under these conditions.
The Phase III of ITCP on Raw Data is defined as the training iterations Ta < Ts. where
T3 - T. (d). At the beginning of this phase, we fix the bias threshold as 6." b.² for
T3 < L T3. Because 1, ((T2)) = (1 + 2) @(dlog d/v) (2:) is easy to know that for /. 2 T2, only when (Ip, Up) and (In, Vn.) contain the true feature j and its corresponding spurious feature j', the indicator functions remain consistently activated for i Sj.uur.
Consequently, using Eq (27 Eq (30) and Eq (3I). the loss function L becomes convex with respect to w; and independently when (Ipp Mp) and (I', Vn) contain the true feature and its corresponding spurious feature j'
At the end Phase II, using Eq 81 we know that (Ta 2 (dd). Consequently, we cannot only consider - (1)(0) ())) (yp)) and the error term Err, becomes non-negligible.
can be observed that the term - (( (ap),, (2) (yp)) is convex and 4,,.1 ||æ,||2|||,, ()) (I)-moohhh. smooth. This ensures that the true features contribute consistently to the optimization process.
Specifically, based on Eq
θ
Additionally, Li,,2 = (L."") is also convex, and we further establish its smoothness to 2T
provide rigorous understanding of its behavior.
To analyze the smootssot,,,,,,, we aim to find an upper bound that satisfies:
V w. L2( v.,1) Vwn. L2 21,2) < 4.,,21l(w,,1 w2,2, U1 2222)
(99)
32

---

The gradient difference for is given by:
(s "'W Viy) =((1.10)
(xTWi ". Viy) (( 7
IVV,, v.1) V Lan V.2)lla
2T
1u 1ux,1 wi.2l12 + 27
1w 2T
||u.1 V1,2ll2,
(100)
where 1we,1 || 112.11/111111111 B1lu,, 112ll, 1|w,, (|,,,l///0,,,1222 ll2) s (d).
O(d) and ww. 1 11 (1 lla 2,2222
Similarly, the gradient difference for 0s is:
[1:11 || w222122 +
1..22
L1,, vt,1) Vo.Lnaaawap 02,2 S
1|0,,1 11
2T
2T
(101)
where l0,,1 S (d) and lu, - O(d).
Combining the results, we find:
12
+
12
w.
12 +
12
(())
(102)
Thus, the total smoothness constant is:
4J = 4J=41 + = -
(103)
These results demonstrate that the loss function L remains convex and 4mmooh for neurons (w,, when (Ip, Up) and (Na, contain the true feature and its corresponding spurious feature j' during Phase III of ITCP on Raw Data, ensuring their convergence to sparse solutions while maintaining consistency in their activation patterns.
We verify that the following inequality holds (t+1) .4 ((+1))
Lg(w0
& Ly(w!"
(*) (1 V [L
(2)
1. (we wi us
(t+1) (1) ((41) - w!())
(104)
4
((+1) (1) (e+1) (1)
W
w es
2
Let I= maxiem (4, (2r)) = O(1) and t to ensure a monotonic decrease, plug Eq 28 and
Eq (2 into Eq 178). have
Ly(wp
(++1) u ((+1)) < L,(w (0 (0)) " [VI
(0) . ((2))12
(105)
Under our data assumptions for S., and conclusion in Eq 96 we define w M, o My, or at + a. Hy. Thus, (ww;, uf) captures both the alignment with the true feature M,. H, and the spurious feature My, Hyr, representing the minimal loss achievable under the influence of both true and spurious features in the optimization process. Using Eq (81). we know (T2) w: = (dd, so [ of)) = (d)).
By the property of smoothness, we have
(w of) )12
2 (L,(w[",e()) Ly(wf,
(106)
Take the telescope of from Tz to Ts, have
1
Σ
2
(1)
(I) + 1,2Do
T3
Ts To
T=
(107)
2 L1 (0(1),,)000 + 0(1)
33

---

where A. [u (21) (())) (wf (d)) In 4 we use T3 T2 - O(d), and L = (11)
Generalized to every j E d, the same convergence holds for all i E S when (Ip, Up) and (#ns ya) contain feature j.j'. For all (Ips Up) and (Izz V/o) in Sw, the following inequality holds:
T3
LT (Ta) & (0" + +(1).
(108)
T3
T2
As result, the relative difference is bounded by:
L(f(Ts), ((Ta)) - L(f" hh) [L((,h")
< 0 2
(109)
F.1 Summary
ITCP trained on raw data S undergoes Stages D After = Od log d) SGD iterations with batch
size B (2(d) and learning rate 1 = O(1). the resulting weights (W, V) minimize the contrastive loss in Eq. 1 up vanishing relative error:
Llf bv) - S 0
(110)
L
However, each neuron pair (w; úz) in (W, V), for i E [m], predominantly encodes a mixture of features indexed by a subset N₂ S [d]. with [N, 2. Specifically, we have:
W;
Σ
Oi Mj + Σ
M
Σ
M
JEN.
jEld
VE[d[MM]
(111)
0
Σ
H
Σ
B H +
Y. H
JEN.
jENIN
JE|d
where a², = 0(1 13 + 1 and the interference from other features is small: Big/oaj I O(1/Vd), n..//a. S O(1/VT)).
Moreover, for every spuriously correlated feature pair (j.j") satisfying Assumption 3.3 there exists
least an (((1) many of neurons i [m] with N, (j.j'), indicating the prevalence of feature mixing due to data misalignment.
G
Captioning
In this stage, the model fine-tunes the pre-trained encoder parameters W and V to obtain the updated parameters W and V through Image-Text Text Contrastive Pre-training (ITCP) on raw data.
Given an image-text pair (Ips yp) in S., the decoder generates synthetic captions 1 VT o (Wz,). where (-) denotes the activation function. The mnoge-Grounded Text Decoder, initialized with W and V from the pre-trained encoders, is fine-tuned on Sj, by minimizing the following loss function:
V (Ww,,) voll2
Le Exy,,,,)s
(112)
where ||
la denotes the Euclidean norm. This fine-tuning process refines the model to generate captions that are more closely aligned with the target text data in Sn-
During the captioning, we sample batch of image-text pairs 5y00) ((Fp, S S,, We perform stochastic gradient descent on Lc. At each iteration, we update as
m w 17 1
[t+1)
(t)
(113)
((+1)
77 v. L0 t- vi
(1)
(114)
34

---

At the beginning of this phase, we set the bias threshold as:
(2:)
1|:0 (11)
12
6 (0)
(115)
During training, the bias threshold is iteratively updated as:
((++1)
(1 2) 6%")
(116)
The gradient of Le with respect to wi
(t) W, and V at iteration is given by:
V ( LC
ui (Up
(t)
v yT Wap)a (u, (0)).)))
(117)
(8) V 10 Lc = wi Tp (Up WT, T (u(" .20) (=)"),>))/2000
(118)
The alignment can be described by the following update rule:
(w) (t) .M,) = (w!") M2) - (Va, M₃) (mo (t) M2) + + tr
(119)
(1)T (2p V WT,,)),, 1 12n
(0)
(++1)
H2} = (0) H } - L.
(120)
(v) H + π tr w (1)) (t)
xp(U0- V Wap) (m.9:.0)(200
G.1 Alignment for i Syru
This section analyzes the alignment growth for neurons i E Sj.erre Specifically, we show that when I < Tc, the alignment with the true feature M, grows exponentially if xp contains the true feature My. In contrast, the alignment with the spurious feature M, exhibits negligible growth, even for neurons i Su.e Specially,
For the true feature M,. based on the result in Eq 96 and the bias threshold in Eq 115 the indicator functions are always activated. This ensures that the neuron can consistently increase its alignment in the direction of the true feature M,.
2. For the spurious feature My, based on the result in Eq (6)) and the bias threshold in Eq (15), the indicator functions remain non-activated. This prevents the neuron from increasing its alignment in the direction of the spurious feature Myr.
The details of proof as follow:
Using Eq
95
we know
1 (2h) 12
(T2)
11 2 (T2) M2 + (mm (Tb) (T)) M2 |! (T)
||0 (Tn(
o
(121)
Using Eq (35) and Eq (36), we have
(w("), M2) (we(*) M2
(a b e)¹²
((w))) M2) + (o H2) (10 ,,,)) - (v(()) Hp))
Frree
2
(122)
Using Eq 44 and (a + b
c)Ti Ti+ >
M(d with high probability 1- O +2) we have.
| (T)) 11
(w (T2) M2 - Iw (T2) M2 > 52(
(123)
Therefore, with high probability 1-0 v/2aa we have
(10 (T)) M2
||00
(T2)
1
w
T
12
Kw
(T))
(2)
.M,)11
(124)
2
35

---

We set 6/00 prove:
/
||w:
-12-11: [ 1
and using Eq 124.. so similarly the proof of Eq 86 we can
2
For i E Surue and Tp contain the true feature Mj, with high probability 1 - 1 (a)) the indicator
functions become consistently activated S & S Tc such that:
Kul" . |26 =1
(125)
2. For E Sy.sue and Tp contain the corresponding spurious aligned feature My, with high probability
1 O a the indicator functions become consistently activated 0 s To such that:
Kuif 22))20000
(126)
3. For i & Sj.pot and M where j E [d1] \ [d], we have:
Kust ..) m>m))=20) 22600) 0
(127)
For the residual loss in Eq (119) and Eq (20 we bound the difference if 1 ( (w/"",,>\/200" = 1:
2j Hy up Up
2 (Up
V w M2
Rus,,,))2:00
(H
22, 2 Σ (0, H (w M2) H oby)) 1 (1000))266
(128)
H _ / ( (v,, H (w,, M, M, 24
In O, we employ the approximation up
M,
= H 2. 2 based on the observation that 24, 2
2) Upp when j A j". In 9, we utilize Eq
38 There
most (dtre)) neurons capable of learning
M,. which satisfy the condition (w..,2))bb 2-
For E Syuur and for Tp contain Mj. using Eq 128
Eq
119 and Eq 126) we have:
(w) (t+1) M2) 2 (w!") (4) M2) 17
U;
(t)
(1
(129)
2 (w!" M2) + C1(1 at) (0 H,),
d
Similar to Eq 35 have
1 C. (1 -1-10 at)
(1-af))
(m (0) M2)
(u. (0)
H,)
(ml M,
(130)
2
Similarly, for E Suser and xn contain the corresponding spurious aligned feature My, because Pr[1 (ut .rp) (uf))<p>)/2 22/0 = 0] 2 1 - O we have
(m)) (++1) M₁) & (w) (t) M₂) + O
Hyy)
(131)
1.5
and
(mo ((o(",,,))) .M, S (1+0(
(m
(Tb)
M,) + (0) (T2)
H₂))
(132)
2
At Tc = (dlog(e)) have:
(1 (+,
To
Kw! M₁
((d))
(133)
(To) M₁y1l
(1+0 (2))) To
36

---

Therefore, we summarize that when t = Te, the alignment with the true feature M, dominates, satisfying:
kw (FC) Kw TT)
M, My²)
S(d),
(134)
highlighting the significant separation between the true feature M, and the spurious feature My for neurons i E Sure. A similar result holds for Ci, where the alignment with the true feature H,; similarly dominates the spurious feature Hy.
G.2 Convergence
For E Suuur when Fp, contains the true feature j. the indicator functions remain consistently activated. Consequently, the loss function Le becomes convex with respect to w, and it independently. We verify that the following inequality holds
Le (t) o((+!)) S (w)( 20(1))
+ (Vc,(W ( (wi (e+1) - (0) 1
((+1) [ (wu (1((t w 2/ (1+1) )
11:0--091 -o(-)) )
(135)
+ 4
where I; = dra /|00001
1/2 10mpl + o0 | = 9(1). This means Las (w: 1/
(t) (t)) is
4-smooth for all i E Sj.sue when Tpp Up contains the true feature j. Let L = maxigm (4:) ((1)
1) = 1 to ensure a monotonic decrease, plug Eq 117 and Eq 118) into Eq 135). we have
[u.
(t+1) 04 ((+1)) - Lus(w) . ((()))
" 2
LI (6) (0 11
(136)
By the property of smoothness, we have
( NE N 2 (Icslup") Lc,((u,, of))
2
(137)
Take the telescope sum from to Te, we have
Tc
(l0) 12D, Te
Σ
(9) ,
Tc t=
Les(w] uf) + )
(138)
_ 2
1
where Ag (w (0) (.())) [c ). In O, we use To = (d), and || ( 10 10
(11). In 12, we use W M, Vi2 o. and Le,, u²) O h) if x, contains the true feature My.
Therefore, for all j E and all (Mip: Up) E Sns when To = (())), we can ensure
1 o
1 s 5
Le = E (e
(139)
2
G.3 Summary
After To iterations, the parameters W and V are updated to W W and TTc V. respeciveely. using the dataset Sr. The generated caption given by:
9/ VT (WW,),
(140)
37

---

where the expected loss satisfies:
11 8p11 = Lo I 0 (2)
(141)
2
1. For i E Sy.sure the alignment strength satisfies:
[(,, (Tc)
M2M1 = 00]) |m0 (T)) 1
(142)
and
Kie (me M2)11 - 01 01
2) 1m 1
(143)
where j' represents the corresponding spurious alignment feature.
2. For i & Sj.pa the alignment strength satisfies:
(w (TL) M2) S x d
w
(To) 2
(144)
2
3. For M where j E [di]] \ [d], we have:
(wo (c+1) M2 < 0
1
10
20
(Tc)
2
(145)
d1
H Filtering
During filtering, we sample the synthetic image-text pair (Ip. Up) in S., and the corresponding image-text pair (Ip, Up) in Sw- The image encoder f and text encoder h trained on raw data are employed to obtain the corresponding embeddings.
Ee,, = f(xp), Eyp = =((ip); = =(yp)
(146)
Then, we calculate the cosine similarity of (Zee, 2gp)) and (24,, > and select the image-text pair with higher cosine similarity denoted as (x, ū). In this way, we replace the noisy pairs in S., with synthetic pairs in Sw. The resulting dataset is denoted as S = S. U U.-
The decoder generates synthetic captions 0p (To(Wop)) Using Eq 141 for each data pair (Ipr Up) which contain feature (M,, H,) in Sn. we have
E(sp.up)
Ejed
[ 2
H H
H, u
[
1
=
s E(pp.up)
[
2 11.
wall2 11 = 1 = Lo S 2
(147)
Therefore, using
H l and Zx, =up in Sr. we have
E jed 1pp = 11 1] 2 21- 2
(148)
Base on Assumption B.1 2 Bernoulli (C²), we have
Pr(z) 9,
= |>, 1) 2 1 01
(149)
Using Eq (134) and Eq
149). we have
Pr(z) 8p
= 1 =1) - e 2
(150)
Therefore, after replace all noisy text yp in S.n by synthetic caption ú S.,
1. for a positive pair (Nip Up), we have
F
= 5,
E
B 1
=
p
1 tj
(151)
for negative pairs (Ip, )q). where # q. we have:
E
2
2
= θ
1
Vj.j' E [d.
(152)
2 yv
38

---

I ITCP on Synthetic (Recaptioned) Data
During ITCP on Raw Data, we use noisy dataset S, Based on SubSection B.2 we have E [2d and 2y both in θ (2). In this scenario, for i Sy,sure w. jointly influenced by M, and My, with both features contributing comparably to the updates. However, during ITCP on recaptioned data, we sample image-text pairs from the dataset S. Using Eq. 151), we find that E % θ (z).
In this case, for i Suyerre w/" features, ensuring purified representations.
(1) is influenced solely by Mj, without interference from spurious
The only difference between ITCP Raw Data and Data lies in the 0p processes remain largely the same. Therefore, we simplify our proof accordingly.
all other training
I.1 Phase I of ITCP on Synthetic Data
The Phase I of ITCP on Data is defined the training iterations f Ti, where Ti =
(dlogd) is
the iteration when all [|w](a]] 1 w? 12 Before Ti, we set (((0) 0. For every neuron i E [m]]
the weights wa i2 will mostly ignore the noise features - H and learn to emphasize the features M. H.
> E 2 and (a bb+b+c) c)* ze (a + b c)². If Pr[12 1 1 = 1) < 0.1, we have E [
this case, wi (++1)) is predominantly influenced by M,, with minimal contributions from My The updates are thus primarily driven by the single feature M,. ensuring that spurious interactions from My are negligible.
(a + b b+cc' 1 ((we0) M2) + (v. H₂)) )
MM
10 m
1012
Σ
2
1=1
(153)
T/C d
21
|MM w
1+
=
i S j.sum::
(m0 (T)) M2
(1 (1+ 17
C2
21
(m: (0)
M, + (u: (0)
H,)
2
C.
27;
logd
|MM T w
(0) 1 +
HH
00)
1 11
d
4
log d.
|MM
W
(Ti)
11 + |HH (T)
(154)
d
logd
(Ti)
11 + 110
(Ti)
||00
(0)
1
d
(1 + log
|/0
(Ti)
11 + 110
(Ti)
Because
1
+
(T))
w
(0)
11
10
1 and e₁
2(1 + co)
39

---

i a Sy.sme:
(uu; (T))
.M,)²
1+ 11 d
C2
22
(w: (0)
M2) + (v) Hyj
2
C2 17
27
C2 logd
|MM T ii 112
(0)
+
HH
(0)
1 11
d
d
4
(155)
02 log d
||M
(T))
(T1)
d
2
logd d
||m (21)
12 +
1105
2
(w (witt)) MH - (
1/0 (T)) 113
1.2 Phase II:
The Phase II of ITCP Synthetic Data is defined the training iterations 7i S t S T2. where T2 - Ti = ( dllgdd is the iteration.
We set ((0) / log d d
and by (++1) (1 + 7) until all I|] (T2) l2 2
((d)[lw? (T) In this phase, the weights (w,, will mostly ignore the features M,, H i $ Sj.ure and the noise features M-, H-, and learn to emphasize the features Mj. H, if Sj.sure
For Sysures using Lemma C.4 the following holds with high probability 1 e- (Q(4) when
T < k T2:
2 5o (wo ())
(156)
dl+co
Under the assumption that, with high probability, the indicator function satisfies the condition when
t Ti:
(1u/")>0)/2000 22012000) 1 (1,0) 22 \20000
(157)
we can ensure that:
C2
E 2/ 1 ((o9" .0)) \20000
w. (,)0())))200)))
(158)
The weight dynamics for (w ((+1)) .M,]] can be expressed as:
kw! ((wo(++)1) 11+1)
.M,11 (1 (1+ (1+n Cs
(w)) (1) M2) + (c. HHg)
(159)
Given that (1 + 17 4 (1+ + 9), and (w)5) (M) Hy)
(1) it follows that:
2
(u
(++1) M2 > 6( g(0+1))
(160)
Thus, with high probability, for 1. < T2, we have:
(ar (x) (20
(161)
so for Ti < T2 we have
(w (1+1) .M,11 =
1 1 (
C2)
(w (T1)
M2) + (T))
(Tt)
H
(162)
2
40

---

For i & Sy,sre, the projection of weights onto a generic feature at iteration T3 satisfies:
Pr (Kuss (uff" 22)) KofP..>> \200 =1 S o d
(163)
We can ensure that:
IE y (wr =p) > . 1 .n)
= ds
(164)
The weight dynamics for Kw, (w/(t1)
((+1) ,M,}] can now be expressed as:
(w) (1) M, (0) H,)
(w
(++1)
M2 = -1+ (12)
(165)
2
Given that (1 + o (2)) < (1 3), and (wf ,M,
Hi)
b, it follows that:
2
((a(tt)) .M,1 < 6,¹h (((+1)
(166)
If (w,, (Ti) M2)
< 0g (T)) then Kw, (9) ,Mj)l < (1) for S T2. Thus, with high probability, for
1 < T2, we have:
( ((uf")>z)\220) (20)/2000) (ud"") (//"),>>)/00) = O.
(167)
(uttt M, - (1 (1+ (22 ))
(
(wu (ao (T)) M2) + (v) (TL) H2)
(168)
There exists T2 = O [dlogd such that the following conditions hold:
(I+ = (1+++)==0(0)
(169)
indicating that (w((-) M, for i Sjsure increase iteratively until:
[ (T)) 2| 2 (dd) (Ti) lz
(170)
while, for i $ Sjaue the updates diminish, such that:
(1
7 T
< $100
(171)
indicating negligible growth in (w!tt M,)1
Thus we have
1mm (Tp)
M, =
2,)1=11100 ||l1 13)1
Σ
(10
(22)
M2
Σ
(m
(T))
M2
jE(d
jE(4]]M
2 || (TunT>113 (1 + (1))
(T))
(1))
(1
(172)
2 (1 a( (1)) (
Finally, for i & jiser we have:
| ( ll2 Vd
| (Th) 12
mml (Tb)
M, 5 5 +
< o
(173)
vd
and for noise components:
|ml! (lo(TTal12
(T))
.M
) )/250 5
(174)
Vdz
41

---

We summarize the results when 71 <ts S T2 follows:
1. For i E Sy,ur the alignment strength satisfies:
(Ta)
1
||u!
11
(uy (Ta) M2 (1-
(1-oo o(1)
(175)
2
without j' that represents the corresponding spurious alignment feature.
For $ Sjppor the alignment strength satisfies:
Kw (Tu MM, =01))
100
(T2)
11
110
(22)
(176)
2
For M where j E [dı] \\ [d]. we have:
(wo 1++1) 2 <
1
100
(T2)
12
+
110
(T2)
(177)
d1
2
Similar results also hold for U/.
1.3 Phase III Convergence of ITCP on Synthetic Data
Similarly to convergence Phase III in ITCP on Raw Data when 7/2 S S T3. using Eq (27 Eq 30 and Eq (31. the loss function becomes convex with respect w; and v; independently when (Tpp Up) and (Ins Un) contain the true feature j.
We verify that the following inequality holds
(t+1) 00 ((++)))
I L, (w,)) 22("))
+ [Ly(w!") (2) w8 (1) (w. (1+1) w/t, o wftt1) -1( )(²))
(178)
+ 4 ( u ((-1) wl( of ((+1)) - ||
Let L maxiem (1,1/ (2T)) = (1) and
i. to ensure a monotonic decrease, plug Eq 28 and
Eq 29 into Eq 178). have
Ly(w (++1)
u ((+1)) < (, (0) 2 [V 2 M1
(179)
Under our data assumptions for Stp and conclusion in Eq 96 we define w = of M,, of = a H Thus, Lay (w, vz)) captures both the alignment with the true feature Mj. H, and the spurious feature My, Hy, representing the minimal loss achievable under the influence of both true and spurious features in the optimization process. Using Eq (11. (dd)..
we know w (Ta) = (dd), Ly of) =
By the property of smoothness, we have
(2) (w 2 11 2 L
(1)
2
(L,( (0(0) LL,(u,
(180)
Take the telescope sum of from T₃ to T3, we have
1 T
(t) ,0 (L)),, + L2DD
(0)
T3
T2
T3 - To
1=
(181)
Ly(wt of) + O(1)
where Ao (w: ,
(Ti)
(T:)
Ly oz)) (1)). In 4. use T2 = O and L= -= O
Generalized every j E d, the same convergence holds for all i Sy.sure when (I,, Un) and (#ns Br) contain feature j.j'. For all (Ips Up) and (I,,, Vn.) in Sw. the following inequality holds:
1
Σ T t=
L(f(Ts) h (Ta)) (()
(182)
T3
42

---

1.4 Summary
ITCP trained on recaptioned data S proceeds according to Eq. After T = ( logd) d) SGD iterations with batch size B (((d) and learning rate 1) ( the returned weights (W, V) achieve a contrastive loss that asymptotically optimal:
i m(y)
T." I {()).
(183)
i.-
Each neuron pair (wi, 0i) in (W, V), for i E [m], primarily encodes a single aligned feature indexed by set N₁ S [d], with [N, 1. Specifically, we have:
W dA M2 +
Σ
M,
Σ
M
JEN,
JE N.
jE(di M4
(184)
0
G H
Σ
H
+
Tr. H
JEN,
jEIdN,
jE(d
where až., (1) O(1/vd1)
| [2) and the residual terms satisfy / S O(L/Vd), Rus/ass S
Moreover, for every feature index j E [d]. there exists 22(1) many of neurons E [mm] such that N {j}, indicating that each semantic concept is distinctly captured by dedicated neuron pairs.
J Downstream Task
We consider the same zero-shot classification task as Section 3.4 where the image x and the class-wise text prompts (y): } are given. Each prompt yı corresponds to one of K class labels, and the goal is to classify x into the class with the best matching prompt.
Each text prompt yiu is generated as:
H + Eger
1 Ilu = @(1),
1 Immux = (1).
(185)
Each test image æ is generated as:
M 2/4 + Ex,
12% = O(1),
| - O(1),
(186)
where M' MP,, and
max
(P
dون S 0(1/Va).
(187)
f.
If a belongs to class k, then:
(=/) Z ||:
(1/1)) Z/y ll2
AK' tk.
(188)
Using Eq. 96 and Eq. 144). let f(x) and h(y) represent the image encoder and text encoder of ITCP on raw data, respectively. Given a data sample containing M, and y containing Hy, where j' is the spurious feature corresponding to j. it holds with high probability that:
f(x)
k(y)
[f [M()))l2 @(1)
(189)
This result implies that the image and text encoders of ITCP raw data struggle to distinguish between features j and j', leading to misclassification caused by spurious correlations.
However, using Eq. (155) and Eq. 176). let f(ar) and ((nx) denote the image and text encoders of ITCP on recaptioned data. Given x containing M, and y containing spurious Hy, it holds with high probability 1 - θ (1) that:
f(x)
((v)
S 0 (2)
(190)
[f(()))2)) 1
43

---

This result implies that the image and text encoders of ITCP on synthetie data are capable of effectively distinguishing the true feature from the spurious feature.
Because K = (1) and || Ilo = (1), we only have constant class classification and constant
features in images. Thus, we have:
For the image encoder f() and text encoder (y)) of ITCP on raw data:
Pr arg (ax((((()))
kz @(1,,
(191)
2. For the image encoder f(x) and text encoder (3))) of ITCP synthetic data:
Pr
arg k (axx) ko o(1).
(192)
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims regarding the improvement of feature purity and zero-shot perfor- mance through recaption are stated clearly in the abstract and introduction, and validated by theoretical and empirical analysis.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in the Conclusion.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification; All theoretical results are stated with full assumptions, and complete proofs provided in the Appendix. Intuitive proof sketches are also included to aid understanding.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We outline all datasets, models, and training procedures in Section 5.1
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will provide public GitHub access post-review.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
44

---

Justification: We provide the details of data generation, dimensionality, noise setup, model size, and SGD training parameters including batch size and learning rate in Section 5.1
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Each reported metric (accuracy, Silhouette Score, feature purity, random selection of neurons) is averaged over 20 random seeds with 1-sigma standard deviation in
Section 5.1
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experimental resources are introduced in Appendix A.2
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpghttttpp///oor.....cop//o//t
Answer: [Yes]
Justification: We comply with all ethical guidelines. All data used is publicly available under appropriate licenses; no human subjects or sensitive data are involved.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve generative models web-scraped datasets with safety concerns.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and respect the licenses of models and datasets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
45

---

Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subject research is conducted.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Answer: [NA]
Justification: No LLMs were used in this work.
46


Reviews:
[{"review_id": "aoNtDYzD85", "reviewer": "Reviewer_XKgJ", "summary": "The paper proposes a theoretical framework to analyze the impact of data misalignment on multimodal contrastive learning models (e.g., CLIP). This framework theoretically characterizes the effects of spurious correlations, recaptioning, filtering, etc., on model performance. The experimental results from simulated experiments and experiments on Food-101, CIFAR-100, and Caltech-101 support the framework.", "questions": "1. In Equation 2, why is $f_i(x)$ defined in the form presented in the paper rather than a simpler form like $f_i(x)=\\sigma(<w_i,x>+b_i)$?\n2. Could you supplement the results of Table 2 on ImageNet?", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength:\n- The issue analyzed in this paper is of great significance.\n- The theoretical analysis of this paper is solid.\n- Real-world experiments have demonstrated the effectiveness of their framework.\n\nweakness:\n\n1. In the theoretical analysis, the model adopted in the paper is somewhat simplified (a one-hidden-layer ReLU network).\n2. The assumptions introduced in the paper are overly strong. For instance, Assumption 3.2 assumes that image-text pairs are under sparse coding.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable time in the evaluation.\n\n## Response to Weakness 1: Model simplicity\nWhile the model studied in this paper is a simplified version of real-world applications, we believe that analyzing the training dynamics of a one-hidden-layer network with nonlinear activation represents the most advanced setting currently allowing a comprehensive theoretical analysis of contrastive learning. Table 1 in the paper shows that our two-encoder architecture achieves the highest modelling fidelity among existing theoretical works.\n\n**Contrastive-learning theory**  \n[R1] analyses only linear encoders.  \n[R2] extends to one hidden layer but still studies a single encoder for one modality.  \nOur work is the first to treat two nonlinear encoders, misaligned image–text multi-modal data, and zero-shot evaluation.\n\n**Supervised and adversarial theory**  \n[R3] establishes feature purification with one hidden layer under adversarial training.  \n[R4] studies nonlinear transformers, but only for supervised model editing, and did not consider contrastive or multimodal objectives.\n\n**Semi-supervised pseudo-labeling**  \n[R5] and [R6] analyse self-training with one-hot class pseudo labels. Generating a single class label is far simpler than producing a full synthetic caption, and pseudo-label methods involve a single encoder with explicit labels. By contrast, our setting requires two nonlinear encoders, high-dimensional text generation, and a non-convex contrastive loss, making the theoretical analysis substantially more challenging.\n\n## Response to Weakness 2: The sparse-coding assumption may be overly strong\nThanks for the observation. Sparse coding is in fact a standard modelling choice for both images and texts and is widely used in recent neural-network theory, rather than a special assumption of our work.\n\n**Image evidence**  \n[R7] shows that Caltech101 images admit sparse codes over learned dictionaries and that sparse coding outperforms vector quantisation.  \n[R11] demonstrates that face images can be reconstructed by a small set of dictionary atoms and that sparsity improves recognition accuracy.\n\n**Text evidence**  \n[R8] models word co-occurrence statistics with sparse latent factors, providing a theoretical basis for skip-gram embeddings.\n\n**Sparse coding in neural-network theory**  \n[R9] analyses sparse text features in multimodal models such as CLIP.  \n[R10] proves adversarial robustness under a sparse latent image model.  \n[R2] tracks how contrastive learning separates true and spurious features using a sparse data model.\n\nThese studies confirm that assuming sparse codes for both modalities is a common and practical basis for rigorous analysis.\n\n## Response to Question 1: Why is $f_i(x)$ defined as in Equation (2)?\nWe adopt the symmetric ReLU $f_i(x)=\\sigma(\\langle w_i,x\\rangle-b_i)-\\sigma(-\\langle w_i,x\\rangle-b_i)$. This form lets us take $b_i>0$ without loss of generality and greatly simplifies analysis. The positive bias makes a neuron activate when $\\langle w_i,x\\rangle>b_i$ and also when $\\langle w_i,x\\rangle<-b_i$, yielding $f_i(x)=-f_i(-x)$. With this symmetry a single indicator function covers both signs of the inner product, streamlining gradient–alignment calculations while preserving the expressive power of the standard ReLU. All results carry over to the usual ReLU $\\max(0,\\langle w_i,x\\rangle)$ after a linear rescaling of weights and biases.\n\nThe same symmetric activation is common in recent theory [R3, R2].\n\n## Response to Question 2: Supplementary ImageNet Results\n\n|Model|Top-1 (%)|Top-5 (%)|Silhouette|\n|---|---|---|---|\n|CC12M CLIP|35.04|62.10|-0.014639|\n|CC12M LaCLIP|**42.62**|**70.17**|**-0.008141**|\n|LAION-400M CLIP|58.34|84.73|-0.029893|\n|LAION-400M LaCLIP|**62.27**|**86.34**|**-0.056593**|\n|RedCaps CLIP|37.66|63.31|-0.022045|\n|RedCaps LaCLIP|**39.66**|**66.06**|**-0.012269**|\n\nThe LaCLIP variants consistently surpass their CLIP counterparts on both Top-1 and Top-5 accuracy. Higher silhouette scores further indicate cleaner feature separation after recaptioning, in line with our theoretical predictions.\n\n## References\n[R1] HaoChen et al., \"Provable Guarantees for Self-Supervised Contrastive Learning,\" NeurIPS 2021.  \n[R2] Wen and Li, \"Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning,\" ICML 2021.  \n[R3] Allen-Zhu and Li, \"Feature Purification: How Adversarial Training Performs Robust Deep Learning,\" FOCS 2022.  \n[R4] Li et al., \"When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers,\" NeurIPS 2023.  \n[R5] D.-H. Lee, \"Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks,\" 2013.  \n[R6] Zhang, Shuai, et al., \"How Does Unlabeled Data Improve Generalization in Self-Training? A One-Hidden-Layer Theoretical Analysis,\" ICLR 2022.  \n[R7] Yang et al., \"Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification,\" CVPR 2009.  \n[R8] Arora et al., \"A Latent Variable Model Approach to PMI-based Word Embeddings,\" TACL 2016.  \n[R9] Chen et al., \"Understanding Transferable Representation Learning and Zero-Shot Transfer in CLIP,\" ICLR 2023.  \n[R10] Allen-Zhu and Li, \"Feature Purification: How Adversarial Training Performs Robust Deep Learning,\" FOCS 2022.  \n[R11] Yang et al., \"Robust Sparse Coding for Face Recognition,\" CVPR 2010."}, {"review_id": "M1hxMrfJcK", "reviewer": "Reviewer_t22h", "summary": "This paper develops a theoretical framework for understanding contrastive learning on noisy, misaligned image–text data and proves how “recaptioning” fixes it. Specifically, it improves zero-shot performance in vision–language models by targeting the data misalignment where texts can include spurious or omit true visual features. Under a sparse-coding data model, it analyzes SGD dynamics for two one-hidden-layer ReLU encoders trained with a spectral contrastive loss, showing that spurious text features entangle representations and degrade generalization. It then proves that a simple ReLU decoder trained on a small high-quality set can generate captions that suppress spurious correlations, yielding purified features and probably better out-of-domain zero-shot accuracy—results are validated by experiments on CLIP and LaCLIP.", "questions": "Q1: How do contrastively pre-trained VLMs align modalities, extract feature representations, and achieve zero-shot capabilities? How does text recaptioning on noisy image-text pairs provably enhance generalization performance?\n\n\nQ2: How well do the theoretical insights extend to deeper or transformer-based encoder architectures commonly used in VLMs, and what modifications (if any) would be needed in your dynamics or misalignment models to capture their behavior?\n\n\n\nQ3: Definition 3.1 uses a ReLU with form σ(⟨w,x⟩−b)−σ(−⟨w,x⟩−b), which differs from standard  ReLU networks. What is the technical necessity of this symmetric activation choice, and can your SGD-dynamics analysis extend to vanilla max⁡(0,⟨w,x⟩)?\n\n\nQ4: Assumption 3.2 models both images and texts via the same sparse coding with orthonormal dictionaries M and H. In practice, text representations (e.g.\\ token embeddings, LLM hidden states) live in very different spaces than dense image features. Could correlated or overcomplete text dictionaries qualitatively break the convergence or purity guarantees?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper demonstrates several strengths across the dimensions of originality, quality, clarity, and significance:\n\n\n\n\nQuality:\n\n\nThis paper gives rigorous theoretical proof of the training dynamics and generalization analysis of nonlinear VLMs, showing the impact of misaligned image-text pairs on pre-training performance. Moreover, they give theoretical justification of enhanced out-of-domain generalization through pre-training with text recaptioning.\n\n\n\nClarity:\n\n\nThis paper gives clear proof and illustrations to set up their theory and training framework. The proof and assumptions are explained clearly. \n\n\n\n\nSignificance:\n\n\n\n\nThis paper is significant because it rigorously links feature purity—the degree to which each neuron encodes a single semantic direction—to downstream generalization in contrastive models. Theorems 4.1–4.2 establish that, although SGD reliably optimizes the nonconvex contrastive objective, training on raw, misaligned pairs inevitably entangles true and spurious features. Theorems 4.3–4.4 then show that recaptioning and filtering purify these representations so each neuron encodes only a single feature, and Theorem 4.5 guarantees that this purification yields vanishing failure rates in zero-shot, out-of-domain classification.\n\n\n\n\n\nOriginality:\nThe originality is good in this paper.", "was_revised": true, "final_justification": "This paper is recommended to be accepted.", "rebuttal": "We thank the reviewer for the valuable time in the evaluation.\n\n## Response to Question 1: How do contrastively pre-trained VLMs align modalities, extract feature representations, and achieve zero-shot capabilities? How does text recaptioning on noisy image-text pairs provably enhance generalization performance?\n\nDuring contrastive pretraining, stochastic gradient descent (SGD) initially causes neurons to align with both true and spuriously correlated features, due to comparable gradient magnitudes (Appendix D). As training progresses, this spurious alignment is reinforced—neurons entangled with mixed features amplify both components simultaneously (Appendix E). Without intervention, the final representation encodes multiple features per neuron, preventing clean separation (Appendix F, Theorem 4.2).\n\nTo improve the pre-trained model, a decoder is fine-tuned on clean data to generate synthetic captions. Theorem 4.3 (Appendix G) proves that neurons aligned with true features enhance their alignment exponentially, while suppressing spurious activations to negligible levels. Appendix H further shows that replacing original captions with synthetic ones filters out spurious correlations. Consequently, purified alignment is restored, and Theorem 4.5 (Appendix J) establishes that the zero-shot error drops to $o(1)$, demonstrating improved downstream generalization. A more detailed description can be found in Appendix B.1 Proof Scratch.\n\n## Response to Question 2: How well do the theoretical insights extend to deeper or transformer-based encoders commonly used in VLMs?\n\nExtending our analysis to deep or transformer-based encoders would require major technical advances.\n\n**Attention and softmax complexity**  \nTransformers rely on multi-head attention, where each layer multiplies token similarities by a softmax weight matrix. This induces dense, data-dependent coupling across all feature dimensions, far beyond the local interactions in a one-hidden-layer ReLU network. Formal convergence proofs must track the evolution of these weights and the resulting non-linear dynamics, a problem still open even in supervised settings.\n\n**Tokenized input and pairwise structure**  \nVision–language transformers operate on sequences of image patches or subword tokens. A rigorous treatment would need a mathematical model for tokenization and a similarity metric for positive pairs that captures spatial or linguistic structure. Such formulations are not yet available in a form amenable to SGD dynamics analysis.\n\n**Current theoretical frontier**  \nExisting theory for transformers is limited to supervised tasks or linear approximations: [R1, R2] studies model editing in nonlinear transformers under supervised objectives. For contrastive learning, the most expressive theoretical results remain at the one-hidden-layer level, as in [R3].\n\nThus our two-encoder, one-hidden-layer nonlinear contrastive model represents the state of the art for provable training dynamics in VLMs. Generalising to full transformers is an important direction for future work.\n\n**Empirical verification on large transformer VLMs**  \nOur theoretical insights, derived from a one-hidden-layer model, are borne out by experiments on deep ViT-based systems:\n\n1. **Purified neurons enhance generalisation.** LaCLIP, which incorporates the recaption–filter stage predicted to raise feature purity, attains higher accuracy and a larger silhouette score than its CLIP counterpart (Table 2).\n\n2. **Data misalignment reduces feature purity.** When we shuffle captions to increase misalignment, the average neuron cosine similarity grows in proportion to the shuffle rate, matching Theorem 4.2.\n\nThese observations show that conclusions proved for a simplified network transfer qualitatively to state-of-the-art transformer VLMs such as CLIP and LaCLIP.\n\n## Response to Question 3: Motivation for the Symmetric ReLU Activation in Definition 3.1\n\nWe adopt the symmetric ReLU $f_i(x)=\\sigma(\\langle w_i,x\\rangle-b_i)-\\sigma(-\\langle w_i,x\\rangle-b_i)$. This form lets us take $b_i>0$ without loss of generality and greatly simplifies analysis. The positive bias makes a neuron activate when $\\langle w_i,x\\rangle>b_i$ and also when $\\langle w_i,x\\rangle<-b_i$, yielding $f_i(x)=-f_i(-x)$. With this symmetry a single indicator function covers both signs of the inner product, streamlining gradient–alignment calculations while preserving the expressive power of the standard ReLU. All results carry over to the usual ReLU $\\max(0,\\langle w_i,x\\rangle)$ after a linear rescaling of weights and biases.\n\nThe same symmetric activation is common in recent theory [R4, R5].\n\n## Response to Question 4: Could correlated or overcomplete text dictionaries break the guarantees?\n\n**Practical incoherence suffices**  \nThis is an interesting question. Since orthogonal or incoherent bases are standard assumptions in learning theory, we follow existing works in adopting them for our analysis. Exact orthogonality is not required; our theoretical guarantees remain valid even if the text dictionary is somewhat correlated, as long as distinct features are sufficiently incoherent—meaning their pairwise inner products are upper bounded by a smaller value. This bounded incoherence limits cross-feature interference, allowing our convergence and feature-purity results to hold. The weaker incoherence yields larger residual terms $r_i$ in Eq.(12) and requires more iterations to achieve convergence.\n\nOn the other hand, if the dictionary is highly coherent, it can lead to non-unique representations of the data. This makes it difficult to clearly define positive pairs and learned features, posing challenges that fall outside the scope of our current analysis. We leave a deeper exploration of this case for future work.\n\n**Extension to overcomplete bases**  \nAn overcomplete dictionary can be reduced to a complete basis by a suitable linear transformation. The latent code $z_x$ therefore need not be limited to the discrete set $\\{0,\\pm1\\}$; any real-valued sparse representation suffices. Allowing arbitrary magnitudes in $z_x$ changes only the scale of the coefficients $\\alpha_{i,j}$ and $\\alpha_{i,j'}$ in Eq.(12), but the qualitative conclusions regarding feature mixing and eventual purification are unchanged.\n\n**Standard theoretical assumption**  \nOrthogonal or incoherent bases are standard in neural-network theory, such as transferability analysis in CLIP embeddings [R6] and sparse-modelled contrastive learning dynamics [R3]. Sparse coding is in fact a standard modelling choice for both images and texts and is widely used in recent neural-network theory:\n\n1. **Image evidence**  \n[R7] shows that Caltech101 images admit sparse codes over learned dictionaries and that sparse coding outperforms vector quantisation.  \n[R8] demonstrates that face images can be reconstructed by a small set of dictionary atoms and that sparsity improves recognition accuracy.\n\n2. **Text evidence**  \n[R9] models word co-occurrence statistics with sparse latent factors, providing a theoretical basis for skip-gram embeddings.\n\n3. **Sparse coding in neural-network theory**  \n[R6] analyses sparse text features in multimodal models such as CLIP.  \n[R4] proves adversarial robustness under a sparse latent image model.  \n[R3] tracks how contrastive learning separates true and spurious features using a sparse data model.\n\nThese studies confirm that assuming sparse codes for both modalities is a common and practical basis for rigorous analysis.\n\n## References\n[R1] Li et al., \"When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers,\" NeurIPS 2023.  \n[R2] Huang, Yu, et al. \"A Theoretical Analysis of Self-Supervised Learning for Vision Transformers,\" ICLR 2025.  \n[R3] Wen and Li, \"Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning,\" ICML 2021.  \n[R4] Allen-Zhu and Li, \"Feature Purification: How Adversarial Training Performs Robust Deep Learning,\" FOCS 2022.  \n[R5] HaoChen et al., \"Provable Guarantees for Self-Supervised Contrastive Learning,\" NeurIPS 2021.  \n[R6] Chen et al., \"Understanding Transferable Representation Learning and Zero-Shot Transfer in CLIP,\" ICLR 2023.  \n[R7] Yang et al., \"Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification,\" CVPR 2009.  \n[R8] Yang et al., \"Robust Sparse Coding for Face Recognition,\" CVPR 2011.  \n[R9] Arora et al., \"A Latent Variable Model Approach to PMI-based Word Embeddings,\" TACL 2016."}, {"review_id": "nrAJGX0lba", "reviewer": "Reviewer_CyKG", "summary": "Contrastive learning has been very successful for learning discriminative representations from image-text pairs. However, how contrastive learning can improve discriminative representations has not been well investigated, especially from theoretical perspectives. This paper theoretically and empirically analyzes contrastive learning with misaligned image-text pairs, showing that spurious alignment leads to feature impurity and generalization degradation. Under a sparse coding model, the authors demonstrate that filtering out misaligned data improves purified representations, and provide SGD convergence guarantees under such noisy supervision.", "questions": "Extension to Deeper/Transformer Architectures:\nYour model uses a 2-layer ReLU network with symmetric activation for tractability.  \n→ Do you expect similar dynamics in deeper or transformer-based encoders? Can empirical observations support this? What are the bottlenecks in theoretical extension?\n\nClarification on Assumptions and Presentation Structure:\nThe paper starts with Assumption 3.2 without presenting Assumption 3.1, which raises concerns about completeness.  \n→ Could you clarify if Assumption 3.1 was omitted intentionally or inadvertently? Moreover, the overall structure of the paper may benefit from better separation between theoretical analysis, empirical validation, and broader discussion. Have you considered re-structuring to better align motivation, analysis, and conclusions?\n\nFeature Purity – Monitoring and Conceptual Clarity:\nThe notion of feature purity plays a central role in both the theory and experiments.  \n→ Could you provide more precise criteria or visualizations for how feature purity is measured and interpreted in practice? Would it be beneficial to isolate this concept in a dedicated section (e.g., a definition box or diagnostic figure)\nto aid clarity and reproducibility?\n\nImpact on Fine-tuning and Transfer Learning:\nYour results emphasize zero-shot generalization.  \n→ Do you expect feature purity to remain beneficial in downstream fine-tuning or domain adaptation scenarios? How might it influence representations during gradient-based adaptation?\n\nEstimability and Practical Relevance of Cs (Spurious Alignment Rate)\nYour theory hinges on a spurious alignment probability Cs that determines the noise level in supervision.  \n→ Can Cs be estimated or approximated in real-world settings, e.g., via proxy metrics on semantic similarity or caption quality? If not, how sensitive are your theoretical guarantees to misspecification of this parameter?", "limitations": "- Real-world robustness: Clarify how the proposed theory holds under real-world caption noise distributions, especially those involving semantic ambiguity rather than total irrelevance.\n- Estimating spurious alignment rate (Cs): Since Cs is central to theoretical guarantees, discussing methods to estimate Cs or proxy metrics in practice would enhance the applicability of the work.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- First theoretical treatment of data misalignment in contrastive learning for VLMs.\n- Provides provable guarantees of SGD convergence under noisy supervision. Introduces the concept of feature purity and connects it to generalization performance.\n- Experimental results strongly support the theoretical claims, including on real-world datasets. \n\nWeaknesses:\n- The theoretical model uses a simplified 2-layer network and does not directly extend to real-world architectures like transformers.\n- Recaptioning mechanism G(x) is not theoretically modeled or analyzed — despite its empirical impact.\n- The notion of spurious alignment (Cₛ) is theoretically assumed but not estimated or measured in real datasets.\n- Random shuffling texts in image-text pairs in the experiments may not reflect real situations since image-text pairs may not that far as random associations. More realistic settings would be to shuffle with similar texts probabilictically.", "was_revised": true, "final_justification": "Sorry for the delayed response, and thank you for the detailed and concrete review. I have carefully read the authors’ rebuttal and considered the clarifications provided. I appreciate the effort the authors put into addressing the concerns raised, and I am satisfied with the explanations. I have engaged in the reviewer discussion and documented my reasoning here. I also acknowledge that my review and conduct are subject to the Responsible Reviewing initiative, and I submit this Final Justification accordingly.\nThat said, I would like to note that this submission is mathematically demanding and outside my primary area of expertise. While I did my best to evaluate the work, I could not carefully verify all technical details, and therefore I assign a lower confidence level to my assessment.", "rebuttal": "We thank the reviewer for the valuable time in the evaluation.\n\n## Response to Weakness 1: The theoretical model uses a simplified 2-layer network and does not directly extend to real-world architectures like transformers\n\nWhile the model studied in this paper is a simplified version of real-world applications, we believe that analyzing the training dynamics of a one-hidden-layer network with nonlinear activation represents the most advanced setting currently allowing a comprehensive theoretical analysis of contrastive learning. Table 1 in the paper shows that our two-encoder architecture achieves the highest modelling fidelity among existing theoretical works published in recent top ML conferences:\n\n**Contrastive-learning theory**  \n[R3] analyses only linear encoders.  \n[R4] extends to one hidden layer but still studies a single encoder for one modality.  \nOur work is the first to handle two nonlinear encoders, misaligned image–text multi-modal data, and zero-shot evaluation.\n\n**Existing theory for transformers is limited to supervised tasks or linear approximations:**  \n[R1, R2] studies model editing in one-hidden-layer transformers under supervised objectives.\n\nIn summary, we believe our work has addressed more complicated and realistic models compared to prior studies. Extending the analysis beyond one-hidden-layer ReLU networks would require fundamentally different tools to handle the highly non-convex loss landscape. Therefore, we leave this direction for future works.\n\n**Empirical validation on transformer VLMs**  \nNotably, our empirical experiments are all conducted using 12-layer ViT-based encoders, and their training behavior aligns well with the theoretical predictions, providing strong empirical support for the theoretical insights:\n\n1. **Recaption improves performance.** LaCLIP achieves higher accuracy and silhouette scores than CLIP (Table 2).\n\n2. **Purified neurons enhance performance.** Lowest-similarity neurons yield best accuracy; highest-similarity worst (Figure 2a–c,e–g).\n\n3. **Misalignment reduces purity.** Caption shuffling increases neuron similarity proportionally, matching Theorem 4.2 (Figure 2d,h).\n\nThese observations show that conclusions proved for a simplified network transfer qualitatively to state-of-the-art transformer VLMs such as CLIP and LaCLIP.\n\nExtending our analysis to deeper or transformer-based architectures would require substantial technical advances. Beyond the nonlinearity introduced by ReLU activations, such extensions must address the multiplicative interactions of the attention mechanism and the highly nontrivial behavior of the softmax function. These components introduce dense, dynamic coupling across features, posing significant challenges for any rigorous analytical framework.\n\n## Response to Weakness 2: Recaptioning mechanism is not theoretically modelled or analysed\n\n**We theoretically analyze a simplified recaptioning mechanism in Appendix G and prove it enhances feature purity.** A simplified recaptioning mechanism $G(x)$ is actually theoretically analyzed in Appendix G, beginning at line 973. We could not include that in the main paper due to space limit, resulting in this confusion. We actually theoretically prove that fine-tuning the decoder on clean data $S_h$ can make the neurons become aligned with true features, i.e., their alignment is increased exponentially see Eq.(130), while spurious alignments remain negligible see Eq.(132). This theoretical analysis of a simplified captioning step serves as a prototype explanation, offering insight into the underlying mechanisms of recaptioning in practice. It supports the effectiveness of $G(x)$ in preserving relevant features and suppressing spurious ones, thereby enhancing feature purity.\n\n## Response to Weakness 3: The notion of spurious alignment (Cₛ) is theoretically assumed but not estimated or measured in real datasets.\n\nSame as Response to Question 5.\n\n## Response to Weakness 4: Additional Experiment - Similarity-Based Shuffling\n\nRandom shuffling may exaggerate misalignment. We therefore ran a more realistic variant: for each image we located the ten captions with highest similarity and randomly selected one of them for shuffling. Feature purity was measured by average absolute cosine similarity of all neuron pairs, identical to Figure 2(d). Results are shown below:\n\n|$C_s$|Random shuffle|Similarity shuffle|\n|---|---|---|\n|0.0|0.0562|0.0562|\n|0.1|0.0762|0.0693|\n|0.3|0.0875|0.0760|\n|0.5|0.1013|0.0847|\n|0.8|0.1270|0.0999|\n\nAlthough similarity-based shuffling yields lower absolute cosine similarity than fully random shuffling, feature purity still degrades as $C_s$ increases. Hence realistic misalignment produces the same qualitative trend, reinforcing our theoretical conclusions.\n\n## Response to Question 1: Extension to Deeper/Transformer Architectures\n\nSame as Response to Weakness 1.\n\n## Response to Question 2: Clarification on Assumptions and Structure\n\nSorry for the confusion. The perceived omission of Assumption 3.1 appears to stem from a misunderstanding. Due to the latex numbering system used in this paper, Definitions and Assumptions are numbered together. Assumption 3.2 appears after Definition 3.1, thus obtaining this numbering. No assumption has been omitted.\n\nWe will provide a clearer separation in the indexing and organization of the theoretical framework and empirical validation to improve the paper's clarity.\n\n## Response to Question 3: Feature Purity – Monitoring and Conceptual Clarity\n\nWe thank the reviewer for raising this point. The formal definition for measuring feature purity can be found in line 340. Specifically, we compute the absolute cosine similarity between every pair of learned neurons $v_j$ and $v_{j'}$ as $|\\langle v_j, v_{j'} \\rangle| / (\\|v_j\\| \\|v_{j'}\\|)$ for all $j, j' \\in \\{1,2,\\dots,512\\}$. A lower average value means neurons are more orthogonal whereas a larger value indicates feature mixing.\n\nFigures 2 apply this metric: (a–c,e–g) show that selecting high-purity (lower average absolute cosine similarity) neurons maximises zero-shot scores, whereas (d,h) confirm that higher shuffle rates reduce purity (average absolute cosine similarity increases)  as theory predicts.\n\n## Response to Question 4: Impact on Fine-tuning and Transfer Learning\n\nPurified representations, which emphasise true semantic features and suppress spurious correlations, offer a stronger starting point for gradient-based adaptation. In principle, this lets fine-tuning specialise relevant features more efficiently instead of spending capacity to correct entangled ones, thereby reducing sample complexity and accelerating convergence.\n\nA systematic empirical study of fine-tuning and domain-adaptation benefits is beyond the scope of the current work; we leave a detailed exploration of these transfer-learning effects as future research.\n\n## Response to Question 5: Estimability and Practical Relevance of $C_s$\n\n**While estimating $C_s$ in real datasets remains challenging, our theory provides meaningful qualitative insights across a broad regime.** The spurious alignment rate $C_s$ plays a central role in our theoretical framework by quantifying the likelihood that an image feature appears spuriously in low-quality image-text pairs. While our theorems assume that $C_s \\in \\left( \\omega(1/\\log d), 1/2 \\right)$, they remain valid throughout this broad regime—from low spurious alignment levels $\\omega(1/\\log d)$, up to moderate noise levels approaching $1/2$. Across this range, our analysis guarantees that recaptioning significantly improves alignment and downstream performance.\n\nThat said, we acknowledge that estimating or approximating $C_s$ in real-world datasets remains a challenging open problem. It may be possible to develop proxy metrics based on semantic similarity scores or caption relevance to approximate the degree of spurious feature alignment, but we leave this for future empirical work. Despite the absence of a direct estimator, the qualitative implications of our theory remain meaningful and broadly applicable in practical VLM training scenarios.\n\n## Response to Limitation 1: Real-world robustness\n\nOur theory considers the worst-case noise of totally irrelevant misalignment because semantic ambiguity is difficult to model analytically, and, to our knowledge, no existing theory fully addresses this form of ambiguity.\n\nDespite this simplification, we validated the key predictions on real datasets such as CC12M and LAION-400M and LaCLIP, which naturally contain ambiguous captions. The empirical results in Table 2 and Figure 2 align with our theory.\n\n## Response to Limitation 2: Estimating spurious alignment rate\n\nSame as Response to Question 5.\n\n## References\n[R1] Li et al., \"When Is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers,\" NeurIPS 2023.  \n[R2] Huang, Yu, et al. \"A Theoretical Analysis of Self-Supervised Learning for Vision Transformers,\" ICLR 2025.  \n[R3] HaoChen et al., \"Provable Guarantees for Self-Supervised Contrastive Learning,\" NeurIPS 2021.  \n[R4] Wen and Li, \"Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning,\" ICML 2021."}, {"review_id": "EtMF6qUl4L", "reviewer": "Reviewer_kXL9", "summary": "This paper studies the following text-image contrastive learning setting:\n- We are given a clean dataset $S_h$ and a noisy dataset $S_w$. \n- We first train a text embedding $h(y)$ and an image embedding $f(x)$ using the union dataset $S = S_h \\cup S_w$.\n- Meanwhile, we train a text generator $G: x \\mapsto \\hat{y}$ only using the clean dataset $S_h$. Here $x$ is an image and $\\hat{y}$ is the generated text caption.\n- We filter the data as follows: For each $x$, if the cosine similarity between $f(x)$ and $g(\\hat{y})$ is greater than that between $f(x)$ and $g(y)$, we replace $y$ with $\\hat{y}$. Then, we train two new embeddings $h'$ and $f'$ on this new filtered dataset.\n\nI suppose that one can continue this process iteratively, but it is not considered in the paper.\n\nThe authors consider a very specific setting of data and model. Under this very specific setting, they prove that the embeddings trained on the original union dataset learn spurious features, whereas the embeddings trained on the new filtered dataset are better. They empirically validate their theoretical result with synthetic experiments.", "questions": "I have some questions about the assumptions made in this paper:\n1. Why do you assume that \"the noise magnitude can dominate the signal\" (line 162)? Why is it a reasonable assumption? Will your results still hold if the noise magnitude is much lower?\n2. In line 172 you mentioned that $|S_h| = \\Theta (d^2)$ is much smaller than $|S_w|$. Where do you state this in your assumptions or results? I couldn't see how your results depend on the relative size of the two datasets. Or is it true that your results are independent of the relative size?\n3. In assumption 3.4, what is $z_{y_p}^j$ if $z_{x_p}^j = 0$? And why do you only consider a single pair of $(j, j')$ (line 186)? Is it true that you are assuming that $z_{y_p}$ and $z_{x_p}$ differ by at most two dimensions?\n4. In line 195, why should $W$ and $V$ be initialized at $\\bar{W}$ and $\\bar{V}$? Why is this initialization good provided that $\\bar{W}$ and $\\bar{V}$ are learned on the noisy union dataset? What would happen if one uses standard initialization?", "limitations": "Limitations are not discussed in the conclusion, though the authors claimed to do so in the checklist.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Overall, the setting of this paper is clear, though the paper is not particularly easy to follow. If we have a clean dataset and a noisy dataset, then it is reasonable to use the clean dataset to purify the noisy dataset before using it. Proving that such purification helps learn better features is useful but not extremely significant. I am more interested in the mechanism behind how such purification helps. Instead of the theorems, I am more interested in the \"key step\" that explains why the filtering reduces the spurious features in the dataset. I couldn't find it in the main body of the paper. I might have missed it but I suggest the authors highlight this key step if it is already in the main body.\n\nThe authors only compare the embeddings trained on the noisy dataset with the embeddings trained on the filtered dataset, but I think there are more embeddings that should be compared:\n- Trained on the clean dataset $S_h$ alone\n- Trained on the clean dataset $S_h$ plus the images $S_w$ with generated text only; that is, we never use the text in $S_w$\n\nThe second setting is a typical semi-supervised learning setting, and there are a bunch of theoretical papers on the effectiveness of semi-supervised learning. This setting is also not limited to vision-language contrastive learning. Pseudo-labeling is a very common technique in semi-supervised learning. Regarding the setting in this paper, I am especially curious whether the text information in $S_w$ is useful at all, since the authors consider \"high-noise conditions\" (line 165).\n\nMoreover, the setting considered in this paper is very specific. There are a number of very strong assumptions, making it questionable whether the proved results still hold for a slightly more general setting. I have some specific questions on the assumptions (see below).\n\nIn summary, I lean towards rejection because I am not sure how useful the proved results are, given that the setting studied in this paper is very specific and limited. Moreover, constructing a specific setting where using a generator trained on clean data (to generate pseudo labels, filter noisy data, etc.) helps improve the performance is not really hard, and has in fact been studied a lot in the context of semi-supervised learning. Instead, how such a generator could help leverage the weak signal in the noisy text captions from the noisy dataset is a more interesting question.\n\nI am willing to raise my score if the AC or the other reviewers find the results in this paper meaningful, though personally I am not a big fan of \"theoretical results\" proved by constructing extremely specific examples.", "was_revised": true, "final_justification": "The rebuttal addresses most of my questions. I have raised my score from 3 to 4.", "rebuttal": "We thank the reviewer for the valuable time in the evaluation.\n## Response to Weakness 1: Clarification on the specific setting and semi-supervised learning\n\nWe clarify the data model adopted in this paper is standard and not designed for our specific setting. Both our learning framework and theoretical analysis are different from those in semi-supervised learning and cannot be regarded as extensions of existing work in that literature.\n\n**The sparse-coding model we use is standard in theory and widely used to represent images and texts [R1, R2, R3].** Shallow networks remain SOTA for analyzing training dynamics and generalization [R4, R5, R6], as deeper models are harder to study. The one-hidden-layer model is still SOTA for theoretical analysis of single-modal contrastive learning.\n\n**Our contrastive setting follows the four-stage structure used by many frameworks.** E.g., BLIP applies the recaption–filter cycle iteratively; LaCLIP does it once. We do not aim to propose a new training method but instead give theoretical foundations for these existing strategies. This paper focuses on theory, not algorithmic novelty.\n\n**Our setting differs from semi-supervised learning [R7, R8, R9], which assumes labeled data is noiseless or nearly clean, and uses unimodal classifiers.** Our “clean” pairs are well-aligned but can be highly noisy (lines 162–169). We clarify that \"clean\" means aligned features, not noise-free. So standard semi-supervised theory cannot provide guarantees in our setting. In addition, contrastive learning aligns two modalities without labels, unlike semi-supervised learning with true or pseudo labels. This alignment requires analyzing coupled encoders under a non-convex contrastive loss, which is far harder than the single-encoder supervised case. As far as we know, no existing work gives full dynamic analysis in this multimodal setting. Should the reviewer request it, we are happy to provide further comparison or elaboration in the next stage of discussion.\n\n## Response to Weakness 2: Clarification of the Key Step in Removing Spurious Features\n\nTo avoid confusion in terminology: in our paper, *recaptioning* means generating synthetic text using \\( G(x) \\), and *filtering* refers to replacing some raw text with this synthetic text.\n\n**Both recaptioning and filtering are essential.** Appendix B.1 (line 626) shows that fine-tuning the decoder \\( G(x) \\) on clean data \\( S_h \\) amplifies alignment with true features exponentially while keeping alignment with spurious features negligible. Thus, the generated captions retain relevant semantics and suppress spurious correlations.\n\nFiltering uses these purified captions to replace misaligned ones, further eliminating spurious content. As shown in Theorem 4.3, the resulting dataset contains significantly fewer spurious features and more well-aligned pairs than the original data.\n\n## Response to Weakness 3: On Embeddings from Noisy vs. Filtered Data\n\nTraining only on $S_h$ reflects limited supervision in practice—$S_h$ is small and high-quality, while $S_w$ provides scale. High-capacity VLMs cannot be trained effectively on $S_h$ alone. Empirically, using only $S_h$ yields downstream accuracy 0.86, far below the $>0.90$ when including $S_w$ (Fig. 1(c)), so we exclude this baseline.\n\nFor the setting $S_h \\cup \\{(x, \\hat{y}) \\mid x \\in S_w\\}$, our theory directly applies. As shown in Lines 1053–1054, we compare cosine similarity between filtered ($\\langle z'_x, \\hat{z}_y \\rangle$) and fully replaced ($\\langle z'_x, z'_y \\rangle$) captions, selecting the higher similarity. We prove filtering is at least as effective as full replacement, and empirically performs better.\n\nPrior work ([R10], Fig. 2) studies how varying the ratio of synthetic/raw captions affects performance. Rather than repeating such ablations, our focus is to explain theoretically why recaptioning plus filtering improves feature purity and downstream results.\n\n## Response to Question 1: Noise Dominates Signal\n\n**The assumption that noise dominates the signal (line 160) is not necessary for our main theorems, but it helps to demonstrate that our main theorems work even in some extreme high-level noise cases.** Our results remain valid in the low-noise regime without any significant modification.\n\nWe highlight the high-noise case to emphasize that even under severe noise, our one-hidden-layer contrastive model can still learn useful features. The lower bound in Eq(160) is only to show robustness in this worst-case regime.\n\nThis assumption is motivated by prior work on supervised learning [R6], to justify the improved performance of nonlinear networks over linear models under high-level noise. Since our goal is not to compare linear and nonlinear models, we do not expand on this in the main text. To avoid confusion, we will remove the lower bound in (160) and add a footnote that our guarantees apply in both low- and high-noise regimes.\n\n## Response to Question 2: Clarification on Whether the Results Depend on the Relative Size of $S_h$ and $S_w$\n\nIn our theoretical analysis, we characterize the required batch size $B$ and the required number of training iterations for the model to converge. As we assume independence of the samples in different batches, the sample size requirement is at least $T \\times B$ (iterations times batch size).\n\n(i) Pretraining ITCP with contrastive loss (Theorem 4.1, line 241) uses $T = \\Theta(d^2 \\log d)$ and $B = \\Omega(d)$. This is Stage (S1) on raw data (line 102) and Stage (S4) on filtered data (line 118) with $S = S_h \\cup S_w$ and $\\tilde{S} = S_h \\cup \\tilde{S}_w$.\n\n(ii) Finetuning the decoder $G$ for recaptioning (Theorem 4.3, line 259) uses $T = \\Theta(d \\log d)$ and $B = \\Omega(d)$. This is Stage (S2) (line 107), where only pairs in $S_h$ are used to finetune the image-grounded decoder.\n\nVLM pretraining typically needs far more samples than decoder finetuning, so we model $|S_w| \\gg |S_h|$ to reflect practice, where human-annotated $S_h$ is small and web data $S_w$ is abundant.\n\nThe results will depend on sample size; specifically, larger samples will lead to faster convergence and smaller generalization error. Our intention was to study the worst-case scenario, aiming to identify the minimum number of samples needed for successful training. Therefore, we did not specifically analyze cases with varying sample sizes.\n\n## Response to Question 3: Assumption 3.4\n\nWhen $z_{x_p}^j = 0$, $z_{y_p}^j$ can still equal 1 if a different feature $j' \\neq j$ is active in the image and spuriously mapped to index $j$ in text. So:\n\n$$\\Pr(z_{y_p}^j = 1 \\mid z_{x_p}^j = 0) = \\Pr(z_{x_p}^{j'} = 1)\\\\Pr(z_{y_p}^j = 1 \\mid z_{x_p}^{j'} = 1) = \\frac{C_z}{d}C_s$$\n\nwhere \\( C_z/d \\) is the activation probability of any other feature \\( j' \\) in the image and \\( C_s \\) is the spurious alignment rate. Thus,\n\n$$\\Pr(z_{y_p}^j = 0 \\mid z_{x_p}^j = 0) = 1 - \\frac{C_z}{d} C_s$$\n\nWe assume a single spurious feature $j'$ per $j$ (forming $d/2$ disjoint pairs) only to simplify presentation. This can be generalized to $K-1$ spurious correlates per $j$, making Eq. (34) a $2K \\times 2K$ matrix (line 701) without altering the insight.\n\nTheorem 4.2 and its guarantees rely only on the total spurious mass (bounded by $C_s$), not on how many $j'$ exist per $j$. So the core mechanism is robust to such generalization.\n\nBoth $z_x$ and $z_y$ are $d$-dimensional sparse latent vectors (line 161). Spurious misalignment can occur on multiple dimensions, not just one pair. If $z_x$ has $k$ nonzeros, then $z_x$ and $z_y$ can differ on up to $2k$ coordinates.\n\n## Response to Question 4: Clarification on Initialization at $\\bar{W}, \\bar{V}$\n\n Although $\\bar{W}, \\bar{V}$ are learned on the noisy dataset $S = S_h \\cup S_w$, Theorem 4.2 shows that each neuron pair learns a mixed representation containing both relevant and spurious features. Despite the contamination, such weights preserve partial structure and are substantially better than random initialization.\n\nWhen fine-tuning the decoder $G$ on clean data $S_h$, initializing from $(\\bar{W}, \\bar{V})$ allows the model to purify relevant features, yielding accurate synthetic captions. This practice is standard in state-of-the-art VLMs such as BLIP [R11], where the decoder is initialized from a pretrained VLM and then fine-tuned on human-annotated image-text pairs.\n\nIt is possible to use standard random initialization, but that would require a larger set of clean data to achieve the same level of representation purity of the learned decoder.\n\n## References\n[R1] Yang et al., \"Linear Spatial Pyramid Matching Using Sparse Coding for Image Classification,\" CVPR 2009.  \n[R2] Yang et al., \"Robust Sparse Coding for Face Recognition,\" CVPR 2011.  \n[R3] Arora et al., \"A Latent Variable Model Approach to PMI-based Word Embeddings,\" TACL 2016.  \n[R4] HaoChen et al., \"Provable Guarantees for Self-Supervised Contrastive Learning,\" NeurIPS 2021.  \n[R5] Wen and Li, \"Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning,\" ICML 2021.  \n[R6] Allen-Zhu and Li, \"Feature Purification: How Adversarial Training Performs Robust Deep Learning,\" FOCS 2022.  \n[R7] D.-H. Lee, \"Pseudo-label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks,\" ICML 2013.  \n[R8] Zhang, Shuai, et al., \"How Does Unlabeled Data Improve Generalization in Self-Training? A One-Hidden-Layer Theoretical Analysis,\" ICLR 2022.  \n[R9] Wei, Colin, et al., \"Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data,\" ICLR 2021.  \n[R10] Nguyen, et al., \"Improving Multimodal Datasets with Image Captioning,\" NeurIPS 2023.  \n[R11] Li, Junnan, et al., \"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,\" ICML 2022."}]

Author rebuttal:

Meta-review: This paper develops a theoretical framework for understanding contrastive learning on noisy, misaligned image–text data, and shows that misalignment leads to mixed representations and reduced generalization, while improving alignment through recaptioning and filtering enhances feature purity and model performance. Although the paper considers strong assumptions and simplified settings (two-layer networks), all reviewers agree that this is a good paper. Therefore, I recommend accepting the paper.
Decision: Accept (poster)