Title: A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Contrastive learning

data augmentation

SimCLR

approximate sufficient statistics
Abstract: Contrastive learning---a modern approach to extract useful representations from unlabeled data by training models to distinguish similar samples from dissimilar ones---has driven significant progress in foundation models. In this work, we develop a new theoretical framework for analyzing data augmentation-based contrastive learning, with a focus on SimCLR as a representative example. Our approach is based on the concept of \emph{approximate sufficient statistics}, which we extend beyond its original definition in~\cite{oko2025statistical} for contrastive language-image pretraining (CLIP) using KL-divergence. We generalize it to equivalent forms and general $f$-divergences, and show that minimizing SimCLR and other contrastive losses yields encoders that are approximately sufficient. Furthermore, we demonstrate that these near-sufficient encoders can be effectively adapted to downstream regression and classification tasks, with performance depending on their sufficiency and the error induced by data augmentation in contrastive learning. Concrete examples in linear regression and topic classification are provided to illustrate the broad applicability of our results.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a statistical framework for augmentation based contrastive learning such as SimCLR through the concept of approximate sufficient statistics. Three equivalent definitions of approximate sufficient statistics for general f-divergences are proposed. The authors show that minimizing a finite-batch InfoNCE loss yields encoders whose sufficiency gap is controlled by the excess empirical risk. \nThey then derive excess risk bounds for downstream regression and classification that depend on the encoder's sufficiency and the augmentation error.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Solid theoretical analysis for self supervised contrastive learning using data augmentation.\n1. Establishes a clear link between downstream risk and approximate sufficient statistics, and provides learning-theoretic bounds for those statistics.\n\n**Weaknesses**\n\n1. Their second claimed contribution (line 44) sounds vague. They assert that random transformations in SimCLR \"introduce additional challenges for theoretical analysis,\" but they do not identify the challenges of applying Oko et al. 2025 to the contrastive learning for a single modality data.\nAt the same time, their first claimed contribution (line 39) of extending approximate sufficient statistics from the KL divergence to any f divergence and proving three equivalent forms remains a modest generalization of Oko et al. 2025.\n\n1. SimCLR defines specific data augmentation schemes that are central to its success, yet the paper did not derive the resulting augmentation bias $\\epsilon_{\\mathcal{G}}$ for any of these transforms.\n\nQUESTIONS:\nI would consider increasing the score after the following questions are at least partly addressed.\n1. Could you explain the technical gap or challenges to Oko et al. 2025 when dealing with a self-supervised contrastive learning for a single modality data?\n1. The paper introduces the augmentation error term $\\epsilon_G$. Do you have any upper bounds for any real augmentation such as random cropping, color jitter, or blur? \n1. As a curiosity, how does the temperature parameter in the InfoNCE loss affect convergence rate?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a new theoretical framework for data augmentation-based contrastive learning, focusing on SimCLR. It generalizes the concept of approximate sufficient statistics from prior work on CLIP to general f-divergences and establishes that minimizing contrastive losses yields encoders that are approximately sufficient. The authors show that minimizing contrastive losses (like InfoNCE) yields encoders that are approximately sufficient, and that the sufficiency of these encoders, together with the error induced by data augmentation, governs their adaptability and performance on downstream tasks. The paper provides concrete examples in linear regression and topic classification to illustrate the applicability of the theory.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper significantly extends the notion of approximate sufficiency from its prior, more limited form (KL-divergence for CLIP) to a general framework encompassing various f-divergences and equivalent mathematical formulations. This generalization is both non-trivial and valuable for the broader contrastive learning community.\n- The definitions of sufficiency (information loss, variational form, conditional Bregman) are carefully presented and shown to be equivalent, providing a solid mathematical foundation.\n- In Sec. 3.1, the paper establishes a sufficiency bound for the ERM estimator in and shows that when f minimizes the ERM, the bound is a function of a generalization error (which goes to 0 as the number of samples tend to infinity) and a constant approximation error.\n- Then, in Sec. 3.2, the paper establishes that the downstream performance of learned encoders on linear regression and topic classification depends on both their sufficiency and the augmentation-induced error, providing a theoretical explanation on previous work focusing on how the data augmentions affect the performances of SimCLR's encoder.\n- In Sec 3.3, the authors generalizes to other loss terms than SimCLR, by generalizing their result of other type of f using general f-sufficiency. In Eq. 9, they notably extend to the chi-squared sufficiency. This generalization is appreciated and is a strength of the paper.\n\nWeaknesses:\n- The paper is primarily theoretical; while it provides illustrative examples, it lacks empirical experiments that would demonstrate the practical impact of the theoretical bounds and insights in real-world settings. Notbaly, it would have been interesting to study the use on real dataset of various f, such as teh chi-squared contrastive loss for ex. Besides, we would have expected an experimental comparison between empirical risk and theoretical risk, using either real-world dataset (if applicable), or a synthetic one.\n- Overall, the paper is hard to read, and the use of the term \"encoder\" to both talk about the neural network and the f of the f-MI makes it confusing.\n- While the paper discuss how the sufficiency of the encoder impact the downstream performances, it eludes past work about identifiability (Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, von Kugelgen et al. NeurIPS 2021, https://openreview.net/pdf?id=4pf_pOo0Dt), even though the connection between identifiability and encoder-sufficiency appears intuitively close.\n- While the theoretical bounds are elegant, they rely on assumptions such as boundedness of the score function and invertibility/Lipschitz continuity of the link function. In practical neural networks, these conditions may not strictly hold. Discussing these assumptions would be beneficial for the paper. Moreover, the theory abstracts away from architectural details (e.g., depth, parameter count), which are known to affect the performance of learned representations. A discussion on how architecture choices influence sufficiency and generalization would strengthen the practical relevance of the results.\n\nQUESTIONS:\n- Is it straightforward to extend your theoretical results to multimodal contrastive learning settings (e.g. image-text), in line with identifiability analyse such as in Daunhawer et al. ICLR 2024 https://arxiv.org/pdf/2303.09166, IDENTIFIABILITY RESULTS FOR MULTIMODAL CONTRASTIVE LEARNING\n- How does your approximate sufficiency framework encompass or relate to alternative contrastive/self-supervised objectives such as VICReg, Barlow Twins, SwAV, and BYOL, DINO, where some of them do not explicitly use negative samples, and yet surpasses SimCLR in most natural imaging experiments ?\n- Could you clarify the practical validity of your assumptions (e.g. bounded score functions, invertibility and Lipschitz continuity of the link function), particularly in the context of deep neural network encoders?\n- Could you clarify the impact of the encoder architecture on your generalization bounds ? Does it impact the approximation error ?\n- Do you have experimental results comparing the empirical performance of encoders trained with different f-divergences (e.g. KL vs. chi-squared contrastive losses) on standard datasets?\n- Could you discuss how your present work connects/relates to identifiability ?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretical framework for analyzing data augmentation-based contrastive learning, particularly SimCLR, by means of approximate sufficient statistics, which extends a prior work using KL-divergence to general f-divergences. Specifically, this work shows that minimizing a contrastive loss produces encoders approximately sufficient, and they can be adapted to downstream regression and classification tasks, where downstream performance depends on their sufficiency and the error induced by data augmentation in contrastive learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n+ The idea of analyzing contrastive learning via approximate sufficient statistics sounds interesting.\n\n+ The claim is supported by comprehensive theoretical derivation.\n\n+ Concrete examples on regression and classification are provided, though they are limited to linear models.\n\n\n**Weaknesses**\n\n- There is no experimental results, raising a concern on the practicality of the proposed idea. At least the prior work by Oko et al. [27] conducted some experiments with CLIP. If the theoretical results deviate from practical nonlinear models, then their contribution is limited accordingly. To address this concern, the authors could first conduct toy experiments with linear models as in the examples in Section 4, and then extend it to nonlinear models and see if their theoretical results empirically hold, or how much they are deviated from practical models.\n\n- L252: It is not clear how \"end-to-end theoretical guarantees for the downstream performance of encoders obtained by minimizing general f-contrastive losses\" is drawn by combining the results from Sections 3.3.1 and 3.3.2. \n\n- There are many \"some constants\" throughout the paper. How are they determined, at least in examples? How much are bounds tight?\n\n- The error on the downstream task induced by data augmentation epsilon_G is an important variable in their analysis but it is not specified well. For example, it would be useful to know how we can measure epsilon_G in practice, how much its value deviates by taking different augmentation strategies, and so on.\n\n- Formatting issue: generally speaking, any citation should not be appeared in Abstract.\n\n- Typo: f-sufficieny in the title of Section 3.3.1\n\nQUESTIONS:\nPlease address concerns in Weaknesses above.\n\nLIMITATIONS:\nDifferent from the checklist, limitations are not provided.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a statistical theory of contrastive learning via the approximate sufficient statistics, following~\\cite{oko2025statistical} for contrastive language-image pretraining (CLIP) using KL-divergence. Specifically, the authors generalize the concept of the approximate sufficient statistics to equivalent forms and general $f$-divergence, and illustrate that minimizing SimCLR and other contrastive losses yields encoders that are approximately sufficient. The key factors influencing these near-sufficient encoders are their sufficiency and the error induced by data augmentation in contrastive learning. Examples, including linear regression and topic classification, are given to illustrate the applicability of results.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n1. Overall, this paper is well-written. The related work and comparison with this work are discussed, especially the work~\\cite{oko2025statistical}.\n2. The claims about the contributions are supported. Specifically, (1) it generalizes the concept of the approximate sufficient statistics in~\\cite{oko2025statistical} for contrastive language-image pretraining (CLIP) using KL-divergence; (2) it provides the theoretical analysis of data augmentation-based contrastive learning following the SimCLR framework, demonstrating the downstream performance of the learned encoder depends on its sufficiency and the error induced by the random transformation.\n3. The proofs seem right, although I have not checked the proof details line by line.\n\n# Weaknesses\n1. Technically, the novelty seems a little limited due to its similarity to the work~\\cite{oko2025statistical}. More details about the additional challenges are suggested to be provided.\n2. The tightness of the bounds in Eq.(4) in Theorem 1, Eq.(7a) and Eq.(7b) in Theorem 2, and Eq.(8) in Theorem 3,  are not discussed. \nBesides, for the classification problem of Theorem 3, why use KL divergence instead of common metrics (e.g., 0-1 loss) to evaluate the classification performance?\n2. There are no experimental results to support the theoretical results. Are the provided bounds non-vacuous or meaningful? Synthetic experiments on examples of linear regression or classification can be conducted to illustrate the validity of theoretical results.\n\nQUESTIONS:\n1. Do the provided bounds in Eq.(4) in Theorem 1, Eq.(7a) and Eq.(7b) in Theorem 2, and Eq.(8) in Theorem 3 tight? Please give more discussions.\n2. Please give experimental results to support the validity of the theoretical results in Theorem 6.\n\nLIMITATIONS:\nPlease see the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}