Title: Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: self-supervised representation learning

transfer learning

adversarial training

error analysis
Abstract: Learning transferable data representations from abundant unlabeled data remains a central challenge in machine learning. Although numerous self-supervised learning methods have been proposed to address this challenge, a significant class of these approaches aligns the covariance or correlation matrix with the identity matrix. Despite impressive performance across various downstream tasks, these methods often suffer from biased sample risk, leading to substantial optimization shifts in mini-batch settings and complicating theoretical analysis. In this paper, we introduce a novel \underline{\bf Adv}ersarial \underline{\bf S}elf-\underline{\bf S}upervised Representation \underline{\bf L}earning (Adv-SSL) for unbiased transfer learning with no additional cost compared to its biased counterparts. Our approach not only outperforms the existing methods across multiple benchmark datasets but is also supported by comprehensive end-to-end theoretical guarantees. Our analysis reveals that the minimax optimization in Adv-SSL encourages representations to form well-separated clusters in the embedding space, provided there is sufficient upstream unlabeled data. As a result, our method achieves strong classification performance even with limited downstream labels, shedding new light on few-shot learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a new self-supervised learning approach named Adversarial Self-Supervised Representation Learning (ASSRL) for unbiased transfer learning. Through an innovative iteration format, ASSRL eliminates the bias between the population risk and its sample-level counterpart. In this paper, the end-to-end theoretical guarantee for the downstream classification task performance is provided. This theoretical understanding also explains the effectiveness of few-shot learning for downstream tasks with small sample sizes. Experiments are presented in this paper to show the effectiveness of the ASSRL method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper proposed a new adversarial contrastive learning method for unsupervised transfer learning.\n2. End-to-end theoretical analysis for the downstream classification task is given in this paper.\n3. Experimental results are given to show the improved accuracy for this new ASSRL method.\n\nWeaknesses:\n1. The experimental results comparison only covers a few existing self-supervised learning methods.\n\nQUESTIONS:\n1. For the question the authors raised in line 187, do the authors aim to answer this question just by the empirical experiments in Table 1? Or can this question also be answered from the theoretical perspective? Please elaborate on this question in more detail.\n\n2. For the experimental results, why did the authors select some of the existing methods from [14] for comparison? Any reason for choosing only these self-supervised learning methods for comparison? \n\n3. Why is the comparison not made with more state-of-the-art self-supervised learning methods? For example, what about those self-supervised learning methods that are mentioned in the introduction of this paper?\n\nLIMITATIONS:\nThe limitations are discussed in the appendix of this paper, mainly about the minimax rates and the lower bound for the ASSRL method. Other limitations see  the question part.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies that many self-supervised learning methods suffer from a biased sample risk, which causes optimization instability and prevents theoretical analysis. To solve this, the authors propose Adversarial Self-Supervised Representation Learning (ASSRL), a novel method that reframes the objective as a minimax problem to create an unbiased estimator. This adversarial approach leads to significantly improved classification accuracy, outperforming both the biased methods it aims to correct and other mainstream techniques on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-\tThe paper clearly identifies a significant and practical problem in existing SSL methods—the bias in the sample-level risk approximation. The proposed solution, reformulating the objective as a minimax problem to create an unbiased estimator, is elegant and directly addresses this core issue.\n-\tA major strength of this work is the end-to-end theoretical analysis. Unlike prior work that often focused only on population-level risk or generalization error without considering approximation error, this paper provides a complete convergence analysis. The theory formally connects the number of unlabeled source samples to downstream task performance, which is a crucial aspect of transfer learning.\n-\tThe theoretical results offer a formal explanation for the success of SSL in few-shot learning contexts. The paper proves that with a sufficient amount of unlabeled data, ASSRL can learn representations that are well-clustered by category, thereby enabling strong classification with very few labeled samples. This bridges an important gap between the empirical success and theoretical understanding of few-shot learning.\n\nWeaknesses\n-\tThe theoretical guarantees rely on a number of assumptions (Section 3.3), particularly regarding data augmentations (Assumption 4) and the shift between source and target domains (Assumption 5). These assumptions, such as the existence of an augmentation sequence where distances between similar samples converge to zero, might be difficult to verify or guarantee in practice. Could the authors provide more intuition on why these assumptions are reasonable for standard computer vision datasets and augmentation pipelines?\n-\tThe paper introduces an alternating optimization algorithm (Algorithm 1) but does not analyze its practical implications in detail. There is no discussion of the computational overhead of the adversarial updates compared to the simpler direct minimization in methods like Barlow Twins. The ablation study on the regularization parameter λ shows that its optimal value varies significantly across datasets (e.g., 5.0×10−2 for CIFAR-10 vs. 1.0×10−2 for CIFAR-100). This suggests the method may be sensitive to this choice, requiring expensive tuning for new applications. A more in-depth analysis of this sensitivity would be valuable.\n-\tThe baselines in the key comparison table (Table 1) are re-implementations by the authors. While the authors state that their results are close to those in the LightlySSL library, a direct comparison to the numbers reported in the original papers (e.g., [22, 38]) would provide a stronger and more objective benchmark. This would help ensure that the observed performance gap is entirely due to the methodological improvement of ASSRL and not differences in implementation.\n-\tThe paper's checklist (Question 7) states that the experimental results report the best performance over three runs, rather than providing error bars or other measures of statistical significance. While the reported performance gains are large, this approach makes it difficult to assess the stability and variance of the results. Standard practice would be to report the mean and standard deviation over multiple runs, which would make the empirical claims much more robust and convincing.\n\nQUESTIONS:\nPlease see the comments above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an improvement (termed ASSRL) in how self-supervised learning methods regularize their optimization. Specifically, they improve upon the regulization term that encourages alignment of the feature covariance matrix with the identity matrix. Prior work relied on a biased estimate of this term, leading to instability in optimization and preventing some theoretical analysis. The author's proposed method reformulates this term as a minimax optimization problem that can be optimized through an alternating optimization algorithm. This new formulation provides an unbiased estimate. While the authors primarily focuse on the theoretical soundness of their method, they also include empirical results that show their method does in fact enable an improvement over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\n\n- Very clearly written.\n- Motivations, background context, and logical flow of paper are very easy to follow.\n- Theory looks good and is clearly explained. Comprehensiveness of the analysis is a big plus.\n- Assumptions used seem reasonable / are similar to assumptions made in prior works. This is important--the theory would be less meaningful and useful if the assumptions were overly restrictive.\n- Empirical evaluations show generalization of theory to practice.\n\n\nWEAKNESSES\n\n- Empirical evaluations are limited in scope. While primary focus is the theory, so I am largely okay with this, evaluations beyond CIFAR and tiny ImageNet would further bolster their results.\n- Additionally, empirical results are all positive, but it is unclear how significant this is in terms of actual utility in situations where self-supervised learning is used in practice--CIFAR and tiny ImageNet are primarily used for small scale testing rather than real world use.\n- Table 2 compares the author's results with results from a prior paper. I think this is mostly fine, but it does slightly detract from the results, since training conditions may not be identical.\n\nQUESTIONS:\n- How much of an impact does the method have on (empirical) training time?\n- How well does training scale with image/model size (the empirical evaluations all use relatively small image and model sizes).\n\nLIMITATIONS:\nLimitations adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the regimen where unlabeled data is used to learn a representation that aligns an image with its augmentations with the hope of learning semantic information useful for a downstream supervised task. Of several methods towards this end surveyed in the introduction, they use a regularization approach where the covariance over augmentations is aligned towards the identity matrix (1) and thus encouraging separability. The authors introduce a modification of this regularizer that leads to a minimax learning algorithm they term Adversarial Self-Supervised Representation Learning (ASSRL). This leads way to a natural numerical alternating procedure, Algorithm 1, which they use in experiments to demonstrate their method; These experiments demonstrate the success of their method in comparison to prior work. Finally, Theorem 1 provides a theoretical guarantee for an RERM version of their algorithm for ReLU neural networks for classification.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The problem is well motivated and the algorithmic setting is valuable in practice. \n* The work is extensive in the sense that it has experiments, practical numerical algorithms, a theoretical guarantee, etc. \n* To my knowledge the work is original. Their modification of the regularizer is reminiscent of other methods that could be viewed as searching over all one dimensional projections.\n* It's a generally well thought out paper. The Appendix has several sections that the reader will find useful, as I did. For example see the sections justifying the assumptions made.\n\n**Weaknesses**\n* Theorem 1's guarantee doesn't hold for the result of Algorithm 1. Of course, this is common in the literature as the problem is non-convex.  A note just specifying that Algorithm 1 likely doesn't return the RERM in Theorem 1 may be useful to the reader. \n* I was confused by their augmentations discussion (see below)\n* To my eye, which doesn't include much familiarity with prior work, I do not find the rate in Theorem 1 very interpretable. I'd appreciate more remarks on interpreting the rate of decay for each term in Theorem 1 perhaps, if possible, an example. Concretely there are with the several $\\mathrm{min}$s that make it to tell which setting achieves which argument therein.\n* Theorem 1's rate is a mixture of asymptotic and finite bounds. Asymptotic for the source task and finite for the target task. \n \nGeneral Writing notes: \n* Although perhaps implicit in context I think the reader would be aided in learning that ReLU neural networks for classification is the model used.\n\nQUESTIONS:\n**About Augmentations**\n* Line 150, the authors say that $m$ could be infinite, yet later we consider a uniform density over all augmentations. Could the authors clarify this? \n* In order to approximate some augmentation classes we require a cover that is exponential in dimension. e.g., rotational transformations. Thus, there can be additional dimensional dependence for augmentations commonly used. How does this affect the rate?\n*  It seems to me that a covering based argument over continuous augmentation classes would be more direct. Can the authors comment on the difficulty or feasibility of adapting their argument to this setting? \n\n**Theorem 1**\n* Is there a reason why a finite-sample bound is used for the source task? \n*  Is the  $1/\\mathrm{min}_k \\sqrt{n_t(k)}$ natural here should we expect something that is more balanced as as we get more data for all classes except for one. \n\n\n**On approach** (my biggest concern)\n* For fixed $f$ and $G$ it is clear that  $\\mathcal{R}(f, G)=\\mathbb{E}_{\\widetilde{D}_s}\\{\\widehat{\\mathcal{R}}(f, G)\\}$.   \n\nYet, as I understand, this isn't true for $\\mathbb{E}_{\\widetilde{D}_s}$$ \\sup _{G \\in \\widehat{\\mathcal{G}}(f)} \\widehat{\\mathcal{R}}(f, G)$. \n\nFor example in (4) on line 174, on the left hand side we see taking the maximum over $G$ in $\\hat{\\mathcal{G}}(f)$ which is equal to a term for which there is no max over $G$. As I understand the latter is ASSRL because it's the adversarial minimax and thus it appears to me to be similarly biased in comparison to prior work. And therefore I'm not sure how Section 3.2 provides a justification for the method.\n\nAm I misunderstanding the method? \n\n**Miscellaneous**\n* Are ReLU networks a Holder class? As their partials may not exist everywhere.\n\nLIMITATIONS:\nRepeating a two points mentioned above,  the function in Theorem 1 doesn't apply to the numerical procedure and the \"mixed bounds\" for Theorem 1 (asymptotic versus finite) .", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an unbiased adversarial self-supervised representation learning framework (ASSRL) that removes the estimation bias of covariance regularization through a min–max formulation. It provides end-to-end risk bounds for transfer and few-shot settings and achieves consistent improvements on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper offers the first end-to-end risk bound for non-contrastive self-supervised learning under limited data and distribution shift, filling a gap in the literature. \n\n2. By leveraging adversarial training, the method corrects the bias inherent in mini-batch estimation, avoiding the cumulative drift observed in Barlow Twins and WMSE.\n\nWeaknesses:\n1. The maximization step is realized by directly setting G equal to the empirical estimate, rather than through a dynamic min–max game. The main paper does not clarify this equivalence, which may lead readers to overinterpret the “adversarial” aspect. In practice, the procedure is not truly adversarial.\n\n2. The empirical improvements, while consistent, are modest. Table 2 should include additional recent decorrelation-based non-contrastive methods, notably VICReg [a] and LogDet [b]. The reported accuracies slightly exceed VICReg but are below LogDet. Moreover, more recent contrastive and non-contrastive baselines should be included.\n [a] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, ICLR 2022.\n [b] Geometric View of Soft Decorrelation in Self-Supervised Learning, KDD 2024.\n\n3. Empirical analyses are insufficient.\n1）The claim that “ASSRL-learned representations cluster downstream data by class” should be supported by visualizations or quantitative metrics.\n2）Assumptions 4 and 5 depend on parameters such as  $\\sigma_{s}^{(n)},\\sigma_{t}^{(n)},\\delta_{s}^{(n)},\\delta_{t}^{(n)}$ for augmentations and $\\epsilon_{\\mathrm{ds}}$ for domain shift. These should be measured in experiments and toy examples to validate the assumptions.\n\n4. The theoretical setting does not fully align with the experiments. The required network width and depth, sample sizes, augmentation constraints, and downstream domain settings are stricter than those used in practice. Assumptions 4 and 5 are particularly stringent and difficult to verify. Additionally, a lower bound on the error is absent. These limitations hinder the theory’s direct applicability to implementation and engineering.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}