Title: Mitigating Spurious Features in Contrastive Learning with Spectral Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Contrastive Learning

Spurious Correlation

Self-supervised Learning

Representation Learning
Abstract: Neural networks generally prefer simple and easy-to-learn features. When these features are spuriously correlated with the labels, the network's performance can suffer, particularly for underrepresented classes or concepts. Self-supervised representation learning methods, such as contrastive learning, are especially prone to this issue, often resulting in worse performance on downstream tasks. 
We identify a key spectral signature of this failure: early reliance on dominant singular modes of the learned feature matrix. To mitigate this, we propose a novel framework that promotes a uniform eigenspectrum of the feature covariance matrix, encouraging diverse and semantically rich representations. Our method operates in a fully self-supervised setting, without relying on ground-truth labels or any additional information. Empirical results on SimCLR and SimSiam demonstrate consistent gains in robustness and transfer performance, suggesting broad applicability across self-supervised learning paradigms. Code: https://github.com/NaghmehGh/SpuriousCorrelation_SSRL


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of spurious feature, also called \"simplicity bias\", in contrastive learning by examining the eigenvalue spectrum of learned representations. The authors argue that low-rank feature spaces, often a byproduct of learning spurious but easy-to-fit features, limit downstream generalization and robustness, notably on underrepresented data groups.\n\nTo address this, the authors introduce a spectrum-flattening regularizer that promotes a more uniform singular value distribution in the feature matrix. This regularization is theoretically motivated by the introduction of a linear classifier generalization bound, and then of a expected inverse energy over random classification tasks which appears to depend on the feature covariance matrix. The authors then showed that the expected inverse energy was minimized whenever the eigenvalues of FF.T are uniform. This simple, architecture-agnostic intervention increases the effective rank of the representation space. Empirical results on SpurCIFAR-10 and other spurious correlation benchmarks demonstrate improvements in worst-group accuracy and overall performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe method is theoretically grounded, offering a clear and intuitive motivation for addressing the spectral collapse caused by spurious features.\n\nThe paper is well-written and easy to follow, with a concise experimental setup that effectively communicates the core contributions.\n\nThe authors conduct a thoughtful and controlled evaluation, notably by subsampling majority groups to construct balanced training sets, avoiding common pitfalls in spurious correlation benchmarks.\n\nWeaknesses\nThe paper includes limited comparisons to related work, such as the implicit feature modification (IFM) of Robinson et al., which is cited but not empirically evaluated against.\n\nIt lacks broader evaluation across self-supervised learning paradigms. In particular, methods like BYOL and DINO, which exhibit implicit spectral regularization, are not considered—even though relevant theoretical work (e.g., Zhuo et al., ICLR 2023, https://arxiv.org/pdf/2303.02387) highlights spectral properties in non-contrastive frameworks.\n\nThe proposed method is not evaluated on standard self-supervised benchmarks such as ImageNet-1k or at least ImageNet-100 or STL-10. Given the method’s robustness to spurious features, one would expect improved generalization performance; such experiments would have helped assess its broader applicability.\n\nQUESTIONS:\nIn Theorem 4, you suppose that F satisfies the trace constraint, how valid is this assumption in practice ? Is it trivial ? Could you discuss it a little more ?\n\nIn Theorem 4, you argue that when F is a scaled orthogonal matrix, the expected inverse energy is minimized, wouldn't it be relevant to verify in practice that F gets close to orthogonality at the end of the training ?\n\nI found hard to understand the Fig. 2, could it be possible to explain it a bit more a make it clearer at his point of the paper ? You talk about dropping eigenvalues but you regularization do not work this way, so I am having trouble to understand.\n\nLIMITATIONS:\nThe authors should discuss a bit more their assumptions and compare with more related works as well as other contrastive and non-contrastive methods.\n\nI am prone to changing my opinion to Accept if these limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that standard contrastive self-supervised learning concentrates variance in a handful of feature directions; these dominant directions often encode spurious shortcuts, hurting robustness. The authors prove that, under a trace constraint, a uniform eigen-spectrum minimizes expected generalization error across random downstream linear tasks. Guided by this, they add a spectrum-flattening regularizer to SimCLR that penalizes unequal eigenvalues, effectively lifting small singular values and raising the feature matrix’s effective rank. This regularizer improves worst-group and average accuracy on five spurious-correlation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is very well written, clear and easy to follow. \n- The method is grounded in solid theoretical principles --  Theorem 4 rigorously shows that a perfectly uniform eigenspectrum minimises average linear-probe error under a trace constraint.\n- Hyper-parameters, optimiser settings, and code snippets for the SVD-based penalty are fully disclosed, making the approach easy to drop into existing contrastive SSL pipelines.\n\n**Weaknesses**\n\n- The proof assumes a square, full-rank feature matrix and purely linear downstream heads. Realistic encoders are tall (feature dim < #samples) and many transfer tasks use non-linear fine-tuning. The paper does not quantify how loose the bound becomes in this common regime, nor whether a flat spectrum remains optimal when features are low-rank.\n\n- Because the penalty uniformly boosts all low-eigenvalue directions, it can inadvertently raise purely noisy components if those directions are not spurious but irrelevant. The paper does not test on datasets where spurious and core signals overlap heavily, so this failure mode is unexamined.\n\n- The authors sub-sample majority groups when training the linear probe (Sec. 6.3). Prior work shows that simple re-balancing alone can recover much of the performance loss caused by group or class imbalance—for instance JTT (Liu et al., 2021), GroupDRO (Sagawa et al., 2020), and “Just Train Twice” (Jiang et al., 2021). Because of this step, the study (i) relies on explicit group labels at probe time, weakening its “label-free” narrative, and (ii) makes it harder to judge how much additional robustness comes from the spectrum-flattening regulariser versus the well-known benefits of balanced sampling. Reporting results with and without sub-sampling—or using natural group imbalance—would clarify the regulariser’s standalone contribution.\n\n- The regularization needs an SVD per mini-batch. At 512dim this is not that significant, but it grows quadratically with dimension. The authors mention possible Hutch++ approximations in Appendix K but does not implement or benchmark them. It is unclear how stable these approximations would be in SSL training.  \n\n- All experiments are only conducted on small scale datasets; no evidence is shown for ImageNet-1k, or ImageNet which are now the defacto for SSL evaluations. Further, more exhaustive and large scale benchmarks from WILDS could also be used. \n\n- The paper claims the framework is broadly applicable to representation learning methodologies, but the main experimental results only show it applied to SimCLR. Adding more baselines with the additional regularization will make the experiments section more complete. \n\n- (Minor) Related works section i.e. Section 3 starts abruptly after a nice motivation to the problem in Section 2. It might be better to either place it before Section 2 or towards the end to maintain and coherent flow of information.\n\nQUESTIONS:\nPlease see the weaknesses section above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that the reason why SSRL (self-supervised representation learning) methods learn spurious features, is that the directions which those features are learned dominates the top eigenvalues of the feature covariance $FF^T$. They formalize this by showing that a downstream linear probe (on top of feature representations) learns fastest along large eigenvector directions, so any label information in small eigenvalues is not being used when training the linear probe.  The paper also generalizes this to unknown tasks in Theorem 4, and shows that the expected loss is minimized when all eigenvalues are close to each other. Given this, they propose a Spectrum Flattening Regularizer for SSRL losses and show that it improves downstream worst-group accuracy across a number of tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1 (theoretical analysis motivated by the empirical observation): The authors start from an empirical observation (that the contrastive features collapse into high variance directions for datasets containing spurious correlations) and which motivates the theory. \n\nS2 (proposed method motivated by the theoretical analysis): The proposed spectrum flattening loss is directly derived from theorem 4's uniform eigenvalue optimum, which makes it principled. \n\nS3: The proposed loss is applicable to any sort of SSRL method that is learning a feature space, and adapting it for other SSRL method would be easy. \n\nWeaknesses:\n\nW1: Paper presentation and writing can be improved. For instance, the problem setup (section 4.1) comes after where these symbols first appear in the paper (section 2.1).\n\nW2: Even though the eigenvalues are flattened using the proposed method, the paper offers no analysis of how the directions itself evolves, which is important for understanding whether spurious vectors are suppressed or rescaled, though the downstream worst-group accuracy improves.  \n\nW3: The random-task model and the reduction to $tr[FF^T]^{-1}$ are presented too abruptly, and the paper’s transition from Corollary 3 to Theorem 4 makes it follow in its current form. The “random task” construction that supports Theorem 4 introduces several new vectors and scalars in a single dense paragraph without a guiding figure.\n\nW4: The theorem itself relies on specific assumptions that is unclear if they hold in practice, e.g. independent Bernoulli labels, a uniform $\\ell_\\infty$, a square full-rank feature matrix, and a trace (rather than Frobenius or spectral-norm) constraint. Especially for overparameterized encoders or correlated downstream tasks.\n\nQUESTIONS:\nQ1: The regularizer flattens the eigenvalues of the mini-batch feature covariance, but Theorem 4 is for the entire dataset. How sensitive is the training signal to the mini-batch estimates of the regularizer? could you plot the eigenvalues that vary across batches during training?\n\nQ2: Do spurious eigenvectors actually vanish when adding the regularizer? Is there a way to quantify this?\n\nQ3: Because eigenvalue estimates improve with larger batches, a fixed regularization $\\alpha$ may have different strengths across batches. Does changing the batch size significantly effect tuned $\\alpha$?\n\nLIMITATIONS:\nL1: Theorem 4 assumes square full-rank $FF^T$, independent bernoulli labels, and a trace constraint. It is unclear how the “uniform spectrum” optimum generalizes to overparameterized encoders or correlated tasks.\n\nL2: The regulariser flattens the eigenvalues of the batch feature covariance, which can diverge from the full dataset covariance when batches are small or highly correlated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of model reliance on spurious correlations in contrastive self-supervised learning. The authors propose a spectral regularization method that encourages the eigenvalues of the feature covariance matrix to be close to the largest eigenvalue, promoting a higher-rank representation. The paper provides a theoretical perspective linking downstream generalization error to the rank of the feature covariance matrix. Empirically, the method is applied to SimCLR and shows improved performance on benchmarks with spurious correlations, outperforming alternative approaches such as LateTVG on metrics such as worst-group and average accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper proposes a regularization method that encourages a high-rank feature covariance matrix that is supported by both theoretical analysis and empirical results, and leads to improved downstream performance.\n- The proposed regularization is architecture-agnostic. \n- The experiments report confidence intervals over five seeds, enhancing the statistical reliability of the findings.\n- The authors provide pseudocode and hyperparameter values which can enhance reproducibility.\n- In the questionnaire, they also mention that implementation of the method would be provided. \n\nRoom for improvement:\n- I believe the paper could benefit from including information about related work on regularization, the computational cost of the proposed method, and sensitivity analysis, some of which are available in the appendix. Please see details in the following section.\n\nQUESTIONS:\n1. **Related Work on Rank-Promoting Regularization.**\nThe related works section would benefit from an additional discussion of related literature that explores regularization techniques aimed at improving the rank or diversity of learned representations such as Barlow Twins, VICReg, Whitening-based approaches. Including these connections would help contextualize the proposed method and clarify its novelty relative to existing spectrum- or correlation-based regularizers.\n\n2. **Computational Overhead of Spectral Regularization.** The paper would benefit from quantifying the training and memory overhead introduced by the spectral regularizer, especially relative to other methods such as VICReg and Barlow Twins. Including this analysis in the main manuscript (rather than supplementary material) would improve transparency and help practitioners assess the cost-benefit tradeoff.\n\n3. **Sensitivity Analysis of the Regularization Strength Hyperparameter.**\nThe method introduces a hyperparameter that controls the strength of the spectral regularization. It would be helpful to include a sensitivity analysis showing how performance varies with different settings of this hyperparameter. This would provide insight into the robustness of the method and guide practitioners on how to tune it effectively.\n\nLIMITATIONS:\nYes, in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}