Mitigating Spurious Features in Contrastive Learning with Spectral Regularization
Authors: Naghmeh Ghanooni
Waleed Mustafa
Dennis Wagner
Sophie Fellenz
Anthony Widjaja Lin
Marius Kloft
Venue: NeurIPS 2025 poster ()
Keywords: Contrastive Learning
Spurious Correlation
Self-supervised Learning
Representation Learning
Primary area: general_machine_learning
TL;DR: We introduce a regularizer that promotes diverse, task-relevant features over spurious ones in contrastive learning.

Abstract: Neural networks generally prefer simple and easy-to-learn features. When these features are spuriously correlated with the labels, the network's performance can suffer, particularly for underrepresented classes or concepts. Self-supervised representation learning methods, such as contrastive learning, are especially prone to this issue, often resulting in worse performance on downstream tasks. 
We identify a key spectral signature of this failure: early reliance on dominant singular modes of the learned feature matrix. To mitigate this, we propose a novel framework that promotes a uniform eigenspectrum of the feature covariance matrix, encouraging diverse and semantically rich representations. Our method operates in a fully self-supervised setting, without relying on ground-truth labels or any additional information. Empirical results on SimCLR and SimSiam demonstrate consistent gains in robustness and transfer performance, suggesting broad applicability across self-supervised learning paradigms. Code: https://github.com/NaghmehGh/SpuriousCorrelation_SSRL

Mitigating Spurious Features in Contrastive Learning with Spectral Regularization
Naghmeh Ghanooni
Waleed Mustafa"
Department of Computer Science, RPTU Kaiserslautern, Germany
Department of Computer Science, RPTU
Kaiserslautem, Germany
aaanooni@rptu..
mustafadrptu.cm
Dennis Wagner
Anthony Widjaja Lin Max-Planck Institute for Software Systems Kaiserslautem, Germany lin@cs. uni-kl.de
Department of Computer Science, RPTU Kaiserslautern, Germany
vagnerdrptu.de de
Sophie Fellenz
Marius Kloft
Department of Computer Science, RPTU Kaiserslautern, Germany fellenz8rptu.de
Department of Computer Science, RPTU Kaiserslautern, Germany kloft@rptu.de
Abstract
Neural networks generally prefer simple and easy-to-learn features. When these features are spuriously correlated with the labels, the network's performance can suffer, particularly for underrepresented classes or concepts. Self-supervised representation learning methods, such contrastive learning, especially prone to this issue, often resulting in worse performance on downstream tasks. We identify key spectral signature of this failure: early reliance on dominant singular modes the learned feature matrix. To mitigate this, we propose a novel framework that promotes uniform eigenspectrum of the feature covariance matrix, encouraging diverse and semantically rich representations. Our method operates a fully self-supervised setting, without relying on ground-truth labels any additional information. Empirical results on SimCLR and SimSiam demonstrate consistent gains in robustness and transfer performance, suggesting broad applicability across self-supervised learning paradigms. Code: GitHub repository
1 Introduction
Neural networks tend to prioritize learning simple and easily detectable features in the early stages of training before capturing more complex patterns, revealing an inherent simplicity bias in neural network optimization [Geirhos et al. 2020 Shah et al.) 2020 Rahaman et al. 2019 Kalimeris 2019 Xue al. [023]. While this inductive bias accelerate learning. it also makes models vulnerable to relying on features that are only superficially correlated with the labels due to dataset-specific artifacts.
al.
These features, known as features,, can dominate model behavior, since they are easier to learn than the true task-relevant signals (Sagawa al. 2019 Qiu al. 2024 (Sagaaa al. 2020 Kirichenko et al. 2022 Murali et al. 2023 For instance, Zech et al. 2018 showed that model
"Equal contribution; authors listed alphabetically.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trained to detect pneumonia in chest X-rays relied on visual artifacts such as hospital-specifi metal tokens- rrater than medical markers of the disease. Although these shortcuts may lead to strong performance on training and validation distribution, they often fail to generalize to rarer or more challenging inputs. Thus, models that appear accurate may perform poorly across the full spectrum of real-world variation.
Recent work shows that self-supervised representation learning (SSRL) not immune to spurious
correlations either, even though trained on unlabeled data (Hamidich et al. 2024 Zhu et al. [023) In
particular, contrastive learning (CL; a popular technique in SSRL) learns features- also referred to as representations -tht are intended to be broadly useful across a wide range of downstream tasks, by encouraging similarity between augmented views of the same instance [Chen et al. [020 He
et al.; 20200] Grill et al., (2020) Chen and He [021]] Caron et al.) 20200 (Zbontar et al.) [021]] However,
CL's objective of maximizing agreement between views can lead the model to rely on patterns that
are predictive within the training distribution but unreliable in unfamiliar or varied conditions. In
doing so, CL. may overfit to spurious signals present in the training data, ultimately limiting the
generalization of the learned representations novel or diverse settings.
Whether feature is considered spurious depends on the downstream task. A features that is irrelevant-or even misleading-ffor one task may be essential for another. This task-dependence highlights key challenge in self-supervised learning: the absence of knowledge about the downstream task during pretraining. Without this information, task-relevant features (often referred to as core features) and spurious features are indistinguishable from the perspective of the unlabeled training data. As result, self-supervised methods can entangle both in their learned representations, limiting their robustness and transferability. Despite progress in SSLR, we lack a theoretical understanding and practical solution for how to mitigate the influence of spurious signals during self-supervised pretraining.
To evaluate the quality of the learned features, it is common practice to assess how well simple linear classifiers perform on downstream tasks. Which raises the question, whether we can train single, general-purpest representation that enables a wide range of downstream tasks to be solved with nothing more than shallow model? We explore this question by theoretically investigating why standard networks tend to prefer spurious features. We show that the generalization performance of a downstream task closely tied to the spectrum of the matrix XXT, where X is the input feature matrix. Dimensions associated with larger eigenvalues are learned first. Since spurious features are typically simple and easy to learn, they dominate the top eigenspectrum, leading the model to focus on them and reinforce them disproportionately.
To counteract this tendency, we introduce a rank-promoting regularizer that flattens the spectrum of XX T_ By lifting the smaller eigenvalues and reducing their disparity with the largest ones, the method encourages the model to allocate capacity more uniformly across all informative directions, promoting the learning of both core and spurious factors. Importantly, the regularizer is architecture- agnostic and can be seamlessly integrated into any SSRL pipeline without modification.
The main contributions of this paper are summarized as follows:
We introduce a synthetic dataset that elucidates why neural networks tend to prioritize spuri- ous features over core features, and demonstrate-via rank analysis-hov such preferences reduce representational diversity during training.
We provide a theoretical analysis showing that generalization in downstream tasks depends the eigenvalue spectrum of X XXT, with uniform spectra yielding optimal transferability.
Based on this insight, we propose a simple, architcttre-agnostic regularizer that flattens the spectrum of XXT, encouraging diverse representations. The regularizer integrates seamlessly into any SSLR with negligible overhead.
Experiments across five challenging gourious-correlan benchmarks show that the method substantially improves performance on the most difficult portion of the dataset-worst-grroon accuracy- while also increasing average accuracy, and achieves new state-of-the-an results on multiple downstream tasks.
2

---

CIPAR100
80-
Average Accuracy
Effective Flank
40-
60-
CIPAR-10
40-
SourCIFAR-00
Epoche
Epochs
(a) The effective rank is lower when training on CIFAR- (b) Classification performance on CIFAR- is worse in the presence of artificial spurious features.
10 with artificial spurious fcatures.
Figure Training a classifier on CIFAR-IO with artificially added, strongly correlated spurious features results in noticeably worse performance and a lower effective rank of the learned representa- tions, suggesting potential connection between representation diversity and generalization.
2 Spectral Imbalance Limits Downstream Flexibility
Neural networks are known to prioritize simple, easily learnable features during training [Geirhos) jtt al.) 2020, Rahaman et al., 2019 Xue et al., [023]. In self-supervised pretraining, this often leads that easy align between augmented views but
to the dominance of spurious features-then are semantically irrelevant and fail to generalize beyond the training distribution. These dominant components shape the spectral structure of the learned representation space. In this section, show how such spectral imbalance constrains downstream learning, and how more uniform spectrum enables better task adaptation.
2.1 Gradient Flow and Spectral Bias
To understand how spectral structure biases downstream learning, we analyze the optimization dynamics of linear predictor trained frozen representations from a pretrained encoder. Let f(-) denote a feature representation. Given a training set {I, J define the corresponding feature matrix as = .., )] E Rexd We train a linear predictor Iw (xx) = (f(x), w) using gradient flow on a loss c;
dw
dg
dΦ Ala
d$
-FFF
dt
dw
dt
dg
dg
where FFT E. N is the eigendecomposition of the feature covariance matrix.
This shows that the evolution of predictions dominated by top eigendirections of FFT as those with larger eigenvalues accelerate learning. Consequently, downstream predictors are biased toward dominant spectral directions- surectons-rrgardlesss. of whether they encode meaningful or spurious information. Since spurious features are often the easiest to learn during pretraining, they are likely to occupy top eigenspaces, constraining downstream learning to brittle, semantically weak directions.
To quantify spectral imbalance, we use the effective rank, principled measure based on the entropy of the singular value spectrum:
Definition (Effective Rank [Roy and Vetterli
2007
Let A R have singular values
T Tr, with r = min(n, d). Define:
Pk
rank (A) exp
Σ k=
Pk log Pk
oj
A high effective rank indicates flatter spectrum and greater diversity of informative directions. A low effective rank implies that variance is concentrated in narrow subspace- often aligned with spurious cettures-limitiee downstream flexibility.

---

50
Accorta (9)
Accuwsa (49
bestgioup Aoorooe 1
SinCLF spectium Razering
avarage accuriity
Number Singuler Removed
(a) Dropping the 400 smallest singular values and flat- (b) Removed (0 to 512) with the remaining spectrum tening doubles worst- worst-group group accuracy (20 - 40 %).
flattened. Progressive truncation improves robustness.
features for SpurCIFAR-10, Removing low-energy directions and flattening the remaining spectrum double worst-group accuracy without any architectural changes. Shaded bands around each curve represent worst- and best- accuracy. Left: single 400-mode cut; right: incremental cuts.
Figure 2: Impact of spectral interventions on SimCLR-train
2.2 Spurious Features Collapse the Spectrum
We illustrate the effect of spectral imbalance by comparing SimCLR- representations on CIFAR-10 and SpurCIFAR-10, the latter introducing spurious correlation of 0.95 via class- dependent overlays. Using a ResNet-18 encoder (512 dimensions, 1000 epochs), we compute the effective rank of the training feature matrix and evaluate test accuracy 5,000 held-out examples.
As shown in Figure 1 the effective rank of the representation matrix is significantly lower on SpurCIFAR-10, accompanied by reduced test accuracy. This confirms that spurious features lead to collapsed spectrum, reducing representational diversity and flexibility.
These observations support our central hypothesis:
Hypothesis 2 (Spectral Diversity and Learnability). The number of significant singular values in the feature matrix governs the learnability of downstream tasks. Representations with higher effective rank are more likely support robust generalization across diverse lasks.
2.3 Spectral Interventions and Robustness
To test this hypothesis, we manipulate the spectrum of the feature matrix learned by SimCLR on SpurCIFAR- 10. Specifically, we truncate the smallest 400 singular values and flatten the remaining spectrum. A linear classifier trained on this modified representation (Figure 2a) shows a substantial gain in worst-group accuracy- from approximately 20% to 40%- 40%- maintanning or slightly im- proving average accuracy and the best-group accuracy. Figure [b shows that progressively removing small singular values yields only modest, incremental gains in worst-group performance. Taken together, findings indicate that low-rank directions offer only marginal benefit to generalization and can even slightly undermine robustness. To isolate the effect of flattening, we conduct an ablation study where only the smallest singular values are removed without flattening the remaining spectrum. As shown in Appendix A Figure 3 this approach also yields significant improvement in worst-group accuracy, showing the importance of a balanced spectrum for robustness.
Our findings complement prior work on the role of rank in deep learning (aannahh et al. 2013 Feng et al. 2022 Andriushchenko al.) 2023), while highlighting a different trade-off: contrastive learning benefits not from aggressive compression but from maintaining balanced spectrum. Even in 512-dimensional space, the effective rank remains relatively low; increasing it improves robustness. Importantly, our goal is not to eliminate spurious features- some may benefit certain tasks-but to ensure that task-relevant signals are preserved. Flattening the spectrum promotes richer, more flexible representations that enable robust generalization- even in the presence of spurious correlations.

---

3 Related Work
Spurious Correlations and Robustness
Several studies have examined how neural networks rely on spurious features [Geirhos et al. 2020 Sagawa et al. 2020 Shah et al. 2020). For instance, Zech al. (2018) showed that models trained on chest X-rays often rely on nospital-speciff artifacts rather than medical content. [Sagawa et al. [019 and [Sagaw et al. [220] introduced worst-group accuracy as a robustness metric to evaluate models under spurious feature reliance. More recently, Kirichenko et al. [2022) and Qiu et al. (2024) analyzed how training dynamics and data complexity affect the emergence of spurious features. Beyond metries, algorithmie approaches include GroupDRO, which directly minimizes worst-group risk by adaptively reweighting groups during training and thus requires group annotations [Sagawa et al., 2019]. Complementarilyy Just Train Twice (JTT) improves group robustness without group labels via two-stage procedure- train a standard ERM model, mark misclassified examples as proxies for minority groups, and retrain with upweighting (Liu et al. 2021a]. Both methods are supervised and label-driven, whereas our approach regularizes representations during self-supervised pretraining. For certified robustness, Mustafa et al. (2024) prove non-vacuous adversarial population-rist bounds via randomized smoothing and PAC- Bayes for stochastic networks- compllss-oomllementtryy to our SSRL regularization.
Self-Supervised Learning and Shortcut Representations Recent work has shown that self- supervised representation learning (SSRL), particularly contrastive methods, is also susceptible to spurious features (Hamidich et al. 2024 Zhu al. 2023 Ye al., 2023]. While contrastive learning is principled route to task-agnostic representations supported by strong empirical results and theory giving generalization guarantees under both i.i.d. and non-i.i.d. sampling regimes, as well
dependence [Chen et al. 2020 He et al. as in adversarial settings with logarithmic negative-sampl
2020 Grill et al. 2020 Chen and He| 2021 Caron et al.) 2020 Zbontar et al., [202]] Hieu et al. [225
Hicu and Ledent 2025 Ghanooni et al. [024] it can nevertheless emphasize dominant directions
in feature space, encoding shortcuts rather than robust, transferable signals. Empirically, the effects of dominant, easy-to-learn features have been widely documented [Liu et al. 2021b Jiang et al.,
2021a|b b) Chen et al.] [02T]. and studies addressing group robustness or fairness SSRL often rely on group information or labeled data [Song et al.) 2019 Tasi et al. 20200 Wang et al. [02] | Bordes)
ctt al. 2022 Scalbert et al., 2023]. In contrast, Robinson et al. [202]] tackled shortcut learning in contrastive setups by employing adversarial feature modification without requiring group labels.
Spectral Structure and Representation Quality The spectral properties of learned representa- tions have been studied in various contexts [He et al. 2024, Bansal et al., 2018 Jing al. [021] Rahaman et al. [2019] and Kalimeris et al. [2019] analyzed how gradient- based optimization favors low-frequency, high-energy components early in training. Xue et al. [2023 examined the role of eigenspectrum bias in the evolution of feature learning. Our work builds on these insights by showing
that spectral imbalance overconcentration of variance in a few directions can amplify spurious
features and degrade generalization.
Spectral Regularizers and Low-Rank Perspectives In representation learning, several recent methods explicitly incorporate regularizers to encourage more diverse, high-rank representations in self-supervised learning. For example, Barlow Twins (Zbontar et al. [222]] introduces redundancy- reduction loss that drives the cross-correlation matrix of twin network embeddings toward the identity, thereby decorrelating feature dimensions and minimizing redundancy. Similarly, VICReg adds variance preservation term and a covariance penalty to maintain per-dimension variance while pushing off-diagonal covariances toward zero, thus preventing informational collapse by decorrelating features Bardes et al.] 202T]. Whitening- bbase approaches go even further by forcing the entire feature covariance to match an identity matrix (full whitening), which equivalent to enforcing full-rank embedding space and comes with theoretical guarantees against dimensional collapse [Ermoloy al. 202T]. Beyond these objectives on the representation statistics, other techniques like Implicit Feature Modification (IFM) actively perturb training examples in feature space to ensure the encoder utilizes a broader set of features, reducing reliance on any single dominant "shortcut" and promoting greater feature diversity [Robinson et al. 2021]. Complementing these regularizers, prior work has explored how low-rank structures relate to generalization. /Sainath et al. (2033) found that low-rank projections can improve speech recognition models, Feng et al. [022 showed that adversarially trained networks tend to learn flatter spectra. However, AAnriushchennoo
al. 2023) demonstrated
that flatness alone does not guarantee robustness. All of these strategies underline the importance
5

---

of promoting feature diversity and high effective dimensionality in learned representations. Our approach differs by emphasizing a balanced cigenspectrum (neither overly flat nor too concentrated) to mitigate spurious feature dominance without the extremes of full whitening or low-rank collapse.
4 Theory
This section analyzes how the spectral structure of learned representations affects generalization in downstream tasks. We show that the downstream generalization error depends critically on the spectrum of the feature covariance matrix. Finally, prove that representations with uniform eigenspectrum minimize expected generalization error over random downstream tasks, providing a theoretical justification for promoting spectral diversity.
4.1 Problem Setup
We consider a learning setup where the inputs x E X C Rd are composed of core and spurious features. Only the core features are causally relevant the ground-truth labels, but spurious features may correlate with labels due to dataset biases.
We assume access to unlabeled data in the form of similar pairs xx) and independent negatives - Learning proceeds over function class F = { X + Rd [xx(x] S RJ,, where f maps inputs to d-dimensional representations. During training, we optimize an encoder ʃ F jointly with projection head h(x) W2 (x)), where Wi E R24 xd W₂ E Rd,xda, and (-)) is nonlinearity (e.g., ReLU). The encoder and projection head are trained using contrastive losses such as InfoNCE (Oord et al. 2018). After training, the projection head h is discarded, and the learned features are represented by the matrix
F [f(x))...... f(x,)] Raxdd
where n is the number of samples.
To evaluate the quality of the learned representations, freeze the encoder f and train linear classifier 9w (f(x)) = (x(), w) with w E R.. Given labeled dataset = ((x,, b/ drawn i..d. from a distribution 2 over X' x V, where x; E IRd and yi E 1-1,+1} The classification downstream task involves minimizing the supervised loss
f(y, Iw (f())) = (1 - ygw / (x))²
This setup captures the realistic setting where similarity is defined semantically but is not perfectly aligned with task-relevant (core) features, allowing spurious correlations to arise during representation learning.
4.2 The Role of Feature Spectra in Downstream Task Generalization
We now investigate how the spectral structure of the feature matrix influences the generalization performance of downstream tasks. First, we analyze the impact of the feature matrix F on the generalization of linear classifiers trained via stochastic gradient descent (SGD). The following corollary, based on (Arora et al. (2019), summarizes this effect for a fixed downstream task.
Corollary 3 (Linear classifier generalization bound under rank-deficient feature matrix), Let S = ((x, yi) yi be drawn i.i.d. from a distribution over inputs and binary labels ys E (+1), and fix a failure probability 8 E (0.1), 1). Let f(x) E Ir be a fixed representation and define the feature matrix F R with rows T( Consider linear predictor 9w 9w(x) {f(x), w) trained with gradient descent. Then, with probability at least 5, the population loss Lo [D((ww)) =
B(xx,))D(((0 (ww.:l(((w((())) (x), (x), y)] satisfies
TeD(ww()) & δ
V
where y (U.....,u. TT and (+)+ denotes the Moore Peerree pseudoinverse, Equivalently, if FFT - X v with > 0 and orthonormal {ve) then
y (FF = Σ (vTy)

---

Here O hides logarithmic factors and dependence on 6.
The formal statement and proof provided in Appendix B The dominant term, y (FFT)*y shows that generalization improves when the label vector y aligns well with the top eigenspaces of FFT. In contrastive learning, however, the downstream task not known during pretraining, is unclear which directions in the feature space will ultimately be important.
To address this, consider downstream tasks that arise by randomly sampling two latent classes ctt E C according to distribution p. For each such pair, we assume the existence class-specific vectors , Ve such that the optimal linear classifier in the feature space is given by Vol Specifically, the class posterior is given by
F(Y; = +1 v) = 1+ (Fv)
P(Y; = -1 1- (Fv);
2
where Ruxd is the feature matrix.
Since downstream tasks are unknown at pretraining time, designing robust representations for contrastive learning requires optimizing for generalization over a distribution of tasks. Assuming p is uniform over class pairs, we study which spectral properties of FF lead to improved average generalization. Specifically, we aim minimize the expected surrogate loss:
C(P) = EE,y |Y (FFFFT)+Y]
where the expectation is over random task vectors and induced labels Y (#1)".
The following theorem shows the optimal structure of F to enhance the generalization on a general downstream task.
Theorem 4 (Optimal structure under trace constraint (informal)). Let F E Raxd be a feature matrix with rank(F) F)) S min(n, d). and ler G == FFT Rnxn, Suppose G has eigenvalues = and a fixed trace constraint E A c. A1 2 A 2 .... 2 Xy > O. with Ar+1 = - An
Then the expected loss L(F) under the random task model is minimized when FFT has uniform spectrum; that is, all non-zero eigenvalues
equal: X1
Xy-
Remark 5. The trace constraint used in Theorem 4 is both theoretically meaningful and prac- tically justified. In modern contrastive learning pipelines-such standard to apply fz-normalization to the feature vectors. This ensures that the overall energy of the representation, captured by tr(FFT) FT) remains approximately constant across batches. Thus, the fixed-trace constraint reflects common empirical practice. Furthermore, since downstream classifiers are ccale-invariant, normalizing the trace removes trivial rescalings of the feature matrix and makes the spectral analysis of feature representations posl-posed.
SimCLR and SimSiam-it is
The formal statement and proof provided in Appendix C Theorem 4 shows that, when the down- stream task unknown, the optimal feature geometry (under a fixed-trace constraint) is attained when G : FFT has uniform eigenspectrum. In the rank-deficient setting this means the eigenvalues constant on the rank-r support support and zero elsewhere. Geometrically, the representation spreads variance evenly across the task-relevant subspace while suppressing degenerate directions, preventing any single mode from dominating. See Appendix]] for an illustrative example with spurious correlations that further motivates the uniform spectrum condition.
5 Algorithms
We now present a simple and effective regularization method to promote spectral diversity during contrastive learning. To justify this approach, we establish a spectral relationship between the singular values of the feature matrix F and the eigenvalues of its covariance feature matrix FFT.
5.1 Increasing the Rank of Feature Matrix with Spectral Regularization
In given neural network model, let F denote the feature matrix at the last layer L, which corresponds to the output of the encoder. Our objective to flatten the spectrum of the matrix FFT during the pretraining stage, thus encouraging the model to equalize the eigenvalues and, as result, increase the effective rank of the feature matrix. To achieve this, we propose regularizer that efficiently facilitates this objective:
7

---

Table 1: Worst-group accuracy (%) for SSRL methods (SimCLR, SimSiam), SinCLR-LateTVG: SimSiam- LVG. and our method, which in this experiment is applied only to SimCLR. Values reported as mean + standard deviation across random seeds for method. Results for other methods are taken from Hamidieh et al. [2224.
DATASET
SIMCLR
SIMSIAM
SIMCLR LATETTV
OURS
CMNIST
81.7
80.7
83.8
83.1
95.10 + 2.90
SPURCIFAR-10
36.5
43.4
40.4
61.4
64.65 + 2.86
76.7
77.5
82.2
83.1
83.96 = 1.65
METASHIFT 45.5
42.3
59.3
79.6
61.14 = 8.02
WATERBIRDS 43.8
48.3
55.4
56.3
50.25 = 0.79
Table 2: Average accuracy (%) for SSRL methods (SimCLR, SimSiam), SimCLR-LateTVG, SimSiam-LateTVel and our method, which in this experiment applied only SimCLR. Val- ues are reported as mean + standard deviation across 5 random seeds for our method. Results for other methods taken from Hamidieh et al. 2024
DATASET
SIMCLR
SIMSIAM
SMMCLR-LATETE
SIMSIAM-LATETVA
OURS
CMNIST
82.5
82.1
80.6
98.08 + 0.26
SPURCIFAR-10
69.3
75.1
76.1
82.85 + 0.82
CELEBA
82.1
81.9
88.9
88.40 1 0.35
METASHIFT
55.1
55.8
70.1
76.57 1 1.01
WATERBIRDS
47.5
50.7
54.8
58.32 1 0.59
Definition 6 (Spectrum Flattening Regularizer). Let G == FFT (with rank G)) and set p, == da/ b/E Az. Define
have eigenvalues A
A, >
Ropec (G)
Σ
p
1=1
This penalizes spectral concentration and is minimized at the uniform spectrum (p, 1/r).
To encourage more uniformly distributed representations, we incorporate the spectrum flattening regularizer into the contrastive learning objective. The resulting loss function is defined as
L LSRRL. + a (FF
(1)
where LSSRL denotes the standard contrastive loss (e.g., SimCLR), and a E R4 weighting coefficient that controls the influence of the regularization term. The addition of Ryee encourages the learned feature representations to maintain a flatter spectral profile, thereby promoting greater feature diversity and mitigating representational collapse. Pseudocode for computing the regularization provided in Appendix
6 Experiments
We evaluate our method by analyzing how flattening the spectrum of the feature matrix improves worst-group accuracy under spurious correlations. Motivated by Lemma Appendix D we apply the regularizer directly to the (centered) mini-batch feature Gram matrix and add it to the SimCLR objective promote robust feature learning without labels or group annotations. Concretely, for each mini-batch B with centered features FB R B|xd let GB = FBFT have nonzero eigenvalues {Ar) and define p: := >a/ E Aj- The spectrum-flattening flattening regularizer is
FB
R Rypec (B)
=
P
1=1
8

---

which penalizes concentration of energy and is minimized at a uniform spectrum. (Equivalently. the "zero-at atrooaatoptimum" form rB - (Pr rB can be used.) Additional details are provided in Appendix B
6.1
Baseline Methods
We compare our approach with SSRL models pretrained using standard SSRL losses, specifically SimCLR [Chen et al. 2020] and SimSiam [Chen and He 202T). SimCLR employs contrastive learning framework, where paired augmented views of the same image are pulled together in the feature space while different images are pushed apart using a contrastive loss. It consists of an encoder (e.g., ResNet), a projection head, and a contrastive loss function that requires negative samples. In contrast, SimSiam a non-contrastive method that avoids negative samples by leveraging a stop-gradient mechanism and a predictor network to prevent collapsed representations. Additionally, we evaluate our method against LVteTVG (Hamidieh et al., 2024). state-of-the of-tte-art SSRL approach designed to mitigate spurious correlations during pretraining, which has been applied to both SimCLR and SimSiam. We include baseline results for Barlow Twins [Zbontar et al. 202T]. DirectDLR [ning
er al. [022]] IFM [Robinson et al.) (02T], BYOL [Grill et al. 2020 and DINO CCron et al. 2021
Appendix for completeness and comparison.
6.2 Datasets
We evaluate all methods on five widely used vision benchmarks designed to study spurious cor- relations. Among them, SpurCIFAR-10 [Nagarajan et al. 2020] and C-MNIST [Arjovsky al. 2019] are synthetic datasets constructed by introducing strong artificial correlations: SpurCIFAR- 10 modifies CIFAR- images by associating the color of horizontal lines with the object class, with spurious correlation strength of 0.95, while CMNIST colors the MNIST digits to create a spurious correlation between digit color and binary label, with a spurious correlation strength 0.99. The remaining datasets are based on real-world imagery. In CelebA [Liu et al., 2015). gender (female or male) spuriously correlated with hair color (blond or non-blond). MetaShifi (Liang and Zou 2022] explores spurious correlations in the Cats and Dogs classes, where the background (indoor or outdoor) is associated with the type of pet (cat or dog). Finally, Waterbirds [Sagawa et al.) 2019 contains spurious correlation between the background (land or water) and the bird species (landbird or waterbird). Together, these benchmarks cover both synthetic and natural settings, allowing us to evaluate robustness across different types and strengths of spurious correlations.
6.3 Implementation Details
SSRL Pretraining For each dataset, we train an encoder (ResNet-18 or ResNet-50) with a pro- jection head composed of two linear layers separated by a ReLU activation. The training follows the contrastive learning framework of SimCLR, using the contrastive loss as the primary objective. Additionally, we incorporate our regularizer from Equation [ to enhance feature representations. To determine the optimal regularization weight (a), we perform grid search over the values 0.001, 0.005, 0.01, 0.05, 1. The regularizer flattens the spectrum of the feature matrix, effectively in- creasing its rank by applying the regularization within each batch. This design ensures seamless integration with the SSRL loss function while preventing the rank from increasing excessively, which could lead to the learning of irrelevant features.
Downstream Task After pretraining, we freeze the encoder and use its output representations for linear probing. For linear probing. a linear classifier is trained on top of these fixed representations using cross-entropy loss, with only the classifier weights being updated. To ensure balanced training dataset, we subsample the majority groups (Sagawa et al., 2020 Idrissi et al. 2022), helping to mitigate geometric biases in the linear classifier [Nagarajan et al. 2020]. Finally, we evaluate the learned representations on the standard test split of each dataset, leveraging group information to report both average accuracy and gorst- accuracy.
6.4 Results
Our method achieves consistently higher worst-group and average accuracies compared to strong SSRL baselines, demonstrating its robustness across diverse datasets without relying on architectural

---

modifications. Table reports worst-group accuracies across all five datasets, averaged over five random seeds, while Table 2 shows the corresponding average accuracies. Results for baselines SimCLR and SimSiam are taken from Hamidieh et al. [024].
Our approach consistently outperforms LateTVV when combined with SimCLR across four datasets (except for waterbirds dataset) and both evaluation metries. When paired with SimSiam, it remains highly campetittie-mmachinn
or exceeding LateTVG on several benchmarks-while retaining simpler architecture. Notably, our SimCLR experiments use 512-dimensional embeddings, whereas LateTVG's SimSiam results are based on 144--dimensional representations. We have also applied our regularizer to SimSiam; the complete results appear in Appendix H We further observe that Hamidieh et al.] [024] does not provide the pruning parameter that determines their reported results, nor does it report the standard deviations of the accuracies. Worst-grous group accuracy can fluctuate substantially, so the absence of variance estimates prevents rigorous assessment of the consistency of their reported gains.
Moreover, while LVG (Hamidich al., 2024 is tailored for architectures with stop-gradient operations, our approach is architecture-agnosiic and achieves comparable or superior performance with simpler 512-dimensional feature space. This highlights the potential of spectral regularization for broad SSRL applications. Hyperparameter settings for our method are provided in Appendix and additional ablation studies are included in Appendices and H
6.5 Sensitivity to Regularization Strength
We evaluate the sensitivity of the spectral regularization strength spec on MetaShift with SimCLR, keeping all other settings and the random seed fixed. The spectral regularizer is added the SimCLR objective (as defined earlier). Table 3 shows that performance peaks at Osspe=0..00 (75.84%), dips as O'qpez increases 0.010 (66.29 ).). partially recovers 0.025 (73.60%), ), and degrades again for 2 0.050 (66-68%). These results indicate sensitivity: small regularization works best, while stronger penalties lead to underperformance; a practical starting point asspee E [0.001, 0.005].
Table 3: Sensitivity of average accuracy (%) on MetaShift to spectral regularization strength Ospec with SimCLR (fixed seeds). Best result are in bold.
O Osee
0.001
0.0025
0.005
0.010
0.025
0.050
0.100
Avg. Accuracy (%)
75.84
71.91
68.54
66.29
73.60
67.42
66.29
7 Conclusion
In this paper, we addressed the challenge of spurious feature reliance in self-supervised represen- tation learning by providing both theoretical insights and a practical solution. We showed that the expressiveness of learned representations is closely tied to the number of significant singular values of the feature matrix, which in turn impacts downstream task performance. To promote richer and more diverse representations, we introduced effective rank of the feature space. Rather than explicitly removing spurious features, our method encourages the learning of a broader set of features beyond spurious correlations, thereby improving generalization to downstream tasks without requiring labels or group annotations.
ppectrum-flatteniigg regularizer that increases the
Acknowledgments and Disclosure of Funding
Part of this work was conducted within the DFG Research Unit FOR 5359 on Deep Learning on Sparse Chemical Process Data (BU 4042/2-1, KL 2698/6-1, and KL 2698/7-1). MK and SF further acknowledge support by the DFG through TRR 375 (ID 51 511263698), SPP 2298 (KL 2698/5-2), and SPP 2331 (FE 2282/1-2, FE 2282/6-1, and KI. (698/11-1), by the Carl-Zeiss Foundation through the initiatives AI-Care, A14ChemRisk, and Process Engineering 4.0, and by the BMFTR award 011S2407 IA.
10

---

References
Robert Geirhos, Körn-Henrit Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1)):6556633 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning. pages 5301-5310. PMLR, 2019.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32, 2019.
Yihao Xue, Siddharth Joshi, Eric Gan, Pin- Yu Chen, and Baharan Mirzasoleiman, Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. In International Conference Machine Learning, pages 38938-38970. PMLR. 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. 08731,, 2019.
arXiv preprint
Guan Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Dynamics of feature learning in the presence of spurious correlations. arXiv preprint arXiv:2403.03375, 2024.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness spurious correlations. arXiv preprint arXiv:2204.029:: 2022.
Nihal Murali, Aahlad Puli, Ke Yu, Rajesh Ranganath, and Kayhan Batmanghelich. Beyond distri- bution shift: Spurious features through the lens of training dynamics. Transactions on machine 2023.
learning research, 1r23:hhtps-ppenreeee
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, (5(1I):e10026))
2018.
Kimia Hamidieh, Haoran Zhang, Swami Sankaranarayanan. and Marzych Ghassemi. Views can be deceiving: Improved ssl through feature space augmentation. arXiv preprint arX1v:2406.18562. 2024.
Weicheng Zhu, Sheng Liu, Carlos Fernandez-Granda, and Narges Razavian. Making self-supervised learning robust to spurious correlation via learning-speed aware sampling. arXiv preprint arXiv:2311. 163611, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pages 1597-1607. PMLR, 2020.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7729,9738, 2020.
Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 -2222444, 2020.

---

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9922, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pages 22310-12320 PMLR, 2021.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606 610. IEEE, 2007.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference acoustics, speech and signal processing, pages 6655-6659. IEEE, 2013.
Ruili Feng. Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing deep neural networks. Advances in Neural Information Processing Systems, 35: 33054-33065, 2022.
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 47032-47051 2023.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021a.
Waleed Mustafa, Philipp Liznerski, Antoine Ledent, Dennis Wagner, Puyu Wang, and Marius Kloft. Non-vacuous generalization bounds for adversarial risk in stochastic neural networks. In International conference on artificial intelligence and statistics, pages 4528-4536. PMLR, 2024.
Haotian Ye, James Zou, and Linjun Zhang. Freeze then train: Towards provable representation learning under spurious correlations and feature noise. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2023.
Nong Minh Hieu, Antoine Ledent, Yunwen Lei, and Cheng Yeaw Ku. Generalization analysis for deep contrastive representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 17186-17194, 2025.
Nong Minh Hieu and Antoine Ledent. Generalization analysis for supervised contrastive representa- tion learning under non-iid settings. arXiv preprint arXiv:2505.00 04937, 2025.
Naghmeh Ghanooni, Waleed Mustafa, Yunwen Lei, Anthony Widjaja Lin, and Marius Kloft. Gener- alization bounds with logarithmic negative-sample dependence for adversarial contrastive learning. Transactions on Machine Learning Research, 2024.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-superviedd learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.05025. 202
Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pages 4927-4939. PMLR, 2021a.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open-world sampling. Advances in neural information processing systems, 34: 5997-6009, 2021b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances Neural Information Processing Systems, 14:11834-1184$ 2021.
12

---

Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2164-2173 PMLR, 2019.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying self-supervised learning: An -ooooationtthooreticall framework. arXiv preprint (rXx::2000605576, 2020.
Tan Wang. Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Processing Systems, 34:18225-18240 2021.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. arXiv preprint arXiv:2206. 2307:32200.,3378, 2022.
Marin Scalbert, Maria Vakalopoulou, and Florent Couzinié-Devy, 'improving domaininvariance in self-supervised learning via batch styles standardization. arXiv preprint arXiv:2303.06088 2023.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Junlin He, Jinxiao Du, and Wei Ma. Preventing dimensional collapse in self-supervised learning via orthogonality regularization. arXiv preprint arXiv:2411.00392 2024.
N Bansal, X Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep cnns? arXiv preprint arXiv: 1810.09102, 2018.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint 10.00221 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invarianee-vvaaa for self-supervised learning. arXiv preprint arXiv:2105. 04906, 2021.
regularization
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International conference machine learning, pages 3015-3024. PMLR, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:/807.08:: /807.03748, 2018.
Sanjeev Arora, Simon Du Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Leurning, pages 322-332. PMLR, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur: Understanding the failure modes of outoff-distribttion generalization. arXiv preprint vrXiv:2010.15775: 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini 1907.0::9070002000 002899., 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. arXiv preprint arXiv:2202.06553, 2022.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balaneing achieves competitive worst-group-accuracyy In Conference on Causal Learning and Reasoning, pages 336-351. PMLR, 2022.
13

---

Yutaka Shigeto, Taiji Suzuki, and Yohei Nishiyama. Learning decorrelation representations efficiently using fast fourier transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20046-20055, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction (Section T claim a spectral regularization
method mitigate spurious features in SSRL,, supported by theoretical analysis (Section and experiments (Section 6). Limitations are noted Section L
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our limitation section in Appendix L discusses computational costs of our regularizer with details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: Corollary 3 and Theorem [ (Section 4.2) include full assumptions with proofs
in Appendices B and
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 6.3 and Appendix provide hyperparameters, encoder details, and training procedures. Pseudocode is in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used in this work are publicly available, and the code will be submitted prior to the supplementary material deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Coeetttii.ss
catessionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtp:/cccppppgguuuiiaeoeae Ctteeee//ooSSSeeeSe..ee.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 6.3 details encoder architectures, hyperparameters (Table 4.. data splits, and optimization. Grid search for is described.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Tables I and 2 report mean + standard deviation over 5 seeds for our method and Barlow Twins, capturing variability (Section 6.4).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix[details the computational resources and runtime associated with our regularizer.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///tt//eieisss...ccco/ http://www.....pp..i..ccccccceee
Answer:[Yor
The research uses standard datasets (Section 6.2) and methods, adhering to ethical guidelines. No human subjects or sensitive data are involved.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work foundational, improving SSRL robustness without direct societal applications risks.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:
[NA]
Justification: No models or datasets are released, and the method poses no misuse risks (Section @).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets (e.g., SpurCIFAR-10, CelebA) are cited with references (Section 6.2 Licenses are standard.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
No new datasets or models are released; the contribution is theoretical and algorithmic.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:
[NA]
Justification:
No crowdsourcing human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLMs are used in the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hett:p//meerrrsssccc///ooeerrr/a for what should or should not be described.
21

---

50-
Accunay (8))
Accorta
best group acourtay
bost-gropp zocuracy
everage acoureay
ovanigg
wecrti-group
Mumber Singular
Removed
Number
Singuler Removed
(a) Worst-group accuracy when progressively remov- ing the smallest singular values and flattening the re- maining spectrum. Flattening consistently improves performance and stabilizes the effect of truncation.
(b) Worst-group accuracy when progressively removing the smallest singular values without flattening. While some gains occur, the effect is unstable and inconsistent across truncation levels.
Figure 3: Effect of spectral manipulations on SimCLR-trained feature representations for SpurCIFAR- 10. We evaluate trained features with linear classifier where singular values are incrementally truncated (from 0 to 512) either with (3a) or without (3b) flattening the remaining spectrum. Flat- tening alone (even with no truncation) significantly improves worst-group accuracy frruraccy-ormmmmm from 30% to 40%- highlighitn the importance of spectral balance for robust representation learning. Shaded bands indicate worst- and best-group accuracies.
A Ablation Study Removing Small Singular Values Without Flattening the Spectrum
To test our hypothesis, we manipulate the spectrum of the feature matrix learned by SimCLR SpurCIFAR-10, Specifically, we compare two interventions: (1) progressively truncating the smallest singular values while flattening the remaining spectrum (Figure Bal the same Figure 2b in the main paper), and (2) progressively truncating the smallest singular values without flattening the remaining spectrum (Figure 3b). We then evaluate the quality of the resulting representations using linear classifier.
While both approaches yield some gains in worst- worst-group accuracy as more low-variance directions are removed, the second approach (truncation without flattening) exhibits instability (see Figure 3b it is unclear which singular values should be removed to consistently improve performance. In contrast,
that truncation combined with flattening leads to more robust and consistent improvements in worst-group accuracy. Notably, even without any truncation, simply flattening the full spectrum significantly boosts worst-group performance, from approximately 30% to 40%.
Taken together, these findings suggest that low-rank directions provide limited benefit for gener- alization. As illustrated in Figures] 3 a balanced spectrum plays a crucial role in enabling robust representations, reinforcing the importance of spectral regularization in the presence of spurious correlations.
B Proof of Corollary 3
Corollary 3 (Linear classifier generalization bound under rank-deficient feature matrix), Let S = y:);, be drawn i. from a distribution over inputs and binary labels ys {+1), and fix a failure probability 8 E (0.1). 1). Let f(x) E Id be a fixed representation and define the feature matrix F Rxx with rows f(x,). Consider linear predictor 9w ( (f(x), w) trained with gradient descent. Then, with probability at least 5, the population loss Lo (gww)))) =
Exx [[(gw (x), 2)] satisfies
(
Lp((ww()) $
22

---

wherey = (yi..... Yoo and (-)+ denotes the Coore-Penrose pseudoinverse. Equivalently, if
FFT = -i=l X v.V. with Xg > 0 and orthonormal (vi)== then
y (FFFF))y = Σ x (v]y)²
Here Ô hides logarithmic factors and dependence on
We now prove the result for full-rank feature matrix.
Proof. Without loss of generality, assume F is appropriately normalized, such that Amax (FFT) 1.
Consider gradient updates of the form
w(k 1) w(k) -1 d$
(g(k) y)
dw
with 17 = =0 (Dmmax(FFT) (FFTT) and w(0) = 0. The outputs of the linear network evolve as
g(k: + 1) g(k) F(w(k: + 1) w(k)) = = (g(k) y)
Thus, the distance of the outputs to the labels evolves as
g(k) y
y = g(k - 1) - nFFFFFFF (g(k 1) - y) y = (I -
1) -) -1- (g(()) - y)
Using the previous result, we can express the change in the weights during training as
K
w(K) - w(O) =
Σ
w(k + wkk)
k=
K-1
nF (g(k) y)
K-1
Σ
nF n/F /F (g(0) y)
k=0
K
K-1
1F
(I-
[I -
Σ
(I-
k=0
k=0
To bound the change in the weights, we bound each term individually.
K-1
K
K-1
IIFE"
Σ
Σ
(I - (I-nFEj
FF
Σ (I
- )FF
k=0
k=0
(1 - n2t))
(1 ndde) K
Σ Aoaryy T
vio??
1
(1 - nr
2
A;
= y" (FFT)-1
K-1
K
Σ
nF (1- (P-TI-PFFT k g00)l2 S snn
Σ
(1
Mamnn (FFT))) ||s(0)]22 S 0
0
k=
23

---

Taking these bounds together yields
[w(K) - w(0)lk V/ (FFT)-Ly
E E {=1)". Then it holds
(s, Xw) (c, X( w(O))) (e, Xw(0)) V (c, Vnl
w(0)ll2
Let FR = [W, ||w ww & RJ.
1
Rs(FR) = E (11)"
sup
(c, Xw)
R =
R
||==w(0]|<RR
/1
log 2
sup JEFR
LD(f) - Ls(s) S (Rs(FR)+ +0
2n
1
log Thd
R 0
vn
2m
YT (FFT)
log 2 2n
2V
+0
Using
g y/122 < [I MFFFFFFyF F 12ll y11a (1 7/ (FF 5
for sufficiently large 2 log /n, we can bound Ls as follows.
Ls(f) lo(f(x))) v1 - yl
1
n
We now prove the case in which the feature matrix rank-deficient.
Proof. Without loss of generality, assume F is appropriately normalized so that Amax (FFF S 1.
Consider gradient updates of the form
d&
w(++1) w(k) -1)
F (g(k) y)
dw
with y7 o 2A and w(0) = 0. The outputs evolve as
g(h+1) g(k) F F(w(k+1) w(k)) - FF (g(k) -y),
hence
y (1= (I- (1- (g(0) - y).
Using this, the weight displacement after steps is
K
K
w(K) w(0) = Σ n F (g(k) - y) = F (I-TI-FFFF n - y)
k=0
K-1
K-1
Σ
( FF (I
Σ
n (I- FFT }g(()).
k=0
k=0
24

---

We bound the two terms separately. For the first,
K-1
K-1 y Ea-nEet) FF 1
K-1
aa-
FF
/ FF
17
km0
Let FFT = A. with X > 0 (the remaining eigenvalues are O; their contributions vanish in the display above). Then
K-1
(1 nAi
Σ
[I - 11-FFF
V.V.
4
k=0
1=1
and
FFT
[())
h
k
A 1 - (1(1-))))))
vVT) TT Σ Ex y y y= ((FFT)yy
=yT
where (.)+ denotes the Moore-Penrose pseudoinverse and we used (1 - (1 n)1 1
For the second term, since w(0) = 0 we have g(0) = Fw(()= O, hence
K 1
Σ
nF (I FFF "g(0) = O.
k=0
Combining the two bounds yields
(w((k- s VXT(FTT)ty.
Let e E {+1}". Then
(c. Xw) = (e, X(w - w(0))) + (c. Xw(O)) S vnllw w(0)ll2
and for FR ((w,x) 1|w w(0)ll < R).
R
Rs(FR) = nl
E,
sup
(e.
Vt
|| w()) SR
Hence, with probability at least - 5,
sup (Ln(f)- Ls(f)) s (RR(FR) + oV JEFn
log(2/8) 2m
2R
{v
log(2/6)
<
+
Vn
and taking R = [[([]] w(0)l22 Vy gives the stated bound.
Finally, for the empirical term, decompose y Yrange + Ynull along the eigenspaces of FFT, so that - FTFF (
mull y Ymul and
lgg y|l [[I - 77 FF ) (g(0) Yrange) Yautlla < (1 - 77 )tin) ||Yrangell2 + ||ynuulle 2
where * is the smallest positive eigenvalue of F Choosing k so that (1 - ttin) S 1/v7 yields
Ls(f) - y1l < 1 n
(1+ | 5
(1+
n
and in particular [.s()) < 1 always; when y lies largely in range(F) (F) (the learnable subspace), this term is o(1) for large
D
25

---

C Proof of Theorem
In this section, we formalize the problem setting, restate Theorem and provide its proof. The dominant term, y (F'"))"y,, shows that generalization improves when the label vector aligns well with the top eigenspaces FFT. In contrastive learning. however, the downstream task not known during pretraining, is unclear which directions in the feature space will ultimately be important.
To address this, consider downstream tasks that arise by randomly sampling two latent classes ct,c E C according to distribution p. For cach such pair, we assume the existence class-specific vectors Val, Ve such that the optimal linear classifier in the feature space is given by Vel Specifically, the class posterior is given by
P(Y, +1|+ v) (Fv)
P(Y, -1 v) 1- (Fvv)
2
where F Raxd is the feature matrix.
Since downstream tasks are unknown at pretraining time, designing robust representations for
contrastive learning requires optimizing for generalization over distribution of tasks. Assuming uniform over class pairs, we study which spectral properties of FF lead to improved average generalization. Specifically, we aim minimize the expected surrogate loss:
C(F) == Ev,y (FF(FF) (FFF)+Y] ytY]
where the expectation over random task vectors v and induced labels Y E {#1}".
The following theorem shows the optimal structure of F to enhance the generalization on general downstream task.
Theorem 4 (Optimality of Uniform Spectrum under Trace Constraint), Let F E Buxdd be feature matrix with rank(F) = r < min(n, d). and define G == FF E Ruxx, where has eigenvalues
d A2... Ar > and X Ei X; = e. for some constant 0. Then the expected quadratic form
= An = O. Suppose the trace is fixed, iee, Tr(FFT)
L(F) = Ey-D [YT (YFFFT) (FFT)-LYY)
under random task model is minimized when FFT Al i.e., when all non-zero eigenvalues are equal.
Proof.
We first simplify the objective. Conditioning v, the second moment of Y satisfies
E|Y v] = (Fv)(Fv) + diag(1 - (Fv)²).
Thus,
((F) = EE ((F))
diag(1 - (Fv)?))]
where (+)+ denotes the Moore Penrose pseudoinverse (equal to the inverse in the full-rank case).
With the rank-r SVD V.T. we have FFT = U,T.UUT with A, diag(A)..... Ax) and X; of > 0 (and Ar+1
An Using the identity
F. FT(FFT)+F V.I.V. Prow(F):
we obtain
Under the isotropic random-task model for v, the expectation of this term depends only on the rank (not on the eigenvalues (x;) and is therefore an additive constant for the minimization.
Expanding the second term gives
tr (FF diag(1 - (Fv)?)) -t:((==(()F)))))) - tr((FFT)* -
diag ((Fv)?)).
Taking expectation over and using same SVD calculus, the v-dependent parts of two displays above cancel up to constant independent of {A:). Hence, up an additive constant that does not affect the minimizer.
C(F) tr((FFT)t)
26

---

Next, we optimize ((F) under the constraint )r(FF") c. Let G == FF and let its non-zero
eigenvalues be A...... .....,,, A > 0 (the rest are zero). Then
((F) Σ
and
tr(G)
X
Define the Lagrangian:
Σ
+
im1
Taking partial derivatives with respect to each ) and setting them to zero:
dc
-
x,
A
for all = ,.
aλ:
Thus, all non-zero X; equal. Substituting into the constraint Ee A gives:
--A==
x==
Theorem suggests that when the downstream task is unknown, learning feature matrix FFT with a uniform spectrum is optimal. See also Appendix correlations, which further motivates the benefits of uniform spectrum.
[ for an illustrative example involving spurious
D Lemma
Lemma 4. Let F E Baxd be feature matrix with singular values 01....,05 Or, where r = min(n, d), and let At....y Dy denote the eigenvalues of FFT, Assume the singular values are normalized so that
Σ (o. 1)2 11
<e for some (0,
Then,
Σ (A: 1) Ce for constant C.
Proof. Let F E goxd have rank r min(n. d), and let its singular value decomposition be F UEVT, where Σ diag( ...... o,) with singular values σ; > Then the eigenvalues of FFT Ruxa are exactly X; = of for = 1, and otherwise.
By assumption, the singular values are normalized and satisfy
(a. 1)2 16.
Our goal is to bound
(Ax - 1)2
To relate these two expressions, we use the identity:
(o 1)2 (a: 1) 1)2.
Since are normalized and (a--1) 1)a < s, we have a; E E vē, + VEJ. Therefore,
(o, (0.+ < (1+ V + (2 v2)² 9 for < 1.
27

---

Then:
(o - 1)² 2 9(; - -1)².
(a;
Summing over i I,..., .....,, obtain:
Σ (A: - 1)1- -
(0
1) 59
59 0i 1)2 E 95. 1=1
Remark 5. Lemma shows that flattening the singular values of F implies a corresponding flattening of the eigenvalues of FFT,, leading to more uniform spectrum.
Remark 6. Thus, to encourage spectrum uniformity of FFT, it suffices regularize the singular values of F, which is often simpler and more efficient in contrastive learning frameworks.
E Algorithmic Details of Contrastive Pretraining with Spectral Regularization
This section presents the pseudocode for our contrastive pretraining framework. Algorithm out- lines the self-supervised training procedure based on SimCLR with optional spectral regularization. Algorithm 2 describes the computation of the spectrum flattening loss.
Algorithm 1 Self-supervised Contrastive Pretraining with Spectrum Regularization
Input: Encoder fo. projection head g, temperature т, augmentation pipeline T, spectral weight Osspee epochs N, batch size B Initialize parameters of fe and for epoch 1 to N do for each mini-batch (x.) do // Stage 1: Data Augmentation Sample two augmentations I' T
x! = t(zz), o = f((x,) for = I,...,B B
// Stage 2: Feature Extraction
= ()f(()))) 2² =g(fo(z²))
Post-L2 normalization: 2 /
x = /
Stack all views: 2 { .., 2D. 2 E 2R
// Stage 3: Loss Computation
La 2B
1
Σ
log exp(sim( /p()))7)))
sim(a, b) Tbb
Ejer exp(sim( (2,, 2/)/7)
if spectral regularization enabled then Espae + SpectrumLoss(2) // A/l.
else
Lapec <- end if
Total loss: L Let. + Oispoe Lapee
// Stage 4: Optimization
Update fe. g via gradient step on VoL end for
end for
Output: Pretrained encoder fo
28

---

Table 4: Hyperparameter settings and encoder architectures for SimCLR pretraining.
Dataset
Encoder
Learning Rate
Batch Size
Weight Decay
Epochs
Regularizer
celebA
ResNet-50
0.01
128
le-4
400
0.01
emnist
ResNet-18
le-3
128
le-5
700
0.01
MetaShift
ResNet-18
0.05
256
le-4
700
0.01
spurcifar- 10
ResNet-18
0.02
128
5e-4
800
0.01
waterbirds
ResNet-18
0.01
le-3
600
0.01
Algorithm 2 Spectrum Flattening Loss Computation (Espec)
Input: Post-L2 batch features Ž IR Mxd (here M-2B)
Output: Spectrum loss Lupec
// Center features (remove batch mean)
<- 1 Σi= 74: Ze t 7 1E
M
// Covariance in feature space (angular statistics on the sphere)
C + - Z Ze Rdxd
Eigenvalues and trace normalization
Compute eigenvalues { of let =
Pi + >v/ Σÿ= Xy for i=ri=1....,
ppectrum--llateniig objective (minimized uniform)
Lypec Σp i=1 Return: Lypoc
F Hyperparameters
We employed the SimCLR framework to train ResNer encoders for our approach. To ensure fair comparison SimCLR, adopted the encoder architectures those used in Hamidieh et al. [2024), using ResNet-18 for all datasets except CelebA, where ResNet-50 was used. Detailed hyperparameter configurations for SimCLR across all datasets are provided in Table To select the regularization strength for the spectral flattening loss, we performed grid search over the values (0.001, 0.005, 0.01, 0.05) using validation performance on the worst-group accuracy the selection criterion. The best-performing value was then fixed for each dataset across all evaluation protocols.
G Closing the Gap to Supervised Pretraining
SSRL demonstrated significant potential in narrowing the performance gap with supervised learning approaches, particularly for general representation learning. Similar to Haamddihh et al. 22024]. we utilized a consistent encoder model and varied only the pretraining strategies, ensuring that other variables, such hyperparameter settings and model selection criteria, remained fixed. Notably, supervised pretraining requires labeled data, whereas SSRL methods do not, reducing the overall annotation cost significantly. While this inherently makes the comparison less direct, the goal of this evaluation is to measure how closely SSRL methods, and specifically our proposed approach, match surpass supervised pretraining strategies.
Table compares the SSLR-base (SimCLR or SimSiam) method, our proposed method, and supervised approach. The results highlight how our method narrows the gap with supervised learning terms of average accuracy. Additionally, worst-group accuracy, our approach outperforms both SimCLR/SimSiam and the supervised method on datasets such CclebA, CMNIST, and Waterbirds.

---

Table 5: Comparison of pretraining strategy (SimCLR spectral regularizer) with supervised models in terms of average and worst-group accuracies (%). Our pretraining strategy achieves comparable performance supervised models, both in terms of average and worst-group accuracies (%), despite not utilizing any ground-truth labels or group information.
DATASET
AVERAGE ACCURACY
WORST-GROUT ACCURACY
SSRL-BASE
OURS
SUPERVISED
SSRL-BAR BASE
OURS
SUPERVISED
CELEBA
82.1
88.40
91,9
77.5
83.96
81.7
CMNIST
82.5
98.08
98.4
81.7
95.10
94.9
METASHIFT
55.8
76.57
89.8
45.5
61.14
83.5
SPURCIFAR- 10
75.1
82.85
89.9
43.4
64.65
79.6
WATERBIRDS
50.7
58.32
67.9
48.3
50.25
41.1
Table 6: gorst-group accuracy (%) comparison between state of the art methods and our spectral regularization applied to SimCLR and SimSiam.
METHOD
CMNIST
SPURCIFAR-10
CELEBA
METASHIFT
WATERBIRDS
BARLOW TWINS
57.05
6.00
39.99
58.33
43.13
DIRECTDLR
90.32
20.58
68.68
53.84
47.32
IFM
96.77
53.23
53.85
50.96
BYOL
95.82
50.67
41.67
46,89
DINO
56.67
11.37
58.33
49.03
SIMSIAM SPEC (OURS)
94.40
50.10
74.17
69.23
49.08
SIMCLR + SPEC (OURS)
95.10
64.65
83.96
61.14
50.25
H
Comparing with More Baselines
We evaluate the effectiveness of spectral regularization method by comparing it against two representative baselines: Barlow Twins [Zbontar et al., [021] and DirectDLR RR [Jing et al. 2021]. In addition, examine how regularizer performs when applied top of SimSiam. We report both average accuracy and worst-group accuracy across five standard spurious correlation benchmarks as shown in 6 and
I Example Motivating Importance of Uniforming the Spectrum
present a simple example to highlight the role of the eigenspectrum of the feature matrix. Let F be fixed feature matrix with orthonormal eigenvectors vt, c and v", corresponding to eigenvalues A+, >- and X², respectively. Here, t and e represent daas-discriminative directions for labels +1 and -1, while o" is spurious direction with spurious correlation strength
Specifically, the label generation process as follows: define = 1 (v+ w-). For each sample 2;, let f: f(z,), and define the perturbed direction:
with probability 1 1 a.
(v o*).
with probability
The label y: E {+1} is sampled according
1+ fTu.
P(y = +1 +1++11 f1)
Let g, = gw (f.,,,, and consider the squared loss:
@ y) Σ (1-1
i=1
Lemma' 7. The expected gradient flow under the randomness of the labels satisfies:
E T Vg = [FFT
(1 (1- ( - -
a -
-
a
2 [1000)
2
30

---

Table 7: Average aceuracy (%) of the state of the art methods and our spectral regularization applied to SimCLR and SimSiam.
METHOD
CMNIST SPURCIFAR-10
CELERA
METASHIFT
WATERBIRDS
BARLOW TWINS
93.10
22.00
84.61
64.60
54.67
DIRECTDLR
96.35
50.20
78.23
75.28
53.46
IFM
98.16
76.70
78.65
55.78
BYOL
96.20
73.30
74.71
53.70
DINO
66.05
26.28
73.60
50.60
SIMSIAM SPEC (OURS)
97.20
71.78
89.12
77.52
60.55
SIMCLR + SPEC (OURS)
98.08
82.85
88.40
76.57
58.32
This result shows that even a weak spurious correlation (a < 1) can dominate the training dynamics if X >> A+ >= In contrast, under flat spectrum (i.e., uniform eigenvalues), the influence of the spurious direction scales linearly with a, making the model more robust to such noise.
Proof. The first step is to compute the loss with respect to each model output 9, which is given by
d$
2y: ye9s).
dg.
The sources of randomness from sampling both y and the random mixing of the spurious feature. By the law of total expectation, the expectation with respect to and is given by
dl
Bu.ls
Eyl
dg:
The inner expectation is given
dΦ
Bell fa.r
2B
vog.1 = 2g, 2f, vi.
dg.
since Ely; fa. us] = f. U/ and
y²
Further observe
E,, [2, Uz] = - (1-a)f; a (2
+2)+))) 2n9)
= -1-
a 2) f:
a ST
2
Combining the above three equations and letting = (1 ②) 1
a 0², we get
2
d$"
En..y.
29, - 2f T..
dgi
Stacking across all samples, let g (o.....,gh!. Then:
V Φ (g FU).
Applying the data covariance operator F FFT gives:
EFFF
v = g- Fū).
By the assumption that u+ u² and nd are orthonormal eigenvectors FFT with eigenvalues A
x and X", and 1/2 fo+ v-). Then:
Fū (1- 2
= Fut (1 (1- 2
a
-
1
Fo Fr.
2
and applying FF
FFT Fü
+t - (1 (1- 0)
- vー vー + Ao."
2
Substituting this expression concludes the proof:
F V
2
FF g
(1 0
x xtut
-1--)
- (1
a)
1 xー
a
n
2
2
2
2
31

---

J Computational Efficiency of Regularization Terms
Several self-supervised learning methods aim to mitigate representation collapse and redundancy by decorrelating feature dimensions. Barlow Twins minimizes the cross-correlation matrix between two views and enforces to be close to the identity, effectively promoting invariance while discouraging Zabonar
redundancy Zbontar et al. (202T]. VICReg combines invariance term with variance and covariance regularizers, penalizing off-diagonal entries in the covariance matrix Bardes et al. [202T). Both methods require computing and differentiating through batch-wise matrices of size d d, incurring cost of Ondd²) to form the matrix, and an additional (d²) for computing the regularization loss.
Whitening- based methods, such as ZCA whitening Ermolov et al. [202]] go further by requiring not only the covariance matrix F F Rdxd but also inverse square root, computed via eigende- composition. This results in a total cost of Ond + d²), making them significantly more expensive high-dimensional settings.
In contrast, our spectral flattening regularizer only requires access to the singular values of the feature matrix. These can be obtained by computing the eigenvalues of either F F e R or ge FFT E R=x" or by directly applying SVD to F E Raxa. Since all three methods yield the same singular values, one can select the most efficient strategy depending on the dimensions of F. Specifically, computing eigenvalues of F F' is preferred when d < n, while FF may be used when n < d. Direct SVD provides balanced alternative with cost (nd?) when n 2 d. This makes our method scalable to large batch sizes and embedding dimensions, while remaining competitive with or cheaper than other decorrelation strategies (Shietoo al. [2023).
All experiments using the spectral flattening regularizer were run on a single NVIDIA A100 GPU with 40 GB memory.
Table 8: Computational complexity of regularization terms for different SSRL methods. Here, is the batch size and d is the feature dimension.
METHOD
FORWARD + BACKWARD CosT
BARLOW TWINS ZBONTAR ET AL.
2021
Ond + dP) Onn de) O(nd² d²) (neel)
VICREG BARDES ET AL.
2021
WHITENING ERMOL .0v ET AL.
2021
SPECTRAL FLATTENING (OurS)
K Random-Task Contrastive Setup (Schematic)
This subsection clarifies the random-task model underlying Section 4.2 and its connection to our main results. Section generalizes Corollary (3 (fixed downstream task) to more realistic contrastive pretraining scenario where the downstream task is unknown at pretraining time. In our random- framework: two latent classes e+ c E C are sampled, (ii) the classifier vector I=(Fv), and (iv) the generalization objective becomes Ev,y (YT (FFT)-1Y) Theorem then shows this objective is minimized when FFT has uniform spectrum, directly motivating spectrum flattening regularizer.
V.- V.-.
(iii) labels are generated by the soft classifier TP(Y, +1 v) = 2
-pectrum-flattening
32

---

Randous task sampting and evaluation
trace cunstmint FF Sample (c*,e")
Cliss pood sampler
Soft labels V, i (#1)
Task loss aMM = "
Expeciaioo over tasks
V Vel
OLP) Ex
Fited features grdd
depends
Uniform
Spiky
lower 2IP)
Nabba 4(F)
Figure 4: Random-task setup (Section 4.2 sample (ct, form induce labels via p; 1+(v and evaluate {(F;Y) = =Y (FF Y.
2
Averaging tasks yields L(F), which (under fixed trace) is minimized when FF has uniform spectrum (Theorem
L Limitations
While our spectral flattening regularizer more efficient than full covariance- methods in many practical settings, it does incur additional computational cost due the need for singular value computation. our implementation, we compute the singular values of the batch feature matrix F Rexd via eigendecomposition of F F, which scales (ndd² when typically lower than that Barlow Twins or VICReg, both which require full covariance matrices and gradients through x d terms. However, the quadratic dependence on d may still pose challenges for extremely high-dimensional embeddings. See Appendix J for detailed comparison of methods and costs.
d. This cost is
33


Reviews:
[{"review_id": "7pqEDPKd9w", "reviewer": "Reviewer_EvGa", "summary": "This paper addresses the challenge of model reliance on spurious correlations in contrastive self-supervised learning. The authors propose a spectral regularization method that encourages the eigenvalues of the feature covariance matrix to be close to the largest eigenvalue, promoting a higher-rank representation. The paper provides a theoretical perspective linking downstream generalization error to the rank of the feature covariance matrix. Empirically, the method is applied to SimCLR and shows improved performance on benchmarks with spurious correlations, outperforming alternative approaches such as LateTVG on metrics such as worst-group and average accuracy.", "questions": "1. **Related Work on Rank-Promoting Regularization.**\nThe related works section would benefit from an additional discussion of related literature that explores regularization techniques aimed at improving the rank or diversity of learned representations such as Barlow Twins, VICReg, Whitening-based approaches. Including these connections would help contextualize the proposed method and clarify its novelty relative to existing spectrum- or correlation-based regularizers.\n\n2. **Computational Overhead of Spectral Regularization.** The paper would benefit from quantifying the training and memory overhead introduced by the spectral regularizer, especially relative to other methods such as VICReg and Barlow Twins. Including this analysis in the main manuscript (rather than supplementary material) would improve transparency and help practitioners assess the cost-benefit tradeoff.\n\n3. **Sensitivity Analysis of the Regularization Strength Hyperparameter.**\nThe method introduces a hyperparameter that controls the strength of the spectral regularization. It would be helpful to include a sensitivity analysis showing how performance varies with different settings of this hyperparameter. This would provide insight into the robustness of the method and guide practitioners on how to tune it effectively.", "limitations": "Yes, in the Appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper proposes a regularization method that encourages a high-rank feature covariance matrix that is supported by both theoretical analysis and empirical results, and leads to improved downstream performance.\n- The proposed regularization is architecture-agnostic. \n- The experiments report confidence intervals over five seeds, enhancing the statistical reliability of the findings.\n- The authors provide pseudocode and hyperparameter values which can enhance reproducibility.\n- In the questionnaire, they also mention that implementation of the method would be provided. \n\nRoom for improvement:\n- I believe the paper could benefit from including information about related work on regularization, the computational cost of the proposed method, and sensitivity analysis, some of which are available in the appendix. Please see details in the following section.", "was_revised": true, "final_justification": "I have increased my score during the rebuttal since the authors have adequately addressed my concerns .", "rebuttal": "Thank you very much for your detailed and thoughtful review. We appreciate your recognition of the theoretical and empirical strengths of our work, as well as the clarity, reproducibility, and architectural generality of the proposed regularizer. We address the specific concerns below.\n\n1. **Related Work on Rank-Promoting Regularization.** \n\nWe thank the reviewer for highlighting the importance of better situating our work among existing rank-promoting approaches in self-supervised learning. We have updated the related work section to explicitly include comparisons to methods such as Barlow Twins, VICReg, whitening-based regularization, and IFM.\n\n**Rank-Promoting Regularization.** Several recent methods explicitly incorporate regularizers to encourage more diverse, high-rank representations in self-supervised learning. For example, Barlow Twins [1] introduces a redundancy-reduction loss that drives the cross-correlation matrix of twin network embeddings toward the identity, thereby decorrelating feature dimensions and minimizing redundancy. Similarly, VICReg [2] adds a variance preservation term and a covariance penalty to maintain per-dimension variance while pushing off-diagonal covariances toward zero, thus preventing informational collapse by decorrelating features. Whitening-based approaches go even further by forcing the entire feature covariance to match an identity matrix (full whitening), which is equivalent to enforcing a full-rank embedding space and comes with theoretical guarantees against dimensional collapse [3]. Beyond these objectives on the representation statistics, other techniques like Implicit Feature Modification (IFM)  [4] actively perturb training examples in feature space to ensure the encoder utilizes a broader set of features, reducing reliance on any single dominant “shortcut” and promoting greater feature diversity. All of these strategies underline the importance of promoting feature diversity and high effective dimensionality in learned representations. Our proposed approach aligns with this principle but differs by emphasizing a balanced eigenspectrum (neither overly flat nor too concentrated), aiming to mitigate spurious feature dominance without the extremes of full whitening or low-rank collapse, thereby supporting both robust and transferable features.\n\n2. **Computational Overhead of Spectral Regularization.**\n \nMany self-supervised learning methods, such as Barlow Twins [1] and VICReg [2], reduce feature redundancy by computing and differentiating through $d\\times d$ matrices. Whitening-based methods, like ZCA whitening [3], are even more computationally expensive due to requiring an eigendecomposition complexity.\nIn contrast, our spectral flattening regularizer only relies on computing the singular values of the feature matrix $F$. \n\n| **Method**                          | **Forward + Backward Cost** |\n| ----------------------------------- | --------------------------- |\n| Barlow Twins (Zbontar et al., 2021) | $\\mathcal{O}(nd^2 + d^2)$   |\n| VICReg (Bardes et al., 2022)        | $\\mathcal{O}(nd^2 + d^2)$   |\n| Whitening (Ermolov et al., 2021)    | $\\mathcal{O}(nd^2 + d^3)$   |\n| **Spectral Flattening (Ours)**      | $\\mathcal{O}(nd^2)$         |\n\nWe provide a comprehensive explanation in the Appendix J and would be glad to include it in the main paper for improved clarity.\n\n3. **Sensitivity Analysis of the Regularization Strength Hyperparameter.**\n\nWe thank the reviewer for requesting a sensitivity analysis, which we now provide below. We evaluated the performance of our method on the MetaShift benchmark using different values of the spectral regularization strength $\\alpha$, using a fixed seed. The results are presented in the following table:\n\n| Regularization Strength \\$\\alpha\\$ | Average Accuracy (%) |\n| :--------------------------------: | :------------------: |\n|                0.001               |         76.40        |\n|               0.0025               |         77.53        |\n|                0.005               |         80.34.       |\n|                0.010               |         76.97        |\n|                0.025               |         75.28        |\n|                0.050               |         77.53        |\n|                0.100               |         76.40        |\n\nWe observe that while the method shows some sensitivity to the hyperparameter, it maintains strong performance in a broad range, particularly from $\\alpha = 0.005$ to $\\alpha = 0.05$. The best performance was achieved at $\\alpha = 0.005$. This provides a practical default for tuning.\n\nWe are currently running additional experiments across multiple seeds to confirm robustness, and we will include those upon acceptance. \n\nYour comments on related work, computational overhead, and sensitivity analysis are especially valuable and have helped us improve the presentation and scope of the paper. If any concerns remain or further clarification is needed, we would be happy to address them. Otherwise, we kindly ask you to consider increasing your score.\n\nReferences:\n \n[1] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pages 12310–12320. PMLR, 2021.\n\n[2] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.\n\n[3] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International conference on machine learning, pages 3015–3024. PMLR, 2021.\n\n[4] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974–4986, 2021."}, {"review_id": "I5orrRar8h", "reviewer": "Reviewer_Dgse", "summary": "This paper shows that the reason why SSRL (self-supervised representation learning) methods learn spurious features, is that the directions which those features are learned dominates the top eigenvalues of the feature covariance $FF^T$. They formalize this by showing that a downstream linear probe (on top of feature representations) learns fastest along large eigenvector directions, so any label information in small eigenvalues is not being used when training the linear probe.  The paper also generalizes this to unknown tasks in Theorem 4, and shows that the expected loss is minimized when all eigenvalues are close to each other. Given this, they propose a Spectrum Flattening Regularizer for SSRL losses and show that it improves downstream worst-group accuracy across a number of tasks.", "questions": "Q1: The regularizer flattens the eigenvalues of the mini-batch feature covariance, but Theorem 4 is for the entire dataset. How sensitive is the training signal to the mini-batch estimates of the regularizer? could you plot the eigenvalues that vary across batches during training?\n\nQ2: Do spurious eigenvectors actually vanish when adding the regularizer? Is there a way to quantify this?\n\nQ3: Because eigenvalue estimates improve with larger batches, a fixed regularization $\\alpha$ may have different strengths across batches. Does changing the batch size significantly effect tuned $\\alpha$?", "limitations": "L1: Theorem 4 assumes square full-rank $FF^T$, independent bernoulli labels, and a trace constraint. It is unclear how the “uniform spectrum” optimum generalizes to overparameterized encoders or correlated tasks.\n\nL2: The regulariser flattens the eigenvalues of the batch feature covariance, which can diverge from the full dataset covariance when batches are small or highly correlated.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nS1 (theoretical analysis motivated by the empirical observation): The authors start from an empirical observation (that the contrastive features collapse into high variance directions for datasets containing spurious correlations) and which motivates the theory. \n\nS2 (proposed method motivated by the theoretical analysis): The proposed spectrum flattening loss is directly derived from theorem 4's uniform eigenvalue optimum, which makes it principled. \n\nS3: The proposed loss is applicable to any sort of SSRL method that is learning a feature space, and adapting it for other SSRL method would be easy. \n\nWeaknesses:\n\nW1: Paper presentation and writing can be improved. For instance, the problem setup (section 4.1) comes after where these symbols first appear in the paper (section 2.1).\n\nW2: Even though the eigenvalues are flattened using the proposed method, the paper offers no analysis of how the directions itself evolves, which is important for understanding whether spurious vectors are suppressed or rescaled, though the downstream worst-group accuracy improves.  \n\nW3: The random-task model and the reduction to $tr[FF^T]^{-1}$ are presented too abruptly, and the paper’s transition from Corollary 3 to Theorem 4 makes it follow in its current form. The “random task” construction that supports Theorem 4 introduces several new vectors and scalars in a single dense paragraph without a guiding figure.\n\nW4: The theorem itself relies on specific assumptions that is unclear if they hold in practice, e.g. independent Bernoulli labels, a uniform $\\ell_\\infty$, a square full-rank feature matrix, and a trace (rather than Frobenius or spectral-norm) constraint. Especially for overparameterized encoders or correlated downstream tasks.", "was_revised": true, "final_justification": "Keeping my score, since the reviewers have addressed my concerns in the rebuttal.", "rebuttal": "Thank you very much for your detailed and insightful review. We deeply appreciate your recognition of both the theoretical motivation and general applicability of our method, as well as your thoughtful comments on the limitations and open questions.  Below, we address your concerns and questions.\n\n**W2:** \n\nWhile our proposed regularizer explicitly flattens the _eigenvalues_ of the feature covariance matrix, it does not directly constrain the _eigenvectors_. Nonetheless, the generalization bound presented in Corollary 3 shows that what ultimately matters is the alignment between the label vector $\\mathbf{y}$ and the top eigenspaces of $FF^\\top$. Thus, the key question is whether some (e.g. spurious) features retain high variance and dominate top eigenspaces, or whether their influence is reduced under the regularizer.\n    \nTo address this, we plan to visualize the behavior of spurious directions via three plots upon acceptance:\nSpecifically, we plan to include the eigenvalue spectrum of $FF^\\top$ and the projection magnitudes $\\langle \\mathbf{y}, \\mathbf{v}_i \\rangle$ of the label vector onto each eigenvector $\\mathbf{v}_i$ for SpurCIFAR-10 trained with SimCLR and trained with SimCLR with our spectral regularizer.\n\nThis comparison highlights that in CIFAR-10, label information aligns more strongly with the top eigenspaces, whereas in SpurCIFAR-10, the alignment is weaker due to dominant spurious directions. \n\n**W3:** \n\nSection 4.2 aims to generalize Corollary 3, which applies to a fixed downstream task, to a more realistic _contrastive learning_ setting where the downstream task is _unknown_ at pretraining time. We introduce a random-task framework in which (i) two latent classes $c^+, c^- \\in \\mathcal{C}$ are sampled, (ii) the classifier vector is $\\mathbf{v} = \\mathbf{v}\\_{c^+} - \\mathbf{v}\\_{c^-}$, (iii) the labels are generated via the soft classifier, $\\mathbb{P}(Y\\_i = \\pm 1 \\mid \\mathbf{v}) = \\frac{1 \\pm (F \\mathbf{v})\\_i}{2}$, and (iv) the generalization objective becomes $\\mathbb{E}\\_{\\mathbf{v}, Y} \\left[ Y^\\top (F F^\\top)^{-1} Y \\right]$.\n\nTheorem 4 shows this is minimized when $FF^\\top$ has uniform eigenvalues, directly motivating our spectrum-flattening regularizer. We plan to clarify this section in the camera-ready with a schematic figure of the random-task setup.\n\n**W4:** \n\nWe appreciate the reviewer’s thoughtful comments regarding the assumptions underlying Theorem 4. Below, we address each assumption and clarify its motivation or generalization.\n\n**1) Trace constraint.**\nThe trace constraint used in Theorem 4 is both theoretically meaningful and practically justified. In modern contrastive learning pipelines—such as SimCLR and SimSiam—it is standard to apply $\\ell_2$-normalization to the feature vectors. This ensures that the overall energy of the representation, captured by $\\operatorname{tr}(FF^\\top)$, remains approximately constant across batches. Thus, the fixed-trace constraint reflects common empirical practice. Furthermore, since downstream classifiers are scale-invariant, normalizing the trace removes trivial rescalings of the feature matrix and makes the spectral analysis of feature representations well-posed. We will make this motivation more explicit in the revised version of the paper.\n\n**2) Uniform $\\ell_\\infty$ constraint.**  \nThe $\\ell_\\infty$ constraint used in our theorem is not essential and can be replaced with any constant $c=c_1, c_2$. The results scale accordingly. In practice, this constraint is naturally satisfied due to common regularization on the feature and classifier norms, such as $\\|f_i\\|2 \\leq c_1$ and $\\|v\\|_2 \\leq c_2$, which are widely adopted in deep contrastive learning setups. We used the uniform $\\ell_\\infty$ constraint for analytical clarity, and the bound remains valid under more general norm bounds.\n\n**3) Independent Bernoulli labels.**  \nOur analysis is designed to **measure the quality of features learned by contrastive learning**. The key intuition is that *good representations* should support *easy downstream classification*, typically through a linear model. To formalize this, we consider a stylized label generation process: given feature $f_i$ and two contrast vectors $v^+$ and $v^-$, the label $+1$ is assigned with probability proportional to $f_i^\\top v^+ - f_i^\\top v^-$, and vice versa. This assumption allows for clean analysis and directly evaluates whether the learned features are linearly discriminative. It serves as a principled surrogate for linear separability, a widely accepted proxy for representation quality in the contrastive learning literature (e.g., SimCLR, MoCo).\n\n**4) Full-rank assumption.** \nWhile our theorem assumes a square, full-rank feature matrix $F$, this is a simplifying assumption and not fundamental to the results. The main requirement is that the Gram matrix $FF^\\top$ is full-rank, a condition commonly used in analyses of overparameterized models (e.g., [Arora et al., 2019]) to ensure that the optimization landscape allows perfect fitting. This condition is satisfied in typical **overparameterized** regimes where $F \\in \\mathbb{R}^{n \\times d} \\) with \\( d \\geq n$, such as in contrastive encoders with high-dimensional features.\n\nIn the underparameterized case $d \\ll n$, our analysis can be naturally extended via the eigendecomposition $FF^\\top = \\sum \\lambda_i v_i v_i^\\top$, assuming that the label vector $y$ lies in the span of the eigenvectors with non-zero eigenvalues (i.e., $y^\\top v_i = 0$ whenever $\\lambda_i = 0$). **Under this milder assumption**, the bound in Corollary 3 becomes:\n$\ny^\\top \\left( \\sum_{i: \\lambda_i \\ne 0} \\frac{1}{\\lambda_i} v_i v_i^\\top \\right) y,\n$\nwhich reflects the contribution of the label only along the directions where the feature matrix has support.\n\n**For Theorem 4, this implies that the spectrum must be uniform (constant) over the first $r$ non-zero eigenvalues, and zero elsewhere**, in order to achieve optimality. This provides a clear geometric interpretation: the features are well-structured if they uniformly capture the relevant directions in the label space, while ignoring degenerate components. This insight remains informative even when the feature matrix is low-rank, and highlights how our results apply beyond the strictly full-rank case.\n\nWe appreciate the reviewer’s concerns and hope this response clarifies the role and flexibility of each assumption. Our goal is to provide a theoretical lens on feature quality in contrastive learning, and while our assumptions are stylized for analytical clarity, they are either standard in related theoretical work or adaptable to broader regimes. We will explicitly incorporate these clarifications and generalizations in the final version.\n\n**Q1:** \n\nWe thank the reviewer for raising this important point. While Theorem 4 analyzes the full-dataset spectrum of the feature matrix $F$, our regularizer operates on mini-batch estimates of $FF^\\top$. To assess the sensitivity of the training signal to this approximation, we will empirically analyze the spectrum across mini-batches during training.\n\nThis will be similar in spirit to the visualization we proposed in **W2**, where we plan to plot the eigenvalue spectrum of $FF^\\top$ and the projection magnitudes $\\langle \\mathbf{y}, \\mathbf{v}_i \\rangle$ of the label vector onto each eigenvector $\\mathbf{v}_i$, computed for multiple mini-batches during training. These plots will help quantify whether the flattening effect is consistent across batches and whether eigendirections are suppressed or simply rescaled.\n\nWe will include these visualizations upon acceptance.\n\n**Q2:** \n\nThe spurious eigenvectors do not vanish when applying the proposed regularizer. Instead, their relative spectral dominance is attenuated, allowing other directions—potentially aligned with core or task-relevant features—to become more prominent. The goal of our approach is not to suppress or discard any specific directions, but rather to mitigate over-reliance on a small number of high-variance components. This aligns with our Theorem 4, which suggests that a more uniform spectrum promotes better generalization across downstream tasks when their identity is unknown at pretraining time. We emphasize that spurious directions may still contain useful information for certain downstream tasks. Since the task is not known during representation learning, it is desirable to maintain a feature space that preserves all directions while distributing variance more evenly. The regularizer achieves this by encouraging the spectrum of $FF^\\top$ to become flatter, thereby reducing the disproportionate influence of dominant eigenvectors without erasing them. To empirically assess this effect, we will (upon acceptance) include visualizations of the eigenvalue spectrum and the corresponding label projection magnitudes $\\langle \\mathbf{y}, \\mathbf{v}_i \\rangle$. \n\n**Q3:**\n \nTo investigate this, we conducted an experiment on SpurCIFAR-10 using SimCLR trained with our regularizer, fixing the regularization weight to $\\alpha = 0.01$ and varying the batch size. Each setting was run with a single random seed to isolate batch-size effects. \n\n| **Batch Size** | **Avg. Acc. (%)** | **WG. Acc. (%)** | **Eff. Rank** |\n| -------------- | -------------------- | ---------------------------- | ------------------ |\n| 512            | 77.93                 | 55.77                         | 30.1               |\n| 256            | 80.1                 | 58.82                         | 58.05               |\n| 128            | 80.94                 | 60.31                         | 68.02               |\n| 64             | 81.74                 | 54.16                         | 48.86               |\n\nWe will include an additional ablation study on the effect of varying batch sizes upon acceptance.\n\nWe hope that the revisions and additional discussions have addressed your concerns. If any issues remain unclear or open, we’d be happy to provide further clarification."}, {"review_id": "iVxcC8lMps", "reviewer": "Reviewer_nT26", "summary": "The paper shows that standard contrastive self-supervised learning concentrates variance in a handful of feature directions; these dominant directions often encode spurious shortcuts, hurting robustness. The authors prove that, under a trace constraint, a uniform eigen-spectrum minimizes expected generalization error across random downstream linear tasks. Guided by this, they add a spectrum-flattening regularizer to SimCLR that penalizes unequal eigenvalues, effectively lifting small singular values and raising the feature matrix’s effective rank. This regularizer improves worst-group and average accuracy on five spurious-correlation benchmarks.", "questions": "Please see the weaknesses section above", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well written, clear and easy to follow. \n- The method is grounded in solid theoretical principles --  Theorem 4 rigorously shows that a perfectly uniform eigenspectrum minimises average linear-probe error under a trace constraint.\n- Hyper-parameters, optimiser settings, and code snippets for the SVD-based penalty are fully disclosed, making the approach easy to drop into existing contrastive SSL pipelines.\n\n**Weaknesses**\n\n- The proof assumes a square, full-rank feature matrix and purely linear downstream heads. Realistic encoders are tall (feature dim < #samples) and many transfer tasks use non-linear fine-tuning. The paper does not quantify how loose the bound becomes in this common regime, nor whether a flat spectrum remains optimal when features are low-rank.\n\n- Because the penalty uniformly boosts all low-eigenvalue directions, it can inadvertently raise purely noisy components if those directions are not spurious but irrelevant. The paper does not test on datasets where spurious and core signals overlap heavily, so this failure mode is unexamined.\n\n- The authors sub-sample majority groups when training the linear probe (Sec. 6.3). Prior work shows that simple re-balancing alone can recover much of the performance loss caused by group or class imbalance—for instance JTT (Liu et al., 2021), GroupDRO (Sagawa et al., 2020), and “Just Train Twice” (Jiang et al., 2021). Because of this step, the study (i) relies on explicit group labels at probe time, weakening its “label-free” narrative, and (ii) makes it harder to judge how much additional robustness comes from the spectrum-flattening regulariser versus the well-known benefits of balanced sampling. Reporting results with and without sub-sampling—or using natural group imbalance—would clarify the regulariser’s standalone contribution.\n\n- The regularization needs an SVD per mini-batch. At 512dim this is not that significant, but it grows quadratically with dimension. The authors mention possible Hutch++ approximations in Appendix K but does not implement or benchmark them. It is unclear how stable these approximations would be in SSL training.  \n\n- All experiments are only conducted on small scale datasets; no evidence is shown for ImageNet-1k, or ImageNet which are now the defacto for SSL evaluations. Further, more exhaustive and large scale benchmarks from WILDS could also be used. \n\n- The paper claims the framework is broadly applicable to representation learning methodologies, but the main experimental results only show it applied to SimCLR. Adding more baselines with the additional regularization will make the experiments section more complete. \n\n- (Minor) Related works section i.e. Section 3 starts abruptly after a nice motivation to the problem in Section 2. It might be better to either place it before Section 2 or towards the end to maintain and coherent flow of information.", "was_revised": true, "final_justification": "I thank the authors for their detailed response. Their rebuttal has addressed most of my concerns, so I will increase my score to 4.", "rebuttal": "We thank the reviewer for the constructive and thoughtful evaluation. We appreciate the positive comments on the clarity of the paper, the strength of our theoretical analysis (especially Theorem 4), and the ease of integrating our method into existing pipelines. We're glad the reviewer found our contributions to be both rigorous and practical. Below, we address each of the reviewer's concerns.\n\n**W1:**\n\nAs noted, real-world encoders typically produce tall, rank-deficient feature matrices $d \\\\ll n$ and are often fine-tuned with non-linear heads. To address the reviewer’s concern more directly, we have extended Theorem 4 to the rank-deficient case where $\\operatorname{rank}(F) = r < n$, and show that the same conclusion holds: the surrogate loss $\\mathcal{L}(F) := \\operatorname{tr}((FF^\\top)^{\\dagger})$ is minimized when the non-zero eigenvalues of $FF^\\top$ are uniform. This result strengthens the connection between spectral uniformity and generalization, even in the realistic regime of low-rank encoders. In the proof, we replace the inverse with the Moore–Penrose pseudoinverse $(FF^\\top)^{\\dagger}$, which is well-defined for all symmetric positive semi-definite matrices and coincides with the true inverse whenever $FF^\\top$ is full-rank. Importantly, this replacement preserves the structure of the objective: the trace of the pseudoinverse penalizes low-energy (i.e., low-eigenvalue) directions in the representation space, while ignoring the nullspace where the model has no discriminatory capacity. From a generalization perspective, $Y^\\top (FF^\\top)^{\\dagger} Y$ continues to quantify the expected error for randomly sampled downstream tasks projected onto the span of the learned representation. We have extended our formal theorem.\n\n**Theorem 4** [Rank-deficient Case]\n\nLet $F \\in \\mathbb{R}^{n \\times d}$ be a feature matrix with $\\operatorname{rank}(F) = r \\le \\min(n,d)$, and let $G := FF^\\top \\in \\mathbb{R}^{n \\times n}$. Suppose $G$ has eigenvalues $\\lambda_1 \\ge \\lambda_2 \\ge \\dots \\ge \\lambda_r > 0$, with $\\lambda_{r+1} = \\dots = \\lambda_n = 0$, and a fixed trace constraint $\\sum_{i=1}^r \\lambda_i = c$. Then the expected loss\n$\n\\mathcal{L}(F) := \\mathbb{E}\\_{\\mathbf{w}, \\mathbf{y}} \\left[ \\| \\mathbf{y} - F \\mathbf{w} \\|^2 \\right]\n$\nunder the random task model is minimized when the non-zero eigenvalues are uniform:\n$\n\\lambda_1 = \\dots = \\lambda_r.\n$\n\nWe also updated **Corollary 3** to reflect the low-rank case, using Moore–Penrose pseudoinverse, in the generalization bound. This ensures consistency with the modified theorem and highlights that our conclusions hold even when $F$ is rank-deficient.\nDue to space limit, we do not restate the full theorem here and the complete theorem and proof will appear in the Appendix upon acceptance. \n\n**Corollary 3** [Rank-deficient Case]\nWith probability at least $1 - \\delta$, the population loss satisfies\n$\nL\\_{\\mathcal{D}}(g\\_{\\mathbf{w}^{(k)}}) := \\mathbb{E}\\_{(\\mathbf{x}, y) \\sim \\mathcal{D}} \\left[ \\ell(g_{\\mathbf{w}}(\\mathbf{x}), y) \\right]\n\\le \\widetilde{O} \\left( \\sqrt{ \\frac{ \\mathbf{y}^\\top (F F^\\top)^{\\dagger} \\mathbf{y} }{n} } \\right),\n$\nwhere $\\mathbf{y} = (y_1, \\dots, y_n)^\\top$, and $\\widetilde{O}$ hides logarithmic factors and dependence on $\\delta$.\n\n**W2:**\n\nWe thank the reviewer for highlighting a potential failure mode: that our regularizer, by uniformly encouraging spectrum flattening, could inadvertently amplify noisy or irrelevant directions. We emphasize, however, that our method does not enforce a hard constraint on the spectrum. The regularizer is applied softly via a tunable hyperparameter $\\alpha$, which balances spectral flattening against the base contrastive objective. In practice, this allows the model to adjust its spectrum to the structure of the data, rather than blindly forcing uniformity.\n\nRegarding the distinction between spurious and core features, we respectfully clarify that in our setting this is not a spatial or subspace-level separation, but an inductive one. Spurious features are defined as those that are _simple and easily learnable_ that are irrelevant to the downstream task at hand. They are picked up early in training due to the optimization bias of neural networks, not because they inhabit distinct eigenvectors. Therefore, the notion of \"overlap\" between spurious and core signals does not strictly apply. Our goal is not to eliminate spurious features entirely, but to **ensure that underrepresented or task-relevant features are not suppressed** by spectral collapse. By flattening the spectrum moderately, we allow the model to preserve both types of features — especially those that would otherwise be underutilized. We will clarify these conceptual points and the role of the hyperparameter in the revised version.\n\n**W3:**\n\nWe thank the reviewer for raising the important question of whether rebalancing strategies could account for the observed robustness gains. We emphasize that our method does not assume access to group labels during self-supervised pretraining, so rebalancing is not performed. However, to test this hypothesis in controlled settings where group information is available, we refer to the systematic evaluation presented in [1]. In Table 2 of their paper, the authors trained SimSiam under three rebalancing strategies applied during pretraining: (i) group-balanced sampling to match the downstream distribution, (ii) downsampling majority groups, and (iii) upsampling minority groups. Across multiple spurious benchmarks including CelebA, MetaShift, CMNIST, SpurCIFAR-10, and Waterbirds, they found that such interventions did _not_ consistently improve worst-group accuracy. These results (in Table 2 of [1]) suggest that simple group reweighting during pretraining does not replicate the robustness benefits observed with spectrum regularization. We will include this discussion in the final version to clarify the distinction.\n\nWe thank the reviewer for pointing to Just Train Twice (JTT) [2] and GroupDRO [3] as related work. We were unable to find a separate \"Just Train Twice\" (JTT) paper by Jiang et al., and would appreciate clarification in case we missed it or if this was intended to refer to Liu et al., 2021. All of these methods operate in a supervised setting and require access to either class labels, group labels, or both during training. GroupDRO explicitly assumes known group annotations and dynamically upweights high-loss groups to optimize worst-group performance. JTT identifies difficult examples by first training a supervised model, then upweighting high-loss samples in a second training pass—relying on access to per-example losses and class labels.\n\nIn contrast, our method is applied in the fully self-supervised regime, where neither class nor group labels are available during training. The spectrum-flattening regularizer is designed to improve representation robustness across downstream tasks by encouraging spectral diversity, without relying on any knowledge of group structure or downstream objectives. While JTT and GroupDRO address robustness through explicit supervision or heuristic reweighting, our approach provides a complementary, label-agnostic mechanism that improves generalization even in the absence of downstream supervision. We will clarify this distinction in the related work of the final version.\n\n\n**W4:**\n\nTo investigate this further, we conducted an ablation comparing the full SVD against approximate randomized SVD. All evaluations used batch size 256 on SpurCIFAR-10 with our regularized SimCLR.\n\nThe results are summarized below:\n\n| SVD Method        | WG Acc. (%) | Avg. Acc. (%) | Effective Rank | GPU Memory (MB) | SVD Time (s) |\n|-------------------|------------------|-------------------|-----------------|------------------|---------------|\n| Full SVD          | 58.34        | 80.34             | 57.93           | 2510         | 0.067        |\n| Approx SVD (k=64) | 54.87            | 80.22             | 57.79           | 2510         | 0.037        |\n| Approx SVD (k=32) | 51.30            | 80.28             | 57.51           | 2483         | 0.035        |\nWe observe that:\n\nThis suggests that the regularizer is robust to mild approximation errors and that SVD approximation approaches are viable in practice. These experiments demonstrate that the method is practical and scalable to higher dimensions, especially when using rank-limited approximations.\n\n**W5:**\n\nWe conducted experiments on Hard ImageNet [4].\nWe report the average accuracy on the standard test split (None) for one random seed: \n\n| Method                     | Avg. Accuracy |\n| ------------------------- | -----------------------------: |\n| SimCLR (baseline)         |                          79.06 |\n| SimSiam (baseline)        |                          79.50 |\n| Ours (SimCLR + Reg.)      |                      79.46 |\n| LATETVG [1] |                          78.00 |\n\n**W6** \n\nWe thank the reviewer for the suggestion. We have also applied our regularizer to SimSiam, as reported in Appendix H (Tables 5 and 6), demonstrating its compatibility beyond SimCLR.\n\nWe believe we have addressed your concerns thoroughly, both theoretically and empirically. We kindly ask you to consider raising your score or let us know if any further clarification is needed.\n\nReferences: \n\n[1] K. Hamidieh, H. Zhang, S. Sankaranarayanan, and M. Ghassemi. Views can be deceiving: Improved ssl through feature space augmentation. ArXiv, 2024.\n\n[2] E. Z Liu, B. Haghgoo, A. S Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn. Just train twice: Improving group robustness without training group information. ICML. PMLR, 2021b.\n\n[3] S. Sagawa, P. W. Koh, T. B Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, 2019.\n\n[4] M. Moayeri, S. Singla, and S. Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. NeurIPS, 2022."}, {"review_id": "31NTwr1JKk", "reviewer": "Reviewer_kaxi", "summary": "This paper addresses the problem of spurious feature, also called \"simplicity bias\", in contrastive learning by examining the eigenvalue spectrum of learned representations. The authors argue that low-rank feature spaces, often a byproduct of learning spurious but easy-to-fit features, limit downstream generalization and robustness, notably on underrepresented data groups.\n\nTo address this, the authors introduce a spectrum-flattening regularizer that promotes a more uniform singular value distribution in the feature matrix. This regularization is theoretically motivated by the introduction of a linear classifier generalization bound, and then of a expected inverse energy over random classification tasks which appears to depend on the feature covariance matrix. The authors then showed that the expected inverse energy was minimized whenever the eigenvalues of FF.T are uniform. This simple, architecture-agnostic intervention increases the effective rank of the representation space. Empirical results on SpurCIFAR-10 and other spurious correlation benchmarks demonstrate improvements in worst-group accuracy and overall performance.", "questions": "In Theorem 4, you suppose that F satisfies the trace constraint, how valid is this assumption in practice ? Is it trivial ? Could you discuss it a little more ?\n\nIn Theorem 4, you argue that when F is a scaled orthogonal matrix, the expected inverse energy is minimized, wouldn't it be relevant to verify in practice that F gets close to orthogonality at the end of the training ?\n\nI found hard to understand the Fig. 2, could it be possible to explain it a bit more a make it clearer at his point of the paper ? You talk about dropping eigenvalues but you regularization do not work this way, so I am having trouble to understand.", "limitations": "The authors should discuss a bit more their assumptions and compare with more related works as well as other contrastive and non-contrastive methods.\n\nI am prone to changing my opinion to Accept if these limitations are adequately addressed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe method is theoretically grounded, offering a clear and intuitive motivation for addressing the spectral collapse caused by spurious features.\n\nThe paper is well-written and easy to follow, with a concise experimental setup that effectively communicates the core contributions.\n\nThe authors conduct a thoughtful and controlled evaluation, notably by subsampling majority groups to construct balanced training sets, avoiding common pitfalls in spurious correlation benchmarks.\n\nWeaknesses\nThe paper includes limited comparisons to related work, such as the implicit feature modification (IFM) of Robinson et al., which is cited but not empirically evaluated against.\n\nIt lacks broader evaluation across self-supervised learning paradigms. In particular, methods like BYOL and DINO, which exhibit implicit spectral regularization, are not considered—even though relevant theoretical work (e.g., Zhuo et al., ICLR 2023, https://arxiv.org/pdf/2303.02387) highlights spectral properties in non-contrastive frameworks.\n\nThe proposed method is not evaluated on standard self-supervised benchmarks such as ImageNet-1k or at least ImageNet-100 or STL-10. Given the method’s robustness to spurious features, one would expect improved generalization performance; such experiments would have helped assess its broader applicability.", "was_revised": true, "final_justification": "Authors have globally responded to my points, I am thus recommending accept.", "rebuttal": "We sincerely thank the reviewer for the thoughtful summary and constructive feedback. We are especially grateful for recognizing the theoretical grounding of our approach, and the careful design of our experiments. Your comments have been very helpful in guiding our revisions. Below we address your concerns and questions. \n\n**W1: More comparison across SSRL baselines.**\n\nTo address the reviewer's concern regarding broader evaluation, we conducted additional experiments with IFM, DINO, and BYOL, beyond the baselines already reported in Table 5 and 6. The results, presented above, show that our spectral regularization maintains competitive or superior worst-group accuracy compared to these methods, especially on SpurCIFAR-10, Waterbirds, and MetaShift. Notably, while IFM performs well on some datasets, our method achieves the highest worst-group accuracy on SpurCIFAR-10 and MetaShift with SimCLR + spec. DINO lags behind on several benchmarks despite its implicit spectral flattening behavior, as noted in prior work.\n\nTable A: Worst-Group Accuracy (%)\n| Dataset      | Barlow Twins | DirectDLR | SimSiam + spec (ours) | SimCLR + spec (ours) | IFM       | BYOL  | DINO      |\n| ------------ | ------------ | --------- | --------------------- | -------------------- | --------- | ----- | --------- |\n| CMNIST       | 57.05        | 90.32     | 94.40                 | 95.10            | **96.77** | 95.82 | 56.67     |\n| SpurCIFAR-10 | 6.00         | 20.58     | 50.10                 | **59.70**            | 53.23     | 50.67 | 11.37     |\n| CelebA       | 39.99        | 68.68     | 74.17                 | **84.20**            | –         | –     | –         |\n| MetaShift    | 58.33        | 53.84     | **69.23**             | 67.40                | 53.85     | 41.67 | 58.33 |\n| Waterbirds   | 43.13        | 47.32     | 49.08                 | **56.70**            | 50.96 | 46.89 | 49.03     |\n\nTable B: Average Accuracy (%)\n| Dataset      | Barlow Twins | DirectDLR | SimSiam + spec (ours) | SimCLR + spec (ours) | IFM       | BYOL      | DINO  |\n| ------------ | ------------ | --------- | --------------------- | -------------------- | --------- | --------- | ----- |\n| CMNIST       | 93.10        | 96.35     | 97.20                 | 97.00                | **98.16** | 96.20     | 66.05 |\n| SpurCIFAR-10 | 22.00        | 50.20     | 71.78                 | **80.10**            | 76.70     | 73.30     | 26.28 |\n| CelebA       | 84.61        | 78.23     | **89.12**             | 88.50                | –         | –         | –     |\n| MetaShift    | 64.60        | 75.28     | 77.52                 | 78.10            | **78.65** | 74.71     | 73.60 |\n| Waterbirds   | 54.67        | 53.46     | 60.55             | 57.90                | 55.78     | 53.70 | 50.60 |\n\nThese results demonstrate that our method is not only effective across contrastive methods (SimCLR, SimSiam, IFM) but also compares favorably to non-contrastive approaches like BYOL and DINO. All experiments used a single seed for consistency. This broader evaluation strengthens the case for the general applicability and robustness of our regularizer across different SSRL paradigms.\n\n**W2: More experiments on large-scale datasets: Hard ImageNet [1] and STL-10 [3].**\nWe conducted experiments on Hard ImageNet [1]. We report the average accuracy on the standard test split (None) for one random seed: \n\n| Method                     | Avg. Accuracy |\n| ------------------------- | -----------------------------: |\n| SimCLR (baseline)         |                          79.06 |\n| SimSiam (baseline)        |                          79.50 |\n| Ours (SimCLR + Reg.)      |                      79.46 |\n| LATETVG [2] |                          78.00 |\n\nMoreover, we conducted additional experiments on STL-10 using SimCLR as the base method and evaluated the impact of our spectral regularizer under three different regularization strengths. While this was done using a single seed for completeness, the results consistently show improvements in both average and worst-group accuracy over the baseline. These results will be included in the final version of the paper upon acceptance.\n\n| Method               | Regularizer α | Avg. Accuracy (%) | Worst-Group Accuracy (%) |\n| -------------------- | ------------- | ----------------- | ------------------------ |\n| SimCLR (baseline)    | –             | 63.72             | 44.25                    |\n| SimCLR + spec (ours) | 0.001         | 64.20             | 44.75                    |\n| SimCLR + spec (ours) | 0.005         | 64.49             | 45.37                    |\n| SimCLR + spec (ours) | 0.010         | **64.26**         | **46.75**                |\n\n**Q1:**\n\nThe trace constraint assumed in Theorem 4 is both theoretically meaningful and practically justified. In modern contrastive learning pipelines, feature vectors are typically $\\ell_2$-normalized (e.g., in SimCLR and SimSiam), which ensures that the overall energy—captured by the trace of $FF^\\top$—is approximately fixed across training batches. This normalization implicitly induces a bounded trace, making our assumption aligned with standard practice. Moreover, since downstream classifiers are scale-invariant, constraining the trace serves to eliminate trivial rescalings of the feature matrix and leads to a well-posed analysis of the spectral structure. We will clarify this motivation more explicitly in the main text.\n\n**Q2:**\n\nWe appreciate the reviewer’s suggestion to empirically verify whether the learned feature matrix $F$ approaches orthogonality during training. To this end, we evaluated the _effective rank_ of $F$, which serves as a proxy for spectral flatness and proximity to scaled orthogonality. We conducted this analysis on SpurCIFAR-10 and reported effective rank and the worst-group accuracy (%). Our regularized method achieved higher effective rank than the SimCLR baseline, supporting our theoretical claim that the proposed regularizer promotes more uniform and orthogonal representations. This result will be included in the final version and appendix as further empirical evidence for the connection between spectral uniformity and improved generalization.\n\n| Method               | Effective Rank ↑ | Worst-Group Accuracy (%) ↑ |\n| -------------------- | ---------------- | -------------------------- |\n| SimCLR (baseline)    | 47.31            | 36.11                      |\n| SimCLR + spec (ours) | **58.05**        | **58.82**                  |\n\nThis table illustrates how our spectral regularization substantially increases the effective rank of the learned representations, which correlates with improved worst-group performance, aligning with our theoretical motivation.\n\n**Q3:**\n\nThank you for pointing this out. We agree that the current presentation of Figure 2 may conflate our regularization method with the spectrum truncation ablation, and we will revise the text to improve clarity. To clarify: Figure 2 does not illustrate the effect of our proposed regularizer directly. Rather, it serves as an empirical motivation by showing how post hoc manipulations of the learned feature spectrum (specifically, truncating small singular values and/or flattening the rest) impact downstream performance. As the reviewer correctly notes, our regularizer does not **drop** eigenvalues. Instead, it softly promotes a balanced (i.e., flatter) spectrum via penalizing variance in the eigenvalues. We will make this distinction explicit and revise the wording around \"dropping eigenvalues\" to avoid confusion. Furthermore, we will refer the reader more clearly to Appendix A, where the separate effects of truncation and flattening are disentangled. We appreciate the feedback and will ensure the final version communicates this more transparently.\n\nThank you again for your thoughtful and constructive feedback. We believe the additional results and clarifications provide, especially the broader set of baselines (including BYOL, DINO, IFM), and Hard ImageNet [1], and STL-10 [3] evaluation, and the effective rank analysis, address the key concerns regarding the method’s generality, comparisons, and empirical scope. We kindly ask you to consider updating your score accordingly, or let us know if any further questions remain; we would be happy to clarify.\n\nReferences: \n\n[1] M. Moayeri, S. Singla, and S. Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. NeurIPS, 2022.\n\n[2] K. Hamidieh, H. Zhang, S. Sankaranarayanan, and M. Ghassemi. Views can be deceiving: Improved ssl through feature space augmentation. ArXiv, 2024.\n\n[3] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. AISTATS, 2011."}]

Author rebuttal:

Meta-review: This paper proposes a method that uses spectral regularization to avoid the spurious features that are often acquired by contrastive learning. After empirically demonstrating a relationship between the learning of spurious features during contrastive learning and the tendency of the feature matrix to become low-rank, the authors theoretically show that training a linear predictor on a fixed feature space is strongly influenced by such low-rank structure. Building on this theory, they propose a spectral regularization scheme and empirically demonstrate that it improves performance on downstream tasks.

The paper offers solid theoretical claims regarding the impact of the spectrum of the feature matrix, and, grounded in that theory, puts forward an architecture-agnostic method. Hyperparameters and other details are reported with sufficient clarity to enable reproducibility. Concerns from reviewers regarding related work and the numerical evaluation were addressed through the rebuttal. For these reasons, I recommend acceptance.
Decision: Accept (poster)