Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling
Authors: Weicheng Zhu
Sheng Liu
Carlos Fernandez-Granda
Narges Razavian
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. The data representations can capture many underlying attributes of data, and are useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. disease findings are usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of conspicuous features relevant to those sensitive attributes, despite the presence of other important predictive features for the downstream tasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (e.g. elder patients without diseases). Motivated by these findings, we propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating the enhanced robustness of pretrained representations on downstream classification tasks.

Published in Transactions on Machine Learning Research (01/2025)
Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling
Weicheng Zhu
jackahu@myn.edi edu
New York University
Sheng Liu Stanford University
shengl@stanford.com edu
Carlos Fernandez-Granda
Syymanda@cims.nyy...ll
New York University
Narges Razavian
Narges, Hazaviaan @nyerynggnnnegn.goor
NYU Grossman School of Medicine
Reviewed on OpenReview: https: // openreview. net/ forum? id- 8mg
Abstract
Self-supervised learning (SSL) has emerged as a powerful technique for learning rich rep- resentations from unlabeled data. The data representations can capture many underlying attributes of data, and are useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. disease findings are usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of conspicuous features relevant to those sensitive attributes, despite the presence of other important predic- live features for the downst.ream lasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (c.g. elder patients without discascs). Motivated by these findings, propose learning-spent speed aware SSL (LA-SSL) approach, in which sample each training data with probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating the enhanced robustness of pretrained representations on downstream classification tasks.
1 Introduction
Self-supervised learning (SSL) which learns data representations without explicit supervision has become a popular approach in various vision tasks (Chen et al.] 2020; He et al. 2020 Caron et al. 2020 Grill et al., 2020 Zbontar et al. 2021 Caron et al., 2021 The learned representations are often used various downst.ream tasks through supervised finc-tuning with task-related labeled data. Despite the overall effectiveness of SSL methods for many downst.ream tasks, it crucial to make sure represenlations learned by SSL are not biased. Understanding and addressing the potential sources of bias is essential for ensuring the robustness and reliability of SSL methods in practice.
One source of bias stems from spurious correlations, where some attributes of the data are correlated to the target labels due to the imbalanced data distribution rather than causal relationships. When spurious correlation present, neural networks may pick up the correlation shortcut, and use features corresponding spurious attributes to achieve good overall performance on the task (Nam et al., 2220).. For instance, the prevalence of pneumonia in different hospitals being different. does not imply that patients in
1

---

Published in Transactions on Machine Learning Research (01/2025)
hospitals have higher risk of pneumonia, but neural networks may make predictions based on hospital- related features in the radiography (Zeh et al. 2018). Spurious correlations can therefore negatively impact
out-of-domain generalization and fairness in real-world applications, especially in healthcare (Liu et al., 2020:
Seyyed-Kalantari et al., 2021 Nauta et al. 2021 Zhu et al., 2022). In principle, one would expect SSL pretraining to be robust to spurious correlations, since lask labels are not directly involved in training and the SSL pre-trained representation could potentially learn diverse features from the data. However, here we show that SSL may still introduce biases representations when the data distribution depends strongly on spurious attributes.
Feature suppression is common phenomenon in SSL, where the model prioritizes learning casy-to-learn features as shortcuts, while neglecting features related to other attributes (Robinson et al. 2021 We observe that when some attributes are correlated with one or several other attributes, the feature suppression nature of SSL can result less discriminative representations for some attributes while being more discriminative for others. This bias in the Iearned representation can hinder the performance of downstream tasks that rely those attribules which the learned representations are less discriminative to. Therefore, understanding and mitigating the biases in SSL is crucial for developing accurate and fair SSL pretrained models.
In this work, we aim to address an understudied problem of SSL: how to improve the robustness of SSL to spurious correlations omong underbying allribules? Previous studies show that sampling conditioning on some predefined attributes in SSL can force the data representations to be less discriminative to them (Tsai jtt al. 2022; Ma et al., 2022). This can potentially mitigate the spurious correlation when leveraging spurious attributes as conditions. However, pretraining with SSL typically only utilizes unlabeled data (i.e. the images) and does not leverage any extra labels, therefore potential attributes may not be known in advance during training. As alternative, we introduce the learning speed as proxy for spurious attributes. We find that the samples whose values do not follow such correlation with the spurious attributes are often learned slower in SSL than the samples that align with the correlation. Based on these observations, we propose a lcarning-speed aware sampling method for SSL (LA-SSL). We evaluate the learning speed of the feature extractor on each example during training and sample each example with a probability that negatively related to its learning speed. This forces the SSL to learn more discriminative features from examples that do not follow the spurious correlation, which helps the model learn reprosentations containing rich and diverse features, improving its generalization on variety of downstream tasks. In summary, contributions are the following:
We investigate the training behavior of SSL methods datasets with spurious correlations and observe that there exists a difference in learning speed between examples aligned and conflicting with such correlations;
We propose novel SSL method, LA-SSL, where we dynamically sample training samples based on their learning speed. This reduces the impact of spurious correlations in the dataset by upsampling the examples that may conflict with spurious correlations.
We show that LA-SST improves the robustness of previous SSL frameworks in terms of the linear probing accuracy in downstream classification tasks on three datasets where spurious correlations among different attributes are present.
2
Related Work
Self-supervised learning Self-supervised learning (SSL) is widely used to learn the representations for large unlabeled image datasets. A popular method, contrastive learning, seeks to learn representations by contrasting positive and negative pairs of samples (Chen et al. 2020 He et al. 2020 Caron et al. 2020). Other non-contrastive methods leverage similarities, cross-correlation, variance, or covariance in the representation space (Grill et al. 2020 Zbontar et al. 2021 Caron et al.) 2021 Bardes et al. 2021), to learn representations such that two augmentations from the same image are close while avoiding mode collapse. Other works learn
the data representation by teaching the network to reconstruct the corrupted images (Bao et al. 2022 He
et al. 2021
2

---

Published in Transactions on Machine Learning Research (01/2025)
Uniform Sampling
LA-SSL Sampling
0.98
0.98
Similarity
0.96
0.96
Conflict
Conflict
0.94
Aligned
0.94
Aligned
500
1000
500
1000
Epochs
Epochs
Figure 1: The similarity between the representations of two augmented views on benchmark dataset (95% correlation-alignod corrupted CIFAR-10) during training. The thick curves represent the mcan of similaritics of correlalion-connieeingg and -aligned examples, while the light curves represent the similarity mean of each class. The comparison highlights the network's faster learning on aligned examples compared to confict examples. The reweighted sampling approach in LA-SSL narrows the gap between confiict and aligned examples by upsampling the examples that learn slower. (Appendix B.1 shows additional examples).
Spurious correlation Several supervised learning methods, such as GroupDRO (Sagawa al. 2019 LIF (Nam et al. 2020), JTT (Liu et al. 2021). SSA (Nam et al. 2022 and LC (Liu et al. 2023a),, address the issue of model bias caused by spurious relationships in the labeled data. However, these methods require either strong prior knowledge about the confounding altribules involved in the spurious correlation (e.g., GroupDRO), or rely on the assumption that there exists an unknown attribute correlated with the target label of a specific task. In this paper, we focus on learning robust representations from unlabeled datasets with self-supervised methods, which is distinct from the de- de-biasing algorithms in the supervised learning setting. While the DFR method demonstrates the ability to mitigate spurious correlation by only fine-tuning the last layer top of biased data representations (Kirichenko et al. 2022 Izmailov et al. 2022). the desire for robust representations in SSL remains crucial for generalization across wide range of downstreem tasks without relying on assumptions of spurions correlation.
De-biased sclf-supervised pretraining Sampling bias is a source of bias in self-supervised lcarning (Arora el al. 2019 Chuang et al. 2020; Assran et al.] 2022 This bias impedes the network's ability learn accurate representations of minority subgroups, which are common in the dataset impacted by spurious correlation. To mitigate this bias, rescarchers have proposed conditional sampling techniques, which improve fairness among subgroups (Ma et al. 2022 Tsai et al. 2022). However, these techniques rely on group information, which is typically unavailable in large unlabeled datasets, contradicting the goal of SSL. Similarly, the unsupervised domain generalization frameworks are constrained by the requirements of the domain information (Zhang et al.) 2022 Liu et al.) 2023b). Mcanwhile, various alternative SSL methods have been introduced to overcome this challenge and improve data representations by avoiding shortcuts. These methods include incorporating hard nogatives (Robinson et al., 2020), adversarially perturbing data representations (Robinson al.] 2021), and employing network pruning or masking techniques (Lin al.) 2022 Hamidieh et al. 2022) during training. However, these methods do not specifically address the data imba laacaaaace..ccc..oa caused by spurious correlation. While network pruning has been utilized for pretraining on a biased dataset (Hamidieh et al. 2022), it is only used in downstream tasks on group-balanced datasets. In contrast, we focus on analyzing and improving the robustness of SSL pretraining for the downstream tasks where spurious correlation persists.
3 Preliminary Settings: Spurious Correlation in Self Self-supervised Learning
A key assumption in the success of self-supervised learning (SSL) that minimizing its loss function enables the network learn representations that are discriminative to the underlying attributes of the data. However, SSL fails to achieve this when certain attributes in the training samples are strongly correlated.
3

---

Published in Transactions on Machine Learning Research (01/2025)
We assume that the training data D has underlying attributes Zi. Zp. where each attribute Z₁ can be discretized into K2 categories and follows distribution Pz. By combining the values (z), E 7; x Z,j, any two attributes Z,, 2, can form K; x Kj subgroups. When there is a strong correlation between Z, and Zy. the training samples will not be evenly distributed among the subgroups. We refer to the samples in subgroups with high joint probabilities Pz (24, as corvelalion-aligned samples, while those with low joint probabilities are referred to as correlation- coniioiiiiniiiiiggggg samples. For instance, in medical datascts, the occurrence of diseases is usually positively correlated with age. Hence, old sick and young healthy patients are often correlation-aligune samples, while young sick and old healthy patients are correlation-couflittnng samples.
The bias in SSL arises primarily from the correlation-aligned samples. Within these correlation-alignee examples, the SSL loss can be minimized by only learning representations discriminative to some easy-to-lean to-learn feature Z while suppressing features for Z, (Robinson et al. 2021). This bias is reinforced when optimizing the loss function via randomly sampled training data, as the correlation-alineed samples constitute the majority of the training set. Although correlation-confletiiii samples, which contain diverse combinations of attribute values for Z, and Z,, could potentially guide the model to be discriminative towards Zj, their scarcity prevents them from mitigating this bias effectively. In such situations, the learned representations may succeed in classifying labels relying on Z; in downstream tasks, but they may fail to classify labels relying on Zz accurately. For instance, in medical datasets, if Z represents patient age and Zj represents the presence of a specific disease, the model might effectively classify age groups but struggle to detect the disease in downstream tasks.
To better understand the gap in the discriminability of the learned representations across attributes, we conduct spectral analysis in Section 5.5 which quantifies the strength of dominating features learned by SSL. It indicates that a few large singular values can greatly weaken the informative signals of eigenvectors corresponding to smaller singular values (Chen et al. 2019), suppressing the discriminability of learned reprosentations for subdominant attributes like Zz.
4 Learning-speen Aware Sampling
In this section, we present a learning-speed aware sampling schema in self-supervised learning (LA-SSL) to improve the robustness of representations to spurious correlations. We first introduce conditional sampling, which facilitates the lcarning of a fair reprosentation based on known attributes in Scction 4.1 Then we leverage observations on learning-speed differences to stratify examples that either aligu or conllict with the spurious correlation in Section 4.2 Based on these two insights, LA-SSL tracks the learning speed of each conflicting subgroups. LA-SSI. incorporates conditional sampling based on learning speed, assigning each example probability that is inversely related to its learning speed (see Section 4.3)
example during training the proxy of correlation-aligned
Most recent SSL approaches (Chen ct al. 2020 He ct al. 2020 Grill ct al. 2020 Zbontar et al., 2021
Caron et al. 2021 adopt a two-branch structure, where two views of an example obtained through random augmentations are encoded, and the representations of them are encouraged be similar. In this section, we mainly present the application of LA-SSL on top of one of these frameworks, SimCLR (Chen et al. 2020).. Then we also discuss the applicability of our method to other two-branch SSL frameworks and generative frameworks with only one branch in Section 5.4
4.1 Conditional sampling in SSL
Conditional contrastive learning proves to be effective in learning fairer representations with respect to specific attributes by excluding information on these attributes in the representations (Ma et al., 2022 Tsai ctt al. 2022). This approach can therefore be employed datasets with spurious correlations to mitigate the influence of confounding attributes. The key idea to minimize the InfoNCE loss on data pairs sampled from all the training data that share the same value in attribute Z, denoted as D Z. Formally, it minimizes
4

---

Published in Transactions on Machine Learning Research (01/2025)
the loss function in Equation !+
sim (x .27 .2qu53
CC.SSI. = Ez
log sim (a (x 2nu5s) + Ei= stm (xqust 2z))
(
The similarity score sim (::-): R x RR + R is defined as sim (x,x²) = exp (f+(z) fx(x')/T) for any 2, x' E IR",, where fo = vb o f, in which f R + Rd is the feature extractor mapping the imput data to representation, s Rd + + is projection head with a ferd-forward network and le normalization, and т is temperature hyperparameter possible values of attribute Z; the second expectation is taken over samples x E R drawn uniformly from the subset of the training set with attribute = z. Minimizing the loss brings the representation two random augmentations _= nng? on an instance x closer, and pushes the representation of x away from representations of 1 other examples {a:li= in the training set.
The first expectation is taken over uniformly drawn from all
While conditional contrastive learning is able to learn fairer representation across various subgroups, it is not feasible in the typical SSL practice as it requires additional attributes. However, only have access to unlabeled image data for SSL pretraining. Therefore, we propose an alternative approach to identify the subgroups of attribute values in spurious correlation.
4.2 Learning speed difference in SSL
Previous studies show that different training samples are learned at different paces in the supervised lcarning [agawa et al.) 2019 Nam et al. 2020). When using empirical risk minimization ion (ERM) with cross-entropy loss to train a classifier, if the labels are biased due to a spurious relationship between some confounding attributes and the labels, the training loss of correlation-aligned samples decreases at a faster rate compared to pnarelaaion-confiittii
samples. Note that variables used labels here can be viewed as one attribute of data. in the self-supervised setting. This phenomenon implies that the network can quickly overfit to these biases, using them shortcuts. Since the features associated with these confounding attributes are easier to learn than the original Lask, the neural network tends to memorize the labels that align with the spurious relationship first.
We observe a similar phenomenon in SSL pretraining even in the absence of labeled supervision. We introduce the concept of learning speed for sample, defined by a function 8 R - R that maps the training example score. In SSL frameworks with two branches, the network is trained to enforce the representations of two randomly augmented versions of the same sample to be closer. Consequently, it is natural to evaluate the learning speed of training example by the similarity between the representations of its two augmentations, such that s(z) = sim 1, 2)
Figure shows the training dynamics of SimCLR and LA-SSL on a benchmark dataset of spurious correlation, corrupted CIFAR-10 (Hendrycks & Dietterich 2019), which simulates a spurious relationship by associating corruption type with the object labels. In this illustration, the training set partitioned into two groups: the correlation-aligned group, whose target labels are perfectly correlated with the corruption types, and the correlation-confiettt group, whose target labels are uncorrelated. The plot shows that the feature extractor in SimCLR learns faster the correlation-aligned examples than on the correlation-conffle Similar to supervised learning, SSL also tends to capture the casily learnable attributes first. Motivated by this observation, we propose a method to leverage the imbalanced learning speed to enhance the quality of representations learned through SSL.
examples.
4.3 Conditional Sampling based on Learning Speed
Since the learning speed varies with subgroups formed by the values of attributes involved in spurious correlation, we use them as the proxy those attributes. Instead of sampling conditioning on the values of attributes directly, we sample training data based on their lcarning speed. The goal is to encourage the model to learn faster for the correlation-conficcin samples in the spurious relationship. This is achieved by dynamically adjusting the probability of sampling each training data.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm Learning-speed Aware Self-supervised Learning (LA-SSL)
Require: Samples #1,-.- In
DD
Require: Scaling function h; smoothing constant yß
1: # Initialize sampling probability with equal weights
2: π t (MI,' ... "",,""",TTl"" ,T), where n; = Vi
3: for 1 = to T do
4:
# Compute the similarity between two views
(1) sim (i!uet ,
auga) Vi
6:
# Smooth the similarities by EMA
(1) (1- --mmss (t-1)
+ nss") Vi
8:
# Updale the probabililies after warmup epochs
if t> > Twarump then # Compute weights from the similarity scores m h(s) hi=1h(s?") h(s(")
10:
11:
12: end if
13:
# Optimize the LA-SSL loss via Eq.
1 (1-1) + argmin, fo (....sS.(5))))
14:
15: end for
Let 1I denote a categorical random variable on sample space {1,-.. ,nn nn and π E R" denote the probability of sampling each training example. During training, we sample the indices examples from and minimize the InfoNCE loss on training data with corresponding indices. The loss function of LA-SSL, is defined as:
sim(21) aug 1 1 ,K
2
CLA-SSL
E log ougg aug sim sim (2 2Kk
(R:))_--11
Σi= sim( (2)) .Kk).
In LA-SSL, aim to upsample the training data with lower learning speed, because the correlation-conffict samples are usually learned slower. Therefore, for each training sample 24, dynamically updale тı with a weight that is inversely related to the learning speed as described in Algorithm [ The algorithm is computationally and memory-efficient, requiring only the additional resources nceded to compute and store similarity scores for each training sample, without introducing extra forward or backward propagation steps.
To monitor the learning speed for each training example I,, we compute the similarity between the represen- fations of its two randomly augmented versions at each epoch. Since the similarity score can be affected by randomness in augmentation strengths, we use exponential moving average (EMA) on s(x,) across previous training epochs to compute stabilized learning speed s; for training example x; 's. Then we compute the weights inversely related to the learning speed. The inverse relationship is set by linear scaling function defined as:
(3) where 8" E TR is a threshold selected as the r-percentile among the EMAs of learning-speed 8,'s from all training data, and y E is a constant that increases the margin between examples with slow and fast learning speed. r and 7 are hyperparameters. We typically choose small values of r and y greater than differentiate the underrepresented samples spurious relationships with lower learning speeds. The choice of hyperparamcter is analyzed by sensitivity analysis in Appendix B.4 The probability is then computed by the weights of inversely scaled learning speed for each training example: update the weights after the warm-up up epochs to ensure that the network has already learned some features instead of being random. As demonstrated in the right panel of Figure [ this sampling scheme indeed results in a better-synchhee setter-synchronized learning speed between correlation-alineed and -confficting examples.
h(s:)) = - - =*)]
h(ss)// Ei= h(s;). We only
5 Experiments
We introduce three datasets used to evaluate the robustness of SSL to spurious correlation in Section 5.1 We
show that. LA-SSL consistently improves the performance of SimCLR the downstream tasks impacted by

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: The classification accuracy (%) evaluated on group balanced test sets of corrupted CIFAR-10 with varying correlation-aligned ratio (k%).. The strength of spurious correlation grows with LA-SSL outperforms the uniform sampling baseline all scenarios.
Correlalion-aligneed proportion (4%)
Method
95%
98%
99%
99.5%
SimCLR
44.08
36.60
29.74
22.99
LA-SSL
48.02
40.49
31.55
24.17
spurious correlations the three datasets in Section 5.2 and particularly, under multiple spurious correlations in Scction 5.3 In Soction 5.4 wc show that LA-SSL can be generalized to other two-views and generative- based SSL methods. In Section 5.5 we explain the robustness of LA-SSL pretrained representations from the perspective of spectral decomposition.
5.1 Datasets and metrics
We evaluate the proposed framework on three datasets that have inherent spurious correlations. Since SSL methods require large amount of data train the data representation, we experiment on datasets with large sample sizes instead of other popular benchmark datasets for spurious correlation problems with limited samples, like Waterbirds Welinder et al.) 2010). More details about the dataset and experiment settings are provided in Appendix A
Corrupted CIFAR-10 (Hendrycks Dietterich 2019 a synthetic dataset generated by corrupting 60,000 images in CIFAR-10 with different types of noises. To simulate the spurious correlation between label and noise type, k% images are corrupted with fixed noise type corresponding to their labels, while the other 1 -% of images in each class corrupted by random noise type. The higher the value of k: is, the stronger the correlation between noise types and labels. The downstream task evaluate is object classification. Note that we avoid using angmentation that is similar to corruption during SSL such as Gaussian blur.
CclebA [Luu et al. 2015) a dataset that contains 202,599 images of celebritics along with various attributes associated with the images. One altribute indicates whether the person has Blond Hair, which exhibits a strong spurious correlation with the Male attribute. Only 2% of males in the dataset have blond hair opposed to 24% in females. The downstream task we evaluate hair color classification.
MIMIC-CXR [Johnson et al. 2019 is dataset of 377,110 chest X-ray images labeled with the diagnosis and demographic information of patients. There is spurious correlation between the No Findings attribute and the Age attribute. The No Findings attribute is more common among younger patients in the dataset, while older patients are more likely to have some diseases. We stratify patients into two sub-cohorts by whether they are younger or older than 90. We also create a synthetic version of MIMIC-CXR where we intentionally simulate a strong gender/diseas, bias by downsampling the no-finding male patients.
Evaluation metrics We evaluate the performance of standard (randomly sampled) SSL and LA-SSL on the test set of these three datasets. The test set of corrupted CIFAR-10 is balanced among 10 classes, and the images in each class are randomly corrupted. Therefore, we report the accuracy of classification on the test set. The Lest sets of CelebA and MIMIC-XRR are still highly imbalanced, so we evaluate the precision and recall together with AUROC on the subgroup that performs worse. For the MIMIC-CXB task, we evaluate no-finding classification as the downstream task.
5.2 Results
In this section, we compare the performance LA-SSL with uniform random sampling based on a common SSI framework, SimCLR (Chen al. (2200) with Resnet-50. Both SSL framework and lincar classifier for downstream tasks are trained on the same training dataset. Table Table reports the held-out test set accuracy of models trained corrupted CIFAR-10 with varying levels of spurious correlations between the type of
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table Downstream task performance on the underperforming subgroups affected by the spurious correlation in CclebA and MIMIC-CXR. LA-SSL demonstrates improved precision and recall on those imbalanced subgroups in both datasets. The comprehensive performance of all the subgroups is shown in Appendix B.3
Dataset
Method
AUC
Precision
Recall
SimCLR
95.14
63.80
37.22
CclebA (Gener-Male)
LA-SSL
95.63
66.66
38.88
SimCLR
73.75
37.40
31.61
MIMIC-CXR (Age-Old)
LA-SST.
74.89
40.00
34.83
Table 3: Performance on the synthetic MIMIC-CXR dataset simulating spurious correlation with gender and age attributes. LA-SSL improves subgroup accuracy under both spurious correlations.
Attribute
Method
AUC
Precision
Recall
Gender-
SimCLR
76.39
55.95
56.13
Male
LA-SSL
77.56
59.53
57.55
SimCLR 69.28
32.70
61.47
Age-Old
LA-SSL
70.68
34.71
61.67
corruption and the target label. The spurious correlation is only present in the craining/validattin set and the test does have any spurious correlation. Both SSL methods exhibit lower performance as the proportion of vorrelation-aligneet samples increases, indicating their susceptibility to spurious correlation. However, LA-SSL demonstrates its capability to learn more robust reprosentation for downstream tasks by achieving relative imppovveeent of 8.93%, 10.63%, 6.07%, and 5.13% at each corruption level, respectively. Additionally, Appendix B.5 reports comparisons between LA-SSL and an robust SSL method to avoid shorteut learning by adversarially perturbing data representations [Robinson et al., 2021), demonstrating LA-SSL still outperforms the robust SSL in the presence of spurions correlation. Notably, LA-SSL pretraining, coupled with simple linear probing on frozen representations, achieves even better performance than existing supervised de-biasing algorithms for spurious correlations, such as JTT (Liu al. is 24.73%, 26.90%, 33.40% and 42.20%, respectively. This shows the robustness of LA-SSL to spurious correlations.
2021) whose accuracy on corrupted CIFAR-10
Table 2 reports the performance on two real-world datasets CelebA and MIMIC-CXR, which are both affected by spurious correlations. The male subgroup in ColebA and the old subgroup in MIMIC-CXR have inferior performance compared to the general population. LA-SSL consistently improves precision and recall these underrepresented subgroups. This further confirms its effectiveness addressing spurious correlations.
5.3 Performance under multiple spurious correlations
Since there are many underlying attributes in the dataset, it common that multiple spurious correlations can oceur simultaneously between many attributes to a target label. A good SSL pretraining method should be robust to all these spurious correlations instead of focusing the robustness of LA-SSL in the presence of multiple spurious correlations. We downsample the no-finding malc patients in the training set of MIMIC-CXR by 90% to create an additional spurious correlation between gender and the symplom findings. We conduct SSL pretraining and linear probing on the synthetic training set, and evaluate the model on the original test set. Table reports the test performance on inferior subgroups in both spurious attributes, male and old. LA-SSL able improve the accuracy in both groups. This demonstrates its capability improve the robustness when multiple spurious correlations coexist in the dataset.
only one of them. In this section, we evaluate
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Accuracy on the corrupted CIFAR-10 dataset of various SSL frameworks trained with and without using learning-spor specd aware sampling (LA-SSL).
Correhation-aligneed ratio (k%)
Method
LA-SSL
95%
98%
99%
99.5%
47.80
37.66
30.43
22.11
Barlow
51.45
40.98
34.29
25.08
X
21.17
17.72
15.52
13.49
SimSiam
24.01
19.38
16.54
14.73
52.79
42.40
36.09
30.33
DINO
54.73
44.93
38.78
32.85
8x10
SinCLL LA-SimCLR
60
Sampling sveitein m
MAE
TA-MAE
40
6 6000
SimCLR
20
Malu
Male
x
Female
Female
20
40
60
80
100
Precision
Recall
Epochs
Figure Left: Comparison of the performances on CelebA between the uniform sampling and LA-SSL versions of SimCLR and MAE. LA-SSL improves the accuracy of the male subgroup in both SSL frameworks. Right: The median of LA-SSL sampling probabilities for males and females with blond hair in CelebA, based on different SSL frameworks. In both francworks, LA-SSL assigns higher probabilities to training examples in the minority group (male blonds).
5.4 Generalization to other SSL frameworks
In previous sections, we build the speed-aware sampling based on the similarity between two augmentations of the same image. This approach is inspired by a contrastive SSL method SimCLR. this section, we show that the proposed method can be generalized to various SST. frameworks.
Two views-based SSL Despite variations in loss functions, most SSL methods with two encoder branches. such SimCLR, BarlowTwins (Zoonaar al. 2021 SimSiam (Chen & He 2020), and DINO (Caron et al. 2021), share common objective to learn augmentationninnaaiatt features maximizing the similarity between representations from two distinct views. Figure 5 in Appendix B.] shows that The learning speed differences persist on other SSL frameworks. Hence, our proposed method can be seamlessly integrated with these SST. frameworks by monitoring similarity scores and implementing learning speed-aware sampling following Algorithm
[ during training. To assess the efficacy of the proposed approach, we conducted comparisons with alternative SSL methods corrupted CIFAR-10 datasets, both with and without resampling. Table 4 shows that the DINO distillation method exhibits greater robustness to spurious correlations in the dataset, while the contrastive learning method without negative samples, Simsiam, is significantly affected. Despite the differences, incorporating learning speed-aware sampling consistently improves the performance under various SST. frameworks, demonstrating its general applicability.
Generative-based SSL The LA-SSL framework can also be adapted to generative-based SSL, such as the Masked autoencoder (MAE) HHe et al. 2021), which uses vision transformer (ViT) (Dosovitskiy al.) 2021 the feature extractor and trains the network to reconstruct the randomly masked out patches in VTT.

---

Published in Transactions on Machine Learning Research (01/2025)
Corrupted Ci (bias level)
Corrupled Cifar-10
Normalized Singular Values
4x107
4<
2
3x
10
10
Preportion
Methods
99.5%
SimC1 R
95%
LA-SSL
1000
1500
2000
500
1000
1500
2000
Dimension
Dimension
Figure 3: Spectral analysis of SSL-pretrained data representations Corrupted CIFAR-10 using normalized singular values. The left. plot demonstrates faster decay of singular values when the representation ion is trained on a dataset with stronger spurious correlation (99.5% compared to 95%). The right plot compares SimCLR with LA-SSL, showing slower decay of singular values in LA-SSL. See Appendix B.2 for the results on CelebA and MIMIC-CXR.
To adapt the LA-SSL framework to MAE, we monitor the learning speed based on the similarity between the representations of two randomly masked versions of the examples during training. We applied the LA-SSL, equipped MAE on the CelebA dataset. The right panel of Figure P demonstrates that the learning speed difference persists within the MAE framework. Uniform sampling hampers training correlational samples, which represent the minority subgroups. This observation supports the hypothesis that the learning speed difference remains relevant factor in MAE, which can be addressed using LA-SSL. The left panel of Figure 2 shows that applying LA-SSL to MAE improves the precision and recall of the hair color classification lask among males.
5.5 Spectral analysis on representations
To understand why LA-SSL pretrained representation can generalize better in the presence spurious correlation, we conduct spectral analysis on the data representations of the training set. The distribution of singular values of pretrained representation is particularly important due to the gradient starving phenomenon
when training the downstream classification tasks (Tachet al. 2020: Pezeshki et. al.) 2021). This phenomenon reveals that minimizing cross-entropy loss captures only subset of attributes relevant to the task, and the attributes captured depend on the distribution of singular valuos, especially when the target is correlated with certain altributes.
Suppose we train a linear classifier to classify target labels y E {0, 1]" as a downstream task of SSL, we define the data representations from the feature extractor by E Raxd where o; = f(;) and a lincar classifier with paraneters 0 e Rd Then we perform linear probing on top of this learned representation for the downstream task. Specifically, we minimize the binary cross-entropy loss LBCE between target labels and model's prediction, defined as:
CACF = Σ y log(d)) - (1 11--::)log log(1 9i)
(4)
where 0 a (BT))) is the model prediction. To optimize the loss with gradient descent, we update the linear coefficients 0 with the gradient of the LECE with respect to 0. Denote the singular value decomposition (SVD) of as + = USVT, the gradient becomes:
HLBCE TT 10 ()-v) VSU (i (i-v)
ae
S is a diagonal matrix, which contains the singular values that indicate the importance singular vectors u; 's. This suggests that the gradient of LBCE is dominated by the directions of singular vectors corresponding to
10

---

Published in Transactions on Machine Learning Research (01/2025)
higher singular values. When there is spurious correlation, the linear classifier quickly picks up the spurious attributes while unable to learn enough about task-related target labels, suggesting that those easy-to-learn attributes lie in the subspace of representations that correspond to large singular values, while the target labels are associated with subspaces corresponding to smaller singular values (Chen et al. 2019 Park et al.) 2022 When training the linear classifier, the gradient becomes biased lowards the subspace of spurious attributes. To reduce such bias in the gradient, smaller gap between singular values on subspaces of representation corresponding to spurious and target attributes desirable. We show that LA-SSL indeed results in flatter distribution of singular values.
Figure B depicts the normalized singular values of data reprosentations pretrained by SSL under various conditions. The left plot compares the singular values of data representations trained on datasets with different levels of spurious correlations. It illustrates that the top few normalized singular values of representations trained with a more biased dataset (99.5% correlation as opposed to 95%) are similar or even greater than those of the less biased dataset. However, the remaining majority of singular values decay significantly faster in biased representations, which strongly suppresses the feature space and weakens the discrininability of other non-dominating attributes. This explains why datasets with more bias make it easier for the classifier to overfit the spurious attributes. The right plot in Figure X and plots in Appendix B.2 compare vanilla Sim SimCLR with LA-SSL. The slower rate of decay in singular values supports that LA-SSL pretrained representation is more robust in classifying attributes that are impacted by the spurious correlation and other non-dominating attributes.
6 Conclusion
In this work, we addressed the challenge of making self-supervised learning (SSL.) robust to spurious correlation. We observed that current SSL methods can be limited in scenarios with the imbalanced distribution of correlated underlying attributes, as uniform sampling may lead the model to overfit to corrclation-alignee examples. This can suppress features related to attribules involved in spurious correlation. To overcome this limitation, proposed novel SSL framework that dynamically adjusts the sampling rates during training based the learning speed of each example. Our method demonstrated improved performance downstream tasks across three datasets affected by spurious correlations, showing consistent improvements across different SSL frameworks.
Limitations While our work highlights LA-SSL effectively mitigates spurious correlations in SSL, there are also some broader challenges for future research. LA-SSL relies on the assumption that dominant attributes are easier to learn than subdominant attributes, which does not always hold. For instance, in scenarios where two correlated attributes are both difficult to learn, their correlation may still influence the learned representations. For example, two diseases might be highly correlated but without causality. In such cases, LA-SSL may struggle to fully address the biases. This is actually a broader challenge in tackling spurious correlations for both SSL and supervised learning setting. Additionally, the lack of diverse datasets hinders better benchmarking and addressing this potential issue. In real-world datasets, the key altributes that contribute to spurious correlations may not be explicitly labeled or easily, making difficult identify broader spurious correlation types. These limitations highlight important directions for future work.
Acknowledgement
This study was supported by the National Institute On Aging of the National Institutes of Health under Award R01 AG079175 and R01AG055617, and NSF grant NRT-1992658.
References
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, 2019.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Mahmoud Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Represenlations, 2022. URL http:://oppareviieennt//oooo
p p-BhZS25904
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianc-iivarianc--vvrinnn regularization for self-supervised learning. ArXio, abs/21 0as/2105.0483 04906, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised
learning of visual features by contrasting cluster assignments. arXiv preprint XrXix:2066. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprinl arXiv:214414229, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05777 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability:
Batch spectral penalization for adversarial domain adaptation. Tn Kamalika Chaudhuri and Ruslan
Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97
of Proceedings of Machine Learning Research, pp. 1081-1090. PMLR, 09-15 Jun 2019. URL https:
//proceedings.o wwtteeeddinn.m........mmmmmmmm ml.. S.lr.press/vv97 chen19i.html ppress//97/ceen11...tmm
Ching Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. ArXiv, Abs/2007.00224 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 21284, 2020.
Kimia Hamidich, Haoran Zhang, and Marzych Ghassemi. Evaluating and improving robustness of self- supervised representations to spurious correlations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022. URL https://openrevierr http://eeeeeee..eeeooooooococooo
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation ion learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision ond Pottern Recognition, pp. 9729-977979999 9738,973,,408 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp.
2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Leurning Representations, 2019.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learning in the presence of spurious correlations, 2022.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. Minic-cxr-jpg, a large publicly available database of labeled chest radiographs, 2019.
P. Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. ArXiv, abs/ 018/2204.002933 2022.
Xinyu Lin, Yi Tian Xu, Wenjie Wang, Yang Zhang, and Fuli Feng. Mitigating spurious correlations for self-supervised recommendation. Machine Intelligence Research, 20:263 275, 2022.
Evan Zheran Liu, Behzad Haghgoo, Annic S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. CoRR, abs/2107.00009 2021. URL https://arxx...rr/aaa/////////
Sheng Liu, Chhavi Yadav, Carlos FernandozGrranda, and Narges Razavian. On the design of convolutional neural networks for automatic detection of alzheimer's disease. In Machine Learning for Health Workshop, pp. 184 234-201. PMIRR, 2020.
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateck Singhal, and Carlos Fernandez-Cranda. Avoiding spurious correlations via logit correction. In ICLR, 2023a.
Yuchen Liu, Yaoming Wang, Yabo Chen, Wenrui Dai, Chenglin Li, Junni Zou, and Hongkai Xiong. Promoting semantic connectivity: Dual nearest neighbors contrastive learning for unsupervised domain generalization. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3510 3519, June 2023b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Martin Q. Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Conditional rastivvvvve learning for improvvng fairness in self-supervised learning, 2022. URL htttp:///azz: wwweziv.org/bbbo. argg/bbs/2106 02866
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Loe, and Jinwoo Shin. Learning from failure: Do-biasing classifier from biased classifier. In Neural Information Processing Systems, 2020.
Jun Hyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurions attribute estimation. ArXiv, abs/2204.02070,, 2022.
Meike Nauta, Ricky Walsh, Adam Dubowski, and Christin Seifert. Uncovering and correcting shortcut learning in machine learning models for skin cancer diagnosis. Diagnostics, 12:40, 12 2021. doi: 10.3390/ diagnostics 120000000 20100400.
Geon Yeong Park, Chanyong Jung, Jong- Chul Ye, and Sang Wan Lee. Self-supervised debiasing using low rank regularization. ArXiv, abs/ 2bs/2210.05248, 2022.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In M. Ranzato, A. Beygelzimer Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 1256 1272. Curran Associates, Inc., 2021. URL httpttttt///prrreeding neurips pdf
Joshna Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanic Jegelka. Contrastive learning with hard negative samples. ArXiv, abs/2010.04592, 2020.
Joshua Robinson, Li Sun, Ke Yu, K. Batmanghelich, Stefanic Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? Advances in neural information processing systems, 34:4974-4988 4986, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. CoRR, abs/1911.08731, 2019. URL
08731
Laleh Seyyed-Kalantari, Haoran Zhang, Matthew B. A. McDermott, Trene Y. Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms appliod to chest radiographs in under-sorved patient populations. Nature Medicine, 27:2176 2182, 2021.
Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics of deep neural networks, 2020. URL https://arxit: httpa///..rr//a/////.0...../
Yao-Hung Hubert Tsai, Tianqi Li, Martin Q. Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kernel. ArXiv, abs 2202.222220 2022.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS--1R-0100-00,, California Institute of Technology, 2010.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.080 03230,, 2021.
John Zech, Marcus Badgeley, Manway Liu, Anthony Costa, Joseph Titano, and Eric Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross- sectional study. PLOS Medicine, 15xe1002683, 11 2018. doi:
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain generalization. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4910 4920, June 2022.
Weicheng Zhu, Carlos Pernandez-Granda, and Narges Razavian. Interpretable prediction of lung squamous cell carcinoma recurrence with self-supervised learning. In International Conference on Medicol Imaging with Deep Learning, 2022.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Supplementary Material
In Appendix A we include additional descriptions of the datasets (Appendix A.1 and implementation details (Appendix A.2
In Appendix we include the additional plots for learning speed analysis (Appendix B.1 and spectral analysis (Appendix B.2 In Appendix B.3 we report the model performances of all subgroups CelebA and MIMIC-CXE In Appendix B.4 we analyze the sensitivity of hyper- parammeersstters in the scaling function h. In Appendix [B.5 compare LA-SSL with another robust SSL baseline which is developed without considering the data imbalance in spurious correlation.
A
Experiments Settings
A.1 Datasets
Corrupted CIFAR-10 Hendrycks & Dietterich, 2019 is a synthetic dataset generated by corrupting 60,000 images in CIFAR-10 with different types of noises, where there is spurious correlation between the label and the noise type in the training and validation set, but not in the test set.
To simulate the spurious correlation in the training and validation set, a certain percentage, denoted as k%, images are corrupted with a specific noise type that matches their label, while the other k% of images in each class are randomly corrupted with different noise types. The higher the value of k is, the stronger the correlation between noise types and labels. The following are the numbers of images corrupted with label-related noise types versus random noise types for different values of k: 44,832 vs 228 for h = 99.5, 44,527 vs 442 for k = 99, 44,145 vs 887 for k = 98, and 42,820 vs 2,242 for k = 95. All the images in the test set are randomly corrupted with different noise types.
In this study, we adopt the simulation scttings from previous paper Nam et al., 2020). We include the following corruption types for Corrupted CIFAR-10 dataset: Brighlness, Contrusl, Gaussian Noise, Frost, Elastic Transform, Goussian Blur, Defocus Blur, Impulse Noise, Saturate, and Pirelate. Each corruption has 5 strength level settings in the original paper (Hendrycks & Dictterich 2019) among which we apply the strongest. level. In the simulation, these types of corruptions are highly correlated with the original classes of CIFAR-10, which are Plane, Car, Bird, Cut, Deer, Dog, Frog, Horse, Ship, and Truck.
CelebA Liu et al. 2015) is a dataset that contains 202,599 images of celebrities along with varions attributes associated with the images. One attribute indicates whether the person has Blond Hair, which cxhibits a strong spurious corrolation with the Male attribute. In the datasct, 38.64% the images are males, and 61.36% are females. However, only 2% of males in the dataset has blond hair as opposed to 24% in females. The hair color classification is the downstream task we evaluate. We use the original training, validation and test set split in the dataset, which results in 162,770, 19,867, and 19,962 samples, respectively.
MIMIC-CXR JJohnson et al. 2019 is a dataset of 377,110 chest X-ray images labeled with the diagnosis. For our analysis, we focused on images with demographic information and anteroposterior (AP) and posteroanterior (PA) positions, resulting in 228,905 X-ray images. We split these samples into the training, validation, and test set by patients at 8:1:1 ratio.
Our downstream task is to classify whether the patients exhibit any medical findings based on their chest X-ray images. The No Findings altribute and the Age altribute are correlated. Among younger patients, the No Findings attribute more prevalent, while older patients are more likely to have some form of disease or abnormality. We divided patients into two sub-cohorts based on whether they were younger or older than 90. Among the younger group, 32.43% of patients have no findings, while only 16.07% of older patients exhibit. findings.
15

---

Published in Transactions on Machine Learning Research (01/2025)
A.2 Implementation details
All experiments were conducted on NVIDIA RTX8000 GPUs and NVIDIA V100 GPUs. The settings of the model training on different datasets are listed below:
Corrupted CIFAR-10
We use SimCI. (Chen et al. 2020 train Resnet50 feature extractor for 2000 epochs, which we notice that the training loss converges. We set the batch size to 1024, the projection head size to 512 and the temperature to 0.5. We use SGD with learning rate of 0.5, weight decay of 10-4, 50 warmup epochs, and cosine annealing scheduler. We apply random data augmentation to each image, including:
Random crop with scale 1.0).
Random Norizontal/vertica flipping with 0.5 probability,
Random (p- 0.8) color jittering: brightness, contrast, and saturation factors are uniformly sampled from [0.8, 1.2], hue factor uniformly sampled from [-0.1,0.1],
Random grayscale (p = 0.2),
Random solarization (p 0.2).
For the LA-SSL method, we adopted the same settings as SimCLR. Additionally, we set the scale 7 = 10, threshold 0.01 and update every 20 epochs.
CelebA We trained SimCLR using the same hyperparameters Corrupted CIFAR-10. In the training process, use batch size of 512, warmup epochs at 10 and train the SSL model for 200 epochs. We apply random data augmentation to each image, including:
random crop with scale [0.2, 1.0),
random horizontal flipping with 0.5 probability,
random (p = 0.8) color jittering: brightness, contrast, and saturation factors uniformly sampled from [0.6, 1.4], hue factor is uniformly sampled from [-0.1,0.1],
random Gaussian blur (p 0.5),
random grayscale (p 0.2),
random solarization (p 0.2).
For the LA-SSL method, we adopted the same settings as SimCLR. Additionally, we set the scale ? = 10, threshold r = 0.1 and update π every epochs.
MIMIC-XR We trained SimCLR using the same hyperparameters Corrupted CIFAR-10. the training process, batch size of 512, warmup epochs 10 and train the SSL model for 100 epochs. We apply random data augmentation to each image, including:
random rotation with a degree uniformly sampled from [0,
random crop with scale [0.7, 1.0] to size at 224 x 224,
random horizontal flipping with 0.5 probability,
random (p 0.8) color jittering: brightness, contrast, and saturation factors are uniformly sampled from [0.6, 1.4), hue factor uniformly sampled from (.0.1,0.1),
random grayscale (p- 0.2),
16

---

Published in Transactions on Machine Learning Research (01/2025)
random Gaussian blur (p 0.5),
random solarization (p= 0.2).
For the LA-SSL method, we adopted the same settings as SimCLR. Additionally, we the scale ? 10, threshold 0.1 and update x every epochs.
B
Additional Results
B.1 Learning speed
In Figure 1 we show the training dynamics of SimCLR and LA-SSL on corrupted CIFAR-10 at varying levels of spurious correlation (k%) in supplementary to Figure [ The plot shows that the feature extractor in SimCLR consistently learns faster the correlation-aligned examples than the correlation-confiittnn examples under different levels of spurious correlation. The reweighted sampling approach in LA-SSL consistently narrows the gap between conflict and aligned examples by upsampling the examples that learn slower. Figure 5 shows that The learning speed differences persist on other SSL frameworks. Leveraging the reweighted sampling mitigates the gap on other SSL mothods as well.
B.2 Spectral analysis
we plot the normalized singular values of data representations pretrained by SSL all three datasets in supplement to Figure 3 They compare vanilla SimCLR with LA-SSI. and demonstrate that the rate of decay in singular values of LA-SSL is slower consistently on all three datasets.
In Figure S
B.3 Results subgroups
Table and report the performances of each subgroup in CelebA and MIMIC-CXR. LA-SSL is able to improve the model performance the group with inferior performance while maintaining the performance on the superior subgroups.
Table 5: Downstream task performance for all the subgroups regarding gender in the CelebA dataset on hair color classification.
Prevalance Female
AUC
Precision
Recall
Method
Male
Female
Male
Female
Male
Female
Male
SimCLR
20.24
2.33
97.91
95.14
81.86
63.80
88.66
37.22
LA-SSL
20.24
2.33
97.89
95.63
81.54
66.66
90.00
38.88
Table 6: Downstream task performance for all the subgroups regarding age in MIMIC-CXR dataset on No
Finding classification
Prevalance Young
AUC
Precision Young
Precision Young
Method
Old
Young
Old
Old
Old
SimCLR
32.43
16.07
82.25
73.75
62.33
37.40
72.25
31.61
LA-SSL
32.43
16.07
83.47
74.89
63.65
40.00
73.42
34.83
B.4 Sensitivity analysis on the scaling function
In Section we define scaling function enlarge the gap of learning speeds among different samples, which include two hyperparameters: scale y and threshold We conduct the analysis how sensitive
17

---

Published in Transactions on Machine Learning Research (01/2025)
Table 7: Comparison among SimCLR, LA-SSL, and contrastive learning with adversarial perturbation on corrupted CIFAR-10 (95%). LA-SSL is able obtain significantly more improvements compared adversarial perturbation technique.
Adversarial Perturbation (e)
Method
SimCLR
LA-SST
0.05
0.1
0.2
Accuracy
44.08
44.72
45.41
45.26
48.02
the model to h on CelebA. We experiment with various values of r while keeping 7 fixed at 10. Similarly, we investigate the effect of different y values when is set to 0.1. Figure 7 illustrates that the model's performance remains relatively consistent across different r values within a reasonable range. However, when is set to 5, the scale becomes too small and the performance deteriorates.
B.5 Comparison with other robust SSL baselines
Some other SSL methods have demonstrated the ability to mitigate the issue of learning shortcuts during training [Rbiinson et al.) 2021 Lin et al., 2022 Hamidich et al., 2022), which could potentially enhance robustness under spurious correlations. our experiments, we explored one such method that has a publicly available implementation. Robinson al. proposed a technique that incorporates adversarial perturbations into the contrastive loss overcome feature suppression (Robinson al. 2021 Table 7 shows the adversarial perturbation does improve the SimCLR baseline. However, compared to LA-SSL, the improvements achieved by this method are relatively limited. Similar findings have also been reported in (Park et al. 2022). This could be attributed to the fact that robust SSL methods designed for general purposes may not specifically address the attribute imbalance among different subgroups.
18

---

Published in Transactions on Machine Learning Research (01/2025)
= 99.5
Uniform Sampling
LA-SSL Sampling
Similarity
0.96
Conflict Aligned
Conflict
Aligned
0.94
0.94
500 Epochs
1000
500 Epochs
1000
f
99
Uniform Sampling
LA-SSL Sampling
0.98
0.98
Similarity
L.MA
Conflict Aligned
Conflict
Aligned
0.94
0.94
500 Epochs
1000
500 Epochs
1000
k
98
Uniform Sampling
LA-SSL Sampling
0.98
0.98
Similarity
0.96
Conflict
Conflict
Aligned
Aligned
500 Epochs
1000
500 Epochs
1000
Uniform Sampling
LA-SSL. LA-SSSLSSamlingg
0.98
0.98
Similarity
0.96
0.96
Conflict
Confliet
Aligned
Aligned
1000
1000
Epuchs
Epochs
Figure The similarity between the representations two augmented views on corrupted CIFAR-10 10 at varying level of spurious correlations during training. The thick curves represent the mean of similarities of correlation-confiictngg and -aligned examples, while the light curves represent the similarity mean of each class.
19

---

Published in Transactions on Machine Learning Research (01/2025)
BarlowTwine
Uniform Sampling
LA-SSL Sampling
0.9
Similarity
0.7
Conflicting Aligned
Conflicting
Aligned
0.5
200 Epochs
200 Epochs
400
Simsiam
Uniform Sampling
LA-SSL Sampling
1.0
1.0
Similarity
0.9
0.9
Conflicting Aligned
Conflicting
Aligned
0.8
0.8
40
120
200
40
120 Epochs
200
Epochs
DINO
Uniform Sampling
LA-SSL Sampling
1.00
1.00
0.98
0.98
Similarity
0.96
0.96
0.94
0.94
0.92
Conflicting
0.92
Conflicting
Aligned
Aligned
0.90
0.90
100
300
500
100
300
500
Epochs
Epochs
Figure 5: The similarity between the representations of two augmented views on corrupted CIFAR-10 under different SSL frameworks. The thick curves represent the mean of similarities of correlation-conficcinn and -aligned examples, while the light curves represent the similarity mean of each class.
20

---

Published in Transactions on Machine Learning Research (01/2025)
Corrupted Cifar-10 (bias level)
CelebA
Normalized Singular Values
10
14
3xt0
12>10
10
Proportion
Methods
99.5%
SimCLR
95%
10
LA-SSL
1500
2000
500
1000
2000
Dimension
Dimension
Corrupted Cifar-10
MIMIC-CXR
Normalized Singular Values
10
10"
9
$=13
10
Methods
Methods
SimCLR
SimCLR
LA-SSI.
LA-SSL
10
500
1000
1500
2000
500
1000
1500
2000
Dimension
Dimension
Figure 6: The left column shows the normalized singular values of representation pretrained on Corrputed CIFAR-10. The top-left plot shows when the representation trained on the dataset stronger spurious correlation (99.5% opposed 95%) between the singular values decays faster. The top-left plot compares SimCLR with LA-SSL. The singular values of LA-SSL decay slower. The right column indicates similar trends two real-world datasets. LA-SSL enables singular values to decay slower.
70
70
60
50
50
40
Precision
Precision
30
Recall
30
Recall
F1
F1
20
0.01
0.05
0.10
0.20
Y
Figure 7: The sensitivity analysis and y in function h on CelebA.
21


Reviews:
[{"review_id": "sLp2KiDHPt", "reviewer": "Reviewer_BNof", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors do a nice job at clearly motivating the problem\n- The related work section is well written and extensive.\n- There are many different experiments that all highlight how LA-SSL outperforms the other method SimCLR.\n- I really enjoy section 5.5 because it provides some intuition above why LA-SSL can generalize better. \n\nWeaknesses:\n- I am not sure I entirely understand the preliminary setting. There are underlying attributes Z_1,..., Z_p, but what does it mean they can be discretized into K_1, .., K_p categories. Is it saying that each Z_i can be discretized into K_i respective categories. Thus each attribute can be discretized but the number of categories can be different for each attribute? Is that correct?\n- Could you elaborate on what this sentence means, \"In such situations, the learned representations may succeed in classifying labels relying on Zi in downstream tasks, but they may fail to classify labels relying on Zj accurately.\n- I think the content of section 5.5 is very compelling, but the placement of it is a little odd. I think it would make more sense earlier in the work? \n- My main concern is that all of these results are empirical. Is there any theory that guarantees this idea of using learning speed is the right one? Are their scenarios where doing this actually results in worse results? If so, could the authors elaborate on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies self-supervised learning (SSL) when there are spurious correlations present. They demonstrate that SSL loss can often be minimized by capturing the subset of conspicuous features that are correlated with sensitive attributes and not identifying the important/discriminative features for downstream tasks. To combat this issue, the paper uses a learning-speed aware approach (LA-SSL) which samples data points with probability inversely proportional to its learning speed. They evaluate their method on 3 datasets and showcase the utility of their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Do the authors think that section 5.5 could be moved to earlier in the work? I find it odd that all the experimental results are presented and then the paper jumps back to some theory.", "broader_impact_concerns": "no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "safOhVKRyQ", "reviewer": "Reviewer_DRo7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I believe all the claims the authors make in their paper are correct and justified with the exception of the theoretical analysis, where there are very minor gaps. I also think that based on the subject of the paper, there will be an audience interested in mitigating spurious correlation for self-supervised methods.  Based on these two criteria I believe this paper should be accepted.\n\n**Strengths:**\n1. Well written, easy to understand and well presented paper.\n2. Simple method, with intuitive motivation. The method due to its simplicity is easy to implement and has negligible computational overhead.\n3. Strong results, where the proposed method consistently improves the metrics across a variety of datasets and using different SSL frameworks.\n\n**Weaknesses:**\n1. Memory overhead: The method requires storing the similarity score for each datapoint in the dataset (which is not terribly bad).\n2. In the spectral analysis, while it is true that singular directions will pick the easy to learn attributes, it is not clear to me that this corresponds uniquely to spurious correlations. In other words, the higher singular values will also pick important features for classification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method to address the presence of spurious correlations in representation learning for self-supervised learning objectives like SimCLR. To do so, the authors use a \"learning-speed aware\" method to upsample data points that are harder to learn, as measured by their similarity scores (when encoded after applying two random augmentations). The method is evaluated on three datasets—Corrupted CIFAR-10, CelebA, and MIMIC-CXR—showing improved robustness and performance in downstream tasks. The approach is also extended to various SSL frameworks beyond SimCLR, and spectral analysis is provided to explain the effectiveness of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors could be more explicit about the role of the singular directions in the spectral analysis, which includes picking up important features for classification (this one is critical ). Overall, I feel that this analysis is very superficial and provides little to no insight on why the method works well. A more precise analysis that might try to distinguish between the two cases (spurious correlations v.s. not) would significantly enhance the paper (this one is a suggestion).\n2. An ablation study on the hyperparameters of the method, specifically the impact of $\\gamma$ and $r$ in the classification accuracy could provide further insights about the robustness of the method (this one is a suggestion).", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vsi1RA1No3", "reviewer": "Reviewer_7D2K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ Novel and generalizable approach across SSL algorithms: The paper proposes a novel learning-speed aware sampling method (LA-SSL) to mitigate the impact of spurious correlations on learned representations in self-supervised learning (SSL). The approach is shown to generalize across multiple SSL frameworks like SimCLR/BarlowTwins/DiNO.\n+ Insightful analysis: The authors provide an insightful analysis of how SSL models can overfit spurious correlations by learning features related to easy-to-learn attributes while suppressing others. They also show that samples defying the spurious correlations are learned more slowly, which is leveraged in the LA-SSL algorithm.\n+ Spectral analysis: The paper includes a spectral analysis of the learned representations, providing a theoretical justification for why LA-SSL enables better discriminative power for attributes impacted by spurious correlations.\n\n\n**Potential Areas for Improvement**\n+ Comparison to more baselines: While the paper compares LA-SSL to standard SSL approaches, it would be informative to include comparisons to other techniques proposed for mitigating spurious correlations or improving SSL robustness to strengthen the core contributions of the paper.\n+ Computational cost: The paper could discuss the computational overhead introduced by LA-SSL compared to standard SSL training, as the dynamic sampling may impact training time. This is particularly important in the context of large-data training. \n+ Limitations and failure cases: Discussing any observed limitations or failure cases of LA-SSL would help provide a more complete picture of the technique's applicability and areas for future improvement.\n+ Scaling to more classes/datasets: The authors already include several benchmarks like CIFAR/Mimic. Including some analysis on a larger number of classes (like Imagenet-100) and more complex spurious relations would help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the problem of improving the robustness of self-supervised learning (SSL) algorithms and dealing with spurious correlations in the training data. In particular, core contributions from the paper include:\n1. **instance-level learning dynamics**: analysis showing that current SSL methods can overfit spurious correlations between underlying attributes in imbalanced datasets. The SSL loss can be minimized by only learning features related to some easy-to-learn attributes while suppressing 'hard to learn' and discriminative features. The authors also observe that samples whose attributes do not align with spurious correlations (\"conflicting\" samples) are learned slower by SSL models compared to \"aligned\" samples. This learning speed difference is leveraged as a proxy for the spurious attributes.\n\n2. **learning-aware SSL**: This is a proposal for a learning-speed-aware sampling method for SSL (LA-SSL). It dynamically adjusts the sampling probability for each example inversely to its learning speed. This upsamples conflicting examples to force learning more diverse features.\n\n3. **benchmarks**: demonstration that LA-SSL improves the robustness of learned representations on three image datasets with spurious correlations:\n   - Corrupted CIFAR-10 (synthetic spurious correlation)\n   - CelebA (spurious correlation between hair color and gender) \n   - MIMIC-CXR (spurious correlation between disease findings and age)\n\n4. Analysis showing LA-SSL is robust to multiple simultaneous spurious correlations and generalizes across SSL frameworks (SimCLR, BarlowTwins, SimSiam, DINO, MAE). Spectral analysis indicates that LA-SSL leads to a slower decay of singular values of the learned representation compared to standard SSL, enabling better discriminative power for attributes impacted by spurious correlation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Adjustments Critical for Acceptance**\n* Clarify the novelty and significance of the contribution in the context of existing work on mitigating spurious correlations and improving SSL robustness. While the paper demonstrates the effectiveness of LA-SSL, it would be essential to discuss how it advances the state-of-the-art and differs from related techniques.\n* Include experiments that use a more complex dataset (in terms of the number of classes/scale), like the ImageNet-100 dataset, or more complex spurious correlations (in terms of aligned vs conflicted subgroups) to demonstrate the scalability and effectiveness of LA-SSL. This is critical to show that the proposed method can handle the challenges of larger-scale datasets and more diverse spurious correlations.\n\n\n**Adjustments to Strengthen the Work**\n* Include comparisons to additional baselines, such as other techniques proposed for mitigating spurious correlations or improving SSL robustness. This would help better situate the contribution and demonstrate the advantages of LA-SSL over existing approaches.\n* Discuss the computational overhead introduced by LA-SSL compared to standard SSL training. While dynamic sampling is a key aspect of the approach, it would be informative to understand its impact on training time and resources.\n* Provide a more comprehensive discussion on the limitations and potential failure cases of LA-SSL. While the experiments demonstrate its effectiveness, understanding the scenarios where it may not perform well would help characterize its applicability and guide future work.", "broader_impact_concerns": "The paper proposes LA-SSL, a technical approach to mitigate the impact of spurious correlations on self-supervised learning (SSL) representations, aiming to improve their fairness and robustness across different subgroups in the data. No significant ethical concerns requiring a separate Broader Impact Statement were identified, but the paper could benefit from a brief discussion on the responsible use and limitations of LA-SSL to guide practitioners and researchers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were overall positive. They raised some concerns related to including more baselines, being clear about potential memory and compute implications of the technique, and providing more thorough consideration of the limitations of this approach and the evidence marshalled for it. The authors did a decent job of addressing the reviewers' concerns, and so, a decision of accept was reached. However, the authors need to provide a properly formatted, camera ready version, incoroporating all of the promised adjustments to the manuscript before final acceptance.
Decision: Accept (poster)