Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation
Authors: Kimia Hamidieh
Haoran Zhang
Swami Sankaranarayanan
Marzyeh Ghassemi
Venue: ICLR 2024 spotlight (2024)
Keywords: Representation Learning
Spurious Correlations
Self-supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We study spurious features in self-supervised learning, find augmentations introduce undesired invariances, and propose late-layer regularization for better downstream performance.

Abstract: Supervised learning methods have been found to exhibit inductive biases favoring simpler features. When such features are spuriously correlated with the label, this can result in suboptimal performance on minority subgroups. Despite the growing popularity of methods which learn from unlabeled data, the extent to which these representations rely on spurious features for prediction is unclear. In this work, we explore the impact of spurious features on Self-Supervised Learning (SSL) for visual representation learning. We first empirically show that commonly used augmentations in SSL can cause undesired invariances in the image space, and illustrate this with a simple example. We further show that classical approaches in combating spurious correlations, such as dataset re-sampling during SSL, do not consistently lead to invariant representations. Motivated by these findings, we propose LateTVG to remove spurious information from these representations during pre-training, by regularizing later layers of the encoder via pruning. We find that our method produces representations which outperform the baselines on several benchmarks, without the need for group or label information during SSL.

Published as a conference paper at ICLR 2024
VIEWS CAN BE DECEIVING: IMPROVED SSL THROUGH FEATURE SPACE AUGMENTATION
Kimia Hamidieh' 'IIT 'Sony Al (hamidi eh, haoranz, swamiviv wwg.asyggaannnnnn
Haoran Zhang'
Swami Sankaranarayanan"
Marzyeh Ghassemi
ABSTRACT
Supervised learning methods have been found to exhibit inductive biases favoring simpler features. When such features are spuriously correlated with the label, this can result in suboptimal performance on minority subgroups. Despite the growing popularity of methods which learn from unlabeled data, the extent which these representations encode spurious features is unclear. In this work, we explore the impact of spurious features Self Superviuee Learning (SSL) for visual representation learning. We first empirically show that commonly used augmentations in SSL. can cause undesired invariances in the image space, and illustrate this with a simple example. We further show that classical approaches in combating spurious correlations, such as dataset re-sampling during SSL, do not consistently lead invariant representations. Motivated by these findings, we propose LATETVG to remove spurious information from these representations during pretraining, by regularizing later layers of the encoder via pruning. We find that our method produces representations which outperform the baselines on several benchmarks, without the need for group label information during SSL.
INTRODUCTION
Standard supervised machine learning models exhibit high overall performance but of- ten perform poorly on minority subgroups (Shah et al., 2020 McCoy et al. 2019; Gu- rurangan al., 2018). One potential cause is the presence
spurious correlations,
which are features that are only correlated with the label for specific subsets of data. For instance, machine learning model tasked with predicting bird species from images across different habitats may use the background the bird commonly appears in as a "shortcut", instead of core features specific the bird such as the shape of their beak or plumage. This results in poor performance on bird groups that appear in unexpected environments (Sagawa et al. 2020a). Identifying spurious correlations in the supervised learning setting has been well studied, where empirical risk minimiza- tion has been shown to exploit spurious correlations and result poor performance for minority subgroups (Hashimoto et al. 2018). J. As downstream tasks are explicitly defined, the label can be used to distinguish between core and spurious features (Liu et al.) 202 Zhang et al. 2022). Recent work has proposed various methods to identify and mitigate the effects of spurious features, such as learning multiple prediction heads (Lee et al., 2022b), causal inference (Creager et al., 202T), data augmentation (Gao et al.] (023) and targeted strategies such as importance weighting (Lahoti et al. 2020), re-sampling Idrissi et al. (202I): ); Tu et al. (2020), approaches based on group distributionally robust optimization (Sagawa et al. 2020at Duchi et al. 2019p.
More recently, self-supervised learning (SSL) has emerged as common form of pre- pre-training for
task-agnostic learning with large, unlabeled datasets (Chen et 2020a| He et al. 2019 Grill et al.
2020 Chen & He. 2020 Caron et al. 22200 Zbontar et al.) [02T]] Chen et al. 2020b). SSL methods learn representations from unlabeled datasets by solving an auxiliary pretext task (Doersch al.
2013), such as inducing invariance between the representations of two augmented views of the same image (He et al. 2019 Chen et al. 2020a). These methods have shown impressive results for wide range of downstream tasks and datasets (Luu et al. 20216: Jaiswal et al. 2020 Tamkin et al. 2021
Capturing core features rather than spurious features is essential for learning effective representa- tions that can be used in downstream tasks, but is particularly difficult in the case of SSL due the absence of labeled data during the pre-training process. Given only unlabeled data, we define spurious features those that strongly correlate with core features for most examples in the training set, but

---

Published as a conference paper at ICLR 2024
not useful for downstream tasks. For example, when training SSL model multi-object images, larger objects may interfere with the learning of smaller objects (Chen et al., 202T). If the downstream task involves only the prediction of smaller objects. the larger (spurious) object may suppress the smaller (core) object from being learned. Large-scale unlabeled datasets that are commonly used in machine learning are inevitably imbalanced (Van Horn et al.] [022]). D, have been found to be biased towards spuriously correlated sensitive attributes (Calude & Longo (207)) such as gender or race (Agarwal et al. 2021), and can also include label-irrelevat features [Torralba & Efros Fan et al. 2014).
2011
In this paper, we investigate the impact of spurious correlations on SSL. pre-training. We first show theoretically that image augmentations used in SSL pre-training can lead to spurious connectivity when learning representations, causing the model fail to predict the label using core features in downstream tasks. We empirically evaluate spurious connectivity, and then show that existing methods for utilizing group information in ERM based approaches do not provide an analogous improvement in SSL pre- training. We then propose Late-layer Transformation- -hased View Generation or LATETVG method that induces invariance spurious features in the representation space by regularizing final layers of the featurizer via pruning. Importantly, since our approach addresses SSL pre-training, we do not assume that model developers know apriori the identity or values of the spurious features that exist in the data. We first evaluate LATETVG on several popular benchmarks for spurious feature learning, and then connect our method to the theoretical analysis by showing that LATETVG models empirically exhibit lower spurious connectivity. Our method demonstrates improved discriminative ability, especially over minority subgroups, for downstream predictive tasks, without access to group or label information. We make the following contributions:
We provide theoretical arguments (Sec3.3) in SSL pre-training affect the model's ability to rely on spurious features, for downstream linear classifiers.
3.3) that illustrate how common augmentations used
We explore the extent of spurious learning in self-supervised representations through the lens of downstream worst-group performance. We empirically show that known techniques for avoiding spurious correlations, such as re-sampling of the training set given group information, do not consistently improve core feature representations (Sec
We propose LATETVG - an approach that corrects for the biases caused by augmentations, by modifying views of samples in the representation space (Sec5.1) We find that LATETVG effectively improves worst-group performance in downstream tasks on four datasets by enforcing core feature learning (Sec 5.2).
2 RELATED WORK
Spurious Correlations. Spurious correlations arise in supervised learning models Koh et al. (2021 I
Joshi et al. (202)); Singla & Feizi (202T) in a variety of domains, from medical imaging Zech et al.
2018: DeGrave et al. 2021) to natural language processing (Tu et al., 2020; Wang & Culottal 2020).
variety of approaches have been proposed to learn classifiers which do not make use of spurious information. Methods like GroupDRO (Sagawa et al.) 2020a) and DFR (Kinichenoo et al. 2022)
require group information during training, while methods like JTT [ et al.
202 la). LiF Nam
et al. 2020). CVaR DRO (Duchi et al. 2019). and CnC (Zhngg et al. 2022) do not. However, all methods require group information for model selection.
Self-supervised Representation Learning. Self-supervised learning methods learn representations from large-scale unlabeled datasets where annotations are scarce. In vision applications, the pretext task is typically to maximize similarity between two augmented views of the same image (ing & Tian 2020). This can be done in a contrastive fashion using the InfoNCE loss (Oord et al.. 2018).
such in Chen et al. (2020a) and Chen al. (2020b), or without the need for negative samples at
all, as in Grill et al. 2020); Caron et al. (020); Chen & He (200)): Caron et al. (02T); Oquab
al. 2023); Zbontar et al. (2021). Prior work has shown that SSL models may learn to spuriously
associate certain foreground items with certain backgrounds (Meehan et al. 2023). In this work, we
explore one potential mechanism for this phenomenon, both theoretically and empirically.
Representation Learning under Dataset Imbalance and Shortcuts. Self-supervised models have demonstrated increased robustness to dataset imbalance (Liu et al. [2021b] Tiang et al.) 2021b a). and the dominance of easier or larger features suppressing the learning of other features (Chen al. 202T). Some prior work has addressed shortcut learning in contrastive learning through adversarial feature modification without group labels (Robinson et al. 202T). However, other approaches to
2

---

Published as a conference paper at ICLR 2024
group robustness or fairness in self supervised learning require group information or labels Tsai et al. 2020 Song et al. 2019, Wang et al.] [021]] [Bordes al.] 2023, Scalbert et al., 2023). This paper focuses learning representations from an unlabeled dataset with spurious correlations, encompassing both dataset imbalance and features of varying difficulty.
Regularization in Self-supervised Learning. The concept of regularizing a specific subset of the network is relatively unexplored in self-supervised learning but finds motivation in recent find- ings from supervised settings, such addressing minority examples (Hooker et al. 2019), out-of- distribution generalization (Zang et al., 2021), late-layer regularizations through head weight-decay (Abnar et al. 202TD, and initialization (Zhou et al.] (022). Additionally, Lee et al. (2222) propose surgically fine-tuning specific layers of the network to handle distribution shifts in particular cat- egories. These studies provide support for the approach of targeting specific component of the network in self- saf--supeeeeed learning.
SPURIOUS CONNECTIVITY INDUCES DOWNSTREAM FAILURES
In this section, we introduce a toy setting to demonstrate that common augmentations used in SSL pre-training affect a model's ability rely on spurious features for downstream linear classifiers. We consider a binary classification problem with a binary spurious attribute, with an equal number of samples per group (Section (.2). We show that augmentations applied during SSL pre-training can introduce undesired invariances in the representation space learned by contrastive objective, making the downstream linear classifier trained representations more reliant the spurious feature (Section 3.3
3.1 BACKGROUND AND SETUP
Setup. BW consider learning representations from unlabeled data space X' generated from an underlying latent feature space Rmm := (Zcore, Espury.... , Zmm where Zeore and Equer are correlated features. For a given downstream task with labeled samples, we assume that each x E X belongs to a class given by the ground-trui -ruth labeling function y X - y where Zeore determines the labels for our downstream task of interest, while Expur determines the spurious attribute, which is easier to learn, and is not of interest for downstream tasks. We can define deterministic attribute function a X' S where each x E X' takes a value in S. Let g = (v(z), a(x)) denote the subgroup of a given sample I, where G = y x S is the set of all possible subgroups. Figure fillustrates the subgroups the Waterbirds dataset, where the background is a spurious feature that correlates with the bird species.
Contrastive learning. We aim to learn representations by bringing together data-augmented views of the same input, which we refer as positive pairs, using a contrastive objective. Let P+ be the distribution of positive pairs, which can be defined as the marginal probability of generating the augmented pair and z' from the same image in the (natural) population data. Thus the distribution P+ relies both on original data distribution and the choice of SSL augmentations. To analyze the representation space learned in contrastive learning and core feature predictivity of the representations, consider a weighted graph with vertex set X' where the undirected edge (x, x²) has weight Waat = P+ (x, x²) similar to augmentation graph in HaoChen et al. 2021
Although the augmentation graph learns semantically similar structures that enables generalization to new domains (Shen et al. 2022). the inductive biases set by these augmentations is not well studied. In this work, we draw attention to cases where augmentations can create sparious connectivities within subgroups of the data, and when and why these connectivities can cause the downstream linear model to rely on the spurious feature.
3.2 SPURIOUS CONNECTIVITY IN A TOY SETUP
In this section, we introduce a setting in which contrastive objectives can learn representations that cause linear downstream models fail on downstream tasks. To start, we investigate how augmentations can transform the samples such that the subgroup assignment changes.
Definition 3.1. Subgroup connectivity. Define the average subgroup connectivity given two disjoint subsets G1, G2 S X as w(),, G2) |.ा|. Ereg, War' where Wzs' is the probability of generating the augmented pair: and x' from the same image in the natural population data.
3

---

Published as a conference paper at ICLR 2024
Spurious feature
(beckgrond))
di Sportons connectivity
Unobuervable Latest Space
Huler
leadted water
>
B
y; Invariant feature
Uppcces)
Invariant connectiiity
landliin
volteberd
waarted water
(a)
(b)
Figure 1: Analysing SSL augmentations. (a) Images generated from a latent space with correlating features. (b) the connectivity induced by SSL augmentations between subgroups with the same spurious features is higher than the ones with the same invariant features, learned representations lead downstream linear model to separate the data based on the spurious feature (red dashed line) instead of the invariant feature (green dashed line). Our empirical evaluation in Table shows that this is indeed the case across different datasets considered in this work.
Intuitively, this subgroup connectivity is the average weight of edges connecting G1 to G2. and is proportional to the probability of sample augmentations. See Appendix C for further details.
G1 being transformed to sample x' E G2 via
We specifically define the following terms to be the expected value of w(G1, G2) from Definition 3.1 when subgroups 1 and G2 have the following properties:
Spurious connectivity (o): G1 and G2 share the same spurious attribute but differ in class
Invariant connectivity (8): G1 and G2 share the same class but differ in spurious attribute
Opposite connectivity (y): G1 and G2 differ both in the spurious attribute and the label
Where a, 3, y are average values estimated across dataset consisting of subgroups.
Toy Setup. We consider downstream classification problem where spurious attribute is present, and both the input and the spurious attribute take binary values. We define the probability of sampling positive pair (ar, x²) based the expected connectivity terms Cnay. Bay- Toys and Ptay as follows:
Oltoy
af + a(x') and y(x) y(x')
Buy
if = a(x') and y(x) # y(x') if + a(x') and y(x) + y(x')
(+ =
Tloy
Play
if = a(x') and y(x) v(x')
Note that the average subgroup connectivity for this setup, would be exactly the same as the corre- sponding connectivity variable. Thus in our running example we have Otay, B 3= Buy: "loy and we can use them interchangeably. For this simplified augmentation graph, the expected connec- tivity terms between groups are a property of the graph, and independent of the model or architecture we use for learning representations. Combined with contrastive objective, the expected connectivity can be a proxy for how close different subgroups are going to be the representation space.
3.3 ANALYSIS OF THE TOY SETTING
In Section empirically show that common augmentations used in contrastive learning can be detrimental to learning invariant representations, as they implicitly encourage samples to cluster primarily based on the spurious feature. Based on this observation, make the following assumption.
Assumption 3.2. Given spurious attribute function X' + |G] which is defined for all E X, we assume that for data point X, the probability of distorting the labeling of the augmented images sampled from the augmentation distribution A(- is greater than the probability of distorting the attribute. More formally,
(u(²) + a( = = 2~A(-22)
Pr
2~Al 2) (u(ø) = a(z), + a(x))
Pr
Lemma 3.3. Consider the set of (unlabeled) population data X in binary-class setting where the spurious attribute takes binary values, consisting of | groups, with the same number of
4

---

Published as a conference paper at ICLR 2024
examples per group. Consider simplified augmentation graph with parameters a, B. p. defined as 3.2 and assume that augmentations are more likely to chunge either class or altribute, than to change neither of the Iwo (ox > 7, 7)) and that augmentations are less likely to change both at the same time (p (p> 20,,,, a. 3).
Under these conditions, the spectral contrastive loss recovers both invariant and spurious fea- tures, and for each sample in the population data, the spurious feature bounded by constant Bep = -3- a + p. while the invariant feature is bounded by Ban = -- in the representation space. Proof in Appendix C.
Corollary 3.4. Given Assumption 3.2 where a B in the simplified augmentation graph, the margin of the spurious classifier is Bap- and is less than the margin of the invariant classifier Bine. and the max-margin classifier trained on representations given by spectral clustering converges to the spurious classifier.
This suggests that even with the same number of samples across different groups during pre-training, downstream linear classifiers can rely the spurious feature to make predictions, where the repre- sentations are determined by the simplified augmentation graph and the spectral contrastive loss.
4 EXPLORING SPURIOUS LEARNING IN REPRESENTATIONS
In this section, we investigate the performance downstream linear models trained on supervisd representations, empirically verify our assumption regarding spurious and invariant connectivity, and show that in practice similar to toy analysis having the same number of examples across groups in the presence spurious connectivity does not lead to performance gains.
4.1 EXPERIMENTAL SETUP
Datasets We evaluate methods on five commonly used benchmarks in spurious correlations CelebA (Lu et al. (013), CMNIST (Arjovsky et al., 2019), MetaShift (Liang & Zou 2022), Spurious CIFAR- 10 Nagarajan al. 2020). and Waterbirds (Wah et al. 2011 (See Appendix D.I for dataset descriptions). For each dataset, we train an encoder with an SSL-based pre-training step followed by supervised training of linear model that probes the representations learned using SSL for the downstream task.
SSL Pre-training For the SSL pre-training, we train SimSiam (Chen & He| (0200) models with ResNet backbone throughout the paper. The training split used during the pre- pre-training stage unbalanced and contain spuriously correlated data. The group/label counts for each dataset and split is shown in Appendix D.I. The backbone network used for most of our experiments are initialized with random weights, unless specified otherwise. We additionally report results for SimCLR (Chen) et al. 2020a) models in Section 5.2.1
Downstream Task For downstream task prediction, train linear layer using logistic regression on top of the pretrained embeddings. Note that the backbone is frozen during this finetuning phase and only the linear layer is updated. We use a balanced dataset for training where the spurious correlation does not hold. To create this downstream training dataset, we subsample majority groups (Sagawa let al. 2020b; Idrissi et al. 202T). to avoid the geometrical skews (Naarajan et al. 2020) of the representations. Then, evaluate the learned representations the standard
linear classifier test split of each dataset, where group information is given. For each run, we report the average and worst-group accuracy.
Empirical Evaluation of Spurious Connectivity To evaluate the connectivity term for each pair of subgroups in datasets exhibiting spurious correlations, we conduct an empirical analysis similar to Shen et al. (2022). Specifically, train classifier distinguish between each pair of subgroups and evaluate its performance on a subset of the data that has been augmented with SSL augmentations. The error of the classifier represents the probability that the augmentation module alters the subgroup assignment for each example between the two subgroups, making them indistinguishable. Figure illustrates this procedure.
The Role of Initialization In representation learning, encoders are not typically trained from scratch but initialized from a model pretrained on larger datasets, such as ImageNet (Deng et al. 2009). Recent work in transfer learning (Geirhos et al. 2018; Salman et al. 2022) has questioned this assumption and pointed out that biases pretrained models linger even after finetuning on downstream target tasks. In this section and more broadly in our work, we focus on performing SSL pre-training from randomly initialized weights. In addition, since the datasets considered in this work
5

---

Published as a conference paper at ICLR 2024
similar to ImageNet, the performance of off- sf-thhe-sheff ImageNet pretrained models is expected to be higher. For completeness, we have added these results to Appendix G.2.
4.2 HIGH LEVELS OF SPURIOUS CONNECTIVITY IN PRACTICE
We measure connectivity across four datasets in Table A and all of them, we find that the average spurious connectivity is higher than invariant connectivity. We also confirm that both these values are higher than the probability of simultaneously changing both spurious attributes and invariant attributes. This means that the samples within the training set are more likely to connected to each other through the spurious attribute, rather than the core feature. This suggests that the contrastive loss prefers alignment based on the spurious attribute instead of the class.
Table 1:
We report the error of classifiers trained to distinguish between two subgroups as a proxy for the probability of augmentations flipping group assignments between each two groups in the dataset, or the connectivity of two subgroups the image space.
Spurious Connectivity
Invariant Connectivity
Opposite Connectivity
Dataset
celebA
10.4
3.7
2.8
cmnist
31.6
8.3
metashift
16.3
13.6
waterbirds 25.3
11.2
We compute the connectivity terms by training classifiers to distinguish augmented data from each combination of the two groups in the dataset and reporting their error rates.
The details of the choice of augmentations and training for this step can be found in Appendix E.
4.3 SSL MODELS LEARN SPURIOUS FEATURES
To measure the reliance of downstream models spurious correlations. measure the accuracy of the downstream model on each group in the test set, and use the worst-performing group ac- curacy a lens to reason about spurious correlations. We find across all datasets, SSL. models exhibit gaps between worst-group and average accuracy when predicting the core feature (Table in Appendix D.3).
These results indicate, that unlike supervised learning (Menon et Rosenfeld et al. 2022). training of the final layer on a balanced set where the spurious corre lation does not hold is not sufficient for improving worst-group accuracy when predicting the core attribute.
202T Kirichenko al. 2022
4.4 RESAMPLING DURING SSL DOES NOT IMPROVE DOWNSTREAM PERFORMANCE
To probe the effect of availability of such group information during the SSL pre-training stage, we examine whether classical approaches for combating spurious correlations, such re-sampling training examples (Idrissi et al. 2021). are effective in removing spurious information during SSL pre-training.
Assuming that group information is available, we train SimSiam on datasets re-sampled using the following strategies: (i) Balancing groups by resampling training examples to match the downstream validation distribution. (ii) Downsampling examples majority groups to have the same number of examples in all groups. (iii) Upsampling minority examples to have the same number of examples in all groups
Table 2: Worst-group accuracy difference (%) between each balancing strategy and the original training set. Original training performance are shown in parentheses below each dataset. Full results can be found in Appendix Table IC
Sampling Strategy
calebA (77.5)
cmnist st (75.4)
metashi spurcifart
waterbi rds
(42.3)
(43.4)
(48.3)
Balancing
-1.7
-8.7
-3.8
-8.3
+3.0
Downsampling
+0.3
-10.6
+3.9
-14.4
+0.5
Upsampling
+4.1
-5.3
+2.7
-19.4
-0.3
We find that re-sampling during self-supervised pre-training does not improve downstream worst- group accuracy in consistent manner in Table
2
We do see minor improvements for metast shi ft

---

Published as a conference paper at ICLR 2024
readdrr /
pesdetor
Red
1-
foge
Sum Res)
cpoodzz
Figure 2: We use model transformation modules to create new views of training examples in the representation space. The introduced of transformations removes the features learned in the final few layers, and provides final representations invariant to such transformations.
and celebA, but contrast this with large drops for spurcifarl and cmnist. Given that the downstream linear model is trained downsampled dataset where such correlations do not exist, this means that re-sampling during self-supervised training does not necessarily improve the linear separability of representations with respect to the core feature, even given balanced finetuning dataset. This is analogous to our findings in the toy setting Section 3.3
CREATING ROBUST REPRESENTATIONS VIA FEATURE SPACE AUGMENTATIONS
In the previous sections, we showed that augmentation mechanisms used in SSL result in poor performance under spuriously correlated features in the training set. Instead of curating specific image augmentations that correct for these biases in the image space, we propose an approach to target spurious connectivity in the representation space by modifying positive pairs. In this section, describe approach, LATETVG that improves the performance of SSL models by introducing pruning based regularization to the later layers the encoder.
5.1 LATE-LAYER TRANSTORMATION-BASS VIEW GENERATION
Motivated by improved SSL model invariance when trained with augmentations in image space (Chen al., 2020a). we propose model transformation module that specifically targets augmentations that modify the spurious feature in representation space. We propose Late-layer T Transformation-uut
View Generation LATETVG, which feature space transformations mitigate spurious learning
in SSL models and improve learning of the core feature.
Formally, propose using model transformation module U, that transforms any given model fo parameterized by 0 (Wi.... Wr, } to fg- At each step, we draw a transformation ΦMM00 ll to obtain the transformed encoder. Each model transformation can be defined with a mask M E {0, 1| where transform the unmasked weights (1 M) @ by o, and keep the rest of the weights M @ the same to obtain 0. Here, we propose specific transformation module 4.
Transformations. For mitigating spurious connectivity. we choose simple transformation targeted towards regularizing the final layers of the encoder. In our experiments, we consider a threshold pruning transformation module, which uses magnitude pruning on a% of the weights in all layers deeper than L. More specifically, we propose a model transformation module 24 Prone, L. with 6(0) = 0, M == Mr,a (M) o Top. (W;) | E [m]} and Top..W:). (Wi)..) = T([ in top a% of 0). Note that this specific setting, the module transformation is deterministic (i.e. [M| = 1), though our formalization also allows for random transformations such as randomized pruning re-initialization.
To learn these representations, given two random augmentations t, T from the augmentation module T, two views x1 = ((x) and xe = ('(x) are generated from an input image x. At each step, given a feature encoder f. and an augmentation module 44, we obtain a transformed model f ((f) with Φ- Ut During training, examples 2 and 22 are respectively passed through the normal encoder U1 = fz1). and the transformed encoder i2 = f(z)). Encoded feature üg is now a positive example that should be close to v, in the representation space. An algorithmic representation of the method can be found in Appendix B
7

---

Published as a conference paper at ICLR 2024
Table 3: gorst-group accuracy (%) of SSL-Base and LATETVG for SimSiam and SimCLR pre- training. Results for average accuracy can be found in Table 8
SimSiam
SimCLR
SSL-BASE
SSL-LATE- TVG
SSL-BASE
ASL-LATE-TVG
celebA cmnist metashi ft
77.5
83.1
76.7
82.2
80.7
83.1
81.7
83.8
42.3
79.6
45.5
59.3
spurcifar10
43.4
61.4
36.5
40.4
waterbirds
48.3
56.3
43.8
55.4
Intuition for LVt.TVG. When learning discriminative process that maps data separable space. the variance among different subpopulations stored in distinct regions the network (Lee et al. 2022ap. As result, both spurious and core features, which describe the high-level data distribution, tend to reside the end of a neural network. Thus, in LATETVG, layers to learn more difficult features, by applying a model transformation that targets these layers, and causing the model to be invariant to final layer transformations. As pruning in supervised models have been shown to affect minority examples more than majority ones (Hooker et al. 2019). we hypothesize that our transformation can be considered minority groups. In particular, pruning would contribute "forgetting" the minority examples from the network, resulting in upweighting the loss for these examples.
to encourage the final
curated view generating operation for the
5.2 EXPERIMENTS
In this section, we demonstrate the efficacy of LATETVG in mitigating the dependence on spurious correlations. We use the same experimental setup as described in Section 4.1 For evaluation of LATETVG,
SSL-LATETVG approach during the pre-training stage. We compare this performance to SSL models pre-trained with the standard SSL-base trained with either SimSiam or SimCLR.
5.2.1 LATETVG IMPROVES SSL WORST-GROUP PERFORMANCE
The goal of this experiment to understand how LATETVG affects worst-group performance in down- stream tasks that use SSL representations. We com- pare the worst group accuracy of two approaches, SSL-Base and SSL-LATETVG Both models used similar hyper-parameter grids and model selection criteria noted previously. The results are presented in Table We show the perfor- mance of the best hyperparameter combination here,
SSL-base SSL-Late-TVG
different datasets.
Worst-greop accuracy
and have provided figures performance gains for all hyperparameters in Appendix D.2. It can be clearly observed that SSL-LATETVG outperforms the base model by large margins across most datasets and for both SimSiam and SimCLR. On cmn i st, our per- formance very close the baseline model and we do not significant improvement. We hypothesize that this is due to the fact that the base encoder on the easier cmnist dataset is already quite perfor- mant. On datasets where the base encoder performs poorly such as met wetashif Et and spurcil ci far O, our approach improves the performance by least 10% over base SimSiam. On a dataset of a larger scale like ce celebA, LATETVG still improves upon a strong encoder baseline.
Perventage of minority downstream train
Figure 3: Downstream worst- accu- racy of LSV- on the met ashift dataset as we vary the percentage of minority group in the downstream training cases except for extreme minority decrement, SSL-Late-TVV outperforms the bascline.
For all
Further, we find that LATETVG closes the gap in performance to supervised pretraining (Table B) We emphasize that this is an unfair comparison begin with, since supervised pretraining requires labeled data whereas SSL. does not, hence reducing the annotation budget drastically. Regardless, find that LATETVG narrows the gap between the SSL baseline and the ERM model significantly
8

---

Published as a conference paper at ICLR 2024
17% relative improvement for cmnist to 50% in the of spurci far10. In the case of ce celebA, we even outperform the ERM baseline.
5.2.2 SSL DOWNSTREAM LINEAR PERFORMANCE LESS RELIANT ON A BALANCED DOWNSTREAM DATASET
Traditional approaches that mitigate spurious correlations in ERM-based settings assume that the downstream training set is balanced (Kincceenko et al. 2022). However, this still requires knowledge of the spurious feature, which we may not always have practice. In this experiment, we challenge this assumption and analyze how SSL models behave when the downstream training set is imbalanced.
We vary the proportion of minority groups in the downstream training set, by first downsampling the training set to have the same number of samples across groups, and second randomly sampling minority groups with weight À (X-axis in Figure 3 and majority groups with weights 1 - X. We measure the worst group accuracy of the trained linear models for each dataset. We show the results on meeaahhie i ft in Figure 3 comparing the performance of SSL-Base and SSL-LATETVG: We can observe that LATETVG outperforms the baseline across a range of minority weights implying that LATETVG is more robust to imbalances in downstream training data. This is a crucial aspect where LATETVG differs from other approaches in the supervised pretraining literature, such as DFR (Kirichenko et al. 2022), which requires a balanced training set for the reweighting strategy to be successful. Similar results for other datasets and linear models provided in in Appendix F.5.
5.2.3 LATETVG REDUCES SPURIOUS CONNECTIVITY IN THE REPRESENTATION SPACE
Finally, we relate our method back to the theoretical analysis presented in Section 3 by computing the connectivity of the representation space learned by the SSL models, using the procedure outlined in Section E In Table 4 we find that LATETVG empirically reduces the spurious connectivity, while increasing the invariant connectivity, for all datasets. Thus, we have shown that LATETVG successfully augments the representation space to induce desired invariances.
Table 4: We report the еπгог of classifiers trained to distinguish between the representations of two subgroups proxy for connectivity terms. We find that LATETVG decreases spurious connectivity while increasing invariant connectivity in comparison to the baseline.
Spurious Connectivity
Invariant Connectivity
Opposite Connectivity
Dataset
Representation Space
celobA
SSL- BASE SSL-LATE-
18.9
15.7
8.3
15.8
17.9
SSL. BASE SSL-LATE-TVG
37.3
3.2
2.7
34.8
3.8
3.0
metashift
SSL-BASE
28.6
21.4
21.8
SSL-LATE LATE-TVC
27.3
27.3
21.3
waterbirds
SSL BASE
44.9
9.4
8.4
LATE
13.5
CONCLUSION
In this paper, we have investigated the impact of spurious correlations self supervieee learning (SSL) pre-training and proposed new approach, called LATETVG address the issue. Our exper- iments demonstrated that spurious correlations caused by data augmentation can lead to spurious connectivity and hinder the model's ability to learn core features, which ultimately impacts down- stream task performance. We have shown that traditional debiasing techniques, such as re-sampling, are not effective in mitigating the impact of spurious correlations in SSL pre-training. In contrast, LATETVG effectively improves the worst-group performance downstream tasks by inducing invariance to spurious features in the representation space throughout training. Our approach does not require access to group or label information during training and can be applied large-scale, imbalanced datasets with spurious correlations. We believe our work will help advance the field of SSL pre-training and encourage future research in developing methods that are robust spurious correlations.

---

Published as a conference paper at ICLR 2024
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint 1rXiv:21 2021.
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108. 02818, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wrXiv:190.022890 2019.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. Transactions on Machine Learning Research, 2023.
Cristian S Calude and Giuseppe Longo. The deluge of spurious correlations big data. Foundations of science, 22:595-611 612, 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprini arXiv:2006.0900 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen. Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 1834-11845, 2021.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. arXiv e-prints, art. arXiv:201 100566, November 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint arXiv:2003 2rX2v92003.04297, 2020b.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Alex DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610 619,,, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1432-1433, 2015.
John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2, 2019.
Jianqing Fan, Fang Han, and Han Liu. Challenges of big data analysis. National science review, 1(2): 293-3144 2014.
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Out-of-domain robustness via targeted augmentations. arXiv preprint arXiv:2302. /1811, 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /8/ 12231, 2018.
10

---

Published as a conference paper at ICLR 2024
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised Learning. arXiv e-prints, art. arXiv:2006.07733, June 2020.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy. Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. arXiv preprini 183:v:1883.02224, 2018.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self sule-vupee deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000- 1,55000-501,,20000 2021.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference Machine Leurning, pp. 1929-1938. PMLR, 2018
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. 1rXiv:1911.05722 2019.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv:1911.05220 2019.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-aceurae arXiv preprint arXiv:2110.145500 2021.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open- open-world sampling. Advances in Neural Information Processing Systems, 34, 2021a.
Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pp. 4927-4939, PMLR, 2021b.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions pattern analysis and machine intelligence, (3(11):40374400 4058, 2020.
Siddharth Joshi, Yu Yang. Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. Towards mitigating spurious correlations in the wild: A benchmark & more realistic dataset. arXiv preprint arXiv:2306. 2023.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.029 02937, 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. al. Wilds: benchmark of in-the-wild wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H. Chi. Fairness without demographics through adversarially reweighted learning, 2020.
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.1 1466, 2022a.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspeci- fied data. arXiv preprint arXiv:2202.03418, 2022b.
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. arXiv preprint arXiv:2202.0665 06523, 2022.

---

Published as a conference paper at ICLR 2024
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021a.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.0502) 2021b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv: 1902.01077 2019.
Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do ssi models have déjà vu? a case of unintended memorization in self-supervised learning, 2023.
Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparamettrissttoo and worst-case generalisation: friend foe? In International Conference Learning Representations, 2021.
URL /ttps::/ net.eeewwwwwewweeeeeeeee...eeeee wor.eewwwwtt/ttuumeemmm
forum? 2id= jphnnwwweeeeee
Mazda Moayeri, Sahil Singla, and Soheil Feizi Hard imagenet: Segmentations for objects with strong spurious cues. Advances in Neural Information Processing Systems, 35:10068-10077 2022.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur: Understanding the failure modes of -uitt--distribution generalization. arXiv preprint arXiv:2010.15775. 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673 23-26473-20684. 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /r8.vv/807.03744 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXhv:2304.07193, 2023.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? arXiv preprint arXiv:2 110.00 2021.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.068856, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020a.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterrization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020b.
Hadi Salman, Saachi Jain, Andrew Ilyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.028422 2022.
Marin Scalbert, Maria Vakalopoulou, and Florent Couzinié- -evy.. Improving domain-invariance in self-supervised learning via batch styles standardization. arXiv preprint arXiv:2303.06088 2023.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv: /61 0wt.:///174444 1.07400, 2016.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585. 2020.
12

---

Published as a conference paper at ICLR 2024
Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang- Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference Machine Leurning, pp. 19847-19878. PMLR, 2022.
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning?
arXiv preprint 0r.iv:210000430,, 2021.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2164-2173. PMLR, 2019.
Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Goodman. Dabs: A domain-agnostic benchmark for self-supervised learning. arXiv preprint arXiv:2111. 2021.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying self-supervised learning: An cneemation-thhortiic. framework. arXiv e-prints, pp. arXiv-2006, 2020.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-68 1613, 2020.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12884-12893, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd 2irds-200-2011 dataset. 2011.
Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Processing Systems,
2021.
Zhao Wang and Aron Culotta. Identifying spurious correlations for robust text classification. arXiv preprint arXiv:2010.02455, 2020.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzych Ghassemi. Change is hard: A closer look at subpopulation shift. In International Conference on Machine Learning, 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Muchine Learning, pp. 12310- 12320. PMLR, 2021.
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(1 1):e 1002688, 2018
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key -iito--distriuution generalization? In International Conference on Machine Learning, pp. 1356-11367. PMLR, 2021.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.015517, 2022.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville, Fortuitous forgetting connectionist networks. arXiv preprini arXiv:2202.0011 2022.
13


Reviews:
[{"review_id": "WguVyMQGLs", "reviewer": "Reviewer_GmKS", "summary": "The paper investigates the impact of spurious features on Self-Supervised Learning (SSL) in visual representation learning. It starts by demonstrating that commonly used augmentations in SSL can inadvertently introduce undesired invariances in the image space, highlighting this issue with a simple example. The authors also examine classical approaches to mitigating spurious correlations, such as dataset re-sampling during SSL, and find these methods inconsistent in leading to invariant representations.\n\nTo address these challenges, the paper proposes a novel method called \"LATE TVG,\" which aims to remove spurious information from representations during pretraining. This is achieved by regularizing later layers of the encoder via pruning. The authors present empirical evidence showing that LATE TVG produces representations that outperform baselines on several benchmarks. Notably, their method does not require group or label information during SSL, marking a significant advantage over traditional approaches.\n\nThe paper provides a thorough examination of the pitfalls of inductive biases in supervised learning, particularly when dealing with minority subgroups and spurious correlations. The proposed solution, LATE TVG, is a promising step towards more robust and fair SSL models, showing effectiveness in various benchmarks without relying on group or label information.", "questions": "Here are some questions and suggestions for the authors:\n\n**1. Clarification on Scalability and Efficiency:**\n- Question: Could you provide more details on the computational efficiency and scalability of the LATE TVG method, especially when applied to very large datasets or extremely deep networks?\n- Suggestion: It would be beneficial if the authors could include a section discussing the computational complexity of their method, possibly comparing it with other SSL methods in terms of training time and resource utilization.\n\n**2. Long-Term Effects of Pruning:**\n\n- Question: Pruning, especially in later layers, might have long-term effects on the learning capabilities of the network. Have you investigated how the pruning process affects the network's ability to learn new tasks or adapt to new data over time?\n- Suggestion: Providing insights or conducting experiments on the long-term effects of pruning could offer a more nuanced understanding of the method's robustness and adaptability.\n\n**3. Robustness to Adversarial Attacks:**\n\n- Question: How robust is the LATE TVG method to adversarial attacks, given that it focuses on mitigating spurious correlations?\n- Suggestion: Including experiments or discussions on the method's robustness to adversarial examples could highlight another dimension of its efficacy, particularly in security-sensitive applications.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Originality:**\nThe paper introduces a novel method, \"LATE TVG,\" to mitigate the influence of spurious correlations in Self-Supervised Learning (SSL) for visual representation learning. This approach is original in its utilization of later layer regularization via pruning to enhance the robustness of SSL models. Unlike conventional methods that often rely on re-sampling or require group or label information, LATE TVG innovatively ensures invariant representations without such dependencies. The idea of addressing undesired invariances introduced by common augmentations in SSL is a creative combination of existing concepts in a unique problem formulation.\n\n**2. Quality:** The empirical evidence presented in the paper is of high quality, showcasing the effectiveness of LATE TVG through comprehensive benchmarks. The authors provide a detailed analysis of the pitfalls of standard augmentations in SSL and demonstrate the superiority of their method over traditional approaches. The experiments are well-designed and executed, offering convincing support for the proposed solution. The quality of the research is further underscored by the rigorous testing on various benchmarks, which highlights the method's robustness and generalizability.\n\n**3. Clarity and Significance:** The paper is clearly written, with a well-structured format that guides the reader through the problem statement, methodology, and findings. The significance of the work is evident, as it addresses a critical challenge in SSL—ensuring fairness and robustness in visual representation learning. By providing a solution that does not require group or label information during SSL, the paper makes a useful contribution to the field, potentially leading to more equitable and effective machine learning models.", "weaknesses": "**1. Scalability and Efficiency:** The paper introduces the LATE TVG method, which involves regularizing later layers of the encoder via pruning. While this approach is novel, the scalability and computational efficiency of the method in large-scale settings are not thoroughly addressed. Pruning, especially in deeper layers, can be computationally intensive and may not scale well with very deep networks or extremely large datasets.\n\n**Actionable Insight:**\nFuture work could focus on optimizing the pruning process to enhance scalability and efficiency. This might involve investigating more computationally efficient pruning techniques or adapting the method to work with sparse neural networks.\n\n**2. Domain Generalization:** Although the paper demonstrates the effectiveness of LATE TVG across several benchmarks, it primarily focuses on visual representation learning. The generalizability of the method to other domains or types of data, such as text or audio, is not explored.\n\n**Actionable Insight:**\nTo strengthen the paper's contributions, additional experiments could be conducted to evaluate the method's performance on non-visual datasets. This would provide a clearer understanding of the method's applicability across different domains and data modalities.\n\n**3. Comparison with State-of-the-Art:** While the paper shows that LATE TVG outperforms traditional methods, it does not provide an extensive comparison with the latest state-of-the-art methods in SSL that address similar challenges. Without this context, it's difficult to gauge the relative progress made.\n\n**Actionable Insight:**\nThe authors could enhance the paper by including a more comprehensive comparison with state-of-the-art methods. This might involve reproducing results from recent papers or directly comparing the proposed method against other cutting-edge techniques in SSL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OerV6OzPr", "reviewer": "Reviewer_H1ZX", "summary": "The paper explores the impact of spurious correlations on Self Supervised Learning (SSL) for visual representation learning. The paper discusses how inductive biases in supervised learning favor simpler features, and when these features are spuriously correlated with the labels, it can result in suboptimal performance, especially for minority subgroups. The paper aims to investigate the extent to which SSL representations rely on spurious features for prediction.\n\nThe paper empirically demonstrates that common augmentations used in SSL can introduce undesired invariances in the image space, which may be problematic. It also finds that traditional approaches, such as dataset re-sampling during SSL, do not consistently lead to invariant representations. To address these findings, the paper proposes a new approach called LATETVG, which removes spurious information from SSL representations during pretraining by regularizing later layers of the encoder through pruning. LATETVG is shown to produce representations that outperform baselines on various benchmarks without requiring group or label information during SSL.", "questions": "Why is the analysis of Section 4 limited to simpler datasets and no results are provided for the datasets discussed above?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n\nTheoretical Insights: By analyzing simpler cases (Section 3.3), the paper provides theoretical arguments that offer a deeper understanding of how common augmentations used in Self Supervised Learning (SSL) pre-training affect the model's reliance on spurious features for downstream linear classifiers. \n\nExperimental evaluation of Spurious Feature Learning: The paper empirically explores the extent of spurious feature learning in self-supervised representations, focusing mainly on downstream worst-group performance. It demonstrates that traditional techniques for avoiding spurious correlations, such as re-sampling the training set with group information, do not consistently lead to improved core feature representations. This empirical analysis exposes the limitations of existing approaches and motivates the need for novel solutions.\n\nLATETVG: The paper introduces LATETVG, a novel approach designed to correct biases introduced by augmentations. LATETVG modifies the views of samples in the representation space, effectively improving worst-group performance in downstream tasks on four datasets. This contribution presents a practical solution to the problem of spurious correlations in SSL pre-training, resulting in better core feature learning.", "weaknesses": "The paper is missing references to some important works on spurious feature learning:\n\n1. Salient ImageNet: How to discover spurious features in deep learning?\n2. WILDS: A Benchmark of in-the-Wild Distribution Shifts\n\n\nMost of the results given in the paper are using smaller datasets. I believe the analysis of Section 4 could have been carried out a large number of publicly available SSL trained models. I would have liked to see some results using models trained on large datasets. \n\nTo evaluate the performance of models, the authors could have used more challenging datasets such as:\n1. FOCUS: FOCUS: Familiar Objects in Common and Uncommon Settings\n2. Hard ImageNet: Segmentations for objects with strong spurious cues\n3. WILDS: A Benchmark of in-the-Wild Distribution Shifts\n\nMost of the results seem to be on relatively simpler datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwRMBR78xQ", "reviewer": "Reviewer_BzfQ", "summary": "This paper studies the problem of spurious correlations in self-supervised learning, in particular, contrastive learning. The paper first formally shows, in a toy setting, that ensuring the groups are balanced is not sufficient to ensure a spurious correlation is not learnt. This is then confirmed empirically as well. The paper then presents the new method to remedy this: \"LATE-LAYER TRANSFORMATION-BASED VIEW GENERATION\". This involves conducting magnitude pruning for the later layers of the network to remove dependence on spurious. This method is able to improve worst-group accuracy on some popular datasets to study spurious correlations in supervised learning.", "questions": "1. The problem of spurious correlations in SSL is actually equivalent to **feature suppression** studied first here: https://proceedings.neurips.cc/paper/2021/hash/628f16b29939d1b060af49f66ae0f7f8-Abstract.html. It will be useful to discuss the equivalence of this problem as well as other relevant literature. \n\n2. In supervised learning, the models trained on Waterbirds etc. are initialized from ImageNet pretrained weights. Is that the case here as well? \n\n3. Experiments on some larger scale datasets provided in packages such as WILDS (https://wilds.stanford.edu/) or SpuCo (https://spuco.readthedocs.io/en/latest/) can strengthen the empirical success of the method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The problem of spurious correlations has not been studied for SSL before and is of importance, considering the lack of group / label information in SSL can make this hard to remedy. \n\n2. The conclusion that balancing the data may not be effective for remedying spurious correlations is extremely interesting. \n\n3. The method proposed is effective in remedying worst-group accuracy despite other standard methods like group balancing failing.", "weaknesses": "1. The intuition / reasoning behind the choice to prune the later layers to forget the spurious feature is not well-explained / discussed sufficiently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68xFjTOG46", "reviewer": "Reviewer_32bS", "summary": "In this paper, the authors explore the self-supervised learning (SSL) in visual representation learning. They emphasize the impact of spurious features on model efficacy. Spurious features are those that are only correlated with the label for specific subsets of data, potentially leading to suboptimal results, especially for minority subgroups. This study delves into the extent to which SSL representations depend on such spurious features and introduces an innovative approach known as LATETVG to alleviate their impact during the pretraining phase.", "questions": "* Which training parameters were utilized in the LaterTVG method?\n* How do the authors identify spurious features? The paper does not provide clear definitions distinguishing between spurious attributes and core features.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The introduced method effectively reduces spurious correlations and enhances the performance of downstream tasks across multiple datasets.\n\n* The author presents a clear depiction of the prevalent issues with SSL arising from spurious features.", "weaknesses": "* The paper's structure and content are not easy to follow. Specifically, the use of symbols like alpha, beta, and gamma appears to be overloaded, making it challenging to distinguish between the connectivity and the P matrix.\n* It's unclear how the correlation in Table 1 is computed. It seems spurious attributes and core features are not defined for real data. \n* The theoretical foundation leans heavily on Spectral Contrastive Learning, yet it is not used in the experimental section. It begs the question of its relevance or applicability. And whether the empirical results are well justified by theory.\n* The presentation would benefit significantly from the inclusion of an algorithmic representation of the proposed Late TVG method, enhancing its comprehensibility for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)