Can contrastive learning avoid shortcut solutions?
Authors: Joshua David Robinson
Li Sun
Ke Yu
kayhan Batmanghelich
Stefanie Jegelka
Suvrit Sra
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
feature suppression
shortcut solutions
self-supervised learning

Abstract: The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via “shortcuts", i.e., by inadvertently suppressing important predictive features.  We find that feature extraction is influenced by  the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. 

Can contrastive learning avoid shortcut solutions?
Joshua Robinson MIT CSAIL LIDS joshrobSmit.. edu
Li Sun University of Pittsburgh lis1 1es111@pttteeem edu
Ke Yu University Pittsburgh yu. wu.ketittt.. edu
Kayhan Batmanghelich University Pittsburgh kayhan@pitt. edu
Stefanie Jegelka MIT CSAIL jteffe@caail wttp:ccsii..ii..... mit.edu
Suvrit Sra MIT LIDS
suvrit@mit. edu
Abstract
The generalization of representations learned via contrastive learning depends crucially on what features the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation some features, the improvement comes the cost of suppressing previously well represented features. In response, we propose impliciffeature feature modification (IFM), method for altering positive and negative samples in order guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and result improves performance on vision and medical imaging tasks. The code is available at: https:// /github.. com/joshr17/IFM
Introduction
Representations trained with contrastive learning adept solving various vision tasks including classification, object detection, instance segmentation, and more [5, 15, 44J. In contrastive learning, encoders are trained discriminate pairs of positive (similar) inputs from selection of negative (dissimilar) pairs. This task is called instance discrimination: It is often framed using the InfoNCE loss [14, 33], whose minimization forces encoders to extract input features that are sufficient to discriminate similar and dissimilar pairs.
However, learning features that discriminative during training does not guarantee model will generalize. Many studies find inductive biases in supervised learning toward simple "shortcut" features and decision rules [16, 21, 32] which result in unpredictable model behavior under perturbations [22, 43] and failure outside the training distribution [2, 37]. Simplicity bias has various potential sources [11] including training methods [8. 29, 41] and architecture design [10, 17]. Bias towards shortcut decision rules also hampers transferability in contrastive learning [4], where it is in addition influenced by the instance discrimination task. These difficulties lead to ask: can the contrastive instance discrimination task itself be modified to avoid learning shortcut solutions?
We upproach this question by studying the relation between contrastive instance discrimination and feature learning. First, we theoretically explain why optimizing the InfoNCE loss alone does not guarantee avoidance of shorteut solutions that suppress (i.e., discard) certain input features [4, 11). Second, despite this negative result, we show that it is still possible to trade off representation of
Correspondence Joshua Robinson (joshrobemit.cm
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Ideal: different because
features
sunny shady dog vs
features
sunny
moving vs still
shady
dog
disrriminatvv fectures
cat
moving
Shortcut: different because sunny vs shady
still
An ideal encoder would discriminate between instances using multiple distinguishing features instead of finding simple shortcuts that suppress features. We show that InfoNCE-trainee encoders can suppress features (Sec. 2.2). However, making instance discrimination harder during training can trade off representation of different features (Sec. 2.3). To avoid the need for trade-offs we propose implicit feature modification (Sec. 3), which reduces suppression in general, and improves generalization (Sec. 4).
Figure 1:
one feature for another using simple methods for adjusting the difficulty of instance discrimination. However, these methods have an important drawback: improved learning of one feature often comes at the cost of harming another. That is, feature suppression is still prevalent. In response, propose implicit feature modification, a technique that encourages encoders to discriminate instances using multiple input features. Our method introduces computational overhead, reduces feature suppression (without trade-offs), and improves generalization on various downstream tasks.
Contributions. In summary, this paper makes the following main contributions:
1. It analyzes feature suppression in contrastive learning, and explains why feature suppression can occur when optimizing the InfoNCE loss.
2. It studies the relation between instance discrimination tasks and feature learning; concretely, adjustments to instance discrimination difficulty leads to different features being learned.
3. It proposes implicit feature modification, a simple and efficient method that reduces the tendency to use feature suppressing shortcut solutions and improves generalization.
1.1 Related work
Unsupervised representation learning enjoying a renaissance driven by steady advances effective frameworks [3, 5, 15, 18, 33, 44, 45, 51]. As well many effective contrastive methods, Siamese approaches that avoid representation collapse without explicitly use of negatives have also been proposed [6, 13, 51J. Pretext task design has been at the core of progress in self-supervised learning. Previously popular tasks include image colorization [54] and inpainting [35], and theoretical work shows are-trained encoders can provably generalize if a pretext task necessitates the learning of features that solve downstream tasks [27, 39]. In contrastive learning, augmentation strategies are a key design component [5, 48, 50], as negative mining techniques [9, 15, 25, 40]. While feature learning in contrastive learning has received less attention, recent work finds that low- and mid-level features more important for transfer learning [55], and feature suppression
occur [4] just as with supervised learning [10, 16]. Combining contrastive learning with an auto-encoder has also been considered [28], but was found to harm representation of some features in order avoid suppression of others. Our work is distinguished from prior work through our focus on how the design of the instance discrimination task itself affects which features learned.
2 Feature suppression in contrastive learning
Feature suppression refers to the phenomenon where, in the presence of multiple predictive input features, model uses only a subset of them and ignores the others. The selected subset often corresponds intuitively "simpler" features, e.g., color as opposed to shape. Such features lead to "shortcut" decision rules that might perform well on training data, but can harm generalization and lead to poor robustness to data shifts. Feature suppression has been identified a common problem in deep learning
[11], and both supervised and contrastive learning suffer from biases induced by the choice of optimizer and architecture. However, contrastive learning bears an additional potential source of bias: the choice of instance discrimination task. Which positive and negative pairs are presented critically affects which features are discriminative, and hence which features are learned. In this work study the relation between feature suppression and instance discrimination.

---

First, we explain why optimizing the InfoNCE loss is insufficient in general to avoid feature suppres- sion, and show how can lead counter-intuitive generalization (Sec. 2.2). Given this negative result, we then ask if it is at least possible to control which features a contrastive encoder learns? We find that this indeed the case, and that adjustments to the instance discrimination task lead to different features being learned (Sec. 2.3). However, the primary drawback of these adjustments is that improving one feature often comes at the cost of harming representation of another. That is, feature suppression is still prevalent. Addressing this drawback is the focus of Sec. 3.
2.1 Setup and definition of feature suppression
Formally, assume that the data has underlying feature spaces 2'...., with distribution pA on each 2 Each j E [n], corresponding to a latent space 2A, models distinct feature. We write the product as - jes Z3, and simply write 2 instead of 2(n) where [n] (1.... n2.. A set of features
(x [e(] E is generated by sampling each coordinate 2¹ E Z' independently, and we denote the measure on 2 induced by z by X. Further, let X(-1=9) denote the conditional measure on 2 for fixed z For S S [n]] we use 2 to denote the projection of onto 2 Finally, an injective map g : Z + X' produces observations I = g(z).
Our aim is train an encoder f X' - S to map input data to the surface of the unit sphere Sd-1 = fu E Rd ||ul|2 =1} in such a way that f extracts useful information. To formally define feature suppression, we need the pushforward (V) v(h (VV) of a measure v on space U for a measurable map h : 1 + V and measurable V C V, where h- h-((v) denotes the preimage.
Consider an encoder f X + gd-1 and features S S [n]] For each 2 2 let u(11=5) (f o 9) ())H)((HHH) xX-12=) be the pushforward measure on g by f fog
g of the conditional X(-1=9).
1. I suppresses S if for any pair 25, =8 2S, we have /u(1==)) m(-1=s).
2. f distinguishes S if for any pair of distinct 25, 5555 25, measures 11-12 mn- (=5) have disjoint support.
Feature suppression is thus captured in distributional manner, stating that S is suppressed if the encoder distributes inputs in a way that is invariant to the value 2.. Distinguishing features, meanwhile, asks that the encoder ff separates points with different features 2 into disjoint regions. to optimize the InfoNCE loss [33, 14],
We consider training an encoder f X + gd-1
e f(a f(xt)/r
L. (f) = E _, [x7]m= - log
(1)
ef(u)) f(x+)// + + ef())
where is known the temperature. Positive pairs x, x are generated by first sampling A, then independently sampling two random augmentations a, a* ~ ~A, a: X' + X from a distribution A, and setting x (=)) and x+ at (s(=)). We assume A samples the identity function a(x) with non-zero probability ("x is similar to itself"", and that there are no collisions: a(x) # a'(x') for
all a, a', and all x # x'. Each negative example 2; is generated as x. = (g(2)))) by independently sampling features 2; ~ X and augmentation d; A.
2.2 Why optimizing the InfoNCE loss can still lead to feature suppression
Do optimal solutions to the InfoNCE loss automatically avoid shortcut solutions? Unfortunately, as we show in this section, this is not the case in general; there exist both optimal solutions of the InfoNCE loss that do and solutions that do not suppress a given feature. Following previous work [40, 49, 56], we analyze the loss as the number of negatives goes to infinity,
C == lim { (fm()) - log m 2 LE (15(x)) - [1 + Ext log [ fatt) f(x [(TM] m-
We subtract log m to ensure the limit finite, and use xF to denote a random sample with the same distribution as x; Prop. 2.2 (proved in App. A) shows that, assuming the marginals pj are uniform, the InfoNCE loss is optimized both by encoders that suppress feature j, and by encoders that distinguish j.
Suppose that P, is uniform on Z S for all j E [n]. Then for any feature j [n]] there exists an encoder faupp that suppresses feature j and encoder faise that discriminates j but both attain min f: measurable L(f).
The condition that pj is uniformly distributed on 2 s is similar to conditions used previous work [56]. Prop. 2.2 shows that empirical observations of feature suppression [4] (see also Fig. 3)
3

---

are not simply due to failure to sufficiently optimize the loss, but that the possibility of feature
suppression is built into the loss. What does Prop. 2.2 imply for the generalization behavior of encoders? Besides explaining why feature suppression can occur, Prop. 2.2 also suggests another counter-intuitive possibility: lower InfoNCE loss may actually lead to worse performance on some
tasks.
To empirically study whether this possibility manifests in practice, we use two datasets
100
with known semantic features: (1) In the Trifeature data, [16] each image is 128 x 128
shape texture color loss
1.00
0.48
0.15
0.61
Pearson correlation
0.5
and has three features: color, shape, and texture, each taking possible 10 values. See
0.48
1.00
loog
1.00
0.98
0..89
correlatton
0.15
-0118
0.
STLZO MUsy
0.96
1.00
-0
81
100
Fig. 10, App. C for sample images. (2) In the STL--igiis data, samples combine MNIST digits and STL10 objects by placing copies of randomly selected MNIST digit on top of an STL10 image. See Fig. 11 App. C for sample images.
-0.61
-0.49
0.44
1.00
105s
color
texture
shape
MNIST
57L10
Trifeature
STL-digits
Figure 2: Linear readout error on different downstream tasks can be negatively correlated. Further, lower In- foNCE loss does not always yield not lower error: error rates on texture, shape and STL10 prediction are nega- tively correlated with InfoNCE loss.
We train encoders with ResNet-18 backbone using SimCLR [5]. To study correlations between the loss value and eπтоr on down- stream tasks, we train 33 encoders on Trifea- ture and encoders on STI--iigits with different hyperparameter settings (see App. C.2 for full details on training and hyperparameters). For Trifeature, we compute the Pearson correlation between In- foNCE loss and linear readout error when predicting {color, shape, texture). Likewise, for STL-digits we compute correlations between the InfoNCE loss and MNIST and STL10 prediction error.
Fig. 2 shows that performance on different downstream tasks is not always positively correlated. For Trifeature, color error is negatively correlated with shape and texture, while for STL--dggits there is a strong negative correlation between MNIST digit error and STL10 error. Importantly, lower InfoNCE loss is correlated with lower prediction error for color and MNIST-digit but with larger error for shape, texture and STI.10. Hence, lower InfoNCE loss can improve representation of some features (color, MNIST digit), but may actually hurt others. This conflict is likely due to the simpler color and MNIST digit features being used as shorteuts. Our observation is an important addition to the statement of Wang and Isola [49] that lower InfoNCE loss improves generalization: the situation is more subtle- whether lower InfoNCE helps generalization on a task depends on the use of shortcuts
2.3 Controlling feature learning via the difficulty of instance discrimination
The previous section showed that the InfoNCE objective has solutions that suppress features. Next, we ask what factors determine which features are suppressed? Is there way to target specific features and ensure they are encoded? One idea is to use harder positive and negative examples. Hard examples are precisely those that are not easily distinguishable using the currently extracted features. So, a focus on hard examples may change the scope of the captured features. To test this hypothesis, consider two methods for adjusting the difficulty of positive and negative samples:
1. Temperature in the InfoNCE loss (Eqn. 1). Smaller places higher importance on positive an negative pairs with high similarity [47].
2. Hard negative sampling method of Robinson et al. [40], which uses importance sampling to sample harder negatives. The method introduces hardness concentration parameter (3, with larger 33 corresponding to harder negatives (see [40] for full details).
Results reported in Fig. 3 (also Fig. 13 in App. C.2) show that varying instance discrimination difficulty-i.e. varying temperature T or hardness concentration 3-enables trade-offs between which features are represented. On Trifeature, easier instance discrimination (large 7, small (3) yields good performance on 'color' an an "easy" feature for which a randomly initialized encoder already has high linear readout accuracy whiule generalization on the harder texture and shape features is poor. The situation reverses for harder instance discrimination (small т, large 3). We hypothesize that the use of "easy" features with easy instance discrimination is analogous to simplicity biases in supervised deep networks [17, 21J. As with supervised learning [10, 17J, we observe a bias for texture over shape in convolutional networks, with texture prediction always outperforming shape.

---

target: color
target: shape
target: texture
top-3 accursyy
inta oater
92 temperature T
temperature
temperature
top-1 accuracy
2.0
02
0.
hardness concentration B
hardness concentration B
hardness concentration
harder instance discrimination
Figure 3: Trifeature dataset [16]. The difficulty of instance discrimination affects which features are learned (Sec. 2.3). When instance discrimination is easy (big т, small 8), encoders represent color well and other features badly. When instance discrimination is hard (small r, big 3), encoders represent more challenging shape and texture features well, at the expense of color.
That there are simple levers for controlling which features are learned already distinguishes contrastive learning from supervised learning, where attaining such control is less easy (though efforts have been made [23]). However, these results show that representation one feature must be sacrificed in exchange for learning another one better. To understand how to develop methods for improving feature representation without suppressing others, the next result (proof in App. A) examines more closely why there is relationship between (hard) instance discrimination tasks and feature learning.
[Informal] Suppose that Pj uniform on 2 g for all j E [n] Further, for S C [n] suppose that x, xx, { }i are conditioned on the event that they have the same features S. Then any f that minimizes the (limiting) InfoNCE loss suppresses features S.
The positive and negative instances in Prop. 2.3 must be distinguished with features in S Relating this point to the above observations, assume that an encoder exclusively uses features S. Any positives and negatives that do not (much) differ in features S are difficult for the encoder. By Prop. 2.3, focusing the training on these difficult examples pushes the encoder to instead use features in S i... to learn new features. But at the same time, the proposition also says that strong focus on such hard negative pairs leads to suppressing the originally used features S, explaining the results in Fig. 3. While the two techniques for adjusting instance difficulty we studied were unable to avoid feature suppression, this insight forms the motivation for implicit feature modification, which we introduce next.
3 Implicit feature modification for reducing feature suppression
The previous section found that simple adjustments to instance discrimination difficulty could signifi- cantly alter which features a model learns. Prop. 2.3 suggests that this ability to modify which features are learned stems from holding features constant across positive and negative samples. However, these methods were unable to avoid trade-offs in feature representation (Fig. 3) since features that are held constant are themselves suppressed (Prop. 2.3).
To avoid this effect, we develop a technique that adaptively modifies samples remove whichever features are used to discriminate a particular positive pair from negatives, then trains an encoder to discriminate instances using both the original features, and the features left over after modification. While a natural method for modifying features is to directly transform raw input data, it is very challenging to modify the semantics of an input in this way. So instead we propose modifying features by applying transformations to encoded samples e = -(x). Since we modify the encoded samples, instead of raw inputs v, we describe our method as implicit.
We set up our notation. Given batch x, x, {x, y we write v = f(x), ut = f(xt), ), and 2 = [(x, ) to denote the corresponding embeddings. As in Eqn. 1, the point-wise InfoNCE loss is,
c" Tut
(v,e* (1, km)) - -log G TTTt+ + Em ct
5

---

[Implicit feature modification] Given budget E E RT, and encoder f X' + S", an adversary removes features from that discriminates batch xt {æ Fi=l by maximizing the point-wise f(u,0* + {o, + + (2)))
InfoNCE loss, (u, ut, {1r } max 1 I..6, eB,,
Here Bg denotes the bz-ball of radius E. Implicit feature modification (IFM) removes components of the current representations that used to discriminate positive and negative pairs. In other words, the embeddings of positive and negative samples are modified to remove well represented features. So. if the encoder is currently using simple shorteut solution, IFM removes the features used, thereby encouraging the encoder to also discriminate instances using other features. By applying perturbations in the embedding space IFM modify high level semantic features (see Fig. 4), which is extremely challenging when applying perturbations in input space. In order to learn new features using the perturbed loss while still learning potentially complementary information using the original InfoNCE objective, we propose optimizing the the multi-task objective min y {C()) + aLe (f) where L. = B&g is the adversarial perturbed loss, and L the standard InfoNCE loss. For simplicity, all experiments set the balancing parameter = unless explicitly noted, and all take et,,, to be equal, and denote this single value by E. Crucially, le can be computed analytically and efficiently. For any v, vt, { 1 E Rd we have,
" 11/
cnTutt
U and volc=
e e0T + Em cuti, 1) v
e0 / + En
In particular, V x v and V - This expression shows that the adversary perturbs vj (resp. u+) in the direction of the anchor u (resp -v). Since the derivative directions are independent of ( and v+, we can analytically compute optimal perturbations in Bg. Indeed, following the constant ascent direction shows the optimal updates are simply o, < v,- +,, and v+ <- vt -.t The positive (resp. negative) perturbations increase (resp. decrease) cosine similarity to the anchor sim(r, oz + f,0) > as & + oo (resp. sim(u, x+ - evv + -1 as e+ + oc). In Fig. 4 we visualize the newly synthesized v7 2 and find meaningful interpolation of semantics. Plugging the update rules for et and u. into the point-wise InfoNCE loss yields.
cl0 -e-
( at, tor ) log el E e( of +e.//T
In other words, IFM amounts to simply perturbing the logits reduce the positive logit by e+ /T and increase negative logits by c//T. From this we see that &e is automatically symmetrized in the positive samples: perturbing instead of ut results in the exact same objective. Eqn. 2 shows that IFM we-whights each negative sample by a factor ge//T and positive samples by e-s"/r
3.1 Visualizing implicit feature modification
With implicit feature modification, newly synthesized data points do not directly correspond to any "true" input data point. However it is still possible to visualize the effects of implicit feature modification. To do this, assume access to a memory bank of input data M = {z, }.s A newly synthesized sample can be approximately visualized by retrieving the 1 1-nearest neighbour using cosine similarity arg min sim (x)) and viewing the image x as an approximation to s.
Fig. 4 shows results using a ResNet-50 encoder trained using MoCo-v2 on ImageNet1 K using the training set as the memory bank. For positive pair v, et increasing causes the semantics of a and ut to diverge. For ε 0.1 a different car with similar pose and color is generated, for = 0.2 the pose and color then changes, and finally for E = 1 the pose, color and type of vehicle changes. For negative pair v, o the reverse occurs. For 5 = 0.1, c- is vehicle with similar characteristics (number of windows, color etc.), and with E = 0.2, the pose of the vehicle a+ aligns with v. Finally for 1 the pose and color of the perturbed negative sample become aligned to the anchor v. In summary, implicit feature modification successfully modifies the feature content in positive and negative samples, thereby altering which features can be used to discriminate instances.
Related Work. Several works consider adversarial contrastive learning [19, 24, 26] using PGD (e.g. FGSM) attacks to alter samples in input space. Unlike our approach, PGD-based attacks require costly inner-loop optimization. Other work takes an adversarial viewpoint in input space for other self-supervised tasks e.g., rotations and jigsaws but uses an maage-to-image network to simulate FGSM/PGD attacks [31]. introducing comparable computation overheads. They note that low-level

---

Mandeed BacuPao
C85
anchor
positive p*
0.1
rpoch
maarary (c) amc. repodt
anchor V
negative V
0.2
larger perturbations
CIAART
CIFAR100
Figure 4: Visualizing implicit feature modification. Top row: pro- gressively moving positive sample away from anchor. Bottom row: progressively moving negative sample away from anchor. In both cases, semantics such color, orientation, and vehicle type are modified, showing the suitability of implicit feature modification for altering instance discrimination tasks.
Figure 5: Comparison be-
tween IFM and ACL(DS). Under standard linear eval- uation IFM performs best. ACL is suited to adversarial evaluation.
top-1 accuracy
100
100
top- 1 accuracy
mill calio
SinCLE
80
top-1
40
40
20
20
20
texture
shape
texture
color
color
temperature
temperature
temperature
Figure 6: Trifeature dataset. Implicit feature modification reduces feature suppression, enhancing
the representation of texture, shape and color features simultaneously. All results average linear readout accuracy over three seeds and use a fixed value = = 0.1 to illustrate robustness to E.
(i.e., pixel-level) shortcuts can be avoided using their method. All of these works differ from ours by applying attacks in input space, thereby focusing on lower-level features, whereas ours aims to modify high-level features. Fig. 5 compares IFM to this family of input-space adversarial methods by comparing to a top performing method ACL(DS) [24]. We find that ACL improves robust accuracy under loo attack on input space (see [24] for protocol details), whereas IFM improves standard accuracy (full details and discussion in Appdx. C.3). Synthesizing harder negatives in latent space using Mixup [53] has also been considered [25] but does not take an adversarial perspective. Other work, AdCo [20], also takes an adversarial viewpoint in latent space. There are several differences to our approach. AdCo perturbs all negatives using the same weighted combination of all the queries, whereas IFM perturbations are query specific. In other words, IFM makes instance discrimination harder point-wise, whereas AdCo perturbation makes the InfoNCE loss larger average (see Fig. 4 for visualizations of instance dependent perturbation using IFM), AdCo also treats negatives as learnable parameters, introducing 1M more parameters and ~ 7% computational overhead, while IFM has no computational overhead and is implemented with only two lines code (see Tab. I for empirical comparison). Finally, previous work makes the connection between suppression of semantic features and adversarial methods in contrastive learning (see Fig. 6).
4 Experimental results
Implicit feature modification (IFM) be used with any InfoNCE- contrastive framework. and we write FEM-SimCLR, IFM-MoCo-v2 etc. to denote IFM applied within a specific framework. Code for IFM will be released publicly, and is also available in the supplementary material.
4.1 Does implicit feature modification help avoid feature suppression?
We study the effect IFM has on feature suppression by training ResNet- 18 encoders for 200 epochs with E {0.05, 0.2, 0.5} on the Trifeature dataset [16]. Results are averaged over three seeds, with IFM using & = 0.1 for simplicity. Fig. 6 shows that IFM improves all three features for all temperature settings. The capability of IFM to enhance the representation of all features i.e. reduce reliance on shorteut solutions is an important contrast with tuning temperature using hard negatives, which Fig. 3 shows only trades-off which features are learned.
linear readout accuracy across
7

---

baseline
4, only, c- 0.1
e 0.05
E 0.1
E- 0.2
6 0.5
top-1 accuracy
92
56
70
92.0
54
91.5
91.0
68
90.5
67
SimCLR MoCo-v2
SimCLR MoCo-v2
SimCLR McCo-v2
90.0
SimCLR MoCo-v2
STL10
tinylmageNet
CIFAR100
CIFAR10
Figure 7: IFM improves linear readout performance all datasets for all E 0.1, compared to baselines. Protocol uses 400 epochs of training with ResNet-50 backbone.
4.2 Performance on downstream tasks
Sec. 3.1 and Sec. 4.1 demonstrate that implicit feature modification is adept at altering high-level features of input, and combats feature suppression. This section shows that these desirable traits translate into improved performance on object classification and medical imaging tasks.
Experimental setup for classification tasks. Having observed the positive effect IFM has on feature suppression, we next test if this feeds through to improved performance on real tasks of interest. We benchmark using both SimCLR and MoCo-v2 [5, 7] with standard data augmentation [S]. All encoders have ResNet-50 backbones and are trained for 400 epochs (with the exception of on ImageNet100, which is trained for 200 epochs). All encoders evaluated using the test accuracy of a linear classifier trained on the full training dataset (see Appdx. C.4 for full setup details).
Classification tasks.
MoCo-v2
AdCo [20]
IFM-MoCo-v2
Results given in Fig. 7
and Tab. find that every
E
N/A
N/A
0.05
0.1
0.2
value of 0 < E S 0.2 improves performance across all datasets using both MoCo-v2 and Sim- CLR frameworks. We find that optimizing Le (76.0% average score across all eight runs in Fig. 7) performs similarly to the standard contrastive loss (75.9% average score), and does worse than the IFM loss (C L22/2. This suggests that L and Lg learn complementary features. Tab. I benchmarks IFM on ImageNet100 [44] using MoCo-v2, observing improvements of 0.9%. We also compare results on ImageNet100 to AdCo [20], another adversarial method for contrastive learning. We adopt the official code and use the exact same training and finetuning hyperparameters as for MoCo-v2 and IFM. For the AdCo-specific hyperparamters negatives learning rate ('neg and negatives temperature Theg we use grid search over all combinations Iraeg E 2, 4} and Tney {0.02, 0.1). which includes the AdCo default ImageNet1 K recommendations Ir leng = 3 and Tneg = 0.02 [20]. The resulting AdCo performance of 78.9% is slightly below MoCo-v2. However using their respective ImageNet K default parameters AdCo and MoCo-v2 achieve 72.4% and 71.8% respectively, suggesting that the discrepancy between AdCo and MoCo-v2 may in part be due to the use of improved hyperparameters tuned on MoCo-v2. Note importantly. IFM is robust to the choice of e: all values (0.05, 0.1, 0.2) were found to boost performance across all datasets and all frameworks. We emphasize that the MoCo-v2 baseline performance of 80.5% on ImageNet100 is strong. Our hyperparameters, which we detail in Appdx. C.4.1, may be of interest to other works benchmarking MoCo-v2 on ImageNet100.
top-1
80.4
78.9
81.1
80.9
80.7
Table I: Linear readout (%) on ImageNet100, averaged over five seeds. IFM improves over MoCo-v2 for all settings of
Medical images. To evaluate our method on a modality differing significantly from abject-based images we consider the task of learning representations of medical images. We benchmark using the approach proposed by [42] which is a variant of MoCo-v2 that incorporates the anatomical context in the medical images. We evaluate our method on the COPDGene dataset [38], which is multi-center observational study focused on the genetic epidemiology of Chronic obstructive pulmonary disease (COPD). See Appdx. C.5 for full background details on the COPDGene dataset, the five COPD related outcomes we use for evaluation, and our implementation. We perform regression analysis for continuous outcomes in terms of coefficient of determination (R-square). and logistic regression to predict ordinal outcomes and report the classification accuracy and the I-off accuracy, i.e., the probability of the predicted category is within one class of true value.
Tab. reports results. For fair comparison we use same experimental configuration for the baseline approach [42] and our method. We find that IFM yields improvements on all outcome predictions.
8

---

Method
lugFEVipp
LogFEV, FVC
CLE
CLE 1-off
Para-septal Para-septal 1-off
mMRC
mMRC 1 1-of
Loss
R-Square
Accurucy (5)
C (haseline)
0.5666.003
0.661
49.6
81.820.5
55.7.0.3
84.4_002
50.4g6.5
72.5+0.3
Ce. 5 = 0.1
0.591 ous
0.68
49. 49.4
81.9203
55.6203
86.1
50.3g08
72.7
IFM. е = 0.1
0.615=.005
0.691: ooe
2
80.6
65.3-20.4
84.7=0.5
50.4.0.5
72.8+00.2
IFM. 5 0.2
0.595
0,683 ooe
48.5
80.5
55.320.a
86.1=0.1
49.826.s
72.0
IFM. 5 0.5
0.607 000
0.653 0os
49.6
82.0g0.3
51.920.2
84.7=002
50.6g0.4
73. 13.12
5
0.5833.003
0.675 006
50.0
82.9
56.3gn.
85.7=0.2
50.3ge.6
71.9
Table 2: Linear readout performance on COPDGene dataset. The values are the average of 5-fold
cross validation with standard deviations. The bold face indicates the best average performance. IFM yields improvements on all phenotype predictions.
Top-1 accuracy
07.4
SimCLR
53.3
42.845.2
65.566.5
89.6
56.950.1
77.1
85.6
73.7
73.1
70.1
16.1
45.4
17
1
17
20R D
10g
10uR
23:
17B
STL10
tinylmageNet
CIFAR100
CIFAR10
Figure 8: Label {D, DR, ]NR indicates which dataset was used to train the linear readout function. Improved performance of IFM on standard data D can be attributed to improved representation of robust features DR. See Sec. 4.3 for construction of robust (Dx) and non-robust (DNK) datasets.
The gain is largest on spirometry outcome prediction, particularly logEVVIpp with improvement of 8.7% with 0.1. We found that at least 0.5 and 1.0 improve performance on all tasks. However, we note that not all features yield a statistically significant improvement with IFM.
4.3 Further study on the impact of IFM on feature learning
This section further studies the effect implicit feature modification has on what type of features are extracted. Specifically, we consider the impact on learning of robust (higher-level) vs. non-robust features (pixel-level features). Our methodology, which is similar to that of llyas et al. [22] for deep supervised learning, involves carefully perturbing inputs to obtain non-robust features.
Constructing non-robust features. Given encoder f we finetune linear probe (classifier) h on- top of f using training data (we do not use data augmentation). Once h is trained, we consider each labeled example (x, y) from training data Durin E (tinylmageNet, STL10, CIFAR1O, CIFAR100) hallucinated target label / is sampled uniformly at random, and we perturb x = To until o f predicts using repeated FGSM attacks [12] 2g F esign( V (( f( 4)). At each step we check if arg max; o fxx) ) = (we use the maximum of logits for inference) and stop iterating and set aadv Fy for the first for which the prediction is t. This usually takes more than few FGSM steps with = 0.01. We form dataset of "robust" features by adding (Iadv, y) to DR. and dataset of "non-robust" features by adding (Iadv, t) D To a human the pair (Iadys will look mislabeled, but for the encoder Tadv contains features predictive of t. Finally, we re-finetune (i.e. re-train) linear classifier g using DR (resp. DNR).
Fig. 8 compares accuracy of the re- f-fnettmed models on a test set of standard Diest examples (no perturbations applied to the test set). Note that Dr. D depend on the original encoder When re- fe-finetuning f we always use datasets DR. D formed via FGSM attacks on itself. So there is set Dr. D for SimCLR, and another for IFM. Fig. shows that IFM achieves superior generalization (D) compared to SimCLR by better representing robust features (Dp). Representation of non-robust features (D RR) is similar for IFM (55.5% average across all datasets) and SimCLR (56.7% average). IFM is juxtaposed to the supervised adversarial training of Madry et al., which sacrifices standard supervised performance in exchange for not using non-robust features [30, 46].
Discussion
This work studies the relation between contrastive instance discrimination and feature learning. While we focus specifically on contrastive learning, would interest to also study feature learning for other empirically successful self-supervised methods [1. 6, 13, 51]. Understanding differences in feature learning biases between different methods may inform which methods are best suited for given task, well as point the way to further improved self-supervised techniques.

---

Acknowledgments SJ was supported by NSF BIGDATA award IIS- NSF Convergence Accelerator Track D 2040636. SS acknowledges support from NSF-TRIPODS+XXRES (1839258). JR was partially supported by a Two Sigma fellowship. KB acknowledges support from NIH (IR01HL111883-001, NSF (1839332 Tripod+X), and a research grant from SAP SE Commonwealth Universal Research Enhancement (CURE) program awards research grants from the Pennsylvania Department of Health. Finally, we warmly thank Katherine Hermann and Andrew Lampinen for making the Trifeature dataset available for our use.
References
[1] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: aariane-invariann--ovvaaanne regular- ization for self-supervised learning. preprint arXiv:2105.04906, 2021.
[2] Sara Beery, Grant Van Hom, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pages 156-473, 2018.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NewrIPS), pages 9912-9924, 2020.
[4] Ting Chen and Lala Li. Intriguing properties of contrastive losses. preprint urXiv:2011.0280. 2020.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Int. Conference on Machine Learning (ICML), pages 10709-10719, 2020.
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. preprint arXiv:2003.044 04297, 2020.
[8] Lenaic Chizat and Francis Bach, Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory (COLT), pages 1308-1338, 2020.
[9] Ching- Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 8765-8775, 2020.
[10] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Int. Conf. on Learning Representations (ICLR), 2019.
[11] Robert Geirhos, Mern-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):6656673, 2020.
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In Int. Conf. on Learning Representations (ICLR), 2015.
[13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to sulf-super learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 21271-21284 2020.
[14] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. Int. Conference on Artificial Intelligence and Statistics (AISTATS), pages 297-304, 2010.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
10

---

[16] Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. In Advances in Neural Information Processing Systems (NeurIPS), pages 9995-10006, 2020.
[17] Katherine L Hermann, Ting Chen, and Simon Komblith. The origins and prevalence of texture bias in convolutional neural networks. In Advances in Neural Information Processing Systems (Neur[PS), pages 19000-19015, 2019.
[18] R Devon Hjelm, Alex Fedorov, Samuel Marcci-Mar,,,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Int. Conf. on Learning Representations (ICLR), 2019.
[19] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In Advances in Neural Information Processing Systems (NeurIPS), pages 17081-17093, 2020.
[20] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[21] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. preprint arXiv:2103. 10427, 2021.
[22] Andrew Iyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), pages 126-136, 2019.
[23] Jöm-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. In Int. Conf. on Learning Representations (ICLR), 2018.
[24] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. In Advances in Neural Information Processing Systems (NeurlPS), pages 16199-16210, 2020.
[25] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 21798-21809, 2020.
[26] Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[27] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self- superviee learning. preprint arXiv:2008.01064 2020.
[28] Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, and Dina Katabi. Information- preserving contrastive learning for self-supervised representations. preprint arXiv:2012.09966. 2020.
[29] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In Int. Conf. on Learning Representations (ICLR), 2020.
[30] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Int. Conf. Learning Representations (ICLR), 2018.
[31] Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. In International Conference Machine Learning, pages 6927-6937, 2020.
[32] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Int. Conf. on Learning Representations (ICLR), 2021.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. preprint ar (iX.v:8807003744 2018.

---

[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[35] Deepak Pathak, Philipp Krahenbuhi, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In IEFE Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 2536 2544, 2016.
[36] Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of machine learning research, pages 2011.
[37] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Int. Conference on Machine Learning (ICML), pages 5389-5400, 2019.
[38] Elizabeth A Regan, John E Hokanson, James R Murphy, Barry Make, David A Lynch, Terri H Beaty, Douglas Curran-Everett, Edwin K Silverman, and James D Crapo. Genetic epidemiology of COPD (COPDGene) study design. COPD: Journal of Chronic Obstructive Pulmonary Disease, 7(1):32-43, 2011.
[39] Joshua Robinson, Stefanie Jegelka, and Suvrit Sra. Strength from weakness: Fast learning using weak supervision. In Int. Conference on Machine Learning (ICML), pages 8127-8136, 2020.
[40] Joshua Robinson, Ching- Cao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. Int. Conf. on Learning Representations (ICLR), 2021.
[41] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2222-2878 2018
[42] Li Sun, Ke Yu, and Kayhan Batmanghelich. Context matters: Graph-based self-supervised representation learning for medical images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4874-4882. 2021.
[43] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. Int. Conf. on Learning Representations (ICLR), 2014.
[44] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Europ. Conference Computer Vision (ECCV), 2020.
[45] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 6827-6839, 2020.
[46] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In Ini. Conf. on Learning Representations (ICLR), 2018.
[47] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[48] Feng Wang, Huaping Liu, Guo, and Fuchun Sun. Unsupervised representation learning by invariance propagation. In Advances in Neural Information Processing Systems (NeurIPS), pages 3520- 2020.
[49] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Int. Conference on Machine Learning (ICMI.) pages 9574-9584, 2020.
12

---

[50] Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2021.
[51] Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Int. Conference on Machine Learning (ICML), 2021.
[52] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Int. Conference Machine Learning (ICML), pages 7472-7482, 2019.
[53] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. Int. Conf. on Learning Representations (ICLR), 2018.
[54] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. Europ. Conference Computer Vision (ECCV), pages 649-666, 2016.
[55] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In Int. Conf. on Learning Representations (ICLR), 2021.
[56] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Ini. Conference on Machine Learning (ICML), 2021.
13


Reviews:
[{"review_id": "nzTEVLHVWeV", "reviewer": "Reviewer_5txB", "summary": "This paper investigates feature learning in contrastive learning. It argues that feature suppression occurs in contrastive learning, where the model ignores a subset of the features. It argues that by varying the difficulty of the contrastive discrimination task, it is possible to incentivize the learning of particular features, but again at the expense of other features. To deal with this feature suppression problem, the authors propose a new method called implicit feature modification, where the basic idea is to continuously remove features from the embedding representation that are being used to discriminate negatives from positives. This encourages the model to learn new, additional features to perform the discrimination task. The authors show that this scheme makes the model learn more robust features (in the sense of Ilyas et al. (2019) --ref. 22--).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a technically sound, novel contribution to the contrastive learning literature. I think the proposed method conceptually makes sense. Some of the experiments seem to be pretty well done. However, some experiments do not have any error bars (e.g. Figure 7-8, Table 1). Are those effects significant? Also, the reported improvements are rather small and the need to tune an additional hyperparameter (epsilon) makes the proposed scheme practically unappealing. I also want to point out that the authors never do a proper cross-validated experiment: they just show the results for different epsilon values (e.g. Figure 7, Table 2), which again makes me wonder if these effects would survive a properly cross-validated experiment. \n\nI also had a more high level objection that I wanted to mention. This is more of an objection to contrastive learning in general, rather than this particular paper, but the point is that what papers like this really show for me is that instance discrimination and similar objectives as self-supervised learning objectives are fundamentally flawed (although they happen to work quite well at the scales where we're currently testing them), because there’s no guarantee that they will learn the “intended” representation. These objectives already require quite a bit of manual data augmentation engineering to cajole the model into learning the “correct” representation and papers like this suggest that even then these methods are not entirely successful at preventing shortcut learning. So, instead of trying to patch these issues with ad-hoc fixes like this paper tries to do, I think it would be more productive over the long term to adopt more principled self-supervised learning objectives in the first place (for example, likelihood-based generative objectives), where at least in the limit of large data, we know that they will learn the right features.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I mentioned my views on the limitations of this paper above. It would be great if the authors could address my concerns about the significance of the effects and cross-validation. I also think that in their experiments, the authors could use stronger baselines to convince the readers that their sophisticated method is really needed for the reported improvements. For example, could simply adding a drop-out layer (or a similar regularizing layer) before the embedding also encourage the model to use more features and lead to similar improvements?  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "N-CcpJl78h", "reviewer": "Reviewer_hQmV", "summary": "This paper provides a theoretical study on the cause of feature suppression in contrastive learning, and proposes \"implicit feature modification\" which is a theoretically grounded, yet simple and effective way of improving (avoiding) feature suppression in contrastive learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n1. Interesting theoretical analysis to explain the feature suppression phenomenon in contrastive learning.\n2. Well-designed experiments to demonstrate claims made in the paper, e.g. more difficult instance discrimination task makes the model learn more difficult features but at the same time suppressing the original learned features.\n3. Simplicity of IFM. No computational overheads, theoretically supported and shown to be effective on a decent variety of datasets. It is also nice to see IFM helps learn robust features.\n\n**Cons:**\n\nThe approach is not tested on larger scale datasets such as ImageNet-1k, probably due to computational constraints. I'm curious about the impact of the approach on larger datasets, where diversity of features and amount of images may naturally help contrastive learning to learn more features. The impact of IFM would be even more if it is shown to be effective for even larger datasets.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KE3Y9ZH9qxG", "reviewer": "Reviewer_yWao", "summary": "This paper studies how to avoid shortcut solutions in contrastive learning.  To this end, this work explores feature suppression in contrastive learning and explains why shortcut solutions happen. Then, this work proposes a simple method to handle this issue based on the observation on the difficulty of instance discriminations. Experimental results show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n1. This paper studies an important and practical issue in contrastive learning: how to avoid shortcut solutions.\n2. This paper theoretically analyzes feature suppression in contrastive learning and explains why it may happen in contrastive self-supervised learning.\n3. This paper proposed a simple and efficient method based on adjusting the difficulty of instance discrimination.\n\nNegative points:\n1. The notations are two complex, which makes the paper hard to follow. Moreover, some notations seem to not be defined well. For example, what is $\\bar{z}$ in Definition 1? What is $S^c$ in Line 194.\n\n2. It is would be better to conduct experiments on ImageNet-1k. All advanced contrastive learning methods are evaluated on ImageNET-1k, which can better evaluate the effectiveness of the proposed method.\n\n3. Whether does the proposed method contribute to siamese architectures, like BYOL and Simsiam?\n\nTypos: \"an\" seems \"and\" in Line 172. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the above main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "5sRAvRXIxFN", "reviewer": "Reviewer_c2ts", "summary": "In this paper, the authors assess whether contrastive instance discrimination can be modified to avoid models that learn “shortcut” solutions and discard important features.  They use a theoretical approach to explain why optimizing the InfoNCE loss that is associated with instance discrimination does not necessarily avoid shortcut solutions, and demonstrate that it is possible to adjust the difficulty of instance discrimination in a relatively straightforward way to tune the tradeoff between which features are learned.  The authors propose a method called Implicit Feature Modification (IFM) which encourages encoders to use multiple different input features to discriminate between instances in a cost-efficient manner that improves generalization and reduces suppression of useful features.  They provide a variety of empirical results supporting both the reasoning behind and the practical significance of their approach.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nWhile I am not the most familiar with all of the related work in this area, the approach appears novel and the differences between the current work and cited work is clearly explained.  The narrative arc taking the reader from analysis of feature suppression through representation tradeoffs to motivating IFM is compelling and the arguments are well-made.  \n\n\nQuality:\n\nThe submission appears technically sound.  The authors use appropriate methods to make their arguments, provide experimental justification for the claims the paper makes, and provide detailed descriptions of their experimental protocols.  The combination of theoretical analysis and empirical results that combine to tell a clear story is compelling.  The authors have an explicit limitations and broader impact section that adequately evaluates the strengths and weaknesses of their work.  I would, however, appreciate an additional discussion of the computational cost of this approach relative to the overall training process, as this seems a critical component of the work’s practical applicability that I do not remember seeing in the paper.\n\nClarity:\n\nI found the paper to be extremely well-written.  However, some of the mathematical exposition is notationally dense, and it may be possible to make the paper more accessible by simplifying the presentation.\n\nSignificance:\n\nAs a practitioner in computer vision, I find these results important and would certainly expect others to use these ideas.  The fundamental problem of models that take shortcuts to achieve high levels of performance on a training set, but perform poorly in practice, is extremely common and often addressed only in ad hoc ways.  The authors present one of the first methods of which I am aware by which this issue can be systematically addressed, they motivate/analyze it theoretically, and provide compelling empirical results to support its practical significance.  The fact that the authors show improvement not only on benchmark datasets, but also on a medical imaging dataset that reflects a real application, is particularly compelling.\n\nDetailed Comments/Questions:\n\nL 71-75: I very much appreciate the authors’ presentation of this point.\n\nL 92: What is the “space U” in “measure \\nu on a space U”?\n\nL 83 - 95: It may be worth describing why the authors set up the problem in this way.  It is not immediately obvious what the benefit of this formulation is at this juncture.\n\nL 112: Nit: “number of negatives m goes to infinity” → as the reader, I did not know what m was without reading the following lines.\n\nL112: It’s not immediately clear to me how the LFS → RHS.  Perhaps worth working out in an appendix or citing directly.\n  \nL 133 - 135: Another explanatory sentence here may be useful to provide intuition.\n\nL 229: Do you evaluate sensitivity to alpha at any point, or is alpha=1 generally a good choice?\nFig. 4 caption: Isn’t this moving the negative sample towards the anchor (rather than away, as the caption says)?\n\nL 294: It seems that the difference in linear readout accuracy b/t IFM and standard SIMCLR is very small (if not overlapping) in many cases.  I would soften the statement that it improves linear readout accuracy across all three features for all temperature settings.\n\nFig. 7: What does L_epsilon only mean?  Is this just not using the standard InfoNCE loss at all?  Why is this better sometimes for MoCo-v2?  Potentially worth discussing.\n\nTable 1: Please show confidence intervals here or some measure of spread.  Point estimates are not sufficient.\n\nL 328: Typo in “previouos”\n\nL 337: This should be Appendix C.5 not C.6, I think.\n\nTable 2: Some of the bolded numbers are likely not statistically significant (e.g. mMRC column).  Please indicate exactly what bold means, and ensure that it accurately reflects statistical significance in some sense.\n\nDiscussion:  Extremely short for a section entitled “discussion.”  I would suggest calling this “conclusions” and adding a bit more material here to summarize the results presented.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss these issues in their Appendix.  One potential addition would be a discussion and/or analysis of if this approach yields different levels of performance gains (or potential losses) on some classes versus others, rather than an analysis based purely on aggregate performance metrics.  Though this addition would be valuable, the authors’ existing discussion is informative.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper addresses and important practical question (feature supression in contrastive learning), and provides a theoretically-motivated and empirically-validated way to address it.
Decision: Accept (Poster)