What Should Not Be Contrastive in Contrastive Learning
Authors: Tete Xiao
Xiaolong Wang
Alexei A Efros
Trevor Darrell
Venue: ICLR 2021 Poster (2021)
Keywords: Self-supervised learning
Contrastive learning
Representation learning

Abstract: Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of representational invariances (e.g., invariance to color), and can perform poorly when a downstream task violates this assumption (e.g., distinguishing red vs. yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.

Published as a conference paper at ICLR 2021
WHAT SHOULD NOT BE CONTRASTIVE IN CONTRASTIVE LEARNING
Tete Xiao
Xiaolong Wang UC San Diego
Alexei A. Efros
Trevor Darrell UC Berkeley
UC Berkeley
UC Berkeley
ABSTRACT
Recent self-super contrastive methods have been able to produce impres- sive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of rep- resentational invariances (e.g., invariance to color), and can perform poorly when downstream task violates this assumption (e.g., distinguishing red yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.
INTRODUCTION
Self-supervised learning, which uses raw image data and/or available pretext tasks as its own super- vision, has become increasingly popular the inability of supervised models to generalize beyond their training data has become apparent. Different pretext tasks have been proposed with different transformations, such as spatial patch prediction (Doersch et al., 2015; Noroozi & Favaro, 2016), colorization (Zhang et al., 2016; Larsson et al., 2016; Zhang et al., 2017), rotation (Gidaris et al., 2018). Whereas pretext tasks aim to recover the transformations between different "views" of the same data, more recent contrastive learning methods (Wu et al., 2018; Tian et al., 2019; He et al., 2020; Chen et al., 2020a) instead try to learn to be invariant to these transformations, while remain- ing discriminative with respect to other data points. Here, the transformations are generated using classic data augmentation techniques which correspond to common pretext tasks, e.g.. randomizing color, texture, orientation and cropping.
Yet, the inductive bias introduced through such augmentations is a double-edged sword, as each augmentation encourages invariance transformation which can be beneficial in some cases and harmful in others: e.g., adding rotation may help with new-indeeendent aerial image recognition, but significantly downgrade the capacity of network solve tasks such detecting which way is up in a photograph for display application. Current self-supervised contrastive learning methods assume implicit knowledge of downstream task invariances. In this work, we propose learn visual representations which capture individual factors of variation in contrastive learning framework without presuming prior knowledge of downstream invariances.
Instead of mapping an image into a single embedding space which is invariant to all the hand- crafted augmentations, our model learns to construct separate embedding sub-spaces, each of which is sensitive to a specific augmentation while invariant to other augmentations. We achieve this by optimizing multiple commentation-sensittee with a shared backbone. Our model aims preserve information with regard to each augmentation in a unified representation, as well as learn invariances to them. The general representation trained then be applied to different downstream tasks, where each task is
contrastive objectives using a multi-head architecture
with these augmentations free to selectively utilize different factors variation in our representation. We consider transfer of either the shared backbone representation, the concatenation of all the task-specific heads; both

---

Published as a conference paper at ICLR 2021
Claser
Augmentations
Rotation
Color
Rotation
Texture
Augmentations
Coarsegrained
Fine-grainee (bird)
Fine-grainee (flower) Tasks
(a)
(b)
Figure 1: Self-supervised contrastive learning relies on data augmentations as depicted in (a) to learn visual representations. However, current methods introduce inductive bias by encouraging neural networks to less sensitive information w.rt. augmentation, which may help or may hurt. As illustrated in (b), rotation invariant embeddings can help on certain flower categories, but may hurt animal recognition performance; conversely color invariance generally seems to help coarse grained animal classification, but can hurt many flower categories and bird categories, Our method, shown in the following figure, overcomes this limitation.
outperform all baselines; the former uses same embedding dimensions as typical baselines, while the latter provides greatest overall performance in our experiments. In this paper, we experiment with three types of augmentations: rotation, color jittering, and texture randomization, as visualized in Figure 1. We evaluate our approach across variety of diverse tasks including large-scale clas- sification (Deng et al., 2009), fine-grained classification (Wah et al., 2011; Van Horn et al., 2018), few-shot classification (Nilsback & Zisserman, 2008), and classification on corrupted data (Barbu et al., 2019; Hendrycks & Dietterich, 2019). Our representation shows consistent performance gains with increasing number of augmentations. Our method does not require send-selection of data aug- mentation strategies, and achieves better performance against state-of-the-ar MoCo baseline (He al., 2020: Chen al., 2020b), and demonstrates superior transferability, generalizability and bustness across tasks and categories. Specifically, we obtain around 10% improvement over MoCo in classification when applied on the iNaturalist (Van Horn et al., 2018) dataset.
BACKGROUND: CONTRASTIVE LEARNING FRAMEWORK
Contrastive learning learns representation by maximizing similarity and dissimilarity over data samples which are organized into similar and dissimilar pairs, respectively. It can be formulated as dictionary look-up problem (He et al., 2020), where given reference image is augmented into two views, query and key, and the query token q should match its designated key k-t over set of sampled negative keys {k" } from other images. In general, the framework can be summarized the following components: (i) A data augmentation module T constituting atomic augmentation oper- ators, such as random cropping, color jittering, and random flipping. We denote a pre-defined atomic augmentation as random variable X,. Each time the atomic augmentation is executed by sampling a specific augmentation parameter from the random variable, i.e., -,~X, One sampled data aug- mentation module transforms image into a random view I, denoted as Ī = T[x1, ......, xn,] (I). Positive pair (g. k+*) is generated by applying
randomly sampled data augmentation on the same reference image. (ii) An encoder network í which extracts the feature of an image Z by mapping it into d-dimensional space Rd. (iii) A projection head h which further maps extracted represen- tations into hyper-spherical (normalized) embedding space. This space is subsequently used for specific pretext task, i.c., contrastive loss objective for a batch of positive/negaatie pairs. A common choice is InfoNCE (Oord et al., 2018):
exp (q-k+/) /T)
C. log
exp (g.k+/T) Eτ- exp (q-k-
where T is temperature byper-paramett scaling the distribution of distances.
As a key towards learning a good feature representation (Chen et al., 2020a), a strong augmentation policy prevents the network from exploiting naïve cues to match the given instances. However, in-
2

---

Published as a conference paper at ICLR 2021
medom netatin jileer
catoelding
postior
(02g)
all all-araaiaa smbenling spuce
caloevoutain
cabbedding sprce
regetive
Figure 2: Framework the Leave-one-out Contrastive Learning approach, illustrated with two types of augmentations, ie., random rotation and color jittering. We generate multiple views with leave-one-our strategy, then project their representations into separate embedding spaces with con- trastive objective, where each embedding space either invariant to all augmentations, or invariant to all but one augmentation. The learnt representation can be the general embedding space V (blue region), the concatenation of embedding sub-spaces 2 (grey region). Our results show that either of our proposed representations are able to outperform baseline contrastive embeddings and do not suffer from decreased performance when adding augmentations which the task is not invariant (i.e., the red X's in Figure 1).
ductive bias introduced through the selection of augmentations, along with their hyper-paramen defining the strength of each augmentation, manifested in Equation that any views by the stochas- tic augmentation module T of the same instance are mapped onto the same point in the embedding space. The property negatively affects the learnt representations: 1) Generalizability and transfer- ability are harmed if they are applied to the tasks where the discarded information is essential, e.g., color plays an important role in fine- ane-grained classification of birds; 2) Adding an extra augmentation is complicated as the new operator may be helpful to certain classes while harmful to others, e.g., rotated flower could be very similar to the original one, whereas it does not hold for a rotated car; 3) The for each augmentation to strike a delicate balance between leaving a short-cut open and completely invalidate one source of information.
which control the strength of augmentations need to be carefully tuned
3 LooC: LEAVE-ONE-OUT CONTRASTIVE LEARNING
We propose Leave-one-out Contrastive Learning (LooC), a framework for multi-augmmntationn con- trastive learning. Our framework can selectively prevent information loss incurred by an augmen- tation. Rather than projecting every view into a single embedding space which is invariant to all augmentations, in our LooC method the representations of input images are projected into several embedding spaces, each of which is not invariant to certain augmentation while remaining invari- ant to others, as illustrated in Figure 2. In this way, each embedding sub-space is specialized to a single augmentation, and the shared layers will contain both augmentation-waryyl formation. We learn a shared representation jointly with the several embedding spaces; we transfer either the shared representation alone, the concatenation of all spaces, to downstream tasks.
and invariant in-
View Generation. Given reference image and atomic augmentations, we first augment the reference image with two sets of independently sampled augmentation parameters into the query view Iy and the first key view Ik,, i.c., [(g = -[ (7. .x) (q.Ko)
2 kg. (I). Additionally, ... x we generate n views from the reference image as extra key views, denoted as Ik.. Vi E (1,... n)- For the ith additional key view, the parameter of 5th atomic augmentation is copied from it of the query view, i.e., ,xth =/. Vi E (1,...,m) n:); whereas the parameter of other atomic augmentations are still independently sampled, i.c., x) ~ X,, Vj # i. For instance, assume that we have a set of two atomic augmentations (random rotatioon ion, color_ color_jitte jess_jiterrrr.r Iy and I,, are always augmented by the same rotation angle but different color jittering; Ig and Ik, are always augmented by the same color jittering but different rotation angle; I, and In, are augmented independently, as illustrated in the left part of Figure 2.
3

---

Published as a conference paper at ICLR 2021
Contrastive Embedding Space. The augmented views encoded by neural network encoder f(-) into feature vectors v²,""o, v'o.................,,.... ...., wdn in a joint embedding space E Rd Subsequently, they are projected into n+1 normalized embedding spaces Zo. 21, 2n E Rdd by projection heads h V + Z, among which Zg is invariant
all types of augmentations, whereas Z; (vi E (1, , is dependent on the tth type of augmentation but invariant to other types single point, whereas in Z; (Vi E (1, ... .n)) only v and vt should be mapped to single point while o Vj + i should be mapped to n-1 separate points, as only In and Zt, share the same jth augmentation.
of augmentations. In other words, in Zu all features a should be mapped
We perform contrastive learning in all normalized embedding spaces based on Equation 1, as shown in the right part of Figure 2. For each query z", denote =²* as the keys from the same instance, and a²" as the keys from other instances. Since all views should be mapped to the single point in Zo. the positive pair for the query =i is 20"
A.5 and the negative pairs are embeddings of other
instances in this embedding space {=00 }; for embedding spaces _,, Zn the positive pair for AT the query 2 is z;" while the negative pairs are embeddings of other instances in this embedding Kt
space { th J. and { | E 1.. .n and j * i}, which are the embeddings of the same instance with different ith augmentation. The network then learns to be sensitive to one type of augmentation while insensitive to other types of augmentations in one embedding space. Denote
k(+--)
F = exp Zz
/T). The overall training objective for q is:
C
log
log
(2)
E0,0
Σx-
E00
Σx- E.,
The network must preserve information w.r.t. all augmentations in the general embedding space V in order to optimize the combined learning objectives of all normalized embedding spaces.
Learnt representations. The representation for downstream tasks can be from the general embed- ding space V/ (Figure 2, blue region), or the concatenation of all embedding sub-spaces (Figure 2, grey region). LooC method returns V: we term the implementation using the concatenation of all embedding sub-spaces as LooC++.
4 EXPERIMENTS
Methods. We adopt Momentum Contrastive Learning (MoCo) (He et al., 2020) as the backbone of our framework for its efficacy and efficiency, and incorporate the improved version from (Chen al., 2020b). We use three types of augmentations as pretext tasks for static image data, namely color jittering (including random gray scale), random rotation (90', 180 or 270"), and texture random- ization (Gatys al., 2016; Geirhos et al., 2018) (details in the Appendix). We apply random-resized cropping, horizontal flipping and Gaussian blur as augmentations without designated embedding spaces. Note that random rotation and texture randomization trastive learning based methods (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b) and for good reason, as will empirically show that naïvely taking these augmentations negatively affects the performance on some specific benchmarks. For LooC++, we include Conv5 block into the projec- tion head h, and use the concatenated features the last layer of Conv5, instead of the last layer of h, from each head. Note than for both LooC and LooC++ the augmented additional keys are only fed into the key encoding network, which not back-propagated, thus does not much increase computation or GPU memory consumption.
not utilized in ofite--f--heerr con-
Datasets and evaluation metries. We train our model on the 100-category ImageNet (IN-100) dataset, subset of the ImageNet (Deng et al., 2009) dataset, for fast ablation studies of the proposed framework. We split the subset following (Tian et al., 2019). The subset contains 1255 images, sufficiently large to conduct experiments of statistical significance. After training. we adopt linear classification protocol by training a supervised linear classifier on frozen features of feature space V for LooC, or concatenated feature spaces Z for LooC++. This allows us to directly verify the quality of features from variation of models, yielding more interpretable results. We test the models on various downstream datasets (more information included in the Appendix): 1) IN- 100 validation set; 2) The iNaturalist 2019 (iNat- 1k) dataset (Van Horn et al., 2018), large-scale classification dataset
4

---

Published as a conference paper at ICLR 2021
Table 1: Classification accuracy on 4-class rotation and IN-100 under linear evaluation protocol. Adding rotation augmentation into baseline MoCo significantly reduces its capacity to classify rota- tion angles while downgrades its performance on IN- In contrast, our method better leverages the information gain of the new augmentation.
Rotation Acc.
IN-100
model
top-1
top-5
Supervised
72.3
83.7
95.7
MoCo
61.1
81.0
95.2
MoCo + Rotation
43.3
79.4
94.1
MoCo + Rotation (same for and k)
45.5
78.1
94.3
LooC + Rotation [ours]
65.2
80.2
95.5
Table 2: Evaluation on multiple downstream tasks. Our method demonstrates superior generaliz- ability and transferability with increasing number of augmentations.
Augmentation
iNat-1k
CUB-200
Flowers-
IN-100
model
Color Rotation
top-1
top-5
top-1
top-5
5-shot
10-shot
top-1
top-5
MoCo
36.2
62.0
36.7
64.7
67.9
0.5) 77.3 (1: 0.1)
81.0
95.2
LooC
41.2
67.0
40.1
69.7
68.2
(= 0.6) 77.6 0.1)
81.1
95.3
/
40.0
65.4
38.8
67.0
70.1
(# 0.4) 79.3
( 0.1)
80.2
95.5
/
44.0
69.3
39.6
69.2
70.9
(+ 0.3) 80.8 (4 0.2)
79.2
94.7
LooC++
46.1
71.5
39.3
69.3
68.1
(# 0.4) (
81.2
95.2
containing 1,010 species. Top-1 and top-5 accuracy this dataset are reported: 3) The Caltech- UCSD Birds 2011 (CUB-200) dataset (Wah et al., 2011), a fine-grained classification dataset of 200 bird species. Top-1 and top-5 classification accuracy are reported. 4) VGG Flowers (Flowers-102) dataset (Nilsback & Zisserman, 2008), a consistent of 102 flower categories. We use the dataset for few-shot classification and report 5-shot and 10-shot classification accuracy over 10 trials within 95% confidence interval. Unlike many few-shot classification methods which conduct evaluation on a subset of categories, we use 102 categories our study; 5) ObjectNet dataset (Barbu et al., 2019), test set collected to intentionally show objects from new viewpoints on new backgrounds with different rotations of real-world images. We only use the 13 categories which overlap with IN-100, termed 1N:13; 6) ImageNet-C dataset (Hendrycks & Dietterich, 2019), a benchmark for model robustness of image corruptions. We use the 100 categories as IN-100, termed as IN-C- 100. Note that ON and IN-C are test sets, we do not train supervised linear classifier exclusively while directly benchmark the linear classifier trained on IN- instead.
Implementation details. We closely follow (Chen et al., 2020b) for most training hyper- parameters. We use a ResNet-50 (He et al., 2016) our feature extractor. We use two-layer MLP head with a 2048-d hidden layer and ReLU for each individual embedding space. We train the network for 500 epochs, and decrease the learning rate at 300 and 400 epochs. We use separate queues (He et al., 2020) for individual embedding space and set the queue size to 16,384. Linear classification evaluation details can be found in the Appendix. The batch size during training of the backbone and the linear layer is set to 256.
Study on augmentation inductive biases. We start by designing an experiment which allows us to directly measure how much an augmentation affects a downstream task which is sensitive to the augmentation. For example, consider two tasks which can be defined on IN-100: Task A is 4- category classification of rotation degrees for an input image; Task B is 100-category classification of ImageNet objects. We train a supervised linear classifier for task A with randomly rotated IN-100 images, and another classifier for task B with unrotated images. In Table we compare the accuracy of the original MoCo (w/o rotation augmentation). MoCo w/ rotation augmentation, and our model w/ rotation augmentation. A priori, with no data labels to perform augmentation selection, we have no way to know if rotation should be utilized or not. Adding rotation into the set of augmentations for MoCo downgrades object classification accuracy on IN-100, and significantly reduces the ca- pacity of the baseline model to distinguish the rotation of an input image. We further implement variation enforcing the random rotating angle of query and key always being the same. Although it marginally increases rotation accuracy, IN- object classification accuracy further drops, which is inline with our hypothesis that the inductive bias of discarding certain type of information intro- duced by adopting an augmentation into contrastive learning objective is significant and cannot be trivially resolved by tuning the distribution of input images. On the other hand, our method with rotation augmentation not only sustains accuracy IN-100, but also leverages the information gain
5

---

Published as a conference paper at ICLR 2021
Table 3: Evaluation on datasets of real-world corruptions. Rotation augmentation is beneficial for ON-13, and texture augmentation if beneficial for IN-C-100.
Aug.
ON-13 3
IN-C-100
(top-1)
IN-100
model
Rot.
Tex.
top-1
top-5
Noise
Blur
Weather
Digital
All
d
3
top-1
top-5
Supervised
30.9
54.8
28.4
47.1
44.9
58.5
47.2
36.5
83.7
95.7
MoCo
29.2
54.2
37.9
38.5
47.7
60.1
48.2
37.2
81.0
95.2
LooC
34.2
59.6
31.3
33.1
42.4
54.9
42.7
31.8
80.2
95.5
30.1
54.1
42.4
39.6
54.0
61.9
51.3
41.9
81.0
94.7
V
33.3
59.2
37.0
35.2
50.2
56.9
46.5
37.2
79.4
94.3
LooC++
32.6
57.3
38.3
37.6
52.0
60.0
48.8
38.9
82.1
95.1
Table Comparisons of LooC vs. MoCo trained with all augmentations.
IN-100
iNat-1k
Flowers- 102
IN-C-100
Model
top-1
top-5
top-1
top-5
5-shot
10-shot
all-top-1
MoCo
77.9
93.7
39.5
65.1
72.1 (# 0.4) 81.1 (= 0.2)
47.4
LooC
78.5
94.0
41.7
67.5
72.1 (# 0.7) 81.4 0.2)
45.4
MoCo++
80.8
94.6
43.4
68.5
70.0 0.8) 80.5 0.3)
48.3
LooC++
82.2
95.3
45,9
71.4
71.0 (# 0.7) 81.9 0.3)
48.0
Table 5: Comparisons of concatenating features from different embedding spaces in LooC++
jointly trained on color, rotation and texture augmentations. Different downstream tasks show non- identical preferences for augmentation-deeendenn or invariant representations.
Variance Head
IN-100
iNat-1k
Flowers-
IN-C-100
Model
Col. Rot.
Tex.
top-1
top-5
top-1
top-5
5-shot
10-shot
all-top-1
LooC++
78.5
94.3
38.5
64.7
68.6
(
48.0
79.7
94.4
42.9
68.7
69.1 0.7) 79.5 (.2))
47.1
/
81.5
94.9
41.4
67.4
70.5
(=.5 0.6) 0.22
52.6
80.3
94.9
43.0
68.6
70.4
(#: 0.5) 80.5
44.1
82.2
95.3
45.9
71.4
71.0 (:7. 81.9 0.3)
48.0
of the new augmentation. We can include all augmentations with our LooC multi-self-supervised method and obtain improved performance across all condition without any downstream labels or a prior knowledged invariance.
Fine-grained recognition results. A prominent application of unsupervised learning is learn features which are transferable and generalizable to a variety of downstream tasks. To fairly evalu- ate this, we compare our method with original MoCo on diverse set of downstream tasks. Table lists the results on iNat-1k, 1k. CUB-200 and Flowers- 102. Although demonstrating marginally supe- rior performance on IN-, the original MoCo trails our LooC counterpart on all other datasets by a noticeable margin. Specifically, applying LooC on random color jiterring boosts the performance of the baseline which adopts the same augmentation. The comparison shows that our method can better preserve color information. Rotation augmentation also boosts the performance on iNat- Ik and Flowers- 102, while yields smaller improvements on CUB-200, which supports the intuition that some categories benefit from rotation- ootation-invariant representations while some do not. The perfor- mance is further boosted by using LooC with both augmentations, demonstrating the effectiveness in simultaneously learning the information w.r.t. multiple augmentations.
Interestingly, LooC++ brings back the slight performance drop on IN-100, and yields more gains on iNat- 1k, which indicates the benefits of explicit feature fusion without hand-crafting what should or should not be contrastive in the training objective.
Robustness learning results. Table 3 compares our method with MoCo and supervised model on ON-13 and IN-C-100, two testing sets for real-world data generalization under a variety of noise conditions. The linear classifier is trained on standard IN-100, without access to the testing distribu- tion. The fully supervised network most sensitive to perturbations, albeit it has highest accuracy on the source dataset 100. We also see that rotation augmentation is beneficial for ON-13, but significantly downgrades the robustness to data corruptions in IN-C-100. Conversely, texture ran- domization increases the robustness on IN-C-100 across all corruption types, particularly significant on "Blur" and "Weather", and on the severity level above or equal to 3, as the representations must be insensitive to local noise to lear texture-invarian features, but its improvement ON-13 is marginal. Combining rotation and texture augmentation yields improvements on both datasets, and LooC++ further improves its performance on IN-C-100.

---

Published as a conference paper at ICLR 2021
query
top retrievals
query
top retrievals
MiCo
MoCo
LavC
Loc MoCo
MoCo
Love'
Figure 3: Top nearest-neighbor retrieval results of LooC vs. corresponding invariant MoCo base- line with color (left) and rotation (right) augmentations on IN-100 and iNat-Ik. The results show that our model can better preserve information dependent on color and rotation despite being trained with those augmentations.
Qualitative results. In Figure 3 we show nearest-neighhor retrieval results using features learnt with LooC vs. corresponding MoCo baseline. The top retrieval results demonstrate that our model can better preserve information which is not invariant to the transformations presented in the aug- mentations used in contrastive learning.
Ablation: MoCo w/ all augmentations vs. LooC. We compare our method and MoCo trained with all augmentations. We also add multiple Conv5 heads to MoCo, termed as MoCo++, for a fair comparison with LooC++. The results are listed in Table 4. Using multiple heads boosts the performance of baseline MoCo, nevertheless, our method achieves better or comparable results compared with its baseline counterparts.
Note that the results in Table 2 to 5 should be interpreted the broader context of Table Table illustrates the catastrophic consequences of not separating the varying and invariant factors of an augmentation (in this case, rotation). It can be imagined that if we add "rotation classification" as one downstream task in Table 4, MoCo++ will perform as poorly as in Table 1. The key of our work is to avoid what has happened in Table and simultancously boosts performance.
Ablation: setneeettatio-ddeedeee embedding spaces vs. tasks. We train a LooC++ with all types of augmentations, and subsequently train multiple linear classifiers with concatenated features from different embedding spaces: invariant, color, rotation and texture. Any additional variance features boost the performance on IN-100, iNat-1k and Flowers- Adding texture -detedee fea- tures decreases the performance on IN-C-100: Textures are (overly) strong cues for ImageNet clas- sification (Geirhos et al., 2018), thus the linear classifier is prone to use texture-dependent features, loosing the gains of texture invariance. Adding cetation-dependent features increases the perfor- mance on IN-C NN-C-100: Rotated objects most classes in IN- 100 are rare, thus the linear classifier is prone to rotation-dependent features, that drops IN-C-100 triggered by rotation-invariant augmentation are re-gained. Using all types of features yields best performance on IN-100, iNat- Ik and Flowers-102: the performance on IN-C-100 100 with all augmentations remains comparable to MoCo, which does not suffer from loss of robustness introduced by rotation invariance.
In Figure 4 we show the histogram of correct predictions (activations x weights of classifier) by each augmentatioon-deeenden head of a few instances from IN-100 and iNat-1k. The classifier prefers eexuueedeeeendett information over other kinds on an overwhelmingly majority of sam- ples from IN-100, even for classes where shape is supposed to be the dominant factor, such as "pickup" and "mixing bowl" ((a), top row). This consistent with the findings from (Geirhos al., 2018) that ImageNet-trainee estingly, when human or animal faces dominant an image ((a), bottom-left), LooC++ sharply prefers rotation-dependent features, which also holds for face recognition of humans. In contrast, on iNat- 1k LooC++ prefers a more diverse set of features, such as color-dependent feature for a dragonfly species, rotation and texture- eeture-deppnnent features for birds, well as rotation-invarian features for flowers. Averaged over the datasets, the distribution of classifier preferences is more balanced on iNat- kk than IN-100, as can be from the entropy that the distribution iNat-1k is close
CNNs are strongly biased towards rexture-like representations. Inter-

---

Published as a conference paper at ICLR 2021
AlF
Rasson
Scalio
Terae
packl
Labe miting howe
Labdl plane 962
Label: Inswos 74
Averge
Averoe
Alten
Allen.
Recton
Temore
Taxkor
Ractie:
Label valler Joand
Entroyy: 1.27
Label bond 240
Emmopy:186 1.86
(a) ImaggeNet
(b) iNaturalist
Figure 4: Histograms of correct predictions (activations x weights of classifier) by each augmentation-depenne head from IN-100 and iNat-1k. The classifier IN-100 heavily relies on texture-dependent information, whereas it much more balanced on iNat- k.. This consistent with the improvement gains observed when learning with multiple augmentations.
bits, whereas it is close to bit on IN-100, it is dominated by only two elements. It corroborates the large improvements on iNat-Ik Ik gained from multi-dependent features learnt by our method.
5 RELATED WORK
Pretext Tasks. In computer vision, feature design and engineering used to be a central topic be- fore the wide application of deep learning. Researchers have proposed utilize cue combination for image retrieval and recognition tasks (Martin et al., 2004; Frome et al., 2007a;b; Malisiewicz & Efros, 2008; Rabinovich et al., 2006). For example, the local brightness, color, and texture features are combined together to represent an image and a simple linear model can be trained to detect boundaries (Martin al., 2004). Interestingly, the recent development of unsupervised represen- tation learning in deep learning is also progressed by designing different self-supervised pretext tasks (Wang & Gupta, 2015; Doersch al., 2015; Pathak al., 2016; Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018; Owens et al., 2016). For example, relative patch predic- tion (Doersch al., 2015) and rotation prediction (Gidaris al., 2018) are designed to discover the underlined structure of the objects; image colorization task (Zhang et al., 2016) is used to learn representations capturing color information. The inductive bias introduced by each pretext task can often be associated with a corresponding hand-crafted descriptor.
Multi-Task Self-Supervised Learning. Multi-task learning has been widely applied in image recognition (Kokkinos, 2017; Teichmann al., 2018; He et al., 2017). However, jointly optimizing multiple tasks are not always beneficial. As shown in Kokkinos (2017), training with two tasks can yield better performance than seven tasks together, some tasks might be conflicted with each other. This phenomenon becomes more obvious in multi-task self-suuervvsed learning (Doersch & Zisserman, 2017; Wang et al., 2017; Pinto Gupta, 2017: Piergiovanni et al., 2020; Alwassel al., 2019) as the optimization goal for each task can be very different depending on the pretext task. To solve this problem, different weights for different tasks are learned optimize for the down- stream tasks (Piergiovanni et al., 2020). However, searching the weights typically requires labels, and is conecconuming and does generalize to different tasks. To train general representationss researchers have proposed to utilize sparse regularization to factorize the network representations to encode different information from different tasks (Doersch Zisserman, 2017; Misra et al., 2016). In this paper, also proposed to learn representation which can factorize and unify information from different augmentations. Instead of using sparse regularization, we define different contrastive learning objective in multi- head architecture.
Contrastive Learning. Instead of designing different pretext tasks, recent work on contrastive learn- ing (Wu et al., 2018; Oord et al., 2018; Tian et al., 2019; He et al., 2020; Misra & van der Maaten, 2020: Chen et al., 2020a) trained networks to be invariant to various corresponding augmentations. Researchers (Chen al., 2020a) elaborated different augmentations and pointed out which augmen- tations are helpful or harmful for ImageNet classification. is also investigated in Tian et al. (2019) that different augmentations can be beneficial to different downstream tasks. Instead of enumerat- ing all the possible selections of augmentations, we proposed a unified framework which captures different factors of variation introduced by different augmentations.
8

---

Published as a conference paper at ICLR 2021
CONCLUSIONS
Current contrastive learning approaches rely specific augmentation-deriied transformation in- variances to learn visual representation, and may yield suboptimal performance on downstream tasks if the wrong transformation invariances presumed. We propose new model which learns both transformation dependent and invariant representations by constructing multiple embeddings, each of which is nor contrastive single type of transformation. Our framework outperforms base- line contrastive method on coarse-gres -ranse,, fine- fine-graing grained, few-shot downstream classification tasks, and demonstrates better robustness of real-world data corruptions.
ACKNOWLEDGEMENT
Prof. Darrell's group was supported in part by DoD, NSF, BAIR, and BDD. Prof. Wang's group was supported, in part, by gifts from Qualcomm and TuSimple. We would like to thank Allan Jabri, Colorado Reed and Ilija Radosavovic for helpful discussions.
REFERENCES
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911.1220 2019. 8
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias controledd dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, pp. 9448-9458, 2019. 2,5
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a. 4, 8
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b. 2, 5, 12
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-254225 255.. leee, 2009. 2,
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017. 8
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015. 1,8 8
Andrea Frome, Yoram Singer, and Jitendra Malik. Image retrieval and classification using local distance functions. In Advances neural information processing systems, pp. 417-424, 2007a.
Andrea Frome, Yoram Singer, Fei Sha, and Jitendra Malik. Learning glosally-consiseent local dis- tance functions for shape-based image retrieval and classification. In 2007 IEEE 11th Interna- tional Conference on Computer Vision, pp. 1-8. IEEE, 2007b.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2413-2423, 2016. 4
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- imagenet-trained trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint /81 2018. 4,
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, 2018. 1,8

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 5
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017. 8
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. 1, 4,
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. Proceedings of the International Conference on Learning Represen- fations, 2019. 5,5
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 8
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European Conference on Computer Vision, pp. 577-593, Springer, 2016.
Tomasz Malisiewicz and Alexei A Efros. Recognition by association via learning per-exemplar distances. 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008. 8
David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning to detect natural image bound- aries using local brightness, color, and texture cues. IEEE transactions on pattern analysis and machine intelligence, 26(5):530-544 54900055---0-- 2004. 8
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa-
tions. In CVPR, 2020. 8
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016. 8
Maria-Elema Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008. 2,5
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Europeun Conference on Computer Vision, pp. 69-84. Springer, 2016. I,
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv:1807.0377 2018. 2,8
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In European conference on computer vision, pp. 801-816. Springer, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. arXiv preprint arXiv: 2020.
Lerrel Pinto and Abhinav Gupta. Learning push by grasping: Using multiple tasks for effective learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2161- 2168. IEEF, 2017.5
10

---

Published as a conference paper at ICLR 2021
Andrew Rabinovich, Tilman Lange, Joachim Buhmann, and Serge Belongie. Model order selection and cue combination for image segmentation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), New York City, 2006. URL /tt:///ccootee///uue/e 201 //9/ 650Rab novi ch. ch. ..f.. 8
Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun. Multi- net: Real-time joint semantic reasoning for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1013-1020. IEEE, 2018. 8
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv preprint 1506::990008444 2019. 1,4,3
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018. 2,
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011. 2,5
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2015.
Xiaolong Wang, Kaiming He, and Abhinav Gupta. Transitive invariance for self-supervised visual representation learning. In ICCV, 2017. 8
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018. 1,8
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer-
ence on computer vision, pp. 649-666. Springer, 2016. 1, 8
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.

---

Published as a conference paper at ICLR 2021
A AUGMENTATION DETAILS
Following (Chen et al., 2020b), we set the probability of color jittering to 0.8, with (bright contrast saturat ion, hue) as (0.4, 0.4, 0.4, 0.1), and probability of random scale 0.2. We set the probability of random rotation and texture randomization as 0.5.
B DATASETS
INat-1k, a large-scale classification dataset containing 1,010 species with combined training and validation set of 268,243 images. We randomly reallocate 10% of training images into the validation set as the original validation set is relatively small.
CUB-200, which contains 5,994 training and 5,794 testing images of 200 bird species.
Flowers-102, which contains 102 flower categories consisting of between 40 and 258 images.
collected intentionally show objects from viewpoints on back- grounds with different rotations of real-world images. It originally has 313-category. We only use the 13 categories which overlap with IN-100.
ObjectNet,
ImageNet-C,, which consists of 15 diverse corruption types applied to validation images of Ima- geNet.
C LINEAR CLASSIFICATION
We train the linear layer for 200 epochs for IN- 100 and CUB-200, 100 epochs for iNat- Ik, optimized by momentum SGD with learning rate of 30 decreased by 0.1 at 60% and 80% of training schedule; for Flowers- -0wvers-102aa we train the linear layer with Adam optimizer for 250 iterations with a learning rate of 0.03.
D LEAVE-ONE-OUT VS. ADD-ONE AUGMENTATION
Table 6: Leave-one-out vs. add-one Augmentation. egy.
Default (none add-one) augmentation strat-
Augmentation
IN-100
model
Color Rotation
top-1
top-5
MoCo
81.0
95.2
79.4
94.1
MoCo + AddOne
74.9
92.5
79.3
94.4
LooC [ours]
81.1
95.3
80.2
95.5
forward alternative for our leave-one-out augmentation strategy is add-one augmentation. Instead of applying all augmentations and augmenting two views in the same manner, add-one strat- egy keeps the query image unaugmentated, while in each augmentation-speciff view the designated type of augmentation is applied. The results shown Table 6. Add-one strategy oversimplifies the instance discrimination task, e.g., leaving color augmentation out of query view makes it very easy for the network to spot the same instance of set of candidates. Our leave-one one-ou out strategy does not suffer such degeneration.
12

---

Published as a conference paper at ICLR 2021
E IMAGENET-LE IK EXPERIMENTS
Table Results models trained on 1000 category ImageNet and fine-tuned on iNat- 1k following linear classification protocol.
iNat- 1k
model
top-1
top-5
MoCo
47.8
74.3
LooC++ [ours]
51.2
76.5
We conduct experiments on 1000 category full ImageNet dataset. The models are trained by self- supervised learning on IN-IK, and fine-tuned on iNat-Ik following linear classification protocol. Our model is trained with all augmentations, i.e., color, rotation and texture. Results are reported in Table
DISCUSSIONS
F.1 THE DIMENSIONS OF MoCo, LooC, LooC++
The representations of MoCo and LooC are of exactly the same dimension (2048); same for MoCo++ and LooC++ (2048 # augmentations). It is specifically designed for fair comparisons.
F.2 ARE THE HYPER- PARAMETERS TUNED SPECIFICALLY FOR OUR SUBSETS?
No, except that we increase the number of training epochs the amount of data increases. We did not specifically tune the baseline so that our method can oulperform it most; on the contrary, we first made baseline as strong as possible, then directly applied the same paper-parameters to our method. The subset of ImageNet100 100 behaviors similarly as ImageNetlk; our baseline already significantly outperforms the best method on the same subset from previous literature (75.8% CMC vs. 81.0% top1 [ours]), and since our method is derived from MoCo, they are directly comparable.
13


Reviews:
[{"review_id": "v9-QOkg7Ye", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Summary ==\n\nThe paper proposes a contrastive learning approach for self-supervised learning in which multiple heads are trained to be invariant to all but one type of data augmentation. The rationale is that different downstream tasks may require different types of invariances (e.g. we may want to be rotation invariant for pictures of flowers, but not for pictures of animals), and one does not know a-priori which kind of invariances will be required. After training multiple representation heads, one can later concatenate them or use the general embedding (the input to all the variant-specific heads) for the downstream task.\n\n== Pros ==\n\n- The authors try to tackle an important problem of self-supervised approaches: how does one decide which data augmentation strategies to use when the downstream task is not known in advance? This question has not been properly addressed in the literature, and can be of great important for the real application of self-supervised strategies beyond academic benchmarks.\n\n- The method that the authors introduce scales well with the number of data augmentations used (linear), and avoid a combinatorial explosion that could arise.\n\n- The paper is generally well written and the algorithm is explained quite clearly, and illustrations are used appropriately to help the reader understand the proposed approach (Figure 1 and Figure 2).\n\n== Cons ==\n\n- Once the self-supervised pre-training finishes, one has to decide whether to use the general embedding space in the downstream task, or a concatenation of the different variant-specific embeddings. However, from the results reported in Table 2, 3 and 4 it's not clear which approach is better, and this is downstream task-dependent. This is unfortunate since it basically introduces another hyperparameter to tune for each downstream task. \n\n- Many experiments do not report any measure of variance or statistical significance, and do not follow an an \"standard\" setting. This makes really hard to tell whether the observed increase in accuracy is statistically significant or not. For instance, the authors use IN-100 and ON-13, two subsets of the ImageNet-1k dataset which this reviewer has never seen before, and thus the results are really hard to interpret. The only results that show some measure of variance (standard deviation) are with Flowers-101.\n*Update after discussion*: Authors pointed out that IN-100 is in fact used in other works. During the discussion they also provided additional standard deviation measures for CUB-200. Although I believe that reporting standard deviation of multiple runs and/or confidence intervals for the results should be the standard practice, I acknowledge the effort made by the authors running additional experiments to accommodate this demand at least for some of the datasets that they use.\n\n- When training using all (3) data augmentations, the results in Table 4 don't suggest that LoCo improves upon MoCo in any significant way. The same applies for LoCo++ vs. MoCo++. Some of the reported accuracies are indeed slightly higher than the baseline, but the differences are small. In addition, for the only dataset for which the authors report confidence intervals, these are greatly overlapping in most of the cases.\n\n- The authors restricted their experiments to a ResNet-50. A few experiments showing that their approach works for other modern architectures would be appreciated (e.g. DenseNet, EfficientNet, Inception). \n*Update after discussion*: The authors provided additional results using a ResNet-101. I appreciate the effort of the authors running these extra experiments.\n\n- There are plenty works using contrastive losses for doing self-supervised learning, yet the authors decided to compare only against the MoCo baseline. Additional baselines would be also appreciated. This reviewer acknowledges that using MoCo would probably be sufficient if the results were significantly better. \n*Update after discussion*: The authors argue that MoCov2 is the strongest baseline at the moment of submission, which is publicly available and can run in affordable resources. This is a perfectly valid point.\n\n== Reasons for score ==\n\nI believe that the authors aim to tackle a very important question in self-supervised learning. However, the proposed approach fails to deliver the expected results, according to the results shown in Table 4. Other tables show LoCo and LoCo++ shine on top of the baseline (MoCo), but this is only \"constrained\" settings. The truth is that Table 4 contains the best numbers for the baseline, thus these are the results that LoCo(++) should improve, but it is not clear that it does so. As mentioned before, the differences are too small in most of the cases, and no measure of statistical significance is reported for most datasets. Only with Flowers-102 some measure of variance is reported (it's not clear whether the +/- show standard deviation or confidence intervals), and the intervals largely overlap in most cases.\n\n*Update after discussion*: The authors have addressed most of my concerns, although not always satisfactorily. They made a considerable effort running additional experiments to provide with additional standard deviations of the accuracy in CUB-200, as well as provided results achieved using a ResNet-101. They also clarified some of my concerns regarding the datasets used. In some situations, the benefit of the proposed approach versus already existing methods is not clear, but in others the experimental evaluation shows clears benefits. Given this, and the fact that the paper is well written and motivated, I am increasing my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oQaLkNrKPao", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors observe that, while effective, contrastive learning unavoidably introduces some bias depending on the choice of augmentations the algorithm is made invariant to, and that deteriorates performance depending on the task. The authors corroborate this hypothesis with experiments with the MoCo baseline and proceed to propose a modification to the usual contrastive learning setup: learning a shared representation and multiple projection heads, each invariant to a different augmentation type. They empirically show the effectiveness of the proposed solution on several tasks, including few-shot learning and data corruption datasets.\n\nGreat:\n* The structured approach to understanding which augmentations help and how to tackle the issue of choosing is addressing a significant issue in the contrastive learning literature. The authors thoroughly motivate their approach. While it's overall clear in the literature that contrastive learning is extremely successful at building generalizable representations, the choice of which augmentations to use is often arbitrary. The authors show empirically how making a model invariant to specific augmentations is  detrimental to some tasks (e.g. adding rotation invariance degrades 100-category ImageNet accuracy). \n* Extensive evaluation on several different tasks, showing consistent improvement on all and highlighting the flexibility of the proposed approach. The authors show results on a coarse-grained task (ImageNet 100 and iNaturalist 2019), fine-grained (CUB-100 and VGG Flowers), an augmentation-specific task (ObjectNet, real-word objects with different views and rotations) and a robustness task (ImageNet-C).\n\nQuestions:\n* The proposed method suggests that the best performance is achieved by choosing more and more augmentations. Do the authors believe there is a limit after which adding more augmentation heads makes the task impossible for contrastive learning, by asking the joint embedding space to account for too much flexibility? \n* In practice, even when using the proposed method, one still has to choose a relatively small set of possible augmentations. In light of their work, do the authors have thoughts on how should someone make that choice?\n\nOverall:\nThe paper is proposing a well empirically motivated modification to current contrastive learning methods, the methods and results are presented clearly, the experiments are extensively described. A clear accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y-rfIIEWV6Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The current mainstream way of doing contrastive learning in Instance discrimination is to train the network to associate  two independently augmented versions of the same image. The augmentation process often consists of several stages such as cropping, blurring, etc.\n\nThe authors argue that some augmentation (such as rotation and texture) are bad when used in a general augmentation pipeline. They propose to use embedding sup-spaces during Instance discrimination learning that would effectively learn from the previously hard-to-learn augmentation. The key is that in each different subspace, the definition of positive/negative samples will change.\n\nThe number of augmentation-specific embedding subspaces are correlated with the kinds of augmentation. With 2 or 3 subspaces, the authors can achieve some gain over MoCo (their baseline method) on fine grain datasets. But I wonder if the method can generalize to more subspaces and achieve gain over MoCo on larger datasets like full ImageNet?\n\nDifference between LooC , LooC++, MoCo++. The key difference is that LooC++ is returning the concat of all embedding sub-spaces features. In Table 4, the comparison between MoCo++ and LooC++ seem to suggest there is little difference in overall performance. Other than the performance difference, what is the key difference between MoCo++ and LooC++? I assume both had momentum encoders, separate queues for each subspaces, etc.\n\ntypo:\nPage 4, penultimate paragraph: “Note that[than] for both LooC and LooC++ ….”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2OoUuI7ZlvH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses potential information loss in contrastive learning, when the invariance to a variety of augmentations may be suboptimal for other downstream tasks that require the model to learn those augmentations as discriminative features (e.g: colour in fine-grained bird classification).\nThe approach, LooC, is built on the backbone of MoCo. The representations of images are projected into multiple embedding spaces, each of which is sensitive to an augmentation and invariant to others. Then a shared representation is jointly learned with multiple embedding spaces and then the shared representation or the concatenation of the learned sub-spaces, are fed to downstream tasks. \n\nStrength:\n+ Simple, yet elegant approach.\n+ Paper is well-written and easy to follow.\n+ The approach is computationally efficient, as the additional keys (compared to MoCo) are not back-propagated, and just fed to the encoder network, but it still outperforms the baselines. \n+ Results are presented on variant downstream tasks, ranging from fine- and coarse- grained classification, few-shot classification, to improved robustness on natural corruptions qualitative retrieval task.\n+ Comprehensive ablation study and experiments. \n\nComments (these points are not part of my decision assessment.):\n+ I am curious how the model can be adapted for the generalized zero shot learning as a downstream task.\n+ a very small typo in Experiments: “Note than” to “Note that” \n#######################################\nAfter rebuttal:\nThanks for your response. \nI read authors response to my question and as well as other reviewers feedback. I will keep my rating as it is.\n\n* no effect on the rating: a point on the question on the *generalized* zero-shot learning as a downstream task, is if employing this approach, improve the performance on unseen classes while is not negatively impacting the performance for seen classes. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a predominantly positive feedback from the reviewers so I recommend acceptance of the paper. It is well-written and well-motivated tackling an important problem: That in self-supervised learning one might encode different invariances by default, even if some of these invariances are useful for downstream tasks (e.g. being rotation invariant may be detrimental to predicting if an image has the correct rotation on a phone). For this, they propose a simple, yet elegant approach and validate it on many downstream tasks. Given the recent interest in self-supervised learning, this appears to be a relevant and interesting paper for the ICLR community.
Decision: Accept (Poster)