Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: == Summary ==

The paper proposes a contrastive learning approach for self-supervised learning in which multiple heads are trained to be invariant to all but one type of data augmentation. The rationale is that different downstream tasks may require different types of invariances (e.g. we may want to be rotation invariant for pictures of flowers, but not for pictures of animals), and one does not know a-priori which kind of invariances will be required. After training multiple representation heads, one can later concatenate them or use the general embedding (the input to all the variant-specific heads) for the downstream task.

== Pros ==

- The authors try to tackle an important problem of self-supervised approaches: how does one decide which data augmentation strategies to use when the downstream task is not known in advance? This question has not been properly addressed in the literature, and can be of great important for the real application of self-supervised strategies beyond academic benchmarks.

- The method that the authors introduce scales well with the number of data augmentations used (linear), and avoid a combinatorial explosion that could arise.

- The paper is generally well written and the algorithm is explained quite clearly, and illustrations are used appropriately to help the reader understand the proposed approach (Figure 1 and Figure 2).

== Cons ==

- Once the self-supervised pre-training finishes, one has to decide whether to use the general embedding space in the downstream task, or a concatenation of the different variant-specific embeddings. However, from the results reported in Table 2, 3 and 4 it's not clear which approach is better, and this is downstream task-dependent. This is unfortunate since it basically introduces another hyperparameter to tune for each downstream task. 

- Many experiments do not report any measure of variance or statistical significance, and do not follow an an "standard" setting. This makes really hard to tell whether the observed increase in accuracy is statistically significant or not. For instance, the authors use IN-100 and ON-13, two subsets of the ImageNet-1k dataset which this reviewer has never seen before, and thus the results are really hard to interpret. The only results that show some measure of variance (standard deviation) are with Flowers-101.
*Update after discussion*: Authors pointed out that IN-100 is in fact used in other works. During the discussion they also provided additional standard deviation measures for CUB-200. Although I believe that reporting standard deviation of multiple runs and/or confidence intervals for the results should be the standard practice, I acknowledge the effort made by the authors running additional experiments to accommodate this demand at least for some of the datasets that they use.

- When training using all (3) data augmentations, the results in Table 4 don't suggest that LoCo improves upon MoCo in any significant way. The same applies for LoCo++ vs. MoCo++. Some of the reported accuracies are indeed slightly higher than the baseline, but the differences are small. In addition, for the only dataset for which the authors report confidence intervals, these are greatly overlapping in most of the cases.

- The authors restricted their experiments to a ResNet-50. A few experiments showing that their approach works for other modern architectures would be appreciated (e.g. DenseNet, EfficientNet, Inception). 
*Update after discussion*: The authors provided additional results using a ResNet-101. I appreciate the effort of the authors running these extra experiments.

- There are plenty works using contrastive losses for doing self-supervised learning, yet the authors decided to compare only against the MoCo baseline. Additional baselines would be also appreciated. This reviewer acknowledges that using MoCo would probably be sufficient if the results were significantly better. 
*Update after discussion*: The authors argue that MoCov2 is the strongest baseline at the moment of submission, which is publicly available and can run in affordable resources. This is a perfectly valid point.

== Reasons for score ==

I believe that the authors aim to tackle a very important question in self-supervised learning. However, the proposed approach fails to deliver the expected results, according to the results shown in Table 4. Other tables show LoCo and LoCo++ shine on top of the baseline (MoCo), but this is only "constrained" settings. The truth is that Table 4 contains the best numbers for the baseline, thus these are the results that LoCo(++) should improve, but it is not clear that it does so. As mentioned before, the differences are too small in most of the cases, and no measure of statistical significance is reported for most datasets. Only with Flowers-102 some measure of variance is reported (it's not clear whether the +/- show standard deviation or confidence intervals), and the intervals largely overlap in most cases.

*Update after discussion*: The authors have addressed most of my concerns, although not always satisfactorily. They made a considerable effort running additional experiments to provide with additional standard deviations of the accuracy in CUB-200, as well as provided results achieved using a ResNet-101. They also clarified some of my concerns regarding the datasets used. In some situations, the benefit of the proposed approach versus already existing methods is not clear, but in others the experimental evaluation shows clears benefits. Given this, and the fact that the paper is well written and motivated, I am increasing my score.
review_title: Tackling an important problem, not achieving the expected results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: 
The authors observe that, while effective, contrastive learning unavoidably introduces some bias depending on the choice of augmentations the algorithm is made invariant to, and that deteriorates performance depending on the task. The authors corroborate this hypothesis with experiments with the MoCo baseline and proceed to propose a modification to the usual contrastive learning setup: learning a shared representation and multiple projection heads, each invariant to a different augmentation type. They empirically show the effectiveness of the proposed solution on several tasks, including few-shot learning and data corruption datasets.

Great:
* The structured approach to understanding which augmentations help and how to tackle the issue of choosing is addressing a significant issue in the contrastive learning literature. The authors thoroughly motivate their approach. While it's overall clear in the literature that contrastive learning is extremely successful at building generalizable representations, the choice of which augmentations to use is often arbitrary. The authors show empirically how making a model invariant to specific augmentations is  detrimental to some tasks (e.g. adding rotation invariance degrades 100-category ImageNet accuracy). 
* Extensive evaluation on several different tasks, showing consistent improvement on all and highlighting the flexibility of the proposed approach. The authors show results on a coarse-grained task (ImageNet 100 and iNaturalist 2019), fine-grained (CUB-100 and VGG Flowers), an augmentation-specific task (ObjectNet, real-word objects with different views and rotations) and a robustness task (ImageNet-C).

Questions:
* The proposed method suggests that the best performance is achieved by choosing more and more augmentations. Do the authors believe there is a limit after which adding more augmentation heads makes the task impossible for contrastive learning, by asking the joint embedding space to account for too much flexibility? 
* In practice, even when using the proposed method, one still has to choose a relatively small set of possible augmentations. In light of their work, do the authors have thoughts on how should someone make that choice?

Overall:
The paper is proposing a well empirically motivated modification to current contrastive learning methods, the methods and results are presented clearly, the experiments are extensively described. A clear accept.

review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: The current mainstream way of doing contrastive learning in Instance discrimination is to train the network to associate  two independently augmented versions of the same image. The augmentation process often consists of several stages such as cropping, blurring, etc.

The authors argue that some augmentation (such as rotation and texture) are bad when used in a general augmentation pipeline. They propose to use embedding sup-spaces during Instance discrimination learning that would effectively learn from the previously hard-to-learn augmentation. The key is that in each different subspace, the definition of positive/negative samples will change.

The number of augmentation-specific embedding subspaces are correlated with the kinds of augmentation. With 2 or 3 subspaces, the authors can achieve some gain over MoCo (their baseline method) on fine grain datasets. But I wonder if the method can generalize to more subspaces and achieve gain over MoCo on larger datasets like full ImageNet?

Difference between LooC , LooC++, MoCo++. The key difference is that LooC++ is returning the concat of all embedding sub-spaces features. In Table 4, the comparison between MoCo++ and LooC++ seem to suggest there is little difference in overall performance. Other than the performance difference, what is the key difference between MoCo++ and LooC++? I assume both had momentum encoders, separate queues for each subspaces, etc.

typo:
Page 4, penultimate paragraph: “Note that[than] for both LooC and LooC++ ….”

review_title: interesting findings regarding augmentation vs embedding subspaces.

final_score: rating: 7: Good paper, accept
review_content: The paper addresses potential information loss in contrastive learning, when the invariance to a variety of augmentations may be suboptimal for other downstream tasks that require the model to learn those augmentations as discriminative features (e.g: colour in fine-grained bird classification).
The approach, LooC, is built on the backbone of MoCo. The representations of images are projected into multiple embedding spaces, each of which is sensitive to an augmentation and invariant to others. Then a shared representation is jointly learned with multiple embedding spaces and then the shared representation or the concatenation of the learned sub-spaces, are fed to downstream tasks. 

Strength:
+ Simple, yet elegant approach.
+ Paper is well-written and easy to follow.
+ The approach is computationally efficient, as the additional keys (compared to MoCo) are not back-propagated, and just fed to the encoder network, but it still outperforms the baselines. 
+ Results are presented on variant downstream tasks, ranging from fine- and coarse- grained classification, few-shot classification, to improved robustness on natural corruptions qualitative retrieval task.
+ Comprehensive ablation study and experiments. 

Comments (these points are not part of my decision assessment.):
+ I am curious how the model can be adapted for the generalized zero shot learning as a downstream task.
+ a very small typo in Experiments: “Note than” to “Note that” 
#######################################
After rebuttal:
Thanks for your response. 
I read authors response to my question and as well as other reviewers feedback. I will keep my rating as it is.

* no effect on the rating: a point on the question on the *generalized* zero-shot learning as a downstream task, is if employing this approach, improve the performance on unseen classes while is not negatively impacting the performance for seen classes. 
review_title: Elegant approach and well-written paper

Meta-review: There was a predominantly positive feedback from the reviewers so I recommend acceptance of the paper. It is well-written and well-motivated tackling an important problem: That in self-supervised learning one might encode different invariances by default, even if some of these invariances are useful for downstream tasks (e.g. being rotation invariant may be detrimental to predicting if an image has the correct rotation on a phone). For this, they propose a simple, yet elegant approach and validate it on many downstream tasks. Given the recent interest in self-supervised learning, this appears to be a relevant and interesting paper for the ICLR community.
Decision: Accept (Poster)