Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper focuses on designing more effective ways for contrastive learning. The author claims that stronger augmentations are beneficial for better representation learning. Different from directly applying the stronger augmentations to minimize the contrastive loss, the author proposes to minimize the distribution divergence between the weakly and strongly augmented images. The experimental evaluations are conducted on ImageNet classification and related downstream tasks, and the results are promising.

Clarity:
1. The method is very simple and straightforward. My main concern is the experimental comparisons. As we all know, contrastive learning algorithms like MOCO and SimCLR benefits from longer training epochs a lot (for example, training with 800 epochs is much better than with 400 epochs). Thus I think the comparisons in Table 2 are not convincing. From algorithm 1, we can find that the equivalent batch size of the proposed CLSA method is two times as classical MOCO method. Thus I would prefer to check the results of CLSA at epochs 100 and 400 for fair comparisons.
2. What is the value of the balancing coefficient? It would be nice if some ablation results are provided.
review_title: The experimental evaluations are not convincing

final_score: rating: 7: Good paper, accept
review_content: 
This paper proposes the better utilization of strong data augmentations for contrastive loss functions in unsupervised learning. In Moco set up, typically, weaker augmentations such as color jittering, cropping is applied to construct positive pairs from the same image. In this study, by proposing a modified objective, the authors leverage stronger data augmentations to construct more challenging positives and negatives pairs to improve the quality of the representations. The paper delivers a novel objective together with leveraging existing strong augmentations to improve downstream performance. The authors can find my questions/concerns listed below.

1. The paper is overall well-written, however, it is disappointing to see many typos grammar mistakes throughout the paper. Some examples are in "Thus we proposed the CLSA (Contrastive Learning with Stronger Augementations)", "to train an unsupervised representation", "The contrastive learning (Hadsell et al. (2006)) is a popular self-supervised idea".

2. In section 3.1, the authors mention that the keys in the memory bank is managed with first in first out method. Is it not supposed to be first in last out? I would like to see some clarification on this.

3.  The numerator in Equation 3 should be z_i' vs. z_i not z_k.

4. The authors claim that in He et al. an input image is resized and cropped to 224×224 pixels. It should be "an image is first cropped from an input image and resized to 224x224 pixels."

5. In the experiments section, the authors list other methods including MoCo, SimCLR, MoCo-v2, BYOL and compare to what they propose. As a baseline, it would be nice to directly use the stronger augmentations in MoCo-v2 objective and perform comparison to their method. Throughout the paper, the authors claim that strong augmentations hurt the learned representations due to distorted images. It would be meaningful to show this experimentally as well.

6. The authors explain that they choose a strong transformation randomly from the given 14 transformations and repeat it 5 times to strongly augment an image. Is the sampling done without replacement? In other words, do the authors choose 5 unique transformations with the corresponding magnitude and apply those transformations to a single image?

7. I like how the authors point the similarity of their objective to knowledge distillation. In this case, strong augmentations are assigned probability of being a positive pair from the positive pair constructed with weak augmentations. It helps to understand the full picture for the proposed method.

8. Finally, I think the figure 3 is confusing rather than being helpful. Both weak and strong augmentations go to the memory bank and it looks like two distributions come out of nowhere in the figure. It would be more clear to point out that there is distribution of the representations from the strong augmentations and weak augmentations and they supervise the assignment for strong augmentations given predictions on the weak augmentations.
review_title: CONTRASTIVE LEARNING WITH STRONGER AUGMENTATIONS

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method to incorporate stronger augmentations into the visual representation contrastive learning framework. Specifically, three correlated views of an image are first generated by using two weak and one strong augmentation operations on the same image. Then, the networks are trained to maximize the agreement between the two weak views and also to minimize the distribution divergence between a weak view and the strong view. The method is evaluated on several visual tasks including classification,  transfer learning, and object detection, with the standard evaluation protocol for self-supervised learning, and the results are promising.

Pros:

1. This paper is well-structured and easy-to-follow.
2. The idea of utilizing strong augmentations for contrastive learning is interesting and novel to me, and the results are promising.
3. The proposed framework seems general which might be easily incorporated into the existing contrastive learning frameworks.

Cons:

1. The motivation about using stronger augmentations is not well justified. Specifically, the authors propose to use stronger augmentations based on two reasons: (1) stronger augmentations can expose some novel useful patterns; (2) the effectiveness of stronger augmentations is proved in the semi-supervised learning and supervised learning field. However, no related papers are provided to support the first point, while the papers (Cubuk et al. (2018)); Qi et al. (2019); Wang et al. (2019)) that are cited to support the second point do not explicitly make relevant conclusions. (Chen et al. (2020a)) even demonstrate that when training supervised models, stronger color augmentation hurts their performance. I would like to see a more comprehensive review of related works to clarify the motivation.

2. In addition, some important ablation studies are missing in the experiment. E.g.,  how does the performance change as the magnitude or usage times of stronger augmentations changes?

3. The proposed DDM loss seems general for different contrastive learning frameworks. I would like to see if it still works when applied to other frameworks, e.g., SimCLR, InfoMin?

Overall, given the novelty and strong results of the proposed framework, I remain positive towards this paper. I will be happy to increase my rating if my concerns are addressed in the rebuttal period.

review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:\
This work investigate the recent popular direction of unsupervised representation learning using contrastive loss between augmented images. Authors propose to minimize the divergence between the distributions of strongly augmented vs. weakly augmented images. The method reaches competitive performance in recognition and object detection.
-----
+Strengths\
+The main idea is well motivated: that strong augmentation reveal useful cues in visual representation learning but has not been successfully exploited in unsupervised learning.\
+The proposed solution is novel within contrastive learning to my best knowledge.\
+Results are extremely strong.
-----
-Concerns\
-The divergence between the two conditional distributions can be a moving target since they are trained jointly. It is not clear if this is will result in stable learning for the unsupervised setting, and what effect that may have on the performance and quality of the representations.\
-Evaluation only focus on final result and lacks analysis of the proposed method, especially when compared to recent paper of similar nature published in top conferences. For example, strong augmentation is a focus of this paper, but there are no ablation regarding the augmentations. Is the performance sensitive to the choice of strong augmentation?\
-The paper could also use some more theoretical analysis to address some of the weaknesses stated above.
-----
Recommendation\
I like the proposed idea. It is novel and interesting and seems to achieve good results. However the lack of both theoretical and empirical analysis beyond results on performance raises many questions. As a result I am on the fence but leaning towards accept.
review_title: Official Blind Review #2

Meta-review: This paper improves MoCo-based contrastive learning frameworks by enabling stronger views via an additional divergence loss to the standard (weaker) views. Three reviewers suggested acceptance, and one did rejection. Positive reviewers found the proposed method is novel and shows promising empirical results. However, as pointed out by the negative reviewer, the paper should have clarified about computational overheads of the method compared to the baseline (MoCo) in several aspects, e.g., their effective batch sizes or training costs, for the readers’ better understanding. As the concern was not fully resolved during the discussion phase, AC is a bit toward for reject. AC thinks the paper would be stronger if the authors include more ablations (and the respective discussions) regarding this point, e.g., CLSA-multi (and -single) vs. MoCo-v2 under the same training time, both at early epochs (~200; as reported in the author response) and longer epochs (after convergence; ~1000 and even more). 
Decision: Reject