Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to add a new consistency loss term to the momentum contrast (MoCo) framework for self-supervised visual representation learning.  A common strategy for self-supervised learning, employed by MoCo as well as others, is to learn invariance to a class of transforms.  Here, a deep network is trained on an instance discrimination task: among distractor images and a transformed (e.g. via data augmentation) version of the input (query) image, correctly identify the transformed example (a classification problem).

Consistency regularization formulates an alternate objective: learn a similarity function between images and, given a gallery of images, encourage the input query and positive example (transformed variant) to have a similar distribution of similarity over the gallery.  The similarity distribution can be treated as a pseudo-label.

The paper trains a variant of MoCo that combines the standard instance discrimination loss with this consistency regularization loss.  For optimal hyperparameters of this combination, experiments show small but consistent accuracy gains over the baseline MoCo in multiple scenarios: classification and semi-supervised learning on ImageNet, as well as classification, object detection, and semantic segmentation on PASCAL.

In terms of overall impact, the contribution of this paper appears to be an incremental improvement to the current self-supervised learning paradigm.  The consistency loss itself is not a new idea, as the paper cites Sajjadi et al. (2016).  In fact, Sajjadi et al. examine consistency loss in the context of self-supervised learning, which appears to further limit the novelty of the current paper to the specific contribution of doing so with MoCo.

As another limitation, the overall gains are small and it is not clear that they necessarily make MoCo+CO2 the top method.  For example, on ImageNet classification, SimCLR outperforms MoCov2+CO2 (69.3 to 68.0 in Table 1), though SimCLR is trained for more epochs.
review_title: incremental improvement to MoCo

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
<Summary>

This paper addresses the problem of unsupervised contrastive learning for visual representation. Its key idea is to use a consistency regularization method to resolve the issue of one-hot labels for instance discrimination on which most previous work have relied. The proposed CO2 method is implemented on top of MoCo and MoCo v2 and improves their representation performance on multiple classification and detection tasks. 

<Strengths> 

1. It borrows  a simple consistency regularize method from semi-supervised learning literature to tackle one issue of recent unsupervised contrastive learning for visual representation – one-hot label that cannot discriminate the semantic closeness to a query between negative samples.

2. The proposed approach is applied to recent SOTA MoCo methods and further improves their performance on image classification, detection and semantic segmentation tasks. 

<Weakness>

1. Although this paper can clearly alleviate one issue of recent practice of contrastive learning, the technical novelty is limited. 

(1) This paper proposes a new use of an existing technique (consistence regularization) to a new problem (unsupervised contrastive learning). Given that the technique is basic and well-known in semi-supervised learning literature, the proposal of its simple use (with no extension) bears little technical novelty.

(2) In my opinion, the proposal could be a good practice for unsupervised contrastive learning, but its contribution may not be sufficient enough to make this work as a legitimate full paper on ICLR as one of the top premier ML conferences. 

2. Experimental evaluation should be improved. 

(1) The proposed approach shows the SOTA performance on multiple computer vision tasks, but it largely attributes to the strong performance of MOCO variants. 

(2) The performance gain of the proposed method over MOCO v2 is rather marginal as shown in Table 2 and 3. 

(3) The proposed CO2 is only implemented on MOCO variants. In order to show the generality of the proposed method, it should be tested with multiple contrastive learning methods and show whether it can consistently improve the methods.

<Conclusion>

My initial decision is ‘reject’ mainly because the contribution is somewhat limited and more empirical justification for the method is required. 

review_title: Review #1

final_score: rating: 7: Good paper, accept
review_content: *Summary*
This paper proposes an extension, coined as CO2, to InfoNCE contrastive loss used in semi/unsupervised methods. CO2 is based on the premise that the query-negative crop similarity distribution and positive-negative crop similarity distribution should be alike. The proposed method yields significant improvements for the linear classification protocol using ImageNet while the improvements for downstream transfer learning tasks such as object detection is marginal. Aside from the performance improvements, the paradigm where the researchers think of the pretext task as a downstream task and improve the pseudo labels by using pseudo-pseudo labels is very interesting. Proposed CO2 method provides a relatively simple way to achieve this. 


Authors build upon the intuition that among the many crops the algorithm uses as negatives, it is highly likely that at least a few positive samples exist. These unknown “positive” crops should yield a high similarity to the query, but this is not possible to enforce as these crops’ labels are, by definition, unknown. Instead the authors suggest that positive-negative similarities and query-negative similarities should be alike. A KL divergence term (between positive-negative similarity distribution and query-negative distribution) is used to implement this constraint.

*Novelty*
To the best of my knowledge this is a novel approach. It is also surprising that using pseudo-pseudo labels to correct the wrong assumptions of pseudo labels is working reasonably well.

*Impact*
I believe this paper will have a significant impact. Accuracy improvements on downstream tasks are diminished when the authors use MoCo-v2, suggesting that their method may not always yield significant benefits. However, aside from the numeric accuracy improvements on the downstream tasks, the proposed idea is very simple, seems easy to incorporate in other methods and likely opens new research directions. 

*Clarity*
The paper is well written and easy to follow. It is also generally clear but for some of my questions/comments please see below.

*Evaluation*
Authors use their loss function with MoCo and MoCo-v2 and report relatively small improvements over MoCo-v2. It is an open question whether the proposed loss function would result in large performance gains for other methods. 
On the transfer learning side, authors report marginal improvements for image classification, object detection and semantic segmentation tasks. It is not possible to form an opinion on how statistically significant these improvements are. Still, the authors provide a comparison with label smoothing which implies that CO2 is a beneficial addition.

*Strengths (Reasons to accept)*
This is a relatively simple loss function extension, applicable to other methods.
Experiments are implying that the method works well and improves upon the state of the art (under reasonable resource constraints).
As pointed out in the discussion, the paradigm of relaxing the pretext task’s label constraints (in a way, a learned label smoothing) is likely to open new research directions.

*Weaknesses (Reasons to reject)*
Transfer learning improvements are marginal (and in the object detection case CO2 results in an unexplained 0.2% drop in AP). I would expect to see a discussion about the reasons behind the discrepancy between the large improvements for semi-supervised learning (or linear classification) vs. the marginal improvements for transfer learning.
Only the object detection and semantic segmentation task experiments have been repeated (3 times). The rest of the experiments are, I believe, single runs. This makes the reader question the significance of the reported results.
A more in depth explanation for the choice of the particular loss in Eq. 4  would benefit the reader. Why use symmetric loss? Please see my questions below.

*Questions and other comments to the authors*
In Figure 1 I would like to see where the labels (both One-hot and Pseudo) come from, at least in the caption. The same is true for the similarity graphs, and the authors should consider adding axis-labels and named ticks. Finally, increasing the arrowhead sizes would make everything easier to follow. 
Even though we can not expect the authors to reproduce 1000 epoch and >4000 batch-size methods common to recent semi-supervised techniques, I would still like to see the impact of CO2 using a vanilla ResNet backbone (without the momentum encoder).
I believe in Eq. 4, the first term alone should be enough to ensure that the “learned” query extractor distribution (Q) matches the “ground truth” distribution (P). In the next few paragraphs authors state P to be dynamic, but with just the first term P would be dynamic as both Q and P depend on the same network. As such, it is not clear to me why the authors chose a symmetric divergence.
Finally, authors should refrain from using the letter P both for the distribution and for the pseudo label. This is an unnecessary overloading of notation.
review_title: A new loss function for smoothing the pseudo labels in unsupervised contrastive loss

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

This paper proposed a consistency regularization for unsupervised visual representation learning. This paper argues that the instance discrimination task performed by most contrastive learning methods merely uses one-hot labels, which cannot reflect the similarities between the query sample and negative samples. In order to tackle this problem, this paper proposed a consistency regularization method to generate pseudo labels and encourages the query sample and its positive sample to have consistent similarities to negative samples. The proposed consistency regularization method is simple and low-cost compared to clustering-based methods.


##########################################################################

Reasons for score: 

Overall, I vote for acceptance, but some concerns on implementations lower my score. The idea of consistency regularization is simple and low-cost, but achieves significant improvement on the popular MoCo baseline. My main concerns are that (1) the improvement on MoCo v2 is much smaller than MoCo while the weight of the consistency regularization term is small (10 for MoCo, 0.3 for MoCo v2), (2) the usage of symmetric KD divergence, (3) the discussion about Mean Teacher [1]. I will change my rating depending on the feedback.

 
##########################################################################

Pros: 

P1: The research problem, contrastive learning for unsupervised visual representation learning, is popular and important in the CV community.

P2: The proposed consistent contrast (CO2) method is simple but effective. Compared to clustering-based methods, CO2 is less time-consuming. 

P3: The performance of linear classification, semi-supervised learning, transfer learning is reported.

P4: The analysis with a smaller backbone and dataset is provided as an ablation study.
 
##########################################################################

Cons: 

C1: Section 3.1 states that CO2 can be easily applied to other contrastive learning mechanisms. It is acceptable to me that CO2 is evaluated with MoCo. However, it is not convincing that the application of other methods is easy. The reason is that MoCo and SimCLR have different a design of key encoders. CO2 works well with the momentum updated encoder. However, there is no evidence that CO2 also works well with the end-to-end encoder, since the query and key encoders are updated simultaneously. I doubt that this conclusion is misleading and overstating.

C2: In Section 3.1, $\alpha$ is set as 10 for MoCo while 0.3 for MoCo v2. The improvement of CO2 over MoCo is 2.9% while only 0.5% for MoCo v2. It is not clear why $\alpha$ is set at different levels and why the improvement over MoCo v2 is not significant.

C3: This paper is closed to Mean Teacher [Tarvainen & Valpola, 2017]. The contrastive loss (instance discrimination) is like the classification cost in Mean Teacher, while the symmetric KL Diverfebce (consistency regularization) is like the consistency cost. The comparison, especially the difference, between CO2 and Mean Teacher should be discussed.

##########################################################################

Questions during the rebuttal period: 

Q1: It is not clear to me why a symmetric KD divergence (i.e., D(P|Q)+D(Q|P)) is used in Eq. (4) rather than asymmetric KD divergence (i.e., D(Q|P)). In MoCo, the encoder for computing keys is momentum-updated rather than updated by back-propagation. Therefore, Eq.(4) becomes

$\mathcal{L}_{con}=\frac{1}{2}\sum^K_i-P(i)\log Q(i)+Q(i)\log Q(i)-Q(i)\log P(i)$

Since the gradient of $P(i)$ is stopped, $P(i)\log P(i)$ is ignored. If the gradient of $Q$ in $D_{KL}(Q||P)$ is stopped, then $D_{KL}(Q||P)$ does not contribute to back-propagation. If not, $Q(i)\log Q(i)$ works like a regularization term as minimizing $Q(i)\log Q(i)$ may encourage a sharp $Q(i)$. I wonder whether the sharpness implicitly boosts the performance. I hope that more explanation about the symmetric KD divergence, especially (1) why "symmetric" rather than only $D_{KL}(Q||P)$. If "symmetric" is necessary, additional ablations and references/citations will help. (2) Whether $Q(i)\log Q(i)$ is minimized. If so, more explanation is expected.

Q2: In section 2.2, Eq. (2) and (3) only consider the similarity between the query and negatives. I wonder why the similarity between the query and positive is ignored. Although the latter is used in Eq.(1), I wonder whether there is any difference if the similarity between the query and positive is also considered in Eq. (2) and (3).
 
#########################################################################

Suggestions:

S1: In Page 3, this paper states that $q$, $p$ and $n_k$ shares the same encoder $f_\theta$. However, $q$, $p$ and $n_k$ are obtained with different encoders using MoCo. This part could be carefully revised to be more general.

#########################################################################

References:

[1] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.






------------------------------------------------------------------------------

Thank the authors for their good job during rebuttal.

Most of my questions have been addressed. However, my main concern is still that the improvement of CO2 on MoCo v2 is incremental. During the rebuttal stage, the authors did not include more empirical evidence on whether CO2 can improve MoCo v2 well, or theoretical analysis on why the improvement is incremental, both of which are accepteable to me. This severely limits the contribution of this paper. In all, I would not change my score.
review_title: Good Submission with Some Concerns

Meta-review: This paper presents a simple yet effective approach to improve self-supervised contrastive approaches like MoCo. There are concerns with respect to novelty/simplicity and low improvements over MoCov2. AC believes that simplicity is good and while gains might not be as huge, they still show usefulness of new loss. It might also provide insights for future papers on self-supervised learning. Overall, the sentiment is that paper is above the bar.
Decision: Accept (Poster)