Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a loss function for unsupervised representation learning. It has two terms: an instance classification loss and a consistency loss. The novel part seems to be the consistency loss. It explicitly penalizes the dissimilarity between different views of the same instance.

Good paper, accept

+The proposed method is supported with extensive experiments. The proposed consistency loss which is basically cos similarity between different views of the same sample seems to be novel.

-The proposed term L_C is kind of redundant. The proposed term L_C explicitly pushes positive samples to come closer based on cos similarity.
But, the other term, L_S, already bring positive samples closer and repels the negative samples. Here by positive, I mean samples that are different views of the same instance. Likewise, by negative I mean samples that are from different instances. That being said, explicitly adding L_c term with hyperparameter \alpha provided more fixability in balancing the push/repel phenomenon by tuning the hyper-parameter \alpha. 

-Compared to the state-of-the-art methods reported in Table 2, the improvements are really really minor. In fact, in most columns, there is no meaningful improvement. 

minor comment:
w/ stronger setup --> "with" stronger setup

review_title: seems to be novel

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the instance classification solution for an unsupervised representation learning problem. Particularly, this paper proposes an additional consistency loss that is simultaneously optimized with classification loss, in order to penalize feature dissimilarity between augmented views of the same instance. Such consistency loss makes classification loss optimization easier and avoids large batch size. Extensive experiments on downstream tasks, e.g., segmentation and detection, show the effectiveness of the proposed method.

Compared with previous instance classification solutions, the major contribution is that this paper introduces a consistency loss on augmented views from the same instance. The idea that compacts the augmented views is not new. And, the theoretical contribution of the proposed consistency loss is limited, which just penalizes the cosine dissimilarity. Thus, it is far from a "decent" ICLR paper.

review_title: The contribution is not significant.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes adding an additional loss term to instance classification, within the context of self-supervised pre-training.  Specifically, in addition to the standard classification loss term that views each image (and its augmentation) as a separate category, a second loss term is added, which is 1 minus the inner product of the representations for two augmentation views of the same image.  In a sense, this is incorporating an aspect of the contrasting learning approach, as two views of the same image are being explicitly pulled toward one another due to the consistency loss.  

Positives:
+ sensitivity analysis with respect to the balancing coefficient $\alpha$ between the classification loss and consistency loss demonstrates that there is an optimal $\alpha$ such that the consistency loss leads to a minimum of the classification loss, even compared with setting $\alpha=0$.  In other words, the consistency loss is helping to optimize the classification loss.
+ related to above, empirical results and ablation study demonstrates the benefit of adding in consistency loss for subsequent downstream tasks
+ qualitatively, adding the consistency loss seems to help the network focus on relevant/textured regions of the images

Negatives:
- Novelty is arguably somewhat limited, as the proposed method boils down to adding a term that encourages similarity between two views of the same image, very similar to contrastive learning.
- Empirical results are mixed - in general on-par with existing methods.  Performance is generally consistent across the explored tasks, but does not appear to be significant better than existing approaches.  One data point that would be of interest is whether the proposed approach does better when training for more epochs, similar to methods such as SwAV or MoCo v2 on ImageNet linear evaluation.

Overall summary: 

Given the negatives mentioned above, I currently see the paper as being borderline/marginally below threshold.
review_title: consistency loss improves instance classification; results on-par with existing methods

Meta-review: This paper introduces a consistency loss for instance discrimination by adding a term to maximize the squared dot product between two views of the same image. The impact of the proposed approach is evaluated on a variety of settings with mixed improvements. While reviewers generally found the proposed method to be interesting, there were concerns regarding the novelty of the approach, the size of the performance improvement, and the choice to focus on instance discrimination vs. more recent approaches based on contrastive instance discrimination. 

While I do not share the reviewer concerns regarding novelty, I am sympathetic to the concerns regarding the size of the improvement and the focus on instance discrimination. As such, I recommend that the paper be rejected in its current form. I would encourage the authors to apply their analysis and method to more recent contrastive instance discrimination approaches such as SimCLR and SWaV as well as non-explicitly contrastive, but high performing methods like BYOL. I would also encourage the authors to provide quantitative empirical analyses demonstrating the impact of the consistency term on large models rather than just toy models to demonstrate the impact of the consistency term in representational space. 
Decision: Reject