Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**:
This paper designs a new loss, called SuNCTt, to speed up the convergence of semi-supervised training. Specifically, the loss involves the computation of similarity between anchor and other images with the same class, and the similarity between anchor and other labeled images. It is claimed to be considered as the form of neighborhood component analysis. Together with the standard contrastive learning loss, it only uses less than half the amount of pre-training and computes to match the accuracy of the previous approaches.

**Pros**:
+ The whole idea makes sense. The comprehensive experiments, also shown in appendix, support claims in the paper that the + introduced loss is helpful for semi-supervised training  
+ Overall, the paper is well written and the results part is well structured.


**Concerns**:
- It mentions some semi-supervised work in the related work. Is it possible to compare the results with theirs?
- It would be good to show the results of using cross-entropy pre-trained on ImageNet. In addition to the saving compute, I am also curious about the final performance on ImageNet after training for 1000(500) epochs. 
- On page 5, it mentioned that, with 1% labeled data, it is not significantly greater than the random accuracy. However, in appendix F (figure 7b), the result of 1% data(CIFAR-10) looks good.  I wonder why it is, if I understand the contexts correctly.
- I also wonder, for the labeled data, what will happen if we use explored metric learning methods, like triplet loss. For the triplet loss, we can leverage labels to define the positive and negative samples. Both triplet loss and SuNCTt may do the similar thing, but in different forms.

Overall, I prefer the rating as above the threshold at the current stage. Hope the authors could address my concerns or questions in the rebuttal period.



review_title: Involving the labeled samples in pre-training to speed up the contrastive semi-supervised learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new method applicable to a specific case in unsupervised learning: having access to a small amount of labels during the training phase where label signal is not used.

First, I would like to say this is a well motivated task. Many recent works on self-supervised learning (image classification using instance discrimination signals) eventually need to use all the labels in the linear evaluation phase to obtain the final performance numbers. The authors also point out a similar view in the last paragraph in Section 3. 

In the related works section, the work by Khosla et al. 2020 is mentioned. I wonder if it makes sense for the authors to add their numbers in the experiment comparison given that Khosla et al also use the original SimCLR as benchmark? In addition, I think a "soft-nearest neighbor loss" paper could be cited and compared (for example: Zhirong Wu, Alexei A Efros, and Stella Yu. Improving generalization via scalable neighbor- hood component analysis. In European Conference on Computer Vision (ECCV) 2018, 2018.)
I understand that most supervised contrastive learning frameworks assume full label during the entire process, which is different from the setting in the paper. But it would be interesting (even critical) to compare the proposed method vs the other supervised contrastive learning methods when SuNCEt is given the full data.

The main contribution of the paper is the proposed SuNCEt loss, which is modified on top of the regular NCE loss in instance discrimination training. However, this “supervised constrastive learning“ only accelerates the SIMCLR learning process (by ~2x in terms of epochs from Table 1 and Table 2) and it does not significantly improve the accuracy over regular SIMCLR.

Overall, I feel that the proposed loss is not a very novel idea (i.e., supervised contrastive learning) and the experiment results are not significantly better than prior arts.

review_title: well motivated but needs more work and comparison

final_score: rating: 4: Ok but not good enough - rejection
review_content: Pros:

1. This submission is well written with lots of experiments. The claim is that supervised contrastive learning can speed up representation learning, which is well supported. 

Cons:

1. Instead of proposing a new idea, this paper adopts supervised contrastive learning and discovers that supervision can help accelerate the pretraining stage. Despite it maybe practical to speed up the experimental cycle, the technique contribution is rather limited. I may understands this paper wrong, so could authors clarify the propsoed SuNCEt loss with supervised contrastive loss? 

2. Since this paper is investigating semi-supervised learning, then some comparisons to existing literature is necessary. For example, comparisons to FixMatch. Right now, all the comparisons are only done on SimCLR, which are basically ablation studies.

3. Performance improvement on CIFAR is marginal. I read the limitation section in Appendix, but not clear how it is related to the final results. 

4. I notice that the proposed SuNCEt loss is turned off after some training epochs. Especially when using 1% of the labeled data, this loss is turned off at epoch 30, which is quite early in the learn stage. Does SuNCEt loss hurt the training and why? And in practice, how do you determine the optimal time to turn off SuNCEt loss if you don't have access to all the labeled validation set? 

In conclusion, despite the paper has some interesting results, it lacks of experiments to show its real contributions.
review_title: This submission discovers that supervised contrastive learning can speed up representation learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper combines the self-supervised contrastive loss with the supervised contrastive loss for semi-supervised learning. By leveraging a small amount of labeled data, this paper shows that the semi-supervised contrastive loss can achieve similar performance as self-supervised contrastive loss (SimCLR) with less than half of the compute.

Strength:
1. This paper is well-motivated. It is a good direction to explore contrastive learning in the semi-supervised learning setting.
2. It is interesting to see that using a small amount of labeled data can reduce computation.
3. The paper is well-written and easy to understand.

Weakness:
1. The proposed semi-supervised contrastive loss seems to be a straight-forward combination of the self-supervised contrastive loss and the supervised contrastive loss [Khosla et al. 2020]. Therefore, the technical novelty is limited. 
2. This paper does not compare with any of the existing semi-supervised learning methods (e.g. FixMatch), even though they are discussed in the related work. Self-supervised learning has been known to be computation expensive, but this paper also needs to justify that the proposed method is more efficient than existing semi-supervised learning methods.
3. Figure 2 shows that combining CE with SimCLR already achieves good performance. Therefore, a simple baseline would be combining FixMatch with SimCLR. I would expect it to be comparable or even better than the proposed contrastive learning method, because previous papers (e.g. S4L) have shown that adding a self-supervised objective helps with semi-supervised learning.
4. The improvement over SimCLR does not seem to be significant under the same training epochs. Given that labeled data is used, this improvement is mostly expected.  Furthermore, the supervised contrastive loss is turned off after some epochs, which means it plays a less important role. If SuNCEt is also trained for 1000 epochs, would it converge to a similar result as SimCLR?
5. In Table 2, it would be interesting to also see the performance of fully-supervised learning.
review_title: Official Blind Review #3

Meta-review: The paper proposes to speed-up self-supervised learning for semi-supervised learning by combining self-supervised pretraining and supervised fine-tuning into a single objective. The proposed supervised loss builds on Neighbourhood Components Analysis and soft nearest neighbor losses. Most reviewers are concerned about the novelty of the approach and the significance of empirical results. I agree with both concerns. I appreciate the comparison between $\log \sum \exp$ and $\sum \log \exp$, but it seems a simpler cross entropy loss also achieves a similar goal (potentially somewhat slower). I believe adding more experiments comparing different supervised loss functions across different architectures can help improve the paper.

Decision: Reject