Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an elegant way of combining pseudo labels (semantic similarity) and instance similarity. This method achieves state of the art performance on 1% and 10% ImageNet settings using ResNet-50 architecture. Transfer learning and OOD generalization experiments show the usefulness of this approach.  
strength_and_weaknesses: **Strengths**  
_S1_. The paper provides an elegant solution to combine pseudo labels (semantic similarity) and instance similarity while learning visual representations. This approach is more elegant than SimMatch (Zheng et al, 2022), beating their performance with lesser number of training epochs.   
_S2_. It achieves state of the art on 1% and 10% ImageNet settings using ResNet-50 architecture.  
_S3_. The paper beats the transfer learning performance w.r.t. SimMatch (Zheng et al, 2022).


**Weaknesses**  
As such the paper is well written with a good set of ablation studies. But certain things aren't clear.  
_W1_. The paper says that is uses the Relic objective (Mitrovich et. al., 2021). It is not clear how important this objective would be in terms of the performance gain. This objective could be used for the competing approaches as well.  
_W2_. It would be good to have the precision and recall plots of the pseudo label quality (Fig. 2) for the competing approaches. That would give good insight into whether there is a correlation with this elegant way of optimizing to the quality of pseudo labels generated. It would also justify the point the authors are marking about the virtuous cycle of learning better representations while improving quality of pseudo labels. Currently, that point has not been justified empirically. We know the approach helps learning better representations. 
_W3_. It would be good to also show how this approach does on limited annotation settings introduced in PAWS (Assran et. al., 2021) and Masked Siamese Networks (Assran et. al., ECCV 2022) with 1-10 annotations per class. 
clarity,_quality,_novelty_and_reproducibility: The paper has written in a clear and precise manner. 

While this idea of combining the pseudo labels while learning visual representations during contrastive learning has been introduced in SimMatch (Zheng et al, 2022), the method discussed in this paper is more elegant and original. 
summary_of_the_review: The paper provides an elegant solution to the idea introduced by SimMatch (Zheng et. al. 2022). It is well written and there is extensive evaluation done w.r.t. semi-supervised learning approaches in the literature, along with transfer learning and OOD generalization benchmarks. The novelty of the idea is not large. But it is an interesting contribution to the literature.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper adapts the contrastive learning from self-supervised learning to semi-supervised learning with two core designs: 1) performing pseudo-labeling based on the similarities to the labeled data in the encoded feature space; 2) select positives for a datapoint based on the pseudo-labels.

Pros: The idea is simple and extensive experiments demonstrate its effectiveness.
Cons: The novelty of the proposed method is relatively limited. 

strength_and_weaknesses: 1) The idea is straightforward and sound.
2) The experiments are sufficient to demonstrate its effectiveness. 
3) paper is written well and easy to follow.
clarity,_quality,_novelty_and_reproducibility: 1) My major concern is that the novelty is limited in that: 1) performing pseudo-labeling using KNN based on semantic similarities between unlabeled data and labeled data has been previously studied in few-shot learning or semi-supervised learning; 2) Using obtained pseudo labels to construct positive training pairs for contrastive learning is reasonable but also not quite novel.
For me, the novelty may lie in the whole framework, which performs three tasks jointly, including learning  the representation, predicting pseudo-labels and selecting semantic positives, to optimize the whole model. 
summary_of_the_review: While the idea is simple yet effective and the experiments are extensive, the novelty is limited, not adequate for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends contrastive learning to semi-supervised settings. To do so, it estimates pseudo-labels for the unlabeled data during the training process of contrastive learning and adds a supervised contrastive loss according to the labels of the labeled data as well as the pseudo-labels of the unlabelled data to the original contrastive loss. Experiments demonstrate that it achieves a new state-of-the-art performance in semi-supervised benchmarks and has better robustness and generalizability. 
strength_and_weaknesses: Pros:

- Interesting topic. It is interesting and important to utilize the supervised information in semi-supervised setting rather than a simple cross-entropy objective. This paper proposes to use that information to help decide the similarity relationships in the embedding space for contrastive learning.
- Good experiments. The proposed method outperforms state-of-the-art semi-supervised learning methods on ImageNet with 1% and 10% labels. The analytical and supplemental experiments are also comprehensive.

Cons:

- The proposed method is naive. It just uses the nearest labeled datapoint as the pseudo-label of each unlabeled datapoint and then adds a supervised contrastive loss using the labels and the pseudo-labels directly. Moreover, the description is confusing. For example, what’s the purpose of first retrieving the k-nearest neighbors and then finding the nearest one from the k datapoints (Equation 2)? In 2.1 Algorithm parameters, what does it mean by |a| = 4 (what is a?) and why are there 16 pseudo-labels for each unlabeled datapoint?

- The diversity of the datasets used is too low. It compared with previous methods for semi-supervised learning only on ImageNet, which makes the empirical evaluation less convincing. Most of the baselines included in the tables of this paper (e.g., DebiasPL [Wang et al., 2022], SimMatch [Zheng et al., 2022], etc) present the results on other datasets (e.g., CIFAR) in addition to ImageNet. This is no reason to exclude those results and compare the results on only one dataset.

- This is no computation cost analysis in the paper. Since the proposed method has much more steps than the traditional contrastive learning framework, it would be helpful to evaluate how much computation overhead it brings. Comparison to other semi-supervised methods (e.g., as SimMatch [Zheng et al., 2022] does) is also necessary.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved. As mentioned above in Cons, there are several confusing descriptions. 

The novelty of the proposed method is marginal. Using the nearest labeled datapoints in the embedding space as pseudo-labels of unlabeled datapoints is a common practice in the semi-supervised setting. 

Most of the details are included in the paper, so the proposed method should be reproducible.
summary_of_the_review: This paper addresses an interesting topic, i.e., how to utilize the supervised information in the semi-supervised setting to help contrastive learning to learn a better embedding space. However, as discussed above, the proposed method lacks novelty and some experiments are missing. Therefore, I feel this paper is below the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: There are two successful directions in DL nowadays to build models based on both labeled and large amount of unlabeled data: pseudo-labeling (semi-supervised) and SSL (self-supervised learning) with further fine-tuning. There are few papers which try to properly combine both SSL and supervised loss right from the beginning of training showing that maybe we should use labeled data earlier to ground ourselves or guide the training. Current paper is extending SSL with positive samples mining: we use supervised data to generate pseudo-labels based on KNN in representational space to find closest labeled data and use their label as pseudo-label. The pair is positive if their labels/pseudo-labels are the same. In this way SSL is grounded by some extra knowledge from labeled data. Also pseudo-labeling is used in non-standard way as we don't compute cross entropy on pseudo-labels but we use them to decide what is positive sample. Authors probe this idea with resnet model and 1/10% of labeled data from ImageNet. They achieve SOTA results with for resnet architecture. In extra ablations authors showed that the method works for large resnet-based models, for out-of-distribution generalization and transfer learning.
strength_and_weaknesses: **Strength**
- I like a new idea proposed in the paper in terms of how to perform positive mining and use some pseudo-labeling for this.
- Clear presentation and extensive ablations of the parameters like queue size, KNN size, OOD and domain transfer, large architecture, etc.
- SOTA results for 1/10% of supervision
- Analysis of pseudo-labels quality and how it changes over the training


**Weaknesses**
Part of weaknesses is just clarification into the main text which should be done to resolve some ambiguities. Others are some extra experiments to better understand the contribution on top of simCLR and strongly demonstrate that adding labeled data from the beginning helps SSL loss.
- there are no ablations on how penalty influences training and what is its contribution in vanilla simCLR v2
- Any thoughts / experiments if method is applicable to ViT? And how it is comparable with recent work of pseudo-labeling https://openreview.net/forum?id=7a2IgJ7V4W&referrer=%5BReviewer%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConference%2FReviewers%23assigned-papers)?
- It is not clear from the text if we do fine-tuning on labeled data of the full network or just linear probing. Thus not clear if the baselines are in the same setting as proposed method. Also not clear if simCLR v2 benchmark is done also with EMA for target branch of the SSL as it serves as the main purely comparable baseline having the same structure of the SSL except additional positive mining.
- I think it is worth to mention some limitations of the proposed method as overall discussion about pseudo-labeling could be misleading. Pseudo-labeling is applicable for **any** data type, including sequential like speech recognition and machine translation. Proposed method is not directly applicable to speech and text domain as there we operate with sequence level labels without segmentation. 
- why did authors decide to use EMA? SimCLR v2 worked without it too. I wonder what is the impact of EMA overall on the method. It should work without it as authors maintain the queue of labeled data representations and there could be the same sample but with representations obtained from different model snapshots - kind of ensembling which performs same as EMA.
- there could be different ways (4 cases) to compute embedding for labeled / unlabeled data and then perform KNN. Why do authors select the way described in formula 2? any ablations or at least motivation (better to add it into the text)?
- Table 3, 8: I like results, this is great! But what are the results for simCLR here? Could it be that the method performs better because it is using SSL loss and not PAWS / other semi-sup. loss?
- Table 5: Why do we not get results of simCLR v2 from Table 1 for the last row in Table? What is the difference then as we removed both pseudo-labels and semantic positives? Also not clear if simCLR v2 baseline is also done with multi-crop strategy 
- There is no any discussion on the proportion of labeled / unlabeled data in the main text and in the experiments? How do we balance data?

clarity,_quality,_novelty_and_reproducibility: **Clarity/Quality**
Paper is written very well, it is simple to follow main ideas, results and settings. Several places can be improved further in the paper. Also some technical clarifications are needed in the text to improve the story and idea description.
- abstract: be consistent with specifying the results first for 1% then for 10% - otherwise hard to parse them.
- introduction, first paragraph, last two sentences: suggestion is to say about "labeled data could do earlier guidance and properly ground us from the beginning of training rather than only at the end at fine-tuning phase."
- introduce OOD notation before first usage in introduction.
- any links in introduction for positive mining? "virtuous cycle" - here good to reference to pseudo-labeling where one model is trained all the time and pseudo-labels continuously re-generated, e.g. fixmatch - it is the same story.
- nice to have small comment on why $l_2$ normalization is important (maybe it will work without it and just do normalization for KNN but not for overall training and SSL propagation through this normalization?)
- there is very limited description of queue / buffer in the main text. No any information about how we fill it at the beginning of training (if there no so labeled samples were seen yet. Do we train with SSL loss until we have filled the queue and then we start to do joint training? Do we have only labeled data in queue or we have both labeled and pseudo-labeled data later? All clarification are needed in the main text.
- Could authors add 1 sentence info about ablations in the main text: keep ablations in appendix, but add references into main text?
- Table 1 - not clear if simCLR has EMA in use. Did authors try other any SSL methods to see the generalization of the method for other SSL variants?
- Sec 3.2 paragraph before Table 3 - rewrite, seems there is some typos and not clear formulation of the content.
- Page 7 - precision-recall discussion please refer to Figure 2 somewhere in the text.
- what is an extra cost on top of SSL to do proposed positive mining? Should be negligible but wonder how it is comparable.
- Section 5 last paragraph on related works about transformer-specific trainings - one of the recent works in this direction is Zhai, S., et al., 2022, June. Position Prediction as an Effective Pretraining Strategy. In International Conference on Machine Learning (pp. 26010-26027). PMLR.
- After formula 10 why $\lambda$ and $c$ are not ones as there is phrase later "as our method is robust to the choice of hyperparameters"? I found weird to have 5 and 0.3 correspondingly for these params.
- Code listing, appendix C - queue_i var is not defined before.
- One suggestion about updating the queue is not to remove the oldest embedding, but rather sample randomly to decide which one to remove, or even to decide to we want to add new embedding into it randomly. This can balance the noise we have in pseudo-label and consistency with small changes in training.
- Some more advanced question: can we decrease batch size having now semantic positives? Do we still need such large batch?
- Table 10: not clear False and True notation here from the text. Maybe writing can be improved.
- Could authors confirm that in queue it is only labeled data representations and there could be one sample with several different representations obtained from different model snapshots for target branch?

**Novelty**
All pieces of the proposed algorithm appeared in prior literature, e.g. KNN, SSL, EMA, augmentations, additional MLP on top of representations, ensembling, pseudo-labeling, positive mining. However authors proposed a novel way of combining them and grounding SSL method on the use of labeled data by mining positive samples. The idea is new and interesting.

**Reproducibility**
A bunch of details are given throughout the paper and also in Appendix C, D.1, E, including the snippet of the code for algorithm, augmentation details and model main parameters.

summary_of_the_review: I think the main strong contribution of the paper is demonstration that self-supervised learning (SSL) (via successful and popular contrastive learning) can be significantly improved if supervised data are used not only at finetuning phase but also at pre-training phase, which is more aligned in the way how babies are learning with additional (weak) supervision and active learning. I hope this paper could push research community in the direction of SSL and semi-supervised learning synergy but not as two-phase training but more natural way, e.g. how this paper proposes.

Overall the idea is presented in a clear way and with extensive ablations. There are several concerns on the clear presentation and discussion of the queue component in the method as well as some extra ablations to strengthen the point "we should use labeled data as earlier as possible and integrate/ground SSL with them".


Update: Based on the revision and additional experiments and analysis I am changing my score from 6 to 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to integrate the idea of pseudo-labeling to the contrastive self-supervised learning paradigm. To be specific, the method uses a subset of labeled data to pseudo-label the unlabeled data point through $k$-NN. The pseudo-labeled data can therefore be used as a semantically positive sample and joint optimized using the contrastive learning objective. The paradigm achieves outperforms previous methods by a noticeable margin on ImageNet with 1% or 10% data. It also shows strong performance on robustness, out-of-distribution and transfer learning settings.
strength_and_weaknesses: ### Pros: 

+ The construction of semantic positives introduces additional class-invariant objective in addition to augmentation-invariant objective as in the instance-level contrastive training.

+ New state-of-the-art performance on semi-supervised learning with 1% and 10% ImageNet dataset.

### Cons:

- In SemPPL, the pseudo-labeling stage relies on a $k$-NN to retrieve semantically close labeled samples. What is the speed of $k$-NN compared with the rest of the training?

- What's the effect of loss weight$\alpha$? I am particularly interested in the case of $\alpha=0$, meaning that you drop the augmentation-invariant self-supervised objective and $\alpha=1$, meaning that you are reproducing a baseline from pure self-supervised representation (since I notice the augmentations used in SemPPL are stronger than the original SimCLR so the baseline might be higher than the reported number as well). 
clarity,_quality,_novelty_and_reproducibility: + Clarity: The paper is in general well written.

+ Quality: Good performance on semi-supervised learning and transfer learning.

+ Novelty: The idea of combining augmentation-invariance and class-invariance from the labeled subset is simple but effective.

+ Reproducibility: The authors promise to open source code and model checkpoints on GitHub.
summary_of_the_review: Excellent performance and well-written technical contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes using pseudolabels to improve representations learnt by contrastive learning in a semi-supervised representation learning setup, which is conceptually interesting even though the method itself is a combination of well-studied components. Improvements are demonstrated through downstream evaluations on semi-supervised learning, robustness and out-of-distribution generalization tasks. Experiments were extensive in terms of downstream tasks and contrastive losses, though largely limited to ResNet-style backbones and representations learnt using ImageNet. 
justification_for_why_not_higher_score: - Lack of technically novel components 
justification_for_why_not_lower_score: - Conceptually novel and interesting idea of semi-supervised representation learning
- Strong empirical results and analyses
Decision: Accept: poster