Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper deals with the problem of semi-supervised representation learning and proposes a method based on the specific previous method called PAWS. More specifically, the previous method PAWS can be regarded as kernel density estimation (KDE) for modeling the distribution of latent variables, and the proposed method called RoPAWS extends it in a semi-supervised way. The proposed method also introduces data-dependent in-domain priors assuming that an example is likely to have the same label if it is close to some labeled example.
strength_and_weaknesses: [Strength]

S1. Representation learning is one of the hot topics at this conference. Technically solid methods for this topic will draw attention from a broad range of researchers and engineers.

S2. The current manuscript is basically well-written and easy to follow.

[Weakness]

W1. I am afraid that the main idea of the proposed method is similar to semi-supervised kernel density estimation (SSKDE) presented in the following paper.
Wang, Hua, Mei, Hong, Qi, Song, Dai, "Semi-supervised kernel density estimation for video annotation," Computer Vision and Image Understanding, Volume 113, Issue 3, 2009. https://doi.org/10.1016/j.cviu.2008.08.003.
I understand that the specific implementation of the proposed method is different from that of the above paper; however, the advantages of the proposed method against SSKDE should be clearly presented both literally and experimentally.
clarity,_quality,_novelty_and_reproducibility: [Clarity]
The current manuscript is no problem in terms of clarity.

[Quality]
I am unsure which kind of quality should be discussed here; however, the current manuscript does not have sufficient quality for acceptance due to the lack of novelty justifications against the previous methods.

[Novelty]
As presented in the Weakness section, the current manuscript fails to justify the novelty of the proposed method against the previous method SSKDE.

[Reproduciblity]
I think that it is no problem for reproducibility.
summary_of_the_review: I have to recommend this paper be rejected since it requires major revisions to justify the novelty of the proposed method. It should conduct detailed bibliographical surveys and experimental comparisons with previous related methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends the semi-supervised image classifier PAWS (Assran et al., 2021) by using unlabeled data in addition to the labeled data to inform the representation. Using the unlabeled data in this way allows the model to learn from out-of-distribution classes rather than assuming unlabeled data is curated to contain the same classes seen in labeled data. Experimental results show that the proposed RoPAWS method performs roughly on-par with baseline PAWS method on curated datasets and improves on PAWS in uncurated (OOD) settings.
strength_and_weaknesses: Strengths: 
*  The work reframes the PAWS method as a generative classifier and uses this probabilistic interpretation to clearly introduce the calibration using the unlabeled data in this setting.
* The method prevents over-confident class predictions for OOD data in the unlabeled set, unlike PAWS, which yields better representations for classification.
* Ablation study is provided to provide a careful study of the components of the method
* Sensitivity of hyper parameters are empirically investigated and show improvements over baseline PAWS for a range of hyperparameter settings.


Weaknesses:
* In the original PAWS, class-balanced sampling plays an important role to avoid collapse of representations. By introducing the unlabeled data, it is not clear if this trivial collapse is now a problem in RoPAWS. Can this point be clarified in the exposition of the method? 

* When applying this method to curated data, it is not clear any improvements in performance should be expected. The model will be learning from more data, but the labels inferred for unlabeled data may not be accurate and therefore hurt performance. Indeed the experiments show only minor, perhaps negligible, improvement over PAWS.  The text does claim superiority in these cases as well but the numbers do not generally support the claim. The performance seems generally equal. An interesting point to investigate is the exception to this pattern where RoPAWS conclusively improves on PAWS in the curated CIFAR-10 dataset with only 25 labels per class. Here RoPAWS does benefit from using the (in-distribution) unlabeled data. This could be a strength of the proposed method if investigated further.

* Likewise, the results on ImageNet (Table 2) show similarly small improvements in performance of RoPAWS over PAWS at both 1% and 10% of labels.  Some discussion of this could improve the empirical analysis.
clarity,_quality,_novelty_and_reproducibility: The text is well written and related works are well contextualized. The experiments are fairly complete and clearly explained. 

The method does provide novelty in the re-framing and modification of an existing method to allow for better utilizing uncurated unlabeled data.

The text provides enough detail for reproducibility and authors provide their code.
summary_of_the_review: This is a good paper overall. The method provides an improvement over an existing method that makes is more applicable in real-world settings where OOD data is common. The text is well written and experiments are clearly laid out. Empirical results are mixed, in some settings performing on-par with existing methods and in some improving on them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper at hand prososes an extension of a recent proposed semi-supervised representation learning approach PAWS. The extension focuses in robustness, hence, RoPAWS, concerning uncurated data from, e.g., out-of-distribution data. This is a relevant problem in many real world applications.
strength_and_weaknesses: + Quite simple extensions to pave the way for real world problems.
+ Experiments a thoughtful and support the main aim of the approach.

clarity,_quality,_novelty_and_reproducibility: + The paper is well written, good motivated and quite easy to follow.
summary_of_the_review: The paper addresses a relevant problem and proposes an extension to "fix" limitations of the recently proposed PAWS method. The extensions seems incremental but I'm still in favor for the paper and think its worth publishing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to make semi-supervised learning robust when unlabeled data contains out-of-class data. This is a practical problem. The authors improve the semi-supervised learning algorithm PAWS by calibrating the prediction of PAWS based on the data densities. Experimental results show the proposal achieves better performance compared with the PAWS algorithm.
strength_and_weaknesses: Strength:
1) This paper focuses on the robust semi-supervised learning problem with out-of-class unlabeled data. This is an important problem.
2) The proposal achieves better performance compared with PAWS.

Weakness:
1) This paper only gives a minor improvement on the PAWS algorithm. The contribution and novelty are quite limited. Can the proposal be applied to the general semi-supervised learning algorithms?
2) There are also some other robust SSL algorithms that consider the out-of-class unlabeled data. More related methods should be compared in the experiments.


---------Update-------
I notice that the authors compared with some robust SSL algorithms on CIFAR-10 datasets. I recommend the authors conduct experiments on more settings, such as different datasets, various labels, various extents of OOD unlabeled data, etc.

clarity,_quality,_novelty_and_reproducibility: 1) The paper is clear and easy to understand.
2) The proposal is a minor revision of the existing PAWS algorithm. The novelty is limited.
summary_of_the_review: Overall, the contribution and novelty of this paper is limited, the experimental results are also not convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper argues that the PAWS is not robust to unlabeled real-world data. This paper tries to tackle this problem by suggesting the RoPAWS. RoPAWS uses in-domain prior to prevent a model from overconfidently predicting unseen class data by adopting regularization utilizing similarity between labeled and unlabeled data. Thereby RoPAWS shows better performance in various settings. RoPAWS especially excels in uncurated settings with fewer labels.
Also, this paper gives a novel probabilistic view of PAWS. 


strength_and_weaknesses: Strengths

- Probabilistic explanation of PAWS that it works as generative classifier and extension to RoPAWS make sense and interesting.
- The problem definition and formulation were clear.
- Straightforward solution for the given limitation (overconfident prediction).
- It obtains SOTA results with moderate additional time cost.
- Fig. 4 shows apparently how RoPAWS works effectively compared to PAWS

Weaknesses 

- The performances of the proposed RoPAWS as well as PAWS will be highly dependent on the choice of data augmentation methods for creating different views. The details are missing.
- This is rather a question: I wonder if the self-supervisory signals between the two views of the labeled data gives better performance.


clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. It appears to be technically sound and gives adequate theoretical explanation and experiments. 

summary_of_the_review: The motivation of RoPAWS is well supported. I believe that sufficient advances in the performance on uncurated settings shows the effectiveness of RoPAWS. Also the analysis using t-SNE demonstrates that RoPAWS works as the authors intended.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents RoPAWS, a robust semi-supervised representation learning method for both curated and uncurated data, e.g. out-of-distribution data. As the coined name implies, the approach is largely built upon the previous approach PAWS (Assran et al., 2021), but introduces several necessary improvements based on the reinterpretation of PAWS as a generative classifier that models densities using kernel density estimation (KDE). RoPAWS is shown to significantly outperform PAWS and other leading SSL methods on several common datasets.

The paper has received five review reports with mixed opinions. During the revision and rebuttal phase, authors addressed most of the concerns on technical novelty and empirical evaluation. There are still two votes of "Weak reject" after the discussion phase. Nonetheless, one of the negative reviewer said in the discussion thread that (s)he would increase the score (although not done so finally). 

AC read the paper as well as all the reviewing materials, and concluded that the most concerns in the "Weak reject" reviews, such as the technical connection to SSKDE, empirical evaluation with more methods on more datasets, have been addressed reasonably well. As a final note, the paper improves PAWS with a new generative interpretation and several modifications which are not technically surprising but empirically beneficial for real-world uncurated data analysis. The presentation of the paper is good, with sufficient supplementary material such as implementation details and source code for release. Therefore, AC recommends to accept the paper in its revised form as a poster.
justification_for_why_not_higher_score: The current paper mainly extends PAWS into RoPAWS, with several natural but somewhat incremental improvements, such KDE for density-based calibration, in-domain prior. The final training objective in Eq. 14 is closely fit into the PAWS loss with modifications in the instance weights and calibrated predictions. Empirical results show that these modifications lead to significant performance increases on some de facto datasets such as CIFAR, but the benefit is minor in ImageNet. Hence, both the technical contributions and the empirical benefits are above the acceptance threshold but do not reach that of the spotlight.
justification_for_why_not_lower_score: The contribution warrants acceptance as a poster. Also the review scores support such a decision. It is not possible to reject a paper with scores (88855) because most of the concerns were addressed while the AC cannot find further flaws.
Decision: Accept: poster