Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a modified way of doing self-training that aims to improve the performance when the unlabelled data is not coming from task-related labelled data with artificially removed labels. The authors validate their approach on reduced CIFAR-10, full CIFAR-10, and full CIFAR-100.  
main_review: I really like the goal of this paper. Most SSL and self-training methods are only evaluated on a contrived dataset where the unlabelled data is actually just labelled data whose labels are removed. For example, a common setup for SSL is to use CIFAR-10, where some images are considered labelled data while rest of the data is considered unlabelled data. The obvious issue here is that the "unlabelled" data in this setup is actually very similar to the labelled data, even with the same distribution of classes etc. So overall I like the goal of this paper. 

The method is not very novel. As far as I can tell, the method is similar to noisy student (which itself wasn't too novel or different compared to previous pseudolabelling/self-training methods), with some modifications of calibration, ranking and filtering of confidences, and then relabelling of unlabelled samples with lower confidence pseudolabels. It's worth noting that the original noisy student paper had additional experiments in appendix on filtering psuedolabelled samples by a confidence threshold, where they considered pseudolabels that are hard or soft. 

It is of course perfectly fine for a method to not be too novel, as long as it leads to important improvements in empirical results or scientific understanding. I think that the presentation of ODST could be significantly more convincing if it was evaluated on a more challenging benchmark such as imagenet, where noisy student had shown a significant improvement. That said, I do think the results on cifar-4k where ODST outperforms fixmatch is impressive. I'm curious if authors could reference any results of fixmatch applied to task-unrelated unlabelled data from other published papers. Currently as a reader I'm unsure how much of the improvement over fixmatch could be due to fixmatch not being tuned as well as ODST is, since fixmatch has lots of different hyperparameters that could need retuning when unlabelled data is from a task-unrelated source. The authors could also add more information about how they tuned fixmatch and other baseline methods for this harder task where unlabelled samples come from a task-unrelated dataset.

Minor: I see the phrase "resp." on several pages. I wasn't sure if that's a latex bug or not.        
summary_of_the_review: Since the methodological novelty is limited, the paper could benefit from more comprehensive empirical experiments. Since ODST is quite similar to Noisy-Student, it could be helpful to see experiments on Imagenet. If that is not possible, experiments on more datasets such as STL-10 or SVHN could help, as currently the only experimental evidence for ODST is on CIFAR-10/100.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose an out-of-distribution-aware self-learning (ODST) framework with a careful sample selection strategy that works well with unlabeled datasets containing a small portion of task-relevant data.
main_review: Semi-supervised learning (SSL) has shown great potential to leverage large amounts of unlabeled data to improve model performance on downstream tasks. However, most of the recent works are hinged on a key assumption that the unlabeled data comes from the same distribution as the underlying task or from the same classes. While some recent works address this by considering the OOD setting, they assume the amount of non-task specific data to be relatively less in the mix. The authors focus on SSL for image classification and consider a more realistic setting where the ratio of non-related images in the unlabeled dataset is large.

To this end, the authors propose a self-training scheme with a careful sample selection strategy. The idea of self-training with unlabeled data has been used in several recent works including Unsupervised Data Augmentation (UDA, NeurIPS 2020) and Noisy Self-training Student (NSST, CVPR 2020) which the authors adopt as the base framework. The authors perform top-K class-based sample selection which has been shown to work well in self-training settings in prior work like Uncertainty-aware Self-training (UST, NeurIPS 2020). They further perform calibration and adapt the self-training loss function to combine (pseduo)-labeled losses from 3 different segments of the data. 

The novel component in this method which sets it apart from prior self-training works with sample-selection (including NSST/UDA which uses confidence-based selection and UST which uses uncertainty-based selection) is the use of a weak form of knowledge distillation using pseudo-labels and the uniform distribution. The authors provide theoretical justification for this choice followed by ablation studies in the Appendix to evaluate of such regularization and other components of the framework on the overall model performance. 

For the experiments, the authors show impressive improvements over several baselines in both standard SSL and open-world SSL settings.  Comparing NSST+ and ODST+ which is the major component that bring the most benefit for the open-world setting?

Overall, the paper is well-written and easy-to-follow with significant improvements over several baselines in the open-world SSL setting.


summary_of_the_review: The authors propose an out-of-distribution aware self-training method with significant improvements over several baselines in the open-world SSL setting with extensive experiments and ablation studies.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper assumes an open world setting for semi-supervised learning approaches. In comparison to OE method, the proposed ODST method further dampness the usage of pseudo labels. Specifically, this work further assumes that unlabeled data with lower confidence do come from a mixture model having two components with equal probabilities: one fully uncertain model with label $\frac{1}{K}$ evenly across all classes; and one model with pseudo label determined by the teacher model. 


main_review: Strengh: The paper considers a novel mixture model for unlabeled data in an open world scenario. Each unlabeled sample is allocated with a lable with 50% possibility of an outlier, and 50% possibility as an inliner with the psuedo label assigned by a teacher model.

W1: Presentation. I am afraid the presentation quality needs improvement. There are many grammar issues in the paper. Take for example, “A standard setting in SSL is to use 4k training images of CIFAR10 together with 5k as validation set”, is not correct in grammar. 

W2: Motivation, I doubt that the pseudo labeling strategy is in fact too noisy, owing to the misalignment of classes between in data and out data. The direct consequence is that the pseudo labeling on out of distribution classes (classes not from the same distribution as from the labeled data) might be seriously wrong. The model would then suffer from the artifact of high confidence (actually, softmax is not capable of measuring any confidence anyway) even if the confidence is already thresholded. If we further take into account the domain gap between in and out data, such plain form of pseudo labeling remains skeptical. The method therefore is sensitive to the threshold selection procedure, subject to the actual domain gap and class difference between in and out data. 

W3: Empirical evidence. Due to the lack of presentation clarity, I am curious how the OE model is implemented under a “labeled only” setup as in Table 1, as OE explicitly encourages uncertainty on unlabeled data. Why is OE not compared with the ODST under the same Open World SSL setup? I think OE itself is suitable for such comparisons scenarios. 

W4: Theoretical support. The supporting theorem does not link to the propagation error introduced by such pseudo labeling strategy. Since classes between labeled dataset and unlabeled datasets are significantly different, softmax function is prone to prediction artifact. I think what theoretically matters is how much is the error bound constrained in this case given the assumed domain gap and class discrepancy/overlapping between labeled and unlabeled data?

W5: Ablations. There is no ablation study against various components in the proposed ODST method. I insist that the ablation is critical in this paper since the proposed method now is packaged with calibration method, pseudo labeling strategy, student teacher architecture, and the proposed loss Eq. (2). It is currently unclear which factor eventually contributes to the eventual performance improvement. 

summary_of_the_review: Given the above considerations, I cannot recommend acceptance at this time, unless the above issues are addressed during rebuttal. I would increase my scores if the above concerns are clarified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the open-set semi-supervised learning problem. The authors propose a modification of the classical self-learning algorithms. Compared with FixMatch, which selects pseudo-label based on a static confidence threshold. This paper proposes to select a threshold based on in-distribution validation data.
main_review: This paper focuses on the open-set semi-supervised learning algorithm and proposes a new pseudo-label selection strategy to improve the robustness of SSL algorithms with open-set unlabeled examples.

Strength:
1) The proposal is simple and clear and the paper is well-written.

Weakness:
1) The key technique of the proposal is that the proposal adopts a new pseudo-label threshold to select pseudo labels and the threshold is calculated via in-distribution validation data. In reality, the in-distribution data set is difficult to obtain in a semi-supervised learning setting as we know SSL algorithms are designed to solve the problem with scarce labeled data. The novelty and contribution of the paper are limited.
2) The authors analyze the proposal with bayesian decision theory. The results are mainly based on Eq.(4) and Eq.(5). However, the proposed algorithm does not optimize Eq.(4) and Eq.(5) directly, it seems that the theory can not demonstrate the effectiveness of the proposal.
summary_of_the_review: This paper focuses on the open-set semi-supervised learning algorithm and proposes a new pseudo-label selection strategy to improve the robustness of SSL algorithms with open-set unlabeled examples. However, the proposed algorithms are mainly a small modification of current pseudo-label selection-based algorithms, the contribution and novelty of the proposal are limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This papers studies semi-supervised learning with OOD samples appearing in unlabeled data as well as at test time, which is practical in real-world tasks. To handle OOD samples, this paper proposes a new framework based on OE and self-trainning. Extensive experiments are conducted to verify the superiority of the proposed method.
main_review: ## strengths
1. This paper studies an important problem which is underexplored in literature;
3. This paper provides analyses on the base classifier based on Bayesian decision theory.
4. The proposed method shows significant performance improvement against several baselines on multiple datasets.


## weaknesses
1. The novelty is limited. The proposed method composes of several parts, i.e., self-training, OE, classifier calibration, confidence-based sample selection, and loss functions are pretty much the same as previous works. which makes the method a combination of existing techniques;
2. The proposed method is complex yet not well understood. More ablation studies are needed to better understand the effectiveness of each component. For example, the last term of the loss function in Eq. (3); the choice of $k$ in sample selection; the classifier calibration; the size of in-distribution validation set;
3. The proposed method is inefficient. In particular, it needs 1000 epochs of training. Additionally, in each epoch, it involves hyper parameter selection on validation set, which adds more computational burden.
summary_of_the_review: As listed in **Main Review**, the novelty seems limited and the proposed method is complex yet not well explained. Therefore, I recommend rejection.
review_title: null

Meta-review: This paper proposes a method for self-training in an open-world setting where a significant portion of unlabeled data might include examples that are not task related. The proposed method (ODST) uses a more accurate OOD detection technique which allows an improved sample selection leading to higher accuracy.

Strong Points:
- This paper studies a very important and impactful problem.
- The paper is well-written.
- The empirical results show that the proposed method improves over prior work.
- To better understand the iterative scheme, authors provide theoretical analysis using Bayesian decision theory.

Weak Points:
- Novelty: Given prior work on different variants of noisy students, this work has limited novelty.
- Dataset diversity: The main results are provided for CIFAR-10 and CIFAR-100 datasets which are very similar to each other. During the discussion period, authors added results on SVHN datasets but the accuracy gap between the proposed method and FixMatch is insignificant (FPR gap is higher but since the main goal is improving performance, I think showing accuracy is a more important measure here).
- Connecting theoretical results to the rest of the paper: The paper can be improved significantly if the theoretical results are more connected to the rest of the paper and in particular with the proposed algorithm.

While 4 out of 5 reviewers are recommending rejection, I think this was a very close decision. Most reviewers were concerned with novelty which I think is a valid point. Given that and the fact that the theoretical results are very limited, showing strong empirical results are required to accept this paper. Even though the provided results on CIFAR datasets are strong, the result on SVHN does not show a significant improvement. I understand that running experiments on ImageNet might not be budget-friendly. However, it is possible to run similar experiments or other datasets to show the robustness of the proposed method to the choice of the dataset. Consequently, I recommend rejecting the paper and propose authors to resubmit after adding more datasets as part of their evaluation.
Decision: Reject