Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript introduces OpenMatch for open-set semi-supervised learning. The OpenMatch integrates a one-vs-all classifier (OVA-classifier, working as an outlier detector) and FixMatch. The main contribution should be the soft open-set consistency regularization (SOCR) in Figure 1 and the OpenMatch Framework in Alg 1. Specifically, the SOCR follows the same self-supervised framework in FixMatch but it uses OVA-classifier to output consistent anomaly score distribution. After using the model in Figure 1 to identify inliers from unlabeled data, the FixMatch is allowed to reach a higher accuracy on the OSSL tasks.
main_review: My main concern is about the experiments. 

1)The studied problem of open-set semi-supervised learning (OSSL) is close to the problem of out-of-distribution (OOD) detection.    
And there are some related works also considering semi-supervised settings for OOD: 
- 2020 Robust Semi-Supervised Learning with Out of Distribution Data 
- 2020 AAAI Semi-Supervised Learning under Class Distribution Mismatch

However, in experiments, only two baselines FixMatch (not OOD method) and MTC are mainly considered. This may not be convincing enough to demonstrate its superiority.  
Thus, more baselines (especially related OOD methods) are encouraged to include. 

2)For the model in Figure 1, it should be an *unsupervised* OOD classifier. As the soft open-set consistency regularization (SOCR) is one of the main contributions in this work, it would be better to compare the figure 1 model with the main-stream OOD methods (as they can also be used to identify OOD samples).      

Overall, this paper is well written and the provided solution of using OVA-classifier to help FixMatch is well motivated. The authors demonstarted that using OVA-classifier with SOCR can help FixMatch on open-set semi-supervised learning (OSSL). However, related OOD baselines are not included in comparisons. 
limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose an idea to improve model's performance under the open-set semi-supervised learning setting. This work proposed to leverage OOD detector, combined with FixMatch, and soft open-set consistency regularization (SOCR) to improve the model training during open-set semi-supervised learning. I think the usage of OOD on ignore outliers is not new and an unsupervised method on learning the consistency between augmentations for unlabeled data is also not new. Thus, I am not sure if I can find too much of novelty from this work itself, but I do appreciate the author's effort in putting them together to be a complete model, which in fact make solid progress on the open-set semi-supervised learning setting. 
main_review: ### Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the NeurIPS community.

- A solid advance on a practical Semi-Sup. Learning setting, though there isn't too much novelty in the proposed method. 
- Setting Benchmark by considering MTC, UASD, and D3SL as baselines. 
- Good experimental results and analysis

### Weaknesses: Explain the limitations of this work along the same axes as above.

- The related work section is hard to follow. I think it lacks proof-read, and its writing quality is much worse than the introduction section. 
- Fig. 2 is quite confusing. Why are the y-axis of (b) and (c) for "P of outliers for samples in circle shape"? Aren't the outliers samples in diamond shape? Also, I am not sure if I follow (C), in which the authors try to depict what the proposed soft consistency will do to the predictions of labeled inliers and unlabeled outliers. I think what Soft Open-set Consistency Regularization (SOCR) is trying to do is basically forcing the model to learn better visual representations of the unlabeled outliers. I would be curious what's the difference here if we were to apply contrastive learning to the unlabeled outliers. 

- I don't think the authors ablated the Entropy Minimization. How important is this component in the proposed work? 

### Please provide an "overall score" for this submission.

Overall, I think this paper made a solid contribution to the semi-supervised learning community. I am less satisfied with the novelty of the paper, though I don't think this necessarily leads to grounds for rejection. Personally, I believe that any work that makes meaningful and solid progress for a practical research problem is worth to be shared with the research community. That being said, I do suggest the authors do a proofread on their paper, especially revise some of the confusing parts in the paper. 
limitations_and_societal_impact: Yes, the authors provided the "Limitations" and "Broader Impact" subsections in the conclusion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a model for open-set semi-supervised learning. As a semi-supervised learning model, it tackles classification where only a portion of the training data is labeled. "Open set" refers to the fact that the unlabeled data are noisy and can contain out of distribution examples, for which the class is not among the known classes. Out of distribution samples are present also during test, and the model has to recognize them and avoid classification on them. The proposed model builds on top of existing works, combining several loss terms and ideas such as one-versus-all classifiers and FixMatch as the main semi-supervised learning engine. The main technical contribution is a loss term enforcing the out-of-distribution score for two augmented versions of the same unlabeled example to be similar. Extensive experiments validate the proposal against recent state-of-the-art models.
main_review: This submission has some solid merits, as it tackles a meaningful problem for the machine learning community, namely training in the presence of noisy labels and out of distribution examples. 
The method is straightforward and embeds a combination of several existing components, such as:
- FixMatch as the main semi-supervised learning engine. As is, it cannot deal well with the open set scenario.
- OVA-classifiers to identify in-distribution and out-of-distribution unlabeled examples, and therefore pseudo-label the formers and reject the latters. Originally, OVA-classifiers were proposed (in the version used here) for universal domain adaptation.
This results in the biggest weakness of the paper: the limited technical contribution. Indeed, the only source of novelty is in a regularization loss (Eq. 3) that enforces, on an unlabeled sample, the out-of-distribution scores of two augmented versions to be closer.

Following on the weak points, the model involves a number of hyperparameters, specifically in weighting the loss function terms. An analysis about sensitivity of performances to those lacks, and it is a bit unclear how one should select their value.

Clarification: in Tab. 2, how do the baseline "Labeled Only" and "FixMatch" perform OOD detection? Are they equipped with OVA classifiers as well? In their original formulation no mechanism allows for OOD. Is it with confidence thresholding?

I consider the ablation study on the proposed open-set consistency regularization loss particularily important, as it validates the main novel contribution of the paper. The benefit of the proposed loss function is demonstrated in Tab. 3, but I find curious that the authors turned off another loss component for this experiment (FixMatch). What is the reason for that? The text mentions "to measure the pure gain from the consistency loss", it is unclear to me. I would find ablating the consistency loss, whilst activating all other components, much cleaner.

Despite these limitations, the experimental section is solid and I think it is the true strength of the paper.
The experimental protocols used seem correct, and tests are carried out on 3 different datasets at different degrees of inlier-outlier splits. Performances of the model are properly reported both in terms of classification accuracy on the known classes and outlier detection. On both tasks, the proposed OpenMatch model outperforms recent competing methods by a significant margin.

---

Overall, I consider this a borderline submission. The weakest point is the minor technical contribution, as the main novel part is the consistency loss and other components are off-the-shelf modules. The strongest point is experimental evaluation, that shows big performance gap with respect to very recent baselines. All things considered, I lean more towards accepting the paper.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles open-set semi-supervised learning (OSSL) where outliers exist in unlabeled data. The key idea for detecting outliers is to learn class-specific, one-vs-all (OvA) classifiers—if all classifiers produce negative outputs for an unlabeled data, the data is considered an outlier. In addition to the classification loss computed on labeled data, the OvA classifiers are also trained on unlabeled data with an entropy minimization loss and a consistency regularization loss—the latter is computed on two views of the same data. The results on CIFAR and ImageNet show that the proposed approach largely outperforms the baselines in terms of both recognition accuracy and outlier detection accuracy.
main_review: **Strength**
- The design of combining FixMatch with the OvA classifiers is simple and effective, which could add value to the research of OSSL.
- The experiments are convincing and extensive, covering both classification and outlier detection tasks.
- The improvements over previous SOTA are impressive.
- The paper is easy to follow.

**Weakness/Questions**
- I am a bit concerned with the novelty. The idea of using the OvA classifiers for detecting outliers is borrowed from [32], which has been acknowledged in this paper. [32] deals with the problem of open-set unsupervised domain adaptation (UDA). Given the closedness between UDA and SSL, it's not too surprised to see that the OvA classifiers work in the OSSL setting. One could argue that UDA and SSL are different problems since the former needs to cope with domain shift while the latter doesn't, but we couldn't ignore many recent studies that have shown that methods developed for one problem (either UDA or SSL) work very well on the other. Nonetheless, I still appreciate the simplicity and effectiveness of the proposed approach.
- Why compute the consistency loss on logits instead of the output probabilities? Is it because the former gives a better result?
- Eq.1 looks a bit weird as it is supposed to be the binary cross-entropy loss.
- The entropy minimization loss $L_{em}$ defined in Eq.2 is not evaluated in the ablation study. One would be wondering if the improvement brought by this loss is significant.
- The motivation for SOCR is not very convincing. This loss is computed for the sub-classifiers. Why this loss could help the closed-set classifier produce more accurate pseudo-labels?

---
Post-rebuttal update:

I have read the authors' responses and other reviewers' comments as well. I keep my rating unchanged.
limitations_and_societal_impact: The authors have properly covered these aspects in Sec.5.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper focuses on semi-supervised learning with outlier unlabeled data. This problem has attracted much attention in the semi-supervised learning community. The authors propose a method that contains two parts: First, detect outliers based on the OVA classifier. Second, propose an open-set soft-consistency regularization loss that can help improve the outlier detection performance.
main_review: This paper focuses on semi-supervised learning with outliers, more accurately, the unlabeled data contains some examples that belong to classes that unseen in labeled data.  The authors propose an outlier detection-based method to detect outliers and help improve the SSL performance. I have the following concerns about this paper:

1) The contribution is limited. The proposal is mainly based on an existing outlier detection method. It seems that if we can detect outliers accurately and delete these outlier examples, the problem becomes an ideal SSL setting. So the open-set semi-supervised learning problem is actually an outlier detection problem?
2) Based on question 1, in this paper, the authors do not compare the proposal with the SOTA OOD detection method. The performance of the combination of OOD detection methods and SSL methods should be reported.
3) What are the differences between outlier detection and open-set learning. In my view, open-set learning should make the model has the ability to classify the new class data.
4) Self-Supervised learning, e.g., SimCLR, can help to learn a separatable representation and help OOD detection tasks. There are many OOD detection algorithms based on contrastive learning that have been proposed. Can we simply pre-train a model with SimCLR using all labeled and unlabeled data, and based on the learned representation to detect outlier unlabeled examples. Have you tried this way in your experiments?
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper proposes a model for open-set semi-supervised learning. It is well-written and well-motivated. As pointed out by almost all the reviewers, there is limited novelty as the proposed model is a straightforward combination of existing components. However, extensive experimental evaluation shows big performance gap with respect to very recent baselines, and shows solid progress on the open-set semi-supervised learning setting studied.

To further make the experimental results more convincing, it will be useful to add sensitivity analysis on the hyperparameters (as the model involves a number of them). Also, as the problem is related to OOD methods, the authors are encouraged to include OOD-related baselines.
Decision: Accept (Poster)