Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the pseudo-label in class-mismatched semi-supervised learning when there are unlabeled out-of-scope data from other classes. The authors propose a model consisting of (1) Re-balanced Pseudo-Labeling, which re-balances pseudo-labels on ID classes to filter out OOD data, and (2) Semantic Exploration Clustering, which uses balanced clustering on OOD data to create pseudo-labels on extra classes.
main_review: Strength

- This paper has an interesting observation on how does OOD data influence pseudo-label and resulting model performance.
- The proposed method also seems to be original.

Weakness

I have several concerns about the experiments.
1. First, as the vanilla semi-supervised learning baselines, stronger methods should be leveraged. Especially, as the problem definition of this paper is related to the pseudo-label in semi-supervised learning, the recent “pseudo-label based” semi-supervised learning methods such as FixMatch [1] and ReMixMatch [2] should be considered. VAT or Mean Teacher seems to be outdated. It will also be interesting to investigate if we combine the proposed r-Model with the existing pseudo-labeling-based semi-supervised learning methods.

[1] Sohn et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS 2020.

[2] ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring. ICLR 2020.

2. It would be better to evaluate the proposed method on other datasets such as CIFAR100, STL, or maybe ImageNet dataset. In order to comprehensively understand the behavior of the proposed method, the proposed method should be evaluated on other datasets (at least on the MNIST dataset).

summary_of_the_review: This paper has an interesting observation and the proposed method seems to be original. However, as mentioned before, it would have been better to have additional training setups and baselines to emphasize the efficacy of the proposed method. I would like to rate this paper to be borderline, but I am open to changing the score after the rebuttal.

=======================

After rebuttal:

After reading the authors' response, I think my concerns both on the compared baselines (FixMatch) and the evaluated datasets (ImageNet) are somewhat addressed. Therefore, I would like to increase the score to be 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper researched on the semi-supervised learning task when there are unlabeled out of distribution data from other classes. Several interesting issues of class mismatch SSL are studied, including the reasons for the performance degradation of PL in OOD data and how to better pseudo-label OOD data to provide a more balanced semantic distribution. To address above-mentioned problems, Y-model, consisting of two components – Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC), are proposed. Authors conducted several experiments to show the effectiveness of the proposed method, such as CIFAR-10 and SVHN. 
main_review: Pros:

(1) OOD SSL is a very interesting topic, and it is also a realistic setting in real world applications.

(2) The insight that pseudo-labeled data is often class-imbalanced is also interesting and can guide future research in this direction.

(3) Authors conducted comprehensive experiments and ablation studies on some small-scale benchmarks.

(4) This paper is well-written and easy to follow.

Cons:

(1) CIFAR-10 and SVHN are much easier than CIFAR-100, TinyImageNet or ImageNet. To show the effectiveness of this method on large-scale dataset, it may be necessary to conduct some experiments on larger benchmarks, especially when the performance improvements on small-scale benchmarks are relatively marginal (about 1.7~2.8% on CIFAR-10 as in Table 1 and ~3% on SVHN as shown in Figure 6). 

(2) The current state-of-the-art method on semi-supervised learning is FixMatch [1], however, the authors only compared against worse-performing methods in SSL like Mean Teacher. Therefore, can we still obtain similar observations and probelms when we are using a better SSL framework like Fixmatch? Will the method still perform better when it is integrated with FixMatch?

(3) The OOD SSL is in fact investigated by a con-current work [2], which was released on arXiv about 10 months ago and was in submission to ICLR 2022 too. The rate of this paper will be not reduced for not comparing [2], however, I am curious about the key differences between this paper and [2]. If the author can provide some comparisons and insights, I would be very grateful.

(4) The idea of simply truncating the number of pseudo-labeled data to the minimum is quite brute force, is there any ablation studies on other popular long-tailed recognition method re-weighting, re-sampling, marginal loss or even multi-expert framework?

(5) Will the code of this method be made public available?

(6) Using the expensive and time-consuming clustering method on large-scale benchmark like ImageNet may be of a big challenge to computational efficiency.

[1] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.

[2] Cao, Kaidi, Maria Brbic, and Jure Leskovec. "Open-World Semi-Supervised Learning." arXiv preprint arXiv:2102.03526 (2021).
Harvard	
summary_of_the_review: I like this paper in general, but I have some concerns on the effectiveness of this method in large-scale benchmarks. Also using the expensive and time-consuming clustering method on large-scale benchmark like ImageNet may be of a big challenge to computational efficiency.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the problem of semi-supervised learning (SSL) with class miss-match. The authors first empirically analyze Pseudo-Labeling (PL) in class-mismatched SSL and proposed a new method that consists of two components – Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC). Experiments show that the proposed method achieves steady improvement over supervised baseline and state-of-the-art performance under all class mismatch ratios on different benchmarks. 

main_review: Strengths:
1. This paper provides the empirical analysis of the Pseudo-Labeling model for ID and OOD data.
2. This paper proposed a novel two-branched Υ-Model to solve the class miss-match issue in SSL.
3. Experiments on different SSL benchmarks empirically validate the effectiveness of the proposed method.

Weakness:
1. The motivation for imbalanced pseudo-labels is not clear.
2. The technical novelty and depth of the proposed approach are limited.
3. The experiments are not extensive.


summary_of_the_review: This paper is well presented and organized. The proposed ideas are interesting. However, there are several main issues as follows:
1. the motivation is not clear and strong. The authors state that " imbalance of pseudo-labels harms the performance”. It lacks justifications for this statement. Fig 3 (b) shows that imbalanced pseudo-labels of OOD would result in many of data with class 1-5 into class 0. However, if the pseudo-labels of OOD is not imbalanced, it may still force the ID data into the incorrect class.
2. Fig 7 (c) shows that Υ-Model is sensitive to the Number of Extra Classes K. However, this paper does not provide a good strategy to choose K.  In the real case, we don’t know the actual number of OOD classes. 

3.  The Υ-Model may not work well in a large-scale dataset since the performance of the proposed SEC technical relies on the deep clustering algorithm. When the dataset becomes more complex or the number of OOD classes is large, then SEC may not work at all. It is better to show some results in large-scale datasets.
4. Can Υ-Model extend to other SSL methods? If not, it could be a limation of the proposed method. It is better to show some empirical results.
5. Since 100% Class Mismatch ratio is an extreme case, it is better to show a 50% ratio result for Fig 7 (c).

6. What is the purpose of Fig 2 (a)?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works on the class-mismatch semi-supervised learning problem, where the assumption is that the unlabeled samples include class labels that do not appear in the labeled data, i.e., out-of-distribution (OOD) data, in addition to the class labels that appear in the labeled data, i.e., in-distribution (ID) data. It is known that if there are OOD data included in the unlabeled data, it can degrade performance of semi-supervised learning algorithms. This paper focuses on one of the semi-supervised learning methods, which is the pseudo-labeling method. The paper first investigates how using pseudo-labels in the class-mismatch semi-supervised learning setup can be problematic. The experiments show that proportions of high-confidence data in ID data is larger than OOD data. Experiments also show that pseudo-labels (w.r.t. class predictions) are more balanced in ID data while it is unbalanced in OOD data at the beginning of training (with a pretrained model). This phenomenon becomes exaggerated at the end of training, where almost all of the OOD data is predicted into a single ID class. Assuming we know the labels of the unlabeled data, the paper performs further experiments that compare a few methods, and finds out that it is ideal if we have access to the underlying single class labels of OOD data, and solve the problem as a multi-class classification problem with K_id + K_ood (number of classes of ID and OOD) classes. Based on these observations, the paper proposes an Upsilon model, that consists of a re-balanced pseudo-labelling (RPL) and semantic exploration clustering (SEC). RPL aims to balance the distributions of pseudo-labels based on the observation that pseudo-labels become balanced/imbalanced with ID/OOD data, respectively. SEC is a workaround for not having ground truth labels of OOD data.  Experiments show comparison with both traditional SSL methods and class-mismatch SSL methods, and show how the proposed Upsilon model works better than these baselines. Ablation study shows why the components within the Upsilon model is necessary.
main_review: ##### Strength

The paper gives insights to the behavior of pseudo-labels under the class-mismatch problem experimentally, and provides an algorithm that can alleviate the issues that are discussed. It performs better than the pseudo-label baseline, but also performs better than two recent class-mismatch SSL baselines (especially with high mismatch ratio), so the benefits shown in the experimental results are one of the main strengths of the paper. 

The empirical comparison of open-set labeling and oracle labeling is interesting and motivates the proposed methodology, and makes the story clear. The ablation study further motivates the design of the Upsilon method.

##### Weaknesses

Since Upsilon is based on the findings and discussion in Section 3, it would be better to have at least one more dataset (for example, SVHN that is used in a later section) to see if the findings are not specific to a single dataset.

In Figure 2, it would be interesting to investigate if similar results hold with random splits for ID/OOD classes, instead of the animal/vehicle split. (I think the SVHN experiments later on somehow show that the splits doesn't matter so much, since the 0-5/6-9 split is not as semantically meaningful as the one in CIFAR-10.)

It is interesting and surprising how open-set labeling is worse than oracle labeling, although both are evaluated with only ID classes.  It would be interesting to have some discussions about potential underlying mechanisms of this experimental result.

##### Minor comments
- In the caption of Figure 3, it says "A lot of ID samples are misclassified into one class." but is this correct? Should it be "OOD samples"?
- Which animal class is class 0 in Fig.2(c) and Fig.3(b)? 
- Table 1: typo "aof"

=======================
##### After rebuttal

Thank you for the additional experiments and for answering my questions. I would like to keep my positive score. The answer to "It is interesting .. experimental result." makes sense. So there may be some implicit transfer learning going on.
summary_of_the_review: The paper investigates the issues of using pseudo-labels in class-mismatch semi-supervised learning, and shows the benefits of the proposed method empirically. It performs better than recent class-mismatch semi-supervised learning methods.
review_title: null

Meta-review: All reviewers agreed that this work on OOD and pseudo-labeling presents interesting and strong results. The authors’ rebuttal has addressed some of reviewers’ concerns. Based on the current review and discussion, there are still several major concerns towards the expensive computational cost introduced by the clustering method, the lack of discussion around how the proposed work can be incorporated into SSL methods, and the sensitivity towards the selection of K.
Decision: Reject