On Pseudo-Labeling for Class-Mismatch Semi-Supervised Learning
Authors: Lu Han
Han-Jia Ye
De-Chuan Zhan
Venue: ICLR 2022 Submitted (2022)

Abstract: Semi-Supervised Learning (SSL) methods have shown superior performance when unlabeled data are drawn from the same distribution with labeled data. Among them, Pseudo-Labeling (PL) is a simple and widely used method that creates pseudo-labels for unlabeled data according to predictions of the training model itself. However, when there are unlabeled Out-Of-Distribution (OOD) data from other classes, these methods suffer from severe performance degradation and even get worse than merely training on labeled data.  In this paper, we empirically analyze PL in class-mismatched SSL. We aim to answer the following questions: (1) How do OOD data influence PL? (2) What are the better pseudo-labels for OOD data? First, we show that the major problem of PL is imbalanced pseudo-labels on OOD data. Second, we find that when labeled as their ground truths, OOD data are beneficial to classification performance on In-Distribution (ID) data. Based on the findings, we propose our model which consists of two components -- Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC). RPL re-balances pseudo-labels on ID classes to filter out OOD data while also addressing the imbalance problem. SEC uses balanced clustering on OOD data to create pseudo-labels on extra classes, simulating the process of training with their ground truths. Experiments show that our method achieves steady improvement over supervised baseline and state-of-the-art performance under all class mismatch ratios on different benchmarks.

Under review as a conference paper at ICLR 2022
ON PSEUDO-LABELING FOR CLASS-MISMATCH SEMI- SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-Supervised Learning (SSL) methods have shown superior performance when unlabeled data are drawn from the same distribution with labeled data. Among them, Pseudo-Labeling (PL) is a simple and widely used method that creates pseudo-labels for unlabeled data according predictions of the training model itself. However, when there are unlabeled Out-Of-Distribuute from other classes, these methods suffer from severe performance degradation and even get worse than merely training on labeled data. In this paper, we empirically analyze PL in class- mismatched SSL. We aim answer the following questions:
(OOD) data
(1) How do OOD data influence PL? (2) What are the better pseudo-labels for OOD data? First, we show that the major problem of PL is imbalanced pseudo- labels on OOD data. Second, we find that when labeled as their ground truths, OOD data beneficial classification performance Based on the findings, we propose our model which consists of two components Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC). RPL re- re-balances ascudo-labeld on ID classes to filter out OOD data while also addressing the imbalance problem. SEC uses balanced clustering on OOD data to create pseudo-labels on extra classes, simulating the process of training with their ground truths. Experiments show that our method achieves steady improve- ment over supervised baseline and sfate-of-the-art performance under all class mismatch ratios on different benchmarks.
In-Distribution (ID) data.
INTRODUCTION
Deep Semi- Supri-sueeeeied Learning (SSL) methods are pro- posed to reduce dependency on massive labeled data by utilizing a number of cheap, accessible unlabeled data. Pseudo-Labeling (Lee et al., 2013) is simple but effec- tive and widely used method that creates pseudo-labels according to predictions of the training model itself. Then SSL can be transformed to standard supervised learning. consistency reg- ularization (Laine Aila 2017] Tarvainen & Valpola 2017 Miyato al.] (2019) holistic methods (Berthelot
labeled data
unlabeled data
calt
dog
cat
dog
plane
car
Other representative SSL methods
data
OOD data
ct al. 2019] SShnn et al. 2020) and generative meth-
Figure 1: Realistic Semi-Supervised Learning may simultaneously contain unlabeled ID and OOD data. ID data come from the same classes as labeled data while OOD data come from classes that are not seen in labeled data.
ods (Kingma et al.] 2014). The recent development of shows that these methods have achieved competitive performance to supervised learning methods.
However, all of these SSL methods achieve their good re- sults based on an assumption that the unlabeled data are drawn from the same distribution as the labeled data. This assumption can be easily violated in real-world applications. One the common cases is that some unlabeled data come from unseen classes. For example, is illustrated in Figure I in image classification, we can collect lot of unlabeled images from the internet but usually they cover broader category concepts than labeled data. Oliver al. (2018) have shown that such class-misne mismatched conditions, performance of tra- ditional SSL methods is damaged. To deal with this problem, several methods have been proposed. These methods include filtering out OOD data (Yu et al. 2020 Chen al. 2020), down weighting

---

Under review as a conference paper at ICLR 2022
OOD data (Chen et al., 2020) and re-use OOD data by neural style transfer (uo et al. 2021) or self-supervised learning (Huang et al. 2021). Although these methods achieve good results, why do OOD data damage performance and how will OOD data help remain unclear. Here, focus on analyzing Pseudo-Labeling (PI.) in class-mismatched SSL and give some answers to these two questions.
In this paper, we empirically analyze PL in class- misssttesmtteee SSL setting. These experiments aim answer the following questions: (1) How do OOD data influence PL? (2) What are the better pseudo-labels for OOD data? For question (1), we investigate pseudo-labels created by PL. The main finding is that pseudo-labels on OOD data tend be imbalanced while on ID data, remains balanced. We further show that PL's performance is damaged due to such imbalance on OOD data. For question (2), several strategies for labeling OOD data are investigated. We conclude that beneficial when labeling OOD data as a class different from ID data, and the performance can be further improved when the pseudo-labels partition unlabeled OOD data into their semantic clusters.
Based on the experimental analysis, we propose two-branched model called Y-Model, which pro- cesses unlabeled data according to their confidence score on classes. The first branch performs re-balanced pseudo-labeling on high-confidence data. It utilizes the property of imbalanced pseudo- labels on OOD data, truncating the number of pseudo-labeled data for each class to their mini- mum. This procedure filters out many OOD data and also prevents the negative effect of imbalanced pseudo-labels. For the other branch, semantic exploration clustering is performed on low-confidence data. They considered OOD data and their semanties will be mined by clustering into different partitions on extra classes. The clustering result provides better pseudo-labels for these OOD data than vanilla PL. Experiments on different SSL benchmarks show that our model achieve steady improvement in comparison to supervised baseline. We summarize our contributions as follows:
We analyze the Pscudo-Labeling model for ID and OOD data. The findings lead to two primary conclusions: (1) Imbalance of pscudo-labels on OOD data damages PL's performance. (2) Best lseudo-labels for unlabeled OOD data are those different from ID classes and partitioning them into their semantic clusters.
We propose our two-branched T-Model. One branch re-balances pseudo-labels on ID classes and filter out OOD data. The other branch explores semantics of OOD data by clustering on extra classes.
Experiments on different SSL. benchmarks empirically validate effectiveness of our model.
PRELIMINARY
2.1
SSL
Similar to the SSL problem, the training dataset of the class-mismatched SSL problem contrains ID labeled samples D; {(x,,, and m unlabeled samples D, {xwi) 1m (usually, m > n.) yis E VID = (I...., }. while different from SSL, the underlying ground truth yu of unlabeled data may be different from labeled data. i.e., Vuy E YID U YOOD, [KID- + 1,..., KID + KOOD}. The goal of class-mismatched SSL is to correctly elassify ID samples into VID using labeled set with ID samples and unlabeled set possibly with OOD samples.
2.2 PSEUDO-LABELING
Pseudo-Labes Labelin (PL) leverages the idea that we can use the model itself to obtain artificial labels for unlabeled data (eee et al. 2013). PL first perform supervised learning on labeled data to get pre-trained model f. which outputs the probability of belonging to each ID class. It then creates the pseudo-labels for each unlabeled sample:
arg max max F()x
c(x) >T
(1)
reject
otherwise
c(x) max f()xx),
(2)
DEj'D
2

---

Under review as a conference paper at ICLR 2022
where c(x) is the confidence score for x. All the pseudo-labeled unlabel data will be treated as labeled data for the next supervised learning generation. PL iteratively performs supervised learning and pseudo-label creation until stop condition.
3 ANALYSIS OF PSEUDO-LABELING IN CLASS-MISMATCHEE SSL
In class class-misman mismatched SSL, vanilla PL can only create pseudo-labelt ID classes for OOD data. We will analyze how these OOD data influence vanilla PL and what are the better pseudo-labels for them in this section.
3.1 SETUP
We use CIFAR-I0 (Krizhevsky et al., 2009) as our experimental dataset. The data set contains 10 categories animal classes and vehicle classes. Following Guo al. (2020), we perform a clas- sification task on animal classes (denoted as class 0-5) and select 400 images per class to construct labeled data set, i.e., 2,400 labeled examples. The other 4 vehicle classes are taken as OOD classes (denoted classes 6-9). 20,000 images are randomly selected from all the 10 classes as the unlabeled data set. We vary the ratio of unlabeled images to modulate class distribution mismatch. For example, the extent is 50% means half of the unlabeled data comes from animal classes and the others come from vehicle classes. We use Wide-ResNet-255 backbone. We also adopt data augmentation techniques including random resized crop, random color distortion and random horizontal flip. We train our network for 400 epochs. For each epoch, iterate over the unlabeled set and random sample labeled data, each unlabeled and labeled mini- batch contains 128 samples. We adopt Adam as the optimization algorithm with the initial learning rate 3 10-3 We report the averaged accuracy of the last 20 epochs, pretending there is reliable (too small) validation set to perform early stop OOliver et al. 2018
(Zagorykko & Komodakis 2016) as
3.2 IMBALANCE OF PSEUDO-LABELS ON OOD DATA
In this section, we analyze the pre-trained model that creates the first set of pseudo-lahels, and the final model trained by Pseudo-Labeling.
Confidence Seare on ID and OOD dats
Praudd-babche
Pscudo label OOD data
6.24
Confidence Score
Class
Class
Figure 2: Analysis of pre-trained model. (a) Histogram of the computed confidence scores c(x) over ID and OOD data. (b) On ID data, pseudo-label distribution is balanced since they share the same distribution with the labeled data. (c) On OOD data, the pseudo-label distribution is imbalanced.
Pretrained model. First, we draw the distribution of confidence score on OOD data and ID data. Figure 2(a) tells that, like what is concluded in OOD detection (Hendrycks & Gimpel| 2017). pro- portion of high-confidence data in ID data is larger than OOD data. However, in cluss-mmsmttched SSL, the unlabeled data are in much larger quantities. When the class mismatch ratio is large, there are quite a few OOD data with high confidence scores. We will show in the final model experiments that these high-confidence OOD data damages performance. Secondly, we study pseudo-labed on both ID data and OOD data. Figure 2(b shows that pseudo-labels ID data is balanced. However, they are rather imbalanced on OOD data (Figure 2(c)). This is attributed to the different distribution they drawn from. Samples with certain pattern bias to certain classes. ID data bias to ID classes uniformly because they are sampled by the same distribution. However, with little probability, OOD data will also bias to ID classes uniformly since they have little relevance to ID data.
3

---

Under review as a conference paper at ICLR 2022
PL model performance
Confasion Matris of Pt model
X4
Azcura (56)
Prediction
20
=
Class MinMatch (%)
Ground Truth
(a) Performance of PL.
(b) Confusion Matrix of PL model (Mismatch Ratio 100%)
Figure 3: (a) PL model degrades the mismatch ratio increases. (b) Confusion matrix of PL model when the mismatch ratio = 100%. It demonstrates that the imbalance of pseudo labels on OOD data affects PL's performance. lot of ID samples with class 1-5 miselassified into class 0. Also, as the PL. process goes on, the imbalance pseudo-labels on OOD data get even worse.
model. As an old saying goes, a good beginning is half done. However, such imbalance of the first set of pseudo-labels starts PL model badly when there large portion of OOD data, putting the model in danger of imbalanced learning. We run vanilla PL and show that the imbalance of pseudo-label. harms the performance. Figure ((a] shows the performance of PL model with different OOD ratios. In accord with Oliver et al. (2018). PL. model degrades as the portion of OOD data gets larger. Figure 3(b) displays the confusion matrix of the PL model on the whole test set containing both ID and OOD data. Since only 6 classes are known to the confusion matrix is a rectangle. We can see almost all the OOD samples (class 6-9) are classified as class O, which means the imbalance effect OOD data gets possible reason is that, unlike labeling on ID data, supervision of labeled data can not help correct pseudo-labels on OOD data also influences classification performance on ID data. Samples of major classes (class 0) overwhelm the loss and gradient, leading a degenerate model (Lin al., 2017). We can see the PL model mistakenly classifies many of data with class 1-5 into class O.
Final Pseudo-Labelin
worse the PL training goes The
OOD data. Thus the imbalance continuously deteriorates. The imbalance
3.3 PSEUDO-LARELING STRATEGY FOR OOD DATA
The previous section shows OOD data hurt performance of vanilla PL. Then here comes the ques- tion: Assuming that we already know which data are OOD, how do we use these OOD data? Is omitting them the best way? If not, what are the better pseudo-labels for them? To answer these questions, investigate four strategies to create pseudo-labels for OOD data:
Baseline. This baseline omits all the OOD data and only trains on the labeled ID data.
Re-Assigned Labeling. This strategy assigns data of each OOD class to an ID class. It ensures that different OOD class is assigned to different ID) class, keeping the semantics unchanged between OOD classes. For example, (ship, trunk, airline, automobile) cat, deer, dog). This strategy can be seen as training a classifier of "super-classes".
be assigned to (bird,
Open-Set Labeling. This strategy is named after the related setting Open-Set Recogni- tion (Scheirer et al., 2013]] Bendale & Boult 2016). This strategy treats all OOD data as one unified class KID + I. Thus this model outputs probability over KID + classes
Oracle Labeling. This strategy probability over KID + KOOD classes.
the ground truth of OOD data. Thus this model outputs
Note that Open-Set Labeling and Oracle Labeling can classify samples into more than KID classes. However, during evaluation, we only classify samples into KID ID classes. For these model, the predicted label ý of a test sample is calculated as:
((x) arg max f(y
yEy'n
The overall comparison of the four strategies is illustrated in Figure We also report test accuracy when class mismatch ratio is 100%. From Figure 4 we can get several important conclusions.
4

---

Under review as a conference paper at ICLR 2022
ID data
OOD data
Decision Boundary
(a) Baseline 76.21% (-)
Re-A Re-Assigued Labeling 75.99% (1)
(c) Open-Set Labeling 77.95%6(1)
(c) Oracle Labeling 79.29% (1)
Figure 4: Four strategies of how to label OOD data. Different shapes represent different ground truths. Data with the same color are labeled the same classes. shape with a solid outline means it is an ID sample while OOD data are represented with dashed line. (a) No Labeling acts baseline where OOD data are omitted. (b) Re-Asnegeed Labeling re-labels OOD data to certain ID classes. (c) Open-Set Sett Labeling labels all the OOD data as unified class. (d) Oracle Labeling uses the ground truths of OOD data. Test accuracy under 100% class mismatch ratio reported.
Re-Assigned Labeling underperforms baseline little] This indicates that assign samples with OOD classes to ID classes does not help the model distinguish between ID classes even we somehow know which OOD data are semantically different. It also reveals that performing vanilla PL on OOD data may never help even if we do it perfectly. (2) Open-Set Labeling outperforms baseline, which indicates improves the performance if we label the OOD data as a class other than classes. (3) We see Oracle Labeling improves the performance and achieves the best result among the four strategies. It mcans that in addition to label OOD data as extra classes, if we can further assign OOD data with different semantics to different classes, the model will achieve better results.
3.4 SUMMARY SECTION
In this section, we study the behavior of Pseudo-Labeling model in class-mismatooe SSL. We summarize several important conclusions here:
Conclusion 1: Classification model trained with labeled ID data creates imbalanced pseudo-labels on OOD data while on ID data, remains balanced.
Conclusion The vanilla PL process makes the imbalance ing the classification performance ID data.
pscudo-labels deteriorate, damag-
Conclusion 3: Labeling OOD data ID classes does not help and may even perform a little worse.
Conclusion 4: It is beneficial to label OOD data as extra classes different from ID classes. If we can further label semantically different OOD data different classes, the perfor- mance can be further improved.
4 METHOD
Based the findings in Section we proposed T-Model (named after its shape) for class- mismatched SSL. T-Model trains a classifier / that will output the posterior distribution over KID + K classes, ie., f(y(x) E k which can be known advance (i.e., K KOOD) or be set as a hyper-parameter Similar to vanilla PL, define confidence with the same form Equation 2 However, this confidence is little different from its original definition in Hendrycks Gimpel| (207)), for we only calculate the maximum probability of the K classes instead of all. Therefore, rename it In-Distribution confidence (ID confidence). For evaluation, we predict labels using Equation 3 T-Model aims to solve the following questions:
f(ylx) 1. K is the number of extra classes,
Problem I: how to avoid imbalanced pseudo-labels in PL model? (Conclusion
Problem 2: how to avoid labeling OOD data as ID? (Conclusion 3)
Problem 3: how to create proper pscudo-labels for unlabeled OOD data? (Conclusion
Asssinn 4 OOD classes to ID classes of 6 causes imhalance. But test the accuracy selected 4 classes and find they show similar result.

---

Under review as a conference paper at ICLR 2022
distribution of pseudo labels
balanced clustering on K classes
< >
c(x) <
ID confident c(x)
12200 deta
Kip +1
poeudo labels for high-cortidenzedde data
possdlollaedd for Jon cow-ccefidence data
Kio
unlabeled data Y-model
Kio
(a) Re-balanced Pseudo-Labeling
(c) Semantic Exploration Clustering
Figure 5: Illustration of T-Model and its two main branches. (b) is the main structure of MModel where we judge by the II confidence if certain unlabeled data belongs IID classes not. The data with high confidence will perform Re-halanced will get their pseudo-labels by Semantic Exploration Clustering. (a) Re-balanced Pseudo-Labeling truncates the number of pseudo-labeled data to the minimum, making the pseudo-labels balanced and filtering out OOD data. (c) Semantic Exploration Clustering simulates the process of learning from ground truth labels of OOD data, creating pseudo-labels on extra K classes by clustering.
while those with low confidence
T-Model consists of two main branches Re-balanced Pscudo-Labeling (RPL) and Semantic Ex- ploration Clustering (SEC). RPL acts high-confidence data to solve Problem T 2
SEC acts low-confidence data to solve Problem 3 We describe the two branches the following sections. The overview of M-Model is illustrated in Figure 3
4.1 RE-BALANCED PSEUDO-LABELING
As is illustrated in Section 3.3 the main problem of vanilla PI. is that a large number of OOD data with high confidence scores have imbalanced pseudo-labels. One possible solution is re-weighting the unlabeled sample (Guo et al. 2020) or using other methods in the imbalance learning field. However, even if we solve the problem of imbalance learning, labeling OOD data as ID classes also may damage the performance (Conclusion 3). In this paper, we use simple method Re- balanced simultaneously solve imbalance (Problem [ and incorrect recognition (Prob- lem 2). It produces set PP of pseudo-labeled samples in three steps:
Pseudo Labeling
min Hx E P.. f(v x) > T),,
(4)
VEYID
Ty Nth.biggest( x) x D,)),
1,2, .... KID,
1,2,...,
(5)
P =
U ((x,)) fly x) 2 xg Duh. yEYID
(6)
where Nth biggest represents the N-th th biggest value of the given set. RPL first calculates the minimum number of pseudo-labeled samples for each class by Equation Then truncates the number of pseudo-labels of each ID class to that number by Equation 5 6 The process of RPL is illustrated Figure
First, it enforces the pseudo labels on ID classes to be balanced, solving Problem T Second, as is shown in Section 3 the set of high-confidence data is mixture of ID and ODD data. Due to Conclusion T the pseudo-label distribution of such a set is a sum of imbalanced and balanced ones, thus still imbalanced. However, by selecting only top-N confident samples for each ID class, we will keep ID data and omit many OOD data since confidence on ID data tends to be higher than OOD data (Hendrycks & Gimpel 2017). This process solves Problem
4.2 SEMANTIC EXPLORATION CLUSTERING
As is demonstrated in Section 3.3 if we know a set of samples is OOD, it will improve the perfor- mance if label them unified class KTD 1. But the best way use their ground truths (Conclusion 4p. However, it is impossible to access their ground truths since they are unlabeled. We resort to using Deep Clustering methods (Caron al.) 2018 Asano et al. (020)) to mine their se- mantics and approximate the process of learning with the ground truths. Here, we use the balanced clustering method Asano et al. (2020): Caron et (2020) to create pseudo-labels for these OOD data. Assuming there are M samples recognized as OOD, we first compute their soft targets:
min (Q, log P), U(K, () = Q E K Q1 = 1,Q QEU(K,M)
1
1 M
(7)
6

---

Under review as a conference paper at ICLR 2022
where P R Psj f(KID + + i f is the normalized posterior distribution extra classes, Le.. algorithm Cuturi (2013) to optimize Q. Once get Q. harden the label by picking the class with the maximum predicted probability and mapping it to the extra K classes:
f(Ki Ex f(KiD + ))x)). .) We use the Sinkhom-Knopp
3 KID + arg max Q.j.
(8)
9 is used as the pseudo-label for xj. We perform SEC on the set of data with ID confidence lower than a threshold 7/, i.e. (x(c(x) < 7}.
RELATED WORK
Class-Mismatched Semi-Supervised Learning. Deep Semi- Suni-Supervised Learning suffers from performance degradation when there are unseen classes in unlabeled data (Oliver et al. 2018). As the proportion of such -itoo---istribution (OOD) data get larger, the performance drop more. To problem, several methods are proposed. [Cenn et al. (2020) formu-
cope with such a class-mismadt late a sequence of ensemble models aggregated accumulatively on-the-fly for joint self-distillation and OOD filtering. Guo et al.] (2020) re- re-weight the unlabeled data by meta-learning to decrease the negative effect of OOD data. Huang et al. (2020) recycle transferable OOD data by means of adversarial learning. Different from all these methods, we conduct a comprehensive study on Pseudo-Labeling (PL) and give useful guidance on how to do better in class-mismatched SSL.
Pseudo-Labeling The method of Pseudo-Labeling, also known as self-training, is simple and effective way for Deep Semi-Supervised Learning (Lee et al.) 2013); [Shi et al.) 2018: Arazo et al. 2020 Iscen et al. 2019). Despite its simplicity, it has been widely applied to diverse fields such as image classification (Xie al. 2020j. natural language processing (He et al.
2020) and object
detection (Rosenberg et al.] The use of a hard label makes Pseudo-Labelle closely related
to entropy minimization (Grandvalet & Bengio, 2004).
Deep Clustering. Deep clustering methods improve ability of traditional cluster methods by leveraging the representation power of DNNs. A common means is to transform data into low- dimensional feature vectors and apply traditional clustering methods (Yang et al. al.| [ Caron et al.) 2018). In Self-Supervised Learning, clustering methods are used to learning meaningful represen- tation for downstream tasks (Caron et al.] 20188] AAsano et al.) 2020;; Caron et al.) [2020). Modern Deep Clustering can learn semantically meaningful clusters and achieves competitive results against supervised learning (Gansbeke et al. 2020).
6 EXPERIMENTS
To validate the effectiveness of our T-Model, we conduct experiments on different benchmarks. Dataset. We test our methods on two datasets as in Oliver et al. g (1) CIFARIO: we use the same configuration in Section 3.1 SVHN: The data set contains 10 categories digits "0"--9". We select the first "0"- as ID c classes and the rest as OOD. For each class, we randomly select 100 images labeled data. Meanwhile, 20,000 images are randomly selected from all the 10 classes as the unlabeled data set. The class-mismatch ratio is set as (0%, 25%, 50,, 75%, 100%].
Implementation Details. We use the same network and training protocol as Section 3.1 We first train a classification model only labeled data for 100 epochs without RPL and SEC. We update pseudo-labels every 2 epochs. For both datasets, we set T 0.95, 0.3,K = 4. We use an exponential moving average model for final evaluation as in Athiwaratkun et al. (2019)
6.1 COMPARE WITH TRADITIONAL SSL METHODS
In this subsection, we compare our methods with four traditional SSL methods Pseudo- Labeling (Lee et al., 2013). II- M-MMdee (Laine & Aila 2017). Mean Teacher Tarvainen Valpola 2017) and VAT (Miyato et al. 2019). Figures 6(a) 6(b) show the results. Traditional methods suffer from performance degradation as the mismatch ratio increases. They usually get worse than the su- pervised baseline when the mismatch ratio is larger than 30% on CIFAR10 and SVHN. In contrast, our methods get steady improvement under all class mismatch ratios. The reasons can be attributed follows. First, our method is aware of the existence of OOD data. We do not treat OOD data
7

---

Under review as a conference paper at ICLR 2022
01444
(a)
(b)
(d)
Figure 6: Comparison with existing methods on CIFARIO and SVHN dataset with Nede-ResNet- 28-2 network. Class mismatch ratios are varied. The shaded regions with the curves indicate the standard deviations of the accuracies over five runs. (a) (b) Comparison with traditional SSL meth- ods. These methods suffer from performance degradation as the mismatch ratio increases. (c) (d) Comparison to two existing class-mismatched SSL methods - UASD and L Our methods per- form better in almost all the experimental setups.
like IDD data, which can hurt performance. Second, we reuse OOD data by exploring their semantics which proves to be useful in Section 3.3] Therefore, even when the class-mismatch ratio gets 100%, performance of T-Model is still better than supervised baseline.
6.2 COMPARE WITH CLASS-MISMATCHEDD SSL METHODS
In this subsection, we compare our method with two existing class-mismatched SSL methods - UASD (Chen et al. 2020) and D (Guo et al. 2020 For fair comparison, use Pseudo- Labeling as the base method of DS From Figures 6(c)
6 we can see our methods are superior to these two methods in all settings. It is noticeable that DS underperforms supervised baseline when all the unlabeled data are drawn from OOD classes. This is attributed to the fact that D uses down weighting strategy alleviate the negative effect of OOD data and does not change the form of unsupervised loss. But we have shown in Section 3.3 that labeling OOD data as ID classes damage performance anyhow. On the contrary, T-Model uses the OOD data in the right way- simulating the process of training them with their ground truth labels. As a result, our method shows superiority especially under a large dlass-mismate ratio. We also notice that the performance curve of T-Model appears a U-shape (obvious on CIFAR10). A possible reason is that RPL and SEC compete with each other. RPL tends to make samples get high prediction on ID classes while SEC tends to make samples get a high prediction on OOD classes. When the class-mismatch ratio reaches 0% (100%), RPL (SEC) dominants the other. In this circumstance, one works without any disturbance of the other. However, when the miimmahh ratio is 50%, they compete fiercely with each other, causing many incorrectly recognized ID or OOD samples.
6.3 ABLATION STUDY
In this section, we validate the functionality of RPL and SEC. We conduct experiments on CIFARIO benchmark as in the analysis section 3
Validation of effectiveness of RPL and SEC. We conduct ablation studies under different class- mismatch ratios and report the averaged test accuracy and standard deviation of five runs. As usual, we vary the class-mismmn mismatch ratio. Table [ displays the results. Firstly, comparing the first line and second line of the table, RPL not only outperforms vanilla PL in high class-mismatch ratio scenario but also improves in low class mass-missatch ratio scenario. This reveals that balanced pseudo-labels always help since once the model creates imbalanced pseudo-labels, it will deteriorate when there are not enough measures to correct it. Secondly, comparing the second and third line, it shows that RPL alone alleviate the performance degradation but it can not prevent it, in accord with Conclusion 3 When using SEC, M-Model gets better results than supervised baseline when the class mismatch ratio is high. Besides, comparing the third and last lines, we see that when cluster OOD) data and exploring their semanties instead of using unified class to label them, the performance improves.
RPL helps filter out OOD data and solve imbalance problem. It is noticeable that the last two lines of Table [ show that without RPL, SEC alone can not achieve better performance than supervised baseline. We will show the reason here. Figure plots the proportion of OOD data that are labule- ID classes. It reveals that without RPI., iee, using vanilla PI., the number
8

---

Under review as a conference paper at ICLR 2022
Table 1: Validation of RPL. and SEC. The experiments conducted on CIFARIO with Wide- ResNet-22-2 backbone. Class-mismatch ratio varies from 0 to 100%. Check RPL or not means using RPL or vamilla PL. K means we do not use SEC. means we label all the low confidence data as unified class KiD + 1.
Ratio (%)
RPL.
SEC
25
50
75
100
78.23 + 0.39
76.50 + 0.30
75.85 = 0.35
74.60 + 0.27
73.97
0.30
0
78.76 + 0.49
76.92 + 0.32
76.80 + 0.31
75.48 + 0.27
75.38 + 0.27
78.86 + 0.37
77.17 + 0.55
77.12 = 0.37
76.88 0.35
77.27 0.39
4
78.11
= 0.45
77.43 + 0.29
77.46 = 0.37
76.38 =
74.18 + 0.32
4
78.98
+ 0.49
78.93 + 0.28
77.78 + 0.31
77.90 + 0.21
78.50 + 0.27
baseline
76.21 = 0.21
of incorrectly recognized OOD data keep increasing the training proceed. While with RPL, this ratio rapidly drops to O. This proves that RPL help filter out OOD data by utilizing the imbalance present the confusion matrix of T-Model on the full test set (all
property of OOD data. Further, the 10 classes) CIFARIO. Compared to vanilla PL in Figure 3(b M-Model does not suffer from imbalance problem, as a result of which, its performance
degraded.
Effect of extra class number K. We vary the number of extra classes K. Figure 7(c) shows the result on CIFARIO with class mismatch ratio 100%. The gray dashed line is the supervised baseline. The red dashed line is the Oracle Labeling strategy in SectionB..3 is the upper bound of Y-Model in this setup. Without SEC (K O). T-Model underperform supervised baseline. Using SEC (K 1), MModel is always better than baseline. Also, it reaches its best performance when K equals the actual number of OOD classes. This demonstrates that by simulating the process of training on OOD data with ground truth, SEC helps classification model on ID data.
Proporiko paradi-taheted
Confusion Matris of Y-Modei
CIFARIS. 180% Clars Mismatrh
Chali Lonkie
Trsperiin (N))
Ground
Spech
Prediction
Number Extre Chaona
(b)
Figure 7: Three experiments on CIFARIO benchmark to validate the functionality of RPL and SEC. (a) The proportion of OOD data pseudo-labeled as ID classes with the training goes on. Vanilla PL makes this ratio keep increasing while RPL. makes drop to It demonstrates that RPL. help filter out OOD data. (b) Confusion matrix on the full test set of CIFAR10 10. RPL solves the imbalance problem. (c) Test accuracy versus the number of extra classes K. reaches the best performance when K equals the actual number of OOD classes
CONCLUSION
In this paper, we analyze Pseudo-Labeling in class-mismatched semi-supervised learning where there are unlabeled OOD data from other classes. We show that Pseudo-Labeling suffers from per- formance degradation due to imbalanced pseudo-labels on OOD data. The correct way to use OOD data is to label them as classes different from ID classes while also partitioning them according to their semantics. Based the amalysis, proposed Modell and empirically validate its effec- tiveness. We believe our findings are not only beneficial to PL methods, but also inspiring to other methods like consistency regularization and holistic methods on how to effectively use OOD data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E O' D'Conno, and Kevin McGuinness. Pseudo-labeling and confirmation bias deep semi-supervised learning. In IJCNN, pp. 1-8. IEEE, 2020.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In ICIR, 2020.
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many con- sistent explanations of unlabeled data: Why you should average. In ICLR, 2019.
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR, pp. 1563-1572. 2016.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, pp. 5050-5060, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, pp.
2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class
distribution mismatch. In AAAI, pp. 3569-3576, 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, pp- 2292-2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. SCAN: learning to classify images without labels. In ECCV, pp. 268-285, 2020.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, pp. 529-536, 2004.
Lan-Zhe Guo, Zhenyu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, pp. 3897-3906, 2020.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting OOD data with cross-modal matching for open-set semi-supervised learning. CoRR, abs/2108.05611 2021.
Zhuo Huang, Ying Tai, Chengjie Wang, Jian Yang, and Chen Gong. They are not completely useless: Towards recycling transferable unlabeled data for class-mismatched semi-supervised learning. CoRR, abs/201 I. 1.1522,, 2020.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In CVPR, pp. 5070-5079, 2019.
Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. NIPS, pp. 3581-3589, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2022
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2999-3007, 2017.
Huixiang Luo, Hao Cheng, Yuting Gao. Ke Li, Mengdan Zhang, Fanxu Meng, Xiaowei Guo, Feiyue Huang. and Xing Sun. On the consistency training for open-set semi-supervised learning. CoRR, abs/2101.0222 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. TPAMI, 11(9):1979-1993, 2019.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurlPS, pp. 3239-3250, 2018.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self- self-training of object detection models. In WACV, pp. 29-36, 2005.
Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open recognition. TPAMI, 35(7):1757-1772. 2013.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In ECCV, pp. 311-327, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NIPS, pp. 1195-1204, 2017.
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR, pp. 10684-10695, 2020.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In ICML, pp. 3861-3870, 2017.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, pp. 438-454, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.

---

Under review as a conference paper at ICLR 2022
A ALGORITHM
Algorithm T-Model algorithm
Input:
Labeled dataset Dj {(x,, Bt)) 1=) and unlabeled dataset D., = {Xus} Classification
model f parameterized with Φ, ID class number KYD, extra class number K, total epoch number E, pretrain epochs Ept, interval to update pseudo-labels Epl, pseudo-labeled set P. confidence calculation function c.
2: N millyErr (x E D,, f(y x) >)T> T]| f,T)
1: function REBALANCED
3:
Ty Ntghbiggest( (ff x) x Dul), y= .I,
4:
UyEViO (xx y) fly x) 2 Ty Du)
5: return
6:
function SSNANTTEEEE.ORATTNNCUSSSTEEOO,,
7:
S [x|c(r) <)
8:
M 1
9:
P.j f [(KID + 11x) Ek=1 J(K + k(x,);
1,,,, 1.2,..., K, j= j=1,2....,
10:
Solve by Sinkhorn -nopp algorithm and get
11:
91 + KID + arg max; Qiy j = 1,2,....
12:
C+ {(x,,
13:
return C
14:
for == to E do
15:
ife < Pept then
16:
train fo with standard supervised learning on Dr L=
b Pre-training phase
18:
else
train fo with standard supervised learning on D, UP
b PL training phase
20:
e S. Ept and %ppp then
P+
22:
P.t REBALANGGDDDSSUDDLLABBLLIIDDDD for T)
Perform RPL Perform SEC
P.+-- SHNANTTCEEFFOOKKTOOOCUUSTEEEEE,D
24:
P P. UPP
return classification model fe
B EMBEDDING VISUALIZATION
We visualize the embedding of supervised baseline, vanilla PI. and u-Model on CIFAR10's test set with all the classes by t-SNE. Figure 8 shows the result. OOD data are mixed with ID data since the supervised baseline does not see unlabeled OOD data. PL mixes OOD data with samples of certain classes (class 0). This attributed to that their pseudo-labels are biased toward this class. Also, we can not clearly distinguish between OOD classes. In contrast, T-Model can not only make ID data distinguishable but also forms meaningful clusters on OOD data.
-133
Bascline
(b) PL.
T M-dodel
Figure t-SNE visualization of supervised baseline, vanilla and M-dodel on CIFAR10's test set. Class 0-5 are ID classes, shown by non-transparent circles. Class 6.9 are OOD classes, represented by semi- cemi-transparent circles.
12

---

Under review as a conference paper at ICLR 2022
C UNIVERSALITY OF ANALYSIS CONCLUSIONS
To illustrate the universality of the conclusion in Section We experiment on totally five kinds of datasets. ( We use (n/m) to represent ID classes and OOD classes.)
CIFARIO (6/4) and CIFARIO (5/5) both created from CIFARIO (Krizhevsky et al, 2009 CIFARII (6/4) takes the 6 animal classes as ID classes and 4 vehicle classes as OOD classes. CIFARI R10 (5/5) has 3 animal and vehicle classes for both ID and OOD classes. We select 400 labeled samples for each ID classes and totally 20,000 unlabeled samples from ID and OOD classes
SVHN (6/4): We select the first "0"-"5" as ID classes and the rest OOD, We select 100 labeled samples for each ID class and totally 20,000 unlabeled samples.
CIFAR100 (50/50): created from CIFARIOU Krizhevsky et al. 2009). The first 50 classes are taken ID classes and the rest as OOD classes. We select 100 labeled samples for each ID classes and totally 20,000 unlabeled samples.
Tiny ImageNet (100/100): created from Tiny ImageNet, which is a subset of Ima- geNet (Deng et al. 2009) with images downscaled to 64 x 64 from 200 classes. The first 100 classes are taken as ID classes and the rest as OOD classes. We select 100 labeled samples for each ID class and totally 40,000 unlabeled samples.
Here we use C represent CIEAR, TIN represent Tiny ImageNet for short. For each dataset, we use the same experimental setup as in Section
C.I IMBALANCE OF PSEUDO-LABELS
To demonstrate the imbalance of pseudo-labed on OOD data, we compute two metrics measure the extent of imbalance.
The KL divergence of pseudo-label distribution q and the uniform distribution u:
kl = KT(glu)) = Σ 4s log q
The ratio of the 'majority class' and 'minority class':
max; 4
min; g
The results on these datasets are displayed in Table
Table 2: Illustration of the extent of imbalance of pseudo-labels on different datasets.
C10 (6/4)
C10 (5/5)
SVHN (6/4)
C100 (50/50)
TIN (100/100)
kl
ID OOD
0.0131 0.3636
0.0169
0.0006 0.1031
0.0771 0.4177
0.5319
0.0429
0.9020
ID
1.5390
1.6235 2,4878
1.1047 4.4797
5.1834 213.7661
107.6684
OOD
18.3183
1279.9238
C.2 PSEUDO-LABELING STRATEGY FOR OOD DATA
We report the test accuracy of the four pseudo-labeling strategies in Section 3.3 The mlasmmmismatch ratio is set to 100%. Table shows the results.
Note that Re-Assigned Labeling has many possibilities. If there are ID classes and OOD classes, A possible assignments exist. It is impossible to experiment on all of them. To deal with it, randomly choose 10 possible assignment and pick the maximum performance among them.
13

---

Under review as a conference paper at ICLR 2022
Table 3: Performance of four different pseudo-laheling strategies different datasets.
C10 (6/4)
C10 (5/5)
SVHN (6/4)
C100 (50/50)
TIN (100/100)
Baseline
76.21 (-)
82.36 (-)
(-)
58.68 (-)
39.08 (-)
Re-Assigned Labeling
75.99 (-0.22)
76.90 (-5.46)
84.40(-3.93)
50.52(-8.16)
34.90 (-4.76)
Open-Set Labeling
77.95(+1.74)
83.54 (+1.18)
88.31(-0.02)
58.76(+0.08)
40.06(+0.86)
Oracle Labeling
79.25(+3.04)
87.26(44.9)
92.07(+3.74)
63.9(+3.737)
45.28(+6.78)
D COMPARISON ON MORE DATASETS
We compare our method with vanilla PL and the two class- mismatched methods in Section 6.2 We the following hyperparameters:
CIFARIO (6/4): т = 0.95, 7 0.3, Ept = 50, E = 2, K =4
SVHN (6/4): = 0.95, 0.3, Ept 50, Ept = 2, 4
CIFAR100 (50/50):7 0.95, 0.18, Ept 50, Fpg = k 20
Tiny ImageNet (100/100): T - 0.9, 0.15, Epe = 50, Ept = =2 =
Table 4: Comparison on four datasets with different class- mismatch ratios. The backbone is Wide- ResNet-28-2 for all experiments.
Mismatch Ratio
0%
25%
50%
75%
100%
CIFAR10 (6/4)
PL
78.23 0.39
76.50 = 0.30
75.85 + 0.35
74.60 0.27
73.97 0.30
UASD
78.80 + 0.46
77.33
+: 0.29
77.28 + 0.33
76.90 + 0.59
76.27 + 0.30
D Y-Model baseline
78.21 = 0.32
77.83 = 0.30
76.78 + 0.41
76.28 = 0.33
75.27 + 0.37
78.98 + 0.49
78.93
+ 0.28
77.78 + 0.31
77.90 + 0.21
78.50 + 0.27
76.21 + 0.21
SVHN (6/4)
PL UASD
92.02 + 0.30
90.12 + 0.30
88.76 + 0.29
87.07 + 0.23
86.10 + 0.79
92.16 + 0.29
91.32 + 0.32
89.34 + 0.30
89.26 + 0.27
88.62 + 0.27
DS -Moddel baseline
92.07 + 0.33
90.35 + 0.30
89.2 + 0.56
88.76 + 0.27
87.43 0.31
92.15 + 0.30
91.48 + 0.27
89.95 + 0.30
89.47 + 0.36
89.58 0.56
88.33 + 0.19
CIFAR100 (50/50)
PL. UASD D T-Model baseline
61.68 + 0.30
60.20 + 0.25
60.12 + 0.30
57.79 + 0.27
57.62 0.57
60.30 + 0.26
59.42 + 0.61
59.92 + 0.53
58.94
58.74 + 0.50
60.68 + 0.67
60.60
= 0.40
59.22 + 0.33
+
59.56 + 0.57
62.10 0.30
61.26 0.40
60.68 + 0.24
60.12 0.27
59.46 0.36
58.68 + 0.25
Tiny ImageNet (100/100)
PL.
43.42 + 1.03
42.88 + 1.51
41.94 + 1.49
39.72 +: 2.30
38.94 2.41
UASD 1S T-Model
43.34 + 0.78
42.34 = 0.61
41.8 + 1.338
41.08 + 1.16
36.16 16 + 1.05
44.42 + 0.43
43.48 + 0.40
42.42 + 0.95
43.22 + 0.60 41.76 + 0.63
baseline
39.66 + 0.52
We cannot finish D on a mid-scale dataset like Tiny ImageNet within a reasonable time.
For CIFARI00 (50/50) and Tiny ImageNet (100/100), we use weight factor to trade off the loss labeled set D; and pseudo-labeled set P. which ramps up with function X
-5 x (1 min 40,000
ster
"))²) where iter is the number of training steps from Eet-
exp
14


Reviews:
[{"review_id": "skEu5hD6Mcu", "reviewer": "Reviewer_xUsz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) OOD SSL is a very interesting topic, and it is also a realistic setting in real world applications.\n\n(2) The insight that pseudo-labeled data is often class-imbalanced is also interesting and can guide future research in this direction.\n\n(3) Authors conducted comprehensive experiments and ablation studies on some small-scale benchmarks.\n\n(4) This paper is well-written and easy to follow.\n\nCons:\n\n(1) CIFAR-10 and SVHN are much easier than CIFAR-100, TinyImageNet or ImageNet. To show the effectiveness of this method on large-scale dataset, it may be necessary to conduct some experiments on larger benchmarks, especially when the performance improvements on small-scale benchmarks are relatively marginal (about 1.7~2.8% on CIFAR-10 as in Table 1 and ~3% on SVHN as shown in Figure 6). \n\n(2) The current state-of-the-art method on semi-supervised learning is FixMatch [1], however, the authors only compared against worse-performing methods in SSL like Mean Teacher. Therefore, can we still obtain similar observations and probelms when we are using a better SSL framework like Fixmatch? Will the method still perform better when it is integrated with FixMatch?\n\n(3) The OOD SSL is in fact investigated by a con-current work [2], which was released on arXiv about 10 months ago and was in submission to ICLR 2022 too. The rate of this paper will be not reduced for not comparing [2], however, I am curious about the key differences between this paper and [2]. If the author can provide some comparisons and insights, I would be very grateful.\n\n(4) The idea of simply truncating the number of pseudo-labeled data to the minimum is quite brute force, is there any ablation studies on other popular long-tailed recognition method re-weighting, re-sampling, marginal loss or even multi-expert framework?\n\n(5) Will the code of this method be made public available?\n\n(6) Using the expensive and time-consuming clustering method on large-scale benchmark like ImageNet may be of a big challenge to computational efficiency.\n\n[1] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.\n\n[2] Cao, Kaidi, Maria Brbic, and Jure Leskovec. \"Open-World Semi-Supervised Learning.\" arXiv preprint arXiv:2102.03526 (2021).\nHarvard\t", "summary_of_the_paper": "This paper researched on the semi-supervised learning task when there are unlabeled out of distribution data from other classes. Several interesting issues of class mismatch SSL are studied, including the reasons for the performance degradation of PL in OOD data and how to better pseudo-label OOD data to provide a more balanced semantic distribution. To address above-mentioned problems, Y-model, consisting of two components – Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC), are proposed. Authors conducted several experiments to show the effectiveness of the proposed method, such as CIFAR-10 and SVHN. ", "summary_of_the_review": "I like this paper in general, but I have some concerns on the effectiveness of this method in large-scale benchmarks. Also using the expensive and time-consuming clustering method on large-scale benchmark like ImageNet may be of a big challenge to computational efficiency. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mduDjnjShTW", "reviewer": "Reviewer_kZv3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##### Strength\n\nThe paper gives insights to the behavior of pseudo-labels under the class-mismatch problem experimentally, and provides an algorithm that can alleviate the issues that are discussed. It performs better than the pseudo-label baseline, but also performs better than two recent class-mismatch SSL baselines (especially with high mismatch ratio), so the benefits shown in the experimental results are one of the main strengths of the paper. \n\nThe empirical comparison of open-set labeling and oracle labeling is interesting and motivates the proposed methodology, and makes the story clear. The ablation study further motivates the design of the Upsilon method.\n\n##### Weaknesses\n\nSince Upsilon is based on the findings and discussion in Section 3, it would be better to have at least one more dataset (for example, SVHN that is used in a later section) to see if the findings are not specific to a single dataset.\n\nIn Figure 2, it would be interesting to investigate if similar results hold with random splits for ID/OOD classes, instead of the animal/vehicle split. (I think the SVHN experiments later on somehow show that the splits doesn't matter so much, since the 0-5/6-9 split is not as semantically meaningful as the one in CIFAR-10.)\n\nIt is interesting and surprising how open-set labeling is worse than oracle labeling, although both are evaluated with only ID classes.  It would be interesting to have some discussions about potential underlying mechanisms of this experimental result.\n\n##### Minor comments\n- In the caption of Figure 3, it says \"A lot of ID samples are misclassified into one class.\" but is this correct? Should it be \"OOD samples\"?\n- Which animal class is class 0 in Fig.2(c) and Fig.3(b)? \n- Table 1: typo \"aof\"\n\n=======================\n##### After rebuttal\n\nThank you for the additional experiments and for answering my questions. I would like to keep my positive score. The answer to \"It is interesting .. experimental result.\" makes sense. So there may be some implicit transfer learning going on.", "summary_of_the_paper": "This paper works on the class-mismatch semi-supervised learning problem, where the assumption is that the unlabeled samples include class labels that do not appear in the labeled data, i.e., out-of-distribution (OOD) data, in addition to the class labels that appear in the labeled data, i.e., in-distribution (ID) data. It is known that if there are OOD data included in the unlabeled data, it can degrade performance of semi-supervised learning algorithms. This paper focuses on one of the semi-supervised learning methods, which is the pseudo-labeling method. The paper first investigates how using pseudo-labels in the class-mismatch semi-supervised learning setup can be problematic. The experiments show that proportions of high-confidence data in ID data is larger than OOD data. Experiments also show that pseudo-labels (w.r.t. class predictions) are more balanced in ID data while it is unbalanced in OOD data at the beginning of training (with a pretrained model). This phenomenon becomes exaggerated at the end of training, where almost all of the OOD data is predicted into a single ID class. Assuming we know the labels of the unlabeled data, the paper performs further experiments that compare a few methods, and finds out that it is ideal if we have access to the underlying single class labels of OOD data, and solve the problem as a multi-class classification problem with K_id + K_ood (number of classes of ID and OOD) classes. Based on these observations, the paper proposes an Upsilon model, that consists of a re-balanced pseudo-labelling (RPL) and semantic exploration clustering (SEC). RPL aims to balance the distributions of pseudo-labels based on the observation that pseudo-labels become balanced/imbalanced with ID/OOD data, respectively. SEC is a workaround for not having ground truth labels of OOD data.  Experiments show comparison with both traditional SSL methods and class-mismatch SSL methods, and show how the proposed Upsilon model works better than these baselines. Ablation study shows why the components within the Upsilon model is necessary.", "summary_of_the_review": "The paper investigates the issues of using pseudo-labels in class-mismatch semi-supervised learning, and shows the benefits of the proposed method empirically. It performs better than recent class-mismatch semi-supervised learning methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lVwvCem_eJJ", "reviewer": "Reviewer_A13X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- This paper has an interesting observation on how does OOD data influence pseudo-label and resulting model performance.\n- The proposed method also seems to be original.\n\nWeakness\n\nI have several concerns about the experiments.\n1. First, as the vanilla semi-supervised learning baselines, stronger methods should be leveraged. Especially, as the problem definition of this paper is related to the pseudo-label in semi-supervised learning, the recent “pseudo-label based” semi-supervised learning methods such as FixMatch [1] and ReMixMatch [2] should be considered. VAT or Mean Teacher seems to be outdated. It will also be interesting to investigate if we combine the proposed r-Model with the existing pseudo-labeling-based semi-supervised learning methods.\n\n[1] Sohn et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS 2020.\n\n[2] ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring. ICLR 2020.\n\n2. It would be better to evaluate the proposed method on other datasets such as CIFAR100, STL, or maybe ImageNet dataset. In order to comprehensively understand the behavior of the proposed method, the proposed method should be evaluated on other datasets (at least on the MNIST dataset).\n", "summary_of_the_paper": "The authors tackle the pseudo-label in class-mismatched semi-supervised learning when there are unlabeled out-of-scope data from other classes. The authors propose a model consisting of (1) Re-balanced Pseudo-Labeling, which re-balances pseudo-labels on ID classes to filter out OOD data, and (2) Semantic Exploration Clustering, which uses balanced clustering on OOD data to create pseudo-labels on extra classes.", "summary_of_the_review": "This paper has an interesting observation and the proposed method seems to be original. However, as mentioned before, it would have been better to have additional training setups and baselines to emphasize the efficacy of the proposed method. I would like to rate this paper to be borderline, but I am open to changing the score after the rebuttal.\n\n=======================\n\nAfter rebuttal:\n\nAfter reading the authors' response, I think my concerns both on the compared baselines (FixMatch) and the evaluated datasets (ImageNet) are somewhat addressed. Therefore, I would like to increase the score to be 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2mcHitzD-86", "reviewer": "Reviewer_ayVg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper provides the empirical analysis of the Pseudo-Labeling model for ID and OOD data.\n2. This paper proposed a novel two-branched Υ-Model to solve the class miss-match issue in SSL.\n3. Experiments on different SSL benchmarks empirically validate the effectiveness of the proposed method.\n\nWeakness:\n1. The motivation for imbalanced pseudo-labels is not clear.\n2. The technical novelty and depth of the proposed approach are limited.\n3. The experiments are not extensive.\n\n", "summary_of_the_paper": "This paper focuses on the problem of semi-supervised learning (SSL) with class miss-match. The authors first empirically analyze Pseudo-Labeling (PL) in class-mismatched SSL and proposed a new method that consists of two components – Re-balanced Pseudo-Labeling (RPL) and Semantic Exploration Clustering (SEC). Experiments show that the proposed method achieves steady improvement over supervised baseline and state-of-the-art performance under all class mismatch ratios on different benchmarks. \n", "summary_of_the_review": "This paper is well presented and organized. The proposed ideas are interesting. However, there are several main issues as follows:\n1. the motivation is not clear and strong. The authors state that \" imbalance of pseudo-labels harms the performance”. It lacks justifications for this statement. Fig 3 (b) shows that imbalanced pseudo-labels of OOD would result in many of data with class 1-5 into class 0. However, if the pseudo-labels of OOD is not imbalanced, it may still force the ID data into the incorrect class.\n2. Fig 7 (c) shows that Υ-Model is sensitive to the Number of Extra Classes K. However, this paper does not provide a good strategy to choose K.  In the real case, we don’t know the actual number of OOD classes. \n\n3.  The Υ-Model may not work well in a large-scale dataset since the performance of the proposed SEC technical relies on the deep clustering algorithm. When the dataset becomes more complex or the number of OOD classes is large, then SEC may not work at all. It is better to show some results in large-scale datasets.\n4. Can Υ-Model extend to other SSL methods? If not, it could be a limation of the proposed method. It is better to show some empirical results.\n5. Since 100% Class Mismatch ratio is an extreme case, it is better to show a 50% ratio result for Fig 7 (c).\n\n6. What is the purpose of Fig 2 (a)?\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this work on OOD and pseudo-labeling presents interesting and strong results. The authors’ rebuttal has addressed some of reviewers’ concerns. Based on the current review and discussion, there are still several major concerns towards the expensive computational cost introduced by the clustering method, the lack of discussion around how the proposed work can be incorporated into SSL methods, and the sensitivity towards the selection of K.
Decision: Reject