OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization
Authors: Kuniaki Saito
Donghyun Kim
Kate Saenko
Venue: NeurIPS 2021 Poster (2021)
Keywords: Novelty Detection
Semi-supervised Learning
Open-set Recognition

Abstract:  Semi-supervised learning (SSL) is an effective means to leverage unlabeled data to improve a model’s performance. Typical SSL methods like FixMatch assume that labeled and unlabeled data share the same label space. However, in practice, unlabeled data can contain categories unseen in the labeled set, i.e., outliers, which can significantly harm the performance of SSL algorithms.
  To address this problem, we propose a novel Open-set Semi-Supervised Learning (OSSL) approach called OpenMatch.
Learning representations of inliers while rejecting outliers is essential for the success of OSSL. To this end, 
OpenMatch unifies FixMatch with novelty detection based on one-vs-all (OVA) classifiers. The OVA-classifier outputs the confidence score of a sample being an inlier, providing a threshold to detect outliers. Another key contribution is an open-set soft-consistency regularization loss, which enhances the smoothness of the OVA-classifier with respect to input transformations and greatly improves outlier detection. \ours achieves state-of-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10. The code is available at \url{https://github.com/VisionLearningGroup/OP_Match}.
  

OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
Kuniaki Saito' Donghyun Kim Kate Saenko
Boston University "MIT-IBM Watson Al Lab
[keisaito, donhk, saenko] s @bu. edu
Abstract
Semi-supervised learning (SSL) is an effective means to leverage unlabeled data to improve a model's s performance. Typical SSL methods like FixMatch assume that labeled and unlabeled data share the same label space. However, in practice, unlabeled data can contain categories unseen in the labeled set, i.e., outliers, which significantly harm the performance of SSL algorithms. To address this problem, we propose a novel Open-set semi-Supervised Learning (OSSL) approach called OpenMatch. Learning representations of inliers while rejecting outliers is essential for the success OSSL. To this end, Open unifies FixMatch with novelty detection based on one-vs-all (OVA) classifiers. The OVA-classifier outputs the confidence score of sample being an inlier, providing a threshold to detect outliers. Another key contribution is an open-set soft-consistency regularization loss, which enhances the smoothness of the OVA-classifier with respect to input transformations and greatly improves outlier detection. OpenMatch achieves state- of-thh-art performance on three datasets, and even outperforms fully supervised model in detecting outliers unseen in unlabeled data on CIFARIO. The code is available at https: : //gi thub. com coon/Vissionnnn www...ieeeaaaainnooooooooooooo/p _PMMachh.
1 Introduction
Semi-supervised learning (SSL) leverages unlabeled data improve model's performance [27, 38, 2, I, 35, 22, 37]. An SSL model can propagate the class information of a small set of labeled data large set of unlabeled data, which significantly improves the recognition accuracy without any additional annotation cost. A common assumption of SSL is that the label spaces of labeled and unlabeled data are identical, but, in practice, the assumption is easily violated. Depending on how was collected, the unlabeled data may contain novel categories unseen in the labeled training data, i.e., outliers. Since these outliers can significantly harm the performance of SSL algorithms [14], detecting them is necessary to make SSL more practical. Ideally, model should classify samples of known categories iee., inliers, into correct classes while identifying samples of novel categories as outlies. This task is called Open-set Semi-supervised Learning (OSSL) [44]. While OSSL is more realistic and practical scenario than standard SSL, has not been as widely explored.
Existing strong SSL methods [35, 41] do not work well for OSSL. For example, FixMatch [35] generates pseudo-labels using the model's predictions on weakly augmented unlabeled images and trains the model to match its predictions on strongly augmented images with the pseudo-labels. This method exploits the advantage of aseudo-labeling as well as regularizing a model with the consistency between differently augmented images. But, in OSSL, this risks assigning pseudo-labed of known categories to outliers, which degrades the recognition accuracy. possible solution to compute the SSL objective only for unlabeled samples considered to be inliers, where confidence thresholding is used to pick inliers. For instance, MTC [44] regards some proportion of samples as outliers by using Otsu thresholding [25]. However, this is not robust to varying proportions of outliers as discussed in
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Anomaly Soore for each class
Augmented View
Unlabeled Example
Feature
Outlier Detector
Consistency Regularization
Augmented View 2
Anomaly Score each
Feature Extractor
Outlier Detector
Figure An illustration of our proposed open-set soft-consistency loss used enhance outlier detection. Two differently augmented inputs are fed into the network to obtain the predictions of the outlier detector, The detector consists of one-vs-all classifiers and is able to detect outliers in an unsupervised way. The consistency loss is computed in a soft manner, i.e., without sharpening logits.
[44]. DS3L [14] proposes meta-optimization that attempts to pick unlabeled data useful to improve generalization performance. But, this method does not have an objective to separate inliers from outliers.
Given the limitations of the existing methods, we aim to learn representations that separate outliers from inliers in feature space and threshold effective for detecting outliers. The challenge is learning such representations with a small number of labeled inliers and no supervision to find outliers. Moreover, choosing an accurate threshold value is not a trivial problem.
We propose new framework, OpenMatch, to address the above drawbacks of OSSL. First, propose to utilize a One- Vs-All (OVA) network [32] that can learn a threshold to distinguish outliers from inliers. A separate OVA-classifier is trained for each class, and sample is labeled outlier all of the classifiers determine to be one. Thus, this technique allows us to identify outliers in unsupervised way. We call this an outler detector. Second, we propose novel open-set sofl-consistency loss to learn more effective representations for detecting outliers. We first transform logits from the outlier detector. Then, minimize the
an unlabeled input in Iwo ways and obtain distance between the two logits to encourage consistency (See Fig. 1). The main difference between SSL and OSSL is that unlabeled outliers do not have any neighboring labeled samples, which makes it risky to perform hard-labeling such as pseudo-labeling. The outlier detector outputs distance from inliers given input, and enhancing the smoothness of this function allows to improve its ability to find outliers. Empirically, this objective provides significant improvements in detecting outliers. Finally, to correctly classify inliers, we propose apply FixMatch [35] to unlabeled samples that are considered to be inliers by the outlier detector.
The resulting framework shows consistent gains over baselines various datasets and settings of OSSL. For example, Open achieves a 10.4% error rate on CIFAR 10 with 300 labeled examples compared the previous wfte-of-theean of 20.3%. Surprisingly. OpenMatch demonstrates good performance in detecting outliers unseen in unlabeled training data. For instance, in the experiments on CIFAR 10 with 100 labels per class, OpenMatch achieves a 3.4% higher AUROC in detecting outliers than a supervised model trained with all training samples. To summarize, our contributions are follows:
A soft open-set consistency regularization (SOCR), improve outlier detection in OSSL.
A framework, Open which combines
SOCR, and FixMatch.
A new state-of-the-art in both correctly classifying inliers and detecting outliers, even when the outliers are unseen in unlabeled training data.
2 Related Work
Semi-supervised Learning. Pseudo-labeling (PL) is strong baseline in semi-supervised learn- ing [23], and methods like FixMatch [35] or UDA [41] combining data augmentation and PI. show the highest performance many benchmark datasets. However, with these strong methods,

---

performance drops when a model is exposed to noisy unlabeled data that includes novel categories. On the other hand, methods based on soft consistency regularization enforce smoothness of the decision boundary with respect to stochastic transformations [34] or models [22, 37]. By "soft", we mean that no sharpening or aseudo-labeling is applied to logits to propagate training signals through soft targets. The soft consistency regularization can be understood as approximated Jacobian regularization method [12, 31J. We propose to apply soft consistency regularization to our outlier de- tector, which we call soft open-set consistency regularization. Training model with self-supervised learning and SSL achieves label-efficient training [5, 46). Self-supervised learning and SSL seem to be complementary to each other to improve the recognition accuracy. But, unlabeled data with outliers will hurt the performance of these approaches since they rely on existing SSL objectives. Since our work focuses on the aspect of SSL, we hope it is complementary to self- self-suuee learning.
Open-set SSL methods include MTC [44], D3SI. [14], and UASD [6]. MTC updates the network parameters and the anomaly score of unlabeled data alternately. In addition, it minimizes SSL loss (MixMatch [2] for a part of unlabeled data that is considered be inliers. D3SL selectively employs unlabeled data to optimize SSL loss as well as optimizing function that selects unlabeled data. UASD generates soft targets of the classifier for inliers (closed-set classifier) by averaging over many temporally ensembled networks' predictions. By contrast, OpenMatch does not have to store the temporal ensembles and applies soft consistency loss to the outlier detector. Our empirical results show that the consistency loss for the outlier detector outperforms that for the dlosed-set classifier by a lot (See Sec. 4.4).
Open-set Domain Adaptation (ODA). OSSI is similar to open-set domain adaptation [3, 33] in that the unlabeled data contains novel categories. A key difference is that the unlabeled and labeled data follow different data distributions. Another key difference is that we aim to train a model from scratch while the domain adaptation task assumes access to models pre-trained on ImageNet [9]. Since the ImageNet contains 1,000 diverse categories, the pre-trained models have discriminative representations useful to detect outliers. In this point, our task is more challenging than ODA, To detect outliers in an unsupervised way, we employ OVA-classifier model [32] proposed for this task.
Novelty Detection. Novelty detection or open-set classification aims to identify outliers that are completely unseen during training [17, 24, 10, 11]. Self-supervised learning such as rotation predic- tion and contrastive learning is shown to be useful to separate outliers from inliers [19, 36, 40). The task assumes plenty of labeled inliers in training data. Padhy et al. employ OVA-classifiers for this task [30]. Hendrycks et al. reveal that exposing a model to outlier data allows to effectively detect anomalies [18] and train it on out-of-distribttion training data in supervised way. But, collecting such useful out-o--distribution data is not always possible. In OSSL, a model cannot be trained on many labeled samples and needs to detect which samples are inlier or outliers in unlabeled data. In this sense, OSSL is more challenging and more realistic.
3 OpenMatch
Problem Setting. Our task is to learn a classifier using open-set semi-supervised learning. For K-way classification problem, let X' = {(z5, 1/6) : b E (I) be a batch of B labeled examples I, randomly sampled from the labeled training set, S, where y/ E (1,...)) is the corresponding label. Let 2/ = {(w)) : b E (1,..,uBB) be a batch of μB data randomly sampled from the unlabeled training set, S., where ja is a hyper-parameter that determines the relative sizes of X and U. Unlike in SSL, which assumes all unlabeled data come from the K known classes, in our OSSL setting they may come from unknown outlier classes. The goal of OSSL is to classify inliers into the correct classes while learning to detect the outliers.
Approach Overview. Our model has three components: (1) a shared feature extractor F(-), (2) an outlier detector consisting of K one-vs-all (OVA) sub-classifiers D³ (-),j E [I,..., K}. and (3) a closed-set classifier C(-) which outputs probability vector E K for K-class classification. At test time, the closed-set classifier is first applied to predict the K-way label, ÿ. If D predicts an inlier, then the output class ÿ. otherwise, the output is "outlier". The main technical novelty comes from the choice of OVA-classifiers for outlier detection as well as training them with soft open-set consistency regularization. We first deseribe the training of the OVA-classifier [32] before describing the remaining details of our framework.
3

---

Labeled Inliers
(8) Semi-supervised Leerning
Open-sel supervised Leaminng (One-vs- classifier score)
(b) Pseudo Label
(c) Cure
updated
updated
Unlabeled Inliers
P of class
P of outiorss for
P of outliers for
Unlabeled Outtiers
updaled
Probabiltt
Correct pseudo labels
Incorrect pscudo labels
Smoothness w.r.t transformation
Figure 2: (a) SSL with pscudo-labels. Hard labels such as pseudo-labels are often used in SSL to propagate training signals from labeled samples to neighboring unlabeled ones. (b) OSSI with pseudo-labels. However, in OSSI, the outlier unlabeled samples do not have any labeled neighbors. Therefore, the pseudo-labels can be highly unreliable. (c) OSSL with soft consistency (ours). To address this, propose to ensure smoothness with respect to data augmentation by minimizing soft consistency loss. This separates the outliers from inliers and avoids using incorrect pseudo-labels.
3.1 One-vs-All Outlier Detector
For open-set [33] or universal domain adaptation [42], Saito et al. [32] propose to train OVA-classifiers to detect outlier samples of an unlabeled target domain. They aim to learn a boundary between inliers and outliers for each class. Each sub-classifier is trained to distinguish if the sample is an inlier for the corresponding class. For example, the sub- classifier for class j, D outputs 2-dimensional vector =f DJ (F(r))) TR?, where each dimension indicates the score of sample being inlier and outlier respectively. We denote p² 0)æz5) and p¹ (2r) as the probability of 24, being the inlier and outlier for the class j. computed by Softmax(22 (2ī)) (Note that p (t 0/2²) + p (t 1|rr) 1). Then, the sub-classifier D trained such that the data from class j treated positives, and the data from all the other classes is treated as negatives. To achieve this, we minimize the following loss for the outlier detector for given batch X = {xr: [3b:9b)6mm1 1
B
Lova (X) =
log(p" m))) min log(p' 1(2))))
ig
Here use the hard-negative sub-classifier sampling technique proposed by [32] to effectively learn the threshold. We refer the reader to [32] for more details. The key is that each sub-classifier outputs distance representing how far the input is from the corresponding class. Therefore, the classifiers are effective to identify unlabeled outliers. The output of D is used to detect outliers as follows. If the probability of sample being an outlier for the class highest-scored by C( is larger than that of sample being an inlier, we consider the input as an outlier. To be specific, we identify the unlabeled arg max, C(F(us) This detection is
sample u, as an outlier if pa (t 0 (4) 0.5. where employed in selecting pscudo-inliers from unlabeled data as explained in Sec. 3.3.
Open-set Entropy Minimization. For unlabeled samples, [32] apply entropy minimization with respect to the OVA-classifiers, denoted by Lem (Lf). The separation between inliers and outliers is enhanced through the minimization of this loss for a given batch 2t == 1
Lem (L)
o' (t 0lus) log = 0lus) - p (t 1 log (t 1/u6). (2)
BB
The difference between setup and [32] is that they employ model pre-trained ImageNet [9], which extracts highly discriminative features, thus, inliers and outliers are well-separated even before fine-tuning. However, aim train model learn representations separating them from scratch.
Thus the inliers and outliers are confused before training, and we need an objective to ensure the separation in addition to Eq. 2.
3.2 Soft Open-set Consistency Regularization (SOCR)
To motivate our approach, we describe the difference between SSI. OSSI. Fig. In SSL., the hard consistency loss such as lscudo-labeling can be effective to propagate label information from

---

Algorithm OpenMatch Algorithm.
Input: Set of labeled data S = ((zb: yb); ): E NI), set unlabeled data S. = (uo E (I..... 1, and set of pseudo-inlier data K = 0.
Data augmentation function T. Model parameters w, learning rate n, epoch Eiz and Emax, iteration Tuax. trade-off parameters. Aem. Aoes fmi
for Epoch / to Emax do for Iteration to Imax do
2: Sample batch of labeled data X' 8 and unlabeled data U Sui
3: Compute Lau = (X) + Xem Lem (I) + Xochoo T);
I/ Eq. 2 and
if Epoch >> Ear then
4: Sample batch of pseudo-inliers I E KC;
Sample pseudo-inliers.
5: Compute Lull += f (I)
// Fix for pseudo-inliers.
end end
Update w = -----jj VVVVVVVVV Vo
Update weights
Epoch 2 Efs then 7: Update K = Sw. Dw);
// Detect outliers and select pseudo-inliers.
end
end
8: Output: Model parameters w.
labeled to unlabeled samples. Through the hard consistency loss, unlabeled samples get training signals from neighboring labeled samples (Fig. 2(a)). By contrast, in OSSL, since outliers are not assigned any labels, labeled samples are far from outliers, which makes the hard labeling unreliable, and unsupervised training signals can be incorrect (Fig. 2(b)). The open-set entropy minimization above can also be considered variant of the hard consistency since minimized if the predicted output class distribution is one-hot. To propagate useful training signals to unlabeled samples, propose to smooth the decision boundary of the outlier detector by minimizing the distance between its predictions on two augmentations of the same image (Fig. 2(c)). We use unsharpened logits, keeping the training signal soft to avoid introducing incorrect pseudo-labels. We call this loss soft open-set consistency regularization (SOCR). Given the smoothed outlier detector, the hope is that the training signals of the open-set entropy minimization will also become more correct.
Specifically, SOCR enhances the smoothness of the outlier detector over data augmentation T, which our experiments is just standard random cropping. Note that the standard cropping is also used in computing Eq. 1 and 2, but we omit it for the simplicity of notation. We obtain two different views of uig Ti (w,) and Ty(ub); where Ti and Tz are data augmentation functions stochastically sampled from T. Let p² (t]Ti(u)) be the output logits of Ti (us) from the jth OVA-classifier. We encourage the consistency of the output logits over T to enhance the smoothness by minimizing the following loss for given batch U = [
BB
LoclU, () = pB
ΣΣ Σ (t)() - - (t)Ta (E(0.)
(3)
This minimizes the distance between two probability outputs ensure smoothness with respect T. Note that the key part the regularization is not applying any sharpening to output logits, keeping them soft. The major difference from the previous soft consistency regularization [12, 37, 22] is that we use the outlier detector to compute the regularization loss.
3.3 Overall Framework
The entire Open algorithm is described in Alg. 1. The unsupervised losses (Eqs. 2, 3) for the outlier detector are effective to detect outliers in unlabeled data. However, these losses are not sufficient to correctly classify unlabeled inliers. Therefore, we propose to introduce learning loss for unlabeled samples classified as inliers. We adopt FixMatch [35] since a simple, yet very strong SSL method. FixMatch first generates pseudo-labels using the model's 's predictions on weakly augmented unlabeled images. The model is then trained to predict these pseudo-labels when fed a strongly augmented version of the same image. "Weak" denotes augmentations such simple
sune-supee
5

---

random cropping while "strong" refers to extensive data augmentation such as RandAugment [8] and CTAugment [1].
For labeled samples, we also compute the standard cross-entropy loss to train C(.), Lets(A)) ) for the closed-set output, C(F((s)). We summarize Lsup(()) as the sum of Lova (X) and Lat (d(). After training model for Eyt epochs with Lsup. Lem. Loc. we begin to select pseudo-inliee from unlabeled data at every following epoch. Specifically, in Alg. 1, Select(w, Du,) denotes the process of classifying unlabeled samples given model parameters w of models ((()),(() and C())), and unlabeled data. The outlier-detectio process is summarized in Sec. 3.1.
Then, the FixMatch loss, ym (I), is computed only for the pseudo-inliers to optimize the model, where I denotes batch of pscudo-inliers. The following is the resulting overall objective of OpenMatch:
Caul (,,( == (X) + dem Cem (X) + XooCoo + XfmLfm(I)
(4)
where Aems Aoe and fm control the trade-off for each objective.
Experiments
Setup. We evaluate the efficacy of OpenMatch on several SSL image classification benchmarks. Specifically, we perform experiments with varying amounts of labeled data and varying numbers of known/unknown classes on CIFAR10/100 [21] and ImageNet [9]. We attempt to cover various scenarios by using these three datasets.
Note that we use an identical set of mayper-prameet Aem is set 0.1 in all experiments. Amm is set to O before ft epochs and then set to I for all experiments. Efrr is set to 10 in all experiments. The hyper-parameters for FixMatch, e.g., data augmentation, confidence threshold, are fixed across all experiments for simplicity. The hyper- parameters are set by tuning on validation set that contains a small number of labeled samples. Note that the validation set does not contain any outliers. A complete list of hyper-paramene parameters is reported the appendix. Each experiment is done with a single 12-GB GPU, such as an NVIDIA TitanX.
except for Aoc. which is tuned on each dataset.
Baselines. For the OSSL baseline, we employ MTC [44] using the author's implementation. In addition, we show the result of a model trained only with labeled samples (Labeled Only). We exclude UASD [6] and D3SL [14] from our baselines since their reported results underperform a model we train only with labeled samples. For instance, in the experiments for CIFARIO 400 labels per class, their reported eгror rates are worse than 20% while the rate of the Labeled Only baseline is 20.0%. comparison with FixMatch [35] reveals the effect of OVA-classifiers trained with the soft consistency loss. The parammparsesters of both OpenMatch and baselines are tuned by maximizing the accuracy on the validation set. Since the validation set does not have any outliers, we choose the validation accuracy as a criterion.
Evaluation. We assume the test contains both known (inlier) and unknown (outlier) classes. For known classes, classification accuracy is used to evaluate performance. To evaluate the separation into inliers and outliers, we use AUROC following the standard evaluation protocol of novelty detection [17]. To compute the anomaly score, the outlier detector's 's score is employed for OpenMatch, Labeled Only, and FixMatch, where we add our outlier detector the latter two models. We report the results averaged over three runs and their standard deviations.
4.1 CIFAR1O and CIFAR100
We compare OpenMatch with several baselines on the standard benchmark datasets of SSL, CIFARIO. and CIFAR 100. Following [44], we use randomly initialized Wide ResNet-28-2 [45] with 1.5M parameters in these experiments. For CIFAR1O, we split the classes into known and unknown classes by defining animal classes known and others unknown, which results in 6 known classes and 4 unknown classes. For CIFAR 100, we first split the classes by their super-classes provided by the dataset so that known and unknown classes do not share super-classes. Then, we split the super-classes into known and unknown. To evaluate performance with different numbers of outliers, run experiments on two settings: 80 known classes (20 unknown classes) and 55 known classes (45 unknown classes).
Tables 1 and 2 describe the error rate on inliers and AUROC values respectively. Note that the number of labeled samples per class shown in each column. With respect to the error rate, Open achieves ofte-of-the-ar performance in all cases, and the gain over the baselines is remarkable in

---

Dataset
CIFARIO
CTFAR 100
CIFAR 100
ImageNet-30
No. of Known / Unknown
6/4
55
80 20
20/ 10
No. of labeled samples
100
400
50
100
50
100
Labeled Only
35.7=1.1
30.5=07
20.0=a33
37.0=08
27.3=005
43.6=us
34.7=04
20,9=1.0
FixMatch [35]
43.211.
29.8_06
16.3105
35,4.007
27.3
41.2.107
34.1100
12.9_04
MTC [44]
20.3
13.7=09
9.0a05
33.5
27.9
40.1
33.6103
13,6=07
OpenMatch
10.4
7.110s
5.9108
27.7:044
24.1:06
33.4.02
29.5.00
10.4.
Table 1: Error rates (%) with standard deviation for CIFARIO, CIFARI00 on 3 different folds. Lower better. For ImageNet, use the same fold and report averaged results of three runs. Note that the number of labeled samples per class is shown each column.
Dataset
CIFARIO
CIFAR100
CIFAR100
ImageNet-30
No. of Known / Unknown
6/4
55 /5
80/
20/ 10
No. of labeled samples
50
100
400
50
100
50
100
10 %
Labeled Only
63.9=0.5
64.7=05
76.8=0.
76.6=09
79.9=a9
70,3+u05
73.9w09
80,3=10
FixMatch [35]
56.1
60.4=0.4
71.8=0.
72.0=1.
75.8=12
64,3+110
66.1
88.6=05
MTC [44]
96.6226
98.2
98.9
81.2=34
80.7=46
79.4+2.5
73.2435
93.8=08
OpenMatch
99.3=0.3
99.7=
99.3=a.2
87.0=1.1
86.5-21
86.2+uu
86.8=14
96.4-a7
Table 2: AUROC of Table 1. Higher is better. Note that the number of labeled samples per class is shown in each column.
many cases, In particular, we see about a 10% improvement in CIFARIO error at 50 labels. In addition, the improvements in AUROC are also clear. MTC [44] does not improve the performance very much compared to the labeled-only model on CIFAR 100 because their method of identifying outliers, i.e., Otsu's 's thresholding [25], not very robust the number of outliers as discussed in their paper. On the other hand, OpenMatch improves the error rate by more than 3% and AUROC by more than 6% in all cases. Simply applying FixMatch [35] also does not always improve the error compared to the labeled-only model. This is because FixMatch confuses inliers and outliers and assigns pscudo-labels to outliers, as the significant decrease in AUROC indicates.
4.2 ImageNet
We evaluate OpenMatch on ImageNet to see behavior on more complex and challenging datasets. Since training on the entire ImageNet is computationally expensive, we choose ImageNet-30 [17], which is a subset of ImageNet containing 30 classes. The subset is useful for evaluation on unseen outliers done in Sec. 4.5, because the classes do not include several super-classed such bird class. We pick the first 20 classes, in alphabetical order, as known classes, and the remaining 10 classes unknown. We utilize the ResNet-18 [16] network architecture. Sypee-parammeten such batch-size, learning rate, data augmentation are the same as ones used in CIFAR 10 and CIFAR100 As shown in the rightmost columns of Tables and OpenMatch achieves the best performance in both accuracy and AUROC. Unlike the results on CIFAR, FixMatch improves the performance over the Labeled Only model. This is probably because outliers easier detect, the AUROC score of the Labeled Only model already shows a higher score than other scenarios. Therefore, even without any outlier detection objectives, FixMatch can correctly select pseudo-inlier inliers with confidence thresholding.
4.3 Ablation Study on Soft Open-set Consistency Regularization
In this section, we validate our claim that the SOCR regularization is effective at separating inliers and outliers. We perform several ablation studies to see the benefit of SOCR and measure the model's ability to detect outliers using AUROC. In summary, both numerical and qualitative evaluations confirm the effectiveness of SOCR.
Table 3 describes the numerical comparison of ablated models. In this evaluation, we do not apply Match to pseudo-inliers to measure the pure gain from the consistency loss. Introducing SOCR increases AUROC by more than 20% in CIFAR 10, 8% in CIFAR 100, and 8% in ImageNet. These results support the effectiveness of smoothing the outlier detector's output.
7

---

Dataset
CIFARIO
CIFARI00
ImageNet-30
No. Known / Unknown
6/4
80 / 20
20 / 10
No. Labeled samples
50
400
50
100
10 %
without SOCR with SOCR
60,5+28
75.8
70.4
73,2+02
81,3+00
81 +:29
96.8+06
78.9
85.0+08
89.3
Table 3: Ablation study of our soft consistency regularization (SOCR, ooc). We report AUROC scores (%) In this study, we do not apply FixMatch to pseudo-inliers to the pure gain from SOCR.
416
W/O Consisteury
3000
With Cansisteury
Full
knvor
Koown
Rzvora
7100
110
1100
1200
100
110
717
733
2212
-13
(a) w/o FixMatch, SOCR.
(b) w/o FixMatch
(c) FULL (OpenMatch)
Inliers, Blue: Outliers. From left to right, a model without FixMatch and SOCR, model without FixMatch, and a model with all objectives. These results show that SOCR ensures separation between inliers and outliers, and FixMatch added to SOCR can further enhance this separation.
Figure 3:
The histograms of the outlier detector's 's scores obtained with ablated models. Red:
Histograms Anomaly Scores. Fig. illustrates the anomaly score of inlier and outlier test samples in the ablation experiment on CIFAR1O. If SOCR is not employed (Fig. 3(a)), inliers and outliers confused while SOCR makes the scores of outliers larger and enhances the separation (Fig. 3(b).. Further applying FixMatch pseudo-inliers decreases the score of inliers 3()),, which separates inliers and outliers better. Considering the results of Table 2, just applying FixMatch to all unlabeled samples is often useful detect outliers, but applying it to the pseudo-inliers can improve the outlier detection.
4.4 Analysis
Feature Visualization. Fig. 4 visualizes the feature distributions with T-SNE [26] in the experiment on CIFARID 10 at 100 labels. OpenMatch (c) separates inliers and outliers well while a model trained only with 100 labeled samples (a) confuses them. The separation of OpenMatch appears be better than model trained with all labeled inliers (b).
Closed-set Consistency Open-set Consistency. The soft consistency loss is computed for the outlier detector in OpenMatch. An alternative is to compute it for the closed-set classifier as done in UASD [6]. We replace SOCR with the soft consistency loss with respect the closed-set classifier and perform the same analysis as in Sec. 4.3. In CIFAR1O at 50 labeled samples, this model achieves 70.6 + 1.3 % AUROC, which is 10.1 % better than model without consistency loss and 10.7 % worse than a model with SOCR. In CIFARIO 10 at 400 labeled samples, this model achieves 88.5 + 2.4 % AUROC, which 12.7 % better than model without consistency loss and 8.3 % worse than model with SOCR. These results indicate that even using soft consistency for the closed-set classifier helps detect outliers. as shown in UASD [6], but SOCR helps much more. Thus, our choice of combining both the outlier detector and soft consistency loss is validated in this analysis.
Comparison with OOD with self-supervised learning. We compare detection methods using self-super learning. Several works [19, 36] have shown that training a model with self superviedd learning objectives such as contrastive loss rotation prediction loss improves OOD performance. We provide a brief analysis of these baselines in open-set SSL.
method with outlier
Does self seff-uuper pre-training help open-set SSL? A model trained with self-supervised learn- ing objectives shows great performance many downstream image recognition tasks. We investigate whether initializing model trained with self-supervised learning (SimCLR [4]) improves perfor-

---

Kig
DNd
(a) Partially labeled model
(b) Fully labeled model
(c) OpenMatch
Figure 4: Feature visualization with t-NN [26]. Different colors indicate different classes. Pink dots represent outliers. (a): A model trained with 100 labeled samples per class, (b): A model trained with all labeled inliers.
(c): A model trained with OpenMatch. (Best viewed in color)
Initialization CIFAR10-50
CIFARION 10-400
CIFAR 100-50
SimCLR Init
19.6 10.4 99.3
5.9
27.1 87.1
Random Init
5,9 99.3
27.7 / 87.0
Table 4: Comparison by different initialization (Error (%) / AUROC). Initializing model trained with
SimCLR [4] for all unlabeled samples does not make clear difference from random initialization.
mance in open-set SSL (Table 4). We pre-train a model with SimCLR loss using all unlabeled data. The initialization does not make a clear difference from random initialization, i.e., default initialization.
Does self-supervised training help open-set SSL? Next, we investigate whether using self- supervised learning loss in addition to the classification loss labeled data helps open-set SSL. In Table 5, we show the results of adding self-supervised loss for labeled data or unlabeled data. SimCLR does not help in any case compared to a model trained only with classification loss for labeled data. Constrative learning basically attempts to uniformly distribute all samples in unit sphere, which does not necessarily help in open-set SSL. The goal of open-set SSL is to keep outliers far from inliers whereas clustering inliers within each class. By contrast, rotation prediction improves performance if applied labeled samples in CIFAR-10 10. If applied to unlabeled data, model can assign the same rotation labels to both inliers and outliers, which can prevent from separating them.
4.5 Novelty Detection
We have seen that OpenMatch can learn to detect outliers from unlabeled data. In this experiment, evaluate how well our OSSL model can separate inliers from outliers unseen in the unlabeled data. This setting is similar to general outlier or out- out-of-disttibution sample detection. Note that the models evaluated in this section are the same as the ones in Tables and We consider the following datasets as out-of-distribution: SVHN [28], resized LSUN [43], ImageNet, and CIFAR 100 for CIFARII 10 experiments (Table 6(a)), and LSUN, CUB-200 [39], Dogs [20], Caltech [13], Flowers [29], and DTD [7] for ImageNet-30 (Table 6(b)). See supplemental material for the details. We utilize AUROC to measure how well inliers and outliers are separated. To see the gap from a supervised model, we train a model using all labeled samples of inlier classes and show the comparison.
As seen in Table 6, Open outperforms OSSL baselines by 5.8% average and even the supervised model by 3.4% on average on CIFARIO. On ImageNet-30, OpenMatch outperforms OSSL baselines by more than 4.1% average. This result verifies that OpenMatch is robust to various out-of-diitrribution data by virtue of being exposed to unlabeled data containing outliers.
5 Conclusion
In this paper, we introduce method for open-set semi- semi-supervised learning (OSSL), where samples of novel categories are present in unlabeled data. To approach the task, we propose a novel framework, OpenMatch, which unifies one-vs-all classifiers and FixMatch. Our proposed objective, open-set soft consistency loss, is shown to be effective to detect outliers from unlabeled data, which allows FixMatch to work well in the OSSL setting. In addition, Open sometimes detects outliers unseen unlabeled data better than a supervised model. We believe that our framework for OSSI. will make label-efficient techniques more practical.

---

Method
CIFARI0-50
CIFAR100-50
Labeled only
63.9 #0.5
70.3 #0.5
SimCLR: labeled samples
63.5
68.6 #1.
SimCLR: unlabeled samples
62.1
68.8
Rot Pred: labeled samples
70.0 110
67.8 =0.5
Rot Pred: unlabeled samples SOCR
64.0 1.3
67.0 #15
81.3
78.9 10.1
Table 5: Study about self-supervised learning loss for labeled unlabeled data (AUROC). Rot Pred indicates the rotation prediction loss. SimCLR [4] either for labeled or unlabeled data slightly decreases the performance detect outliers whereas rotation prediction for labeled data shows some gain in CIFAR-10.
Unseen Out-liers
Method
CIFARIO
SVHN
LSUN
ImageNet
CIFARI00
MEAN
Labeled Only
64.7110
83.61110
78.9 18.9
80.5
80.4
80.8108
FixMatch [35]
60.4+0.4
79.9+1.0
67.7
76.9
71.3=1.1
13,9
MTC [44]
98.2++03
87.6+0.5
82.8=0.6
96.5+00.
90.0=0.3
89,2+104
OpenMatch 99.7+0.1
93.0+0.4
92.7=0.3
98.7=t011
95.8=0.4
95.0=0.3
Supervised
89.4+1.0
95.640..
89.510.7
90.810.4
90.4=1.0
91 6.166.66
(a) Model trained on CIFARIO (100 labeled data per class and unlabeled data.)
Unseen Out-liers
Method
ImageNet-30
LSUN
DTD
CUB
Flowers
Caltech
Dogs
MEAN
Labeled Only
80.3=0.5
85.9+14
75.4+10
77.9+08
69.0+1.5
78.7
84.8=10
78.6+11
FixMatch [35]
88.6205
85.710.1
83.1
81.0148
81.9
83. 13.1
86.4_32
83.0
MTC [44]
93.8=088
78,0+10
59.5+15
72,2+09
76.4 #2.1
80.9
78.0=08
74.2
OpenMatch
9.7
89.9
84.4
87.7
80.8119
87.7.0.9
92.1
87.1 111
Supervised
92.8=08
94.4+0.
92.7+0.4
91.5+00
88.2+1.0
89.9
92.3-0a
91.3+07
(b) Model trained on ImageNet-30 (10 % labeled data and unlabeled data).
Table 6: Evaluation of outlier detection on outliers unseen in unlabeled training data (AUROC).
Higher is better. Supervised models use the same batch size, learning rate as OpenMatch, but are trained with fully labeled inliers.
Limitations. OpenMatch can have difficulty in detecting outliers similar to inliers, though any methods for outlier detection have the same limitation. If the outliers have similar visual characteristics, separating them from inliers samples is difficult. A potential solution is to introduce self-supervised learning [15, 4] for unlabeled data.
Broader Impact. OpenMatch is an effective tool for semi-supervised learning with noisy unlabeled data containing outliers. It would benefit projects with small budgets or hard-to-label data. At the same time, it is possible for malicious actors to exploit the advantages of learning from limited data. Broadly speaking, any progress semi-supervised learning can have these consequences.
6 Acknowledgement
This work was supported by Honda, DARPA LwLL and NSF Award No. 1535797.
References
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi superviee learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.0978. 2019.
10

---

[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint 1rXi::19050022999 2019.
[3] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Int. Conf. Comput. Vis., 2017.
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Int. Conf. Mach Learn., pages 1597-1607. PMLR, 2020.
[5] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton, Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[6] Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, volume 34, pages 3569-3576, 2020.
[7] Mireea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Deseribing textures in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3606-3613, 2014.
[8] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quae VL. Randaugment: Practical automated data augmentation with a reduced search space. In JEFE Conf. Comput. Vis. Pattern Recog,, pages 702-703, 2020.
[9] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
[10] Terrance VeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint 10Xiv:802..0465. 2018.
[11] Chuanxing Geng, Sheng-jun jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
[12] Atin Ghosh and Alexandre Thiery, On data- bata-augmentation and consistency baasse semi supersupereieed learning. arXiv preprint arXiv:2101.0661 06967,, 2021.
[13] Gregory Griffin, Alex Holub, and Pictro Perona. Caltech-256 object category datasct. Tech Report, 2007.
[14] Lan-Zhe Guo, Zuen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi-supervised learning for unseen-class unlaheled data. Int. Conf. Mach Learn., pages 3897-3906, 2020.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conf. Comput. Vis. Puttern Recog,, pages 9729-9738. 2020.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Puttern Recog., 2016.
[17] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and -fittr-diitiiuution examples in neural networks. arXiv preprint arXiv:1610.02136. 2016.
[18] Dan Hendryeks, Mantas Mazcika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint
1188.8442066... 2018
[19] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. arXiv preprint arXiv: /906. 12340, 2019.
[20] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-graine image categorization: Stanford dogs. In Proc. CVPR Workshop on Grainnr Visual Categorization (FGVC), volume 2. Citeseer, 2011.
[21] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. arXiv preprint, 2009.
[22] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv, 2016.
[23] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
[24] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv: (tt:::706.02999 2017.
[25] Dongju Liu and Jian Yu. Otsu method and k-means. In 2009 Ninth International Conference on Hybrid Intelligent Systems, volume 1, pages 344 349. IEEE, 2009.

---

[26] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, ((1):2579-2605 2008.
[27] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. arXiv, 2015.
[28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, 2011.
[29] M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Conf. Comput. Vis. Puttern Recog,, volume pages 1447-1454, IEEE, 2006.
[30] Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Liu, Jasper Snock, and Balaji Lakshminarayanan Revisiting one-vs-all all classifiers for predictive uncertainty and out-of-distribution detection in neural networks. arXiv preprint arXiv:2007.05134 2020.
[31] Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. Adv. Neural Inform. Process. Syst., 24:2294-2302, 2011.
[32] Kuniaki Saito and Kate Saenko. Ovanet: One-vs-all network for universal domain adaptation. arXiv preprint wxXi::200.03344,, 2021.
[33] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Eur. Conf. Comput. Vis., 2018.
[34] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. arXiv preprint 1601v:1606.04550 2016.
[35] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001 006685, 2020.
[36] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. arXiv preprint arXiv:2007.08176 2020.
[37] Antti Tarvainen and Harri Valpola. Mcan teachers are better role models: Weight-avcraged consistency targets improve semi-supervised deep learning results. arXiv preprint arXivv1703.000 017000 2017.
[38] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv preprint arXiv: 1903.09823 2019.
[39] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. Tech Report, 2011.
[40] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R I Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved -uito--iistriuutinn detection. arXiv preprint arXiv:2007.05566, 2020.
[41] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmenta- tion for consistency training. arXiv preprint arXiv: 12848, 2019.
[42] Kaichao You, Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Michael I. Jordan. Universal domain adaptation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[43] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.033666 2015.
[44] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In Earopean Conference on Computer Vision, pages 438-454. Springer, 2020.
[45] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 160.:1605.07166 2016.
[46] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-superrvsed learning. In Int. Conf. Comput. Vis., pages 1476-1485, 2019.
12


Reviews:
[{"review_id": "sRfoOgmsohh", "reviewer": "Reviewer_YUGo", "summary": "The authors propose an idea to improve model's performance under the open-set semi-supervised learning setting. This work proposed to leverage OOD detector, combined with FixMatch, and soft open-set consistency regularization (SOCR) to improve the model training during open-set semi-supervised learning. I think the usage of OOD on ignore outliers is not new and an unsupervised method on learning the consistency between augmentations for unlabeled data is also not new. Thus, I am not sure if I can find too much of novelty from this work itself, but I do appreciate the author's effort in putting them together to be a complete model, which in fact make solid progress on the open-set semi-supervised learning setting. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the NeurIPS community.\n\n- A solid advance on a practical Semi-Sup. Learning setting, though there isn't too much novelty in the proposed method. \n- Setting Benchmark by considering MTC, UASD, and D3SL as baselines. \n- Good experimental results and analysis\n\n### Weaknesses: Explain the limitations of this work along the same axes as above.\n\n- The related work section is hard to follow. I think it lacks proof-read, and its writing quality is much worse than the introduction section. \n- Fig. 2 is quite confusing. Why are the y-axis of (b) and (c) for \"P of outliers for samples in circle shape\"? Aren't the outliers samples in diamond shape? Also, I am not sure if I follow (C), in which the authors try to depict what the proposed soft consistency will do to the predictions of labeled inliers and unlabeled outliers. I think what Soft Open-set Consistency Regularization (SOCR) is trying to do is basically forcing the model to learn better visual representations of the unlabeled outliers. I would be curious what's the difference here if we were to apply contrastive learning to the unlabeled outliers. \n\n- I don't think the authors ablated the Entropy Minimization. How important is this component in the proposed work? \n\n### Please provide an \"overall score\" for this submission.\n\nOverall, I think this paper made a solid contribution to the semi-supervised learning community. I am less satisfied with the novelty of the paper, though I don't think this necessarily leads to grounds for rejection. Personally, I believe that any work that makes meaningful and solid progress for a practical research problem is worth to be shared with the research community. That being said, I do suggest the authors do a proofread on their paper, especially revise some of the confusing parts in the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors provided the \"Limitations\" and \"Broader Impact\" subsections in the conclusion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {"ethical_concerns": "I think one potential issue for semi-supervised learning is always that the model could be biased since we are leveraging a smaller number of samples that could already be biased. This essentially is what most of the semi-supervised learning papers tried to address, yet I don't think any of us actually achieved that yet. "}}, {"review_id": "PsiJmkrM2I", "reviewer": "Reviewer_fHpH", "summary": "This manuscript introduces OpenMatch for open-set semi-supervised learning. The OpenMatch integrates a one-vs-all classifier (OVA-classifier, working as an outlier detector) and FixMatch. The main contribution should be the soft open-set consistency regularization (SOCR) in Figure 1 and the OpenMatch Framework in Alg 1. Specifically, the SOCR follows the same self-supervised framework in FixMatch but it uses OVA-classifier to output consistent anomaly score distribution. After using the model in Figure 1 to identify inliers from unlabeled data, the FixMatch is allowed to reach a higher accuracy on the OSSL tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is about the experiments. \n\n1)The studied problem of open-set semi-supervised learning (OSSL) is close to the problem of out-of-distribution (OOD) detection.    \nAnd there are some related works also considering semi-supervised settings for OOD: \n- 2020 Robust Semi-Supervised Learning with Out of Distribution Data \n- 2020 AAAI Semi-Supervised Learning under Class Distribution Mismatch\n\nHowever, in experiments, only two baselines FixMatch (not OOD method) and MTC are mainly considered. This may not be convincing enough to demonstrate its superiority.  \nThus, more baselines (especially related OOD methods) are encouraged to include. \n\n2)For the model in Figure 1, it should be an *unsupervised* OOD classifier. As the soft open-set consistency regularization (SOCR) is one of the main contributions in this work, it would be better to compare the figure 1 model with the main-stream OOD methods (as they can also be used to identify OOD samples).      \n\nOverall, this paper is well written and the provided solution of using OVA-classifier to help FixMatch is well motivated. The authors demonstarted that using OVA-classifier with SOCR can help FixMatch on open-set semi-supervised learning (OSSL). However, related OOD baselines are not included in comparisons. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "LWZp03IPkVi", "reviewer": "Reviewer_6PWa", "summary": "This paper focuses on semi-supervised learning with outlier unlabeled data. This problem has attracted much attention in the semi-supervised learning community. The authors propose a method that contains two parts: First, detect outliers based on the OVA classifier. Second, propose an open-set soft-consistency regularization loss that can help improve the outlier detection performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on semi-supervised learning with outliers, more accurately, the unlabeled data contains some examples that belong to classes that unseen in labeled data.  The authors propose an outlier detection-based method to detect outliers and help improve the SSL performance. I have the following concerns about this paper:\n\n1) The contribution is limited. The proposal is mainly based on an existing outlier detection method. It seems that if we can detect outliers accurately and delete these outlier examples, the problem becomes an ideal SSL setting. So the open-set semi-supervised learning problem is actually an outlier detection problem?\n2) Based on question 1, in this paper, the authors do not compare the proposal with the SOTA OOD detection method. The performance of the combination of OOD detection methods and SSL methods should be reported.\n3) What are the differences between outlier detection and open-set learning. In my view, open-set learning should make the model has the ability to classify the new class data.\n4) Self-Supervised learning, e.g., SimCLR, can help to learn a separatable representation and help OOD detection tasks. There are many OOD detection algorithms based on contrastive learning that have been proposed. Can we simply pre-train a model with SimCLR using all labeled and unlabeled data, and based on the learned representation to detect outlier unlabeled examples. Have you tried this way in your experiments?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.6 hours"}, "extra_text": {}}, {"review_id": "7IsyuF9muc3", "reviewer": "Reviewer_5gr3", "summary": "This paper tackles open-set semi-supervised learning (OSSL) where outliers exist in unlabeled data. The key idea for detecting outliers is to learn class-specific, one-vs-all (OvA) classifiers—if all classifiers produce negative outputs for an unlabeled data, the data is considered an outlier. In addition to the classification loss computed on labeled data, the OvA classifiers are also trained on unlabeled data with an entropy minimization loss and a consistency regularization loss—the latter is computed on two views of the same data. The results on CIFAR and ImageNet show that the proposed approach largely outperforms the baselines in terms of both recognition accuracy and outlier detection accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n- The design of combining FixMatch with the OvA classifiers is simple and effective, which could add value to the research of OSSL.\n- The experiments are convincing and extensive, covering both classification and outlier detection tasks.\n- The improvements over previous SOTA are impressive.\n- The paper is easy to follow.\n\n**Weakness/Questions**\n- I am a bit concerned with the novelty. The idea of using the OvA classifiers for detecting outliers is borrowed from [32], which has been acknowledged in this paper. [32] deals with the problem of open-set unsupervised domain adaptation (UDA). Given the closedness between UDA and SSL, it's not too surprised to see that the OvA classifiers work in the OSSL setting. One could argue that UDA and SSL are different problems since the former needs to cope with domain shift while the latter doesn't, but we couldn't ignore many recent studies that have shown that methods developed for one problem (either UDA or SSL) work very well on the other. Nonetheless, I still appreciate the simplicity and effectiveness of the proposed approach.\n- Why compute the consistency loss on logits instead of the output probabilities? Is it because the former gives a better result?\n- Eq.1 looks a bit weird as it is supposed to be the binary cross-entropy loss.\n- The entropy minimization loss $L_{em}$ defined in Eq.2 is not evaluated in the ablation study. One would be wondering if the improvement brought by this loss is significant.\n- The motivation for SOCR is not very convincing. This loss is computed for the sub-classifiers. Why this loss could help the closed-set classifier produce more accurate pseudo-labels?\n\n---\nPost-rebuttal update:\n\nI have read the authors' responses and other reviewers' comments as well. I keep my rating unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have properly covered these aspects in Sec.5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "01MSOc_shlp", "reviewer": "Reviewer_Gu7C", "summary": "The paper proposes a model for open-set semi-supervised learning. As a semi-supervised learning model, it tackles classification where only a portion of the training data is labeled. \"Open set\" refers to the fact that the unlabeled data are noisy and can contain out of distribution examples, for which the class is not among the known classes. Out of distribution samples are present also during test, and the model has to recognize them and avoid classification on them. The proposed model builds on top of existing works, combining several loss terms and ideas such as one-versus-all classifiers and FixMatch as the main semi-supervised learning engine. The main technical contribution is a loss term enforcing the out-of-distribution score for two augmented versions of the same unlabeled example to be similar. Extensive experiments validate the proposal against recent state-of-the-art models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission has some solid merits, as it tackles a meaningful problem for the machine learning community, namely training in the presence of noisy labels and out of distribution examples. \nThe method is straightforward and embeds a combination of several existing components, such as:\n- FixMatch as the main semi-supervised learning engine. As is, it cannot deal well with the open set scenario.\n- OVA-classifiers to identify in-distribution and out-of-distribution unlabeled examples, and therefore pseudo-label the formers and reject the latters. Originally, OVA-classifiers were proposed (in the version used here) for universal domain adaptation.\nThis results in the biggest weakness of the paper: the limited technical contribution. Indeed, the only source of novelty is in a regularization loss (Eq. 3) that enforces, on an unlabeled sample, the out-of-distribution scores of two augmented versions to be closer.\n\nFollowing on the weak points, the model involves a number of hyperparameters, specifically in weighting the loss function terms. An analysis about sensitivity of performances to those lacks, and it is a bit unclear how one should select their value.\n\nClarification: in Tab. 2, how do the baseline \"Labeled Only\" and \"FixMatch\" perform OOD detection? Are they equipped with OVA classifiers as well? In their original formulation no mechanism allows for OOD. Is it with confidence thresholding?\n\nI consider the ablation study on the proposed open-set consistency regularization loss particularily important, as it validates the main novel contribution of the paper. The benefit of the proposed loss function is demonstrated in Tab. 3, but I find curious that the authors turned off another loss component for this experiment (FixMatch). What is the reason for that? The text mentions \"to measure the pure gain from the consistency loss\", it is unclear to me. I would find ablating the consistency loss, whilst activating all other components, much cleaner.\n\nDespite these limitations, the experimental section is solid and I think it is the true strength of the paper.\nThe experimental protocols used seem correct, and tests are carried out on 3 different datasets at different degrees of inlier-outlier splits. Performances of the model are properly reported both in terms of classification accuracy on the known classes and outlier detection. On both tasks, the proposed OpenMatch model outperforms recent competing methods by a significant margin.\n\n---\n\nOverall, I consider this a borderline submission. The weakest point is the minor technical contribution, as the main novel part is the consistency loss and other components are off-the-shelf modules. The strongest point is experimental evaluation, that shows big performance gap with respect to very recent baselines. All things considered, I lean more towards accepting the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a model for open-set semi-supervised learning. It is well-written and well-motivated. As pointed out by almost all the reviewers, there is limited novelty as the proposed model is a straightforward combination of existing components. However, extensive experimental evaluation shows big performance gap with respect to very recent baselines, and shows solid progress on the open-set semi-supervised learning setting studied.

To further make the experimental results more convincing, it will be useful to add sensitivity analysis on the hyperparameters (as the model involves a number of them). Also, as the problem is related to OOD methods, the authors are encouraged to include OOD-related baselines.
Decision: Accept (Poster)