The Needle in the haystack: Out-distribution aware Self-training in an Open-World Setting
Authors: Maximilian Augustin
Matthias Hein
Venue: ICLR 2022 Submitted (2022)

Abstract: Traditional semi-supervised learning (SSL) has focused on the closed world assumption where all unlabeled samples are task-related. In practice, this assumption is often violated when leveraging data from very large image databases that contain mostly non-task-relevant samples. While standard self-training and other established methods fail in this open-world setting, we demonstrate that our out-distribution-aware self-learning (ODST) with a careful sample selection strategy can leverage unlabeled datasets with millions of samples, more than 1600 times larger than the labeled datasets, and which contain only about $2\%$ task-relevant inputs. Standard and open world SSL techniques degrade in performance when the ratio of task-relevant sample decreases and show a significant distribution shift which is problematic regarding AI safety while ODST outperforms them with respect to test performance, corruption robustness and out-of-distribution detection.


Under review as a conference paper at ICLR 2022
THE NEEDLE IN THE HAYSTACK: OUT-DISTRIBUTION AWARE SELF-TRAINING IN AN OPEN-WORLD SETTING
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional semi-supervised learning (SSL) has focused on the closed world as- sumption where all unlabeled samples are task-related. In practice, this assump- tion is often violated when leveraging data from very large image databases that contain mostly non-task-relevant samples. While standard self-training and other established methods fail this open-world setting, we demonstrate that our out- distribution-awann self-learning (ODST) with careful sample selection strategy can leverage unlabeled datasets with millions of samples, more than 1600 times larger than the labeled datasets, and which contain only about 2% task-relevant inputs. Standard and open world SSL techniques degrade in performance when the ratio of task-relevant sample decreases and show significant distribution shift which is problematic regarding AI safety while ODST outperforms them with re- spect test performance, corruption robustness and out-of-distribution detection.
INTRODUCTION
In past years we have seen tremendous progress in image recognition based deep learning (Krizhevsky et al. 2012 He et al. 2016). However, this success required large labeled datasets that expensive to generate. On the other hand, large amounts of unlabeled data are broadly available, in particular in image recognition. The promise of semi-supervised learning (Chapelle) al. 2006) to leverage unlabeled data in order to improve prediction performance. However, the underlying assumption of traditional and modern (Berthelot et al.) 2019 Sohn et al. (0200) SSL algorithms is that the unlabeled data comes from the same distribution or at least contains data from the same classes (closed world assumption). Oliver et al. (2018) criticized this as being unrealistic as the assumption is hard to check when retrieving large amounts of unlabeled data from the web. SSL in an open world setting. where the unlabeled data contains task-relevant but also non-related out-of-distributio (OOD) images, has recently attracted attention as more realistic approach to SSL (Guo et al. 2020 Chen et al., 2020; Yu et al., 2022).. While these methods could outperform standard SSL techniques when the unlabeled data contains out-of-distribution samples, they have mostly been evaluated in settings where the unlabeled data contains relatively few non-task-related samples. It is thus an open question whether existing open world SSL methods can scale to large unlabeled data bases where the ratio of task-relevant non-related samples is much smaller.
In this paper, we demonstrate that existing methods suffer from severe performance degradations when the ratio of non-related images in the unlabeled dataset increases. Moreover, even if they achieve high accuracy, they are not OOD aware, that is they systematically assign high confi- dence non-related inputs, e.g. a CIFARIO model classifies images containing humans as dogs or horses. Our contributions are: i) we propose our scalable, iterative out-distribution-awar self- training (ODST) which enforces low-confidence predictions on non -on-task-related samples and selects unlabeled samples for the in ne-distribution task via novel class-adaptive selection scheme. This con- fidence based selection scheme allows us deal with strongly unbalanced in in-distribution classes in the unlabeled dataset, ii) we outperform state-of-the-art SSL techniques (Berthelot et al.) 2019 (Shn ct al., 2020), as well as recent open world SSL techniques (Guo et al. 2020 Yu et al.) (020) in terms of prediction performance and oul-of-distribution detection and show that all other approaches suf- fer from distribution shifts, iii) we show strong performance gains when using ODST with the full training sets of CIFAR10 and CIFAR100 labeled data together with the 80 Million Tiny Image (80MTI) dataset as unlabeled dataset. Thus we show that even with large amounts of labeled data, ODST can still leverage unlabeled data to improve prediction and OOD performance.

---

Under review as a conference paper at ICLR 2022
1.00
Tug-165
CB--2.0
(a) ODST
(b) MTCF
(c) FixMatch
Figure 1: Random unlabeled samples considered to be task-relevant by the SSL method (confidence above in-distribution thresholds for ODST, above 95% confidence for Fixmatch and above the Otsu threshold for MTCF) are shown together with their confidence and predicted label (mistakes are marked red). MTCF and Fixmatch show severe distribution shift and only our ODST is able to select the correct samples. All methods are trained on CIFARIO with 4k labeled images and an unlabeled set consisting of 41k CIFARIO 10 training images and IM images from 8OMTI.
RELATED WORK
Semi-supervised learning (SSL) is an established technique (Zhu 2005
Zhu & Goldberg, 2
for leveraging information from unlabeled data to improve predictive performance. In self traiming (Riloff 1996, Riloff & W 2003, Scudder] 1965), teacher model is trained in a fully-supervised fashion on a labeled dataset. The teacher model is then used to label a set of unlabeled examples, typically drawn from the original data distribution (closed world assumption), which is then used in combination with the labeled samples to train new student model. Various extensions of this protocol have been proposed, including the use of an ensemble of teacher models (Zhou et al. 2018 and Co- Co-training Bllum & Mitchell 1998).
Recently, Xie et al. (2020) and Yalniz et al. 2019)) used self-training to improve performance on ImageNet (Russakovsky et al. 2015) by using large image databases consisting of millions of task- relevant and out-of-
samples. On CIFAR1O, Carmon et al. (2019) were able to signif- icantly improve model robustness to adversarial perturbations by adding unlabeled samples from 80 million tiny images (80MTI) (Torralba et al. 2008) using self-training. The distinctive feature of self-training in comparison other SSL methods is that the training of the teacher model is separated from the labeling process. In contrast, in pseudo-labeling
(Lee 2013 Iscen et al.) 2019 Shi et al. 2018) labels are generated during training by the model itself. Similarly, consistency- based mSetmhhsodds like II-models (Laine & Aila 2016 [Sajjadi et al. (016), mean-teacher (Tar- vainen & Valpola| 2017) and virtual adversarial training (Miyato et al. 2018) enforce an invariance of the model's output on the unlabeled data under specific set of perturbations. Methods like MixMatch (Berthelot et al., 2019) and FixMatch (Sohn et al./ 2020 combine consistency regular- ization with strong augmentation e.g. RandAugment (Cubuk et al. 2020). A related technique is entropy-minimizatin (Granvvalet & Bengio 2005). which penalizes low-confidence predictions on unlabeled samples during training. [Olier al. (2018) found that SSL can improve the model's performance in the traditional SSL setting where the unlabeled data is sampled from the same dis- tribution as the training data (closed world assumption) but can degrade the performance when the unlabeled data contains non-task task-related samples (open world setting).
The open world SSL setting has has been explored only recently, by combining elements of consis- tency regularization with online OOD detection and sample filtering (Yu et al.) 2020 Chen et al.) 2020) soft per-sample weighting (Guo et al. 2020). While they demonstrate performance im- provements when the unlabeled data contains non non-task-rn task-related samples, their evaluation is restricted to settings where the unlabeled data contains mostly task-related samples. We later demonstrate that in more challenging settings, where the ratio of task-related samples in the unlabeled data is small, these methods show severe performance degradations.
Deep Neural networks (DNN) have empirically and theoretically been shown to produce overconfident predictions for inputs not related to the task, e.g. noise or other classes not contained in the labeled dataset (Nguyen et al. 2015 Hendrycks & Gimpel| 2017 Hein et al. 2019), i.e. the confidence of a DNN is not reliable for the detection of of (OOD) samples. Approaches for OOD detection include ODIN (Liang et al., 2018) or using the Mahalanobis distance of higher-order features (Lee et al. 2018). Hendrycks et al. (011) introduced Outlier exposure (OE), see Hein et al. (2019) for the related CHDAA and show that the confidence
Dut-of-distribution detection:
2

---

Under review as a conference paper at ICLR 2022
can be used reliable OOD detector when enforcing low confidence OOD training set, even when tested on other OOD test datasets. OOD detection is also related to open set recognition (Boult et al. 2019) which is beyond the scope of this paper.
3 METHOD
In contrast to previous works which focused on problems where the ratio of non-task-rellted to task- related images small, goal is to show that our ODST in combination with careful sample selection strategy yields a self-training scheme that can leverage large unlabeled datasets to improve performance on the CIFARIO 10 and CIFAR 100 test set over fully-supervised baseline trained on the entire training set and additionally has excellent OOD detection performance.
First, we introduce self-training as in the noisy student self-training (NSST) of Xie et al. (020 which serves a baseline. Then we introduce our out-distribution aware self-training ODST.
In the following T (Ii;; y= denotes the of labeled examples, where x, E Rd and y {l..... } and U (z,);= is collection of unlabeled samples. The traditional SSL literature makes the assumption that the unlabeled samples U are drawn from the same distribution as the labeled examples T, or at least belong to the same set of classes which we explicitly do not do in this paper. Given the logits of a neural network f Rdd - the predicted probability distribution for a point x is computed via the softmax as: (5|z)) = efs(z) Σ edi(M), The confidence in the decision for x: is then given by max,=1....,K K P (x), and the cross-entropy loss between (soft)-labels p E IRK
PE P4 = 1. Pi 2 0) and prediction ş is defined as: Llp --E Pi log
3.1 NOISY STUDENT SELF-TRAING (NSST)
In NSST (Xie al. 2020 the first teacher is base model f(() obtained by minimizing the cross- entropy loss on the labeled set only. The iterative scheme of NSST starting at t = is:
1) pseudo-label all unlabeled samples in U with current teacher f(()
2) select a subset I C U of the pseudo-labeled examples by choosing per class the top-k with highest confidence and which are above fixed threshold
3) train model f(++1) by minimizing the loss labeled and pscudo-labeled samples in I with AutoAugment (Cubuk et al. 2019) as data augmentation:
1 Lyi (2:)) + Σ Ze1
(0
4) t - 1+ 1 and go back to step
Xie et al. (2000) call their self-training noisy due the very strong data augmentation which avoids overfitting via noise injection. The main difficulty in self-training in a closed world setting is the propagation of labeling mistakes which leads degradation of prediction performance which taken care of by step 2). While Xie et al. (2020) chose a fixed threshold, we choose it according the false positive rate on an di-iistriuttion validation set. However, in an open- setting, an equally severe problem is that a large fraction of the unlabeled samples is not task-relevant such that including them leads to distribution shift which happens for NSST (see Figure 2 and other SSL methods (see Figure []. However, more importantly, the classifier is highly confident on unre- lated classes which is a problem for AI safety and this undesired behavior is even hard to notice as predictive performance might appear to improve when evaluated only the test set.
3.2 OUT-DISTRIBUTION AWARE SEL.P-TRAINING (ODST)
We first provide an overview over our algorithmic scheme and then explain the individual steps in more detail. The base ODST model is initialized with an out-distribution aware base teacher model f()), trained by minimizing:
Σ L ( (221, Pyco (z)). 2EU
LVii Pooo (2:)) +
(1)
U
3

---

Under review as a conference paper at ICLR 2022
We then iterate the following steps starting from t 0:
A) calibrate ( the in-distribution validation set
B) pscudo-label all unlabeled samples in U with current teacher f(()
C) for each class select the top-k unlabeled instances with highest confidence classified as
c that lie above the threshold. The selected samples for all classes are denoted as I
D) determine new pseudo-labes labels for the unlabeled instances. We use q(z) = -pyc (2) for 2 E (selected samples in step C)) and
<=z) 2/2 2 K - -)oo(z)), for Z ZEUT EUU
E) train new model f(t+1) by minimizing the loss labeled and pseudo-labeled samples with AutoAugment (Cubuk et al. 2019) as strong data augmentation:
1
L Oyr
> [1(g(2).
72
Σ ((((3):rr(=((3) SEUII
U
F) t 1+ and go to step A)
The Base classifier is essentially an Outlier Exposure (OE) model (Hendrycks et al. 2019) (see also (Hein et al. 2019 Papadopoulos et al. 2019) for related losses) where the unlabeled set U is our training out-distribution on which we enforce uniform confidence. OE is known to be one of the best methods for out-of-distribution detection. As in our case, crucial assumption is that the unlabeled samples are partially task-related, it might appear odd to enforce uniform confidence on all of U. However, we show in Section 3.3 that this asymptotically only leads the confidence for task-related samples but preserves the Bayes optimal decision and, in particular, enforces coss--to-unifomm confidence for all unrelated samples.
down-weighting of
while normal neural networks are known to be overconfident on in-(Guo et al., 2017) and out-distribution (Nguyen al. 2015 Hein ing low confidence on unlabeled points (such as ОE) tend to be underconfident on the in-distribution. As we use the predictions of the teacher r(() as new pseudo-labels for the unlabeled data, we cali- brate (() by minimizing the expected calibration error using temperature rescaling Guo et al. 2017)) to its predictions in-distribution sam- ples which improves pseudo-labe quality and and stabilizes the training procedure.
A) Calibration:
2019), the models resulting from enfore-
Thus the teacher model assigns the correct uncertainty
The most important problem in self trff-irainin is to integrate the right samples into the pseudo-labeled set I. While our out-distribution aware teacher is better at discriminating between the in- and out-distribution based on confidence, there still many samples with highly confident predictions due to the sheer size of the unlabeled dataset (> 10%). Note that we select most the top-k samples (where k 5t + 1)/K), but this might still be too much if not sufficiently many task-related examples of class exist in the unlabeled dataset. We thus need to determine class-specific confidence thresholds limit the selection. Class spesi-speifiic selection has been neglected in the literature but is particularly important in practice as the number of task-related examples the unlabeled dataset typically varies widely between the different classes.
C) Sample Selection:
Using the in-distribution validation set, we define the in-distribution threshold for class as the smallest predicted probability for class e such that the precision for images which are above this threshold is greater than or equal to a (binary classification problem: class versus all other classes). An in-distribution precision threshold is especially important if the classification task contains sim- ilar classes, learning from wrongly labeled in in-distribution images (a dog that is classified in CIFAR10) is likely to hurt predictive performance on the in-distribution task even more than the inclusion of an unrelated out-distribution image.
cat
If there exist less than l samples above the threshold for a particular class, we randomly repeat the accepted samples above the threshold maintain class-balanced training scheme. Note that it is
4

---

Under review as a conference paper at ICLR 2022
much easier and also more interpretable to fix a precision value rather than the choice of a confidence threshold (in particular if the model is not calibrated) as done in Xie et al. (2020).
D) Pseudo Labels: for the original labeled dataset we always use one-hot labels. For unlabeled data points that have been selected in I, we determine pseudo-labels according to the predicted probability distribution over the classes by the calibrated teacher model. Due the calibration, this should reflect the "correct" uncertainty about these labels. For all remaining images in our unla- beled dataset U we use weak form of knowledge distillation by defining pseudo-labels as the average of the predicted probability distribution of the teacher model and the uniform distribution, given in (2). This has two reasons: i) a purely uniform distribution on U which in the first it- erations might still contain a lot of task-relevant images, leads to bias as it does not distinguish between reskvrel and irrelevant images, ii) only using soft-labels from the teacher model leads to overconfident predictions as one can observe in the non-out-distribun Thus trade-off between these opposing goals is their average which leads heavy damping of the confidence (note that the pseudo-labels have a maximal confidence of 2 + K on U.D.
aware NSST method.
For the final objective in (3), the selected pseudo-labeled samples in I and the original samples in T are assigned the same weight. This is quite aggressive as for iteration we add up to 51 as much pseudo-labeled data as labeled training data. However, this also enables larger performance gains given that the sample selection process is successful. Note that the losses on I U T and on U.I have equal weight as the damping of confidences on U is crucial for the sample selection process. As pseudo-labels are computed non-augmented images and we use heavy augmentation during training, our scheme can be regarded as offline consistency learning, where the model is encouraged replicate the teacher's output independent of the randomly selected augmentation. This allows us to limit the number of passes through the entire unlabeled pool to the number of self-training iterations, do model calibration and results in more stable targets. The entire scheme is repeated multiple times until the validation accuracy starts to degrade or a fixed maximum number of iterations is reached.
E) Training:
ODST+: In addition to ODST, we provide the variant ODST+ for scenarios where Al safety is crit- ical. It differs from ODST only in step C) where we calculate the final threshold as maximum over ODST's in-distribution threshold and an additional out-distribution threshold. This class-specific out-distribution threshold controls the number of task-irrelevant samples that are falsely added into our apeudo-labeled sample pool I. This is done using an extra out-distribution validation set, i.c. a set of natural images that does not contain any class relevant images (we discuss this choice in Section 44). For each class c, we compute the o-quantile of the predicted probabilities for class on the out-iistribution images which we define as the out-distribution threshold for class c.
3.3 BAYESIAN DECISION THEORY OF SELF-TRAINES
In this section, we analyze our iterations in the framework of Bayesian decision theory. We show that the base classifier that enforces uniform confidence on the unlabeled points still leads to opti- mal decisions on the in-distribution. Moreover, we show that the iterative scheme with soft-labels ultimately reaches the optimal classifier which is Bayes optimal on the in-distribution task and max- imally uncertain elsewhere. Proofs can be found in the Appendix [A]
In this section we assume that the labeled examples (I;; V are drawn i.i.d. from Pim (x, y) (joint distribution on Rd x (1,..., K)) and the unlabeled data (=x) are drawn i.i.d. from Pan on Idd In the open-world setting we think of Pall as the marginal distribution of mixture of very large number of classes (much larger than K). including the in-distribution ones (see Appendix) in which case it naturally holds for the marginal distribution (in(2) that Pin > 0 implies Pan (z) > O.
In expectation, the ODST base classifier (Bq. ( optimizes (we omit the index 0 in (()):
E E(x,Y)~p ((X) + Ezzpa [ 1(2))
(4)
Lemma 3.1 Let p(k(2)) = R efaca then the Bayes optimal prediction for the loss is given for any x with (a) + (ir( > 0 as
p(k())) = P ( + Pan(())
k 1,...,K.
Pi (m) + Pa)(x)
5

---

Under review as a conference paper at ICLR 2022
We have directly provided the optimal predictive probability distribution instead of expressing it in terms of the classifier f. Note that p(k]a)) ()) is a monotonic transformation of Pin(k(z) and thus preserves for each point the ranking of the classes according to (in(k)a) and thus the optimal decision does not change. However, the ordering of the confidence max; Pin (k((r) across different inputs a is influenced significantly by the ratio of Pin (2:) to Pall (z). Non-task relevant instances where Pan (2r) is larger than (x) are significantly
and thus will not be selected, whereas if Pin (2) is much larger than (z) the confidence max; ((k [x)r)) almost equal max; Pin (k]z), Note that the
latter case is in particular true for task-relevany images (Pin(z) large) as Pail is a much more spread out distribution and thus the density value (=) will be small. This justifies our OOD aware base teacher and also our post-training calibration step A) is under-confident on the in-distribution.
The mathematical treatment of our sample selection strategy is difficult, but it instructive to check the case where at each iteration ++1 1 we impose on all unlabeled points soft-labels, fr (k|zr) defined by the Bayes optimal teacher f()) at iteration t. Then we get the total expected loss at iteration t + 1:
P(x [2,Y, (())))))))))))))) (())) + Rz-pan [L (z)))
(5)
Lemma 3.2 The Bayes optimal prediction for 55) at iteration & for t 2 is given for any x with Par(r) + (i)( > 0 and k = 1,..., K as
Pan(()
1+1 1
(c(k))z)= = (kn(k) + (x)( + Par((x)
-mm((k()))
K
In particular, for any with (x) + Pan( we get:
[ Pin (k</)z)
if Pin (m)(x) > if Pin(2) = 0.
lim (e(k)))) 1
= [ K
This is the perfect out-distribution aware classifier: Bayes optimal for the in-distribution and maxi- mal uncertain on non-task-related regions (Pin(x)) = 0). This justifies our approach from a decision- theoretic perspective. In the finite sample case, neural networks are overconfident on far away regions (Hein et al. 2019) and thus damp the pseudo-labels on the unlabeled part in step D).
4 EVALUATION
First, we evaluate ODST/ODST+ on CIFAR10 in the standard SSL setting with 4k labeled images but now using open world unlabeled data with up to 10M unlabeled images and compare it to ex- isting (open-world) SSL methods. Then in our main evaluation, we use the full training sets of CIFAR10/100 together with the full 80 million unlabeled images from 80MTI and show that we can improve performance in this challenging setting. Moreover, we identify that existing (open-world) SSL methods show a strong distribution shift, which is problematic when using these methods for safety-critical applications.
Training of ODST and NSST and (open world) SSL baselines: For the small scale experiments, we use a WedeResNNe28x22 for all methods and for the full S0M experiments a standard ResNet50 He et al. (011)) and also the larger PyramidNet272 (Han et al. (017) with ShakeDrop regularization (Yamada et al. 2019). In each self-training iteration, due computational restrictions, we use fine- tuning to train the PyramidNets and train all other architectures from scratch. We perform three iterations for the large-scale experiments and five iterations for CIFAR10-4k and always report the iteration with the best in-distribution validation error.
In the sample selection step of iteration t, we select the top-k predictions on the full unlabeleld dataset per class as potential candidates for the labeled set, where h 5tN K (N is the size of the labeled set, K the number of classes), Among these points, we select the ones which are above the in-distribution threshold (NSST and ODST) respectively in- and out-distribution threshold (NSST+ and ODST+). This difference in sample selection is the only difference to the plus version and note that NSST and ODST do not need access to any additional data compared to existing approuches. NSST also serves as ablation study, as ODST only differs in two aspects: i) the OE base model ii) the train loss 3 instead of (1) (See Appendix for a detailed discussion).

---

Under review as a conference paper at ICLR 2022
No SSL
SSL
Open
World SSL
plain
OE
MM
FM
NSST
NSST+
MTCF
DS3L
ODST
ODST+
4k I.
Acc. 1
86,62
84.91
81,50
89.03
88. 70
87.75
91.86
78.40
93.89
93.41
1M TI
FPR
78.48
31.17
84.16
66.71
82.20
80.60
62.17
83.51
10.51
16.71
4k L
Ace.
86.62
86.57
79.92
85,43
88.15
87.88
86.76
92.14
92.21
10M TI
FPR 4
78.48
27.13
82.28
82.03
75.23
77.48
73.43
19.33
13.78
4k L IM LSUN
Acc. 1
86.62
86.30
81.26
89.53
88.30
87.82
90.39
94.31
94.31
FPR 1
78.48
43.98
85.27
75.50
77.47
75.73
$4.08
23.26
25.61
Table 1: CIFAR10-4K:
Results for 4k labeled images and an unlabeled dataset containing 41k CIFARIO training and million respectively 10 million unlabeled images from 80MTI or IM LSUN images. MixMatch (MM) and DS3L perform worse than the "plain" baseline (shown in red). ODST+ and ODST outperform all methods in terms of accuracy. For out-of-distribution de- tection, we report the average false positive rate (FPR) at 95% TPR. ODST+ has a FPR more than 30% better than the closest competitors FixMatch (FM) and MTCF.
As standard SSL baselines, we use the state-of-the-art MixMatch (Berthelot et al., 2019) and Fix- Match (Sohn et al. 2020). For open world SSL we use the recent DS3L (Guo et al., 2020) and MTCF Yu al., 2020). We only compare to existing approaches in the small scale setting as DS3L and MTCF do not scale to larger unlabeled datasets due to excessive memory consumption (several terabytes) and MTCF's domain training requiring 5 days for 10M datapoints. Self-training methods are superior in this regard as the number of full passes through the unlabeled data is limited by the number of training iterations and labeling can be parallelized arbitrarily to scale to large datasets.
We use 80 million tiny images Torralba et al. (2008) (80MTI) as unlabeled dataset, which has been created by querying 53,464 different nouns from the wordnet hierarchy. Note that CIFAR10 and CIFAR 100 are subsets of 80MTI (Krizhevsky & Hinton] (009), and we remove (near)-duplicates from 8OMTI, see Appendix [.5 details. For the small scale experiments we use randomly selected subsets of IM and 10M images and additionally a IM LSUN subset.
Unlabeled dataset:
Evaluation metries: In an open-world setting, it is important to not suffer from a distribution shift. Thus, in addition to test accuracy, we evaluate the out-of-distribution (OOD) detection per- formance by reporting average false positive rate over several OOD datasets: test set of CIFAR100
or CIFARIO, 10, SVHN, LSUN-CR(Yu et al. 20155) with Flowers Flowers(Nilshack & Zisserman
2008) plus
Food-101 (Bossard et al. 2014) for CIFARIO and FGVC-Aircrat(Maan et al. (013) for CIFAR100.
4.1 CIFARIO-4K
In the SSL iiterature(Berhhllo al., 2019; Yu et al., 2020). it is common to use 4000 labeled CIFAR1O samples as train data and 5000 as validation set. The remaining 41k training images used as unlabeled data and mix them with different out-of distriiiitiiiioonn datasets to create challenging open-world scenario: i) 1M respectively ii) 10M unlabeled images from 80MTI or iii) 1M LSUN images. Note that 80MTI contains further task relevant samples for CIFAR10.
The class-specific in-distribution threshold is set to a = 0.98 (meaning that we accept maximally 2% false positives per class on the in-distribution validation set) and for ODST+ and NSST+ we use the same value as out-distribution threshold. For the creation of an out-distribution validation set for ODST+ and NSST+, there exist two strategies. The first one is use an existing dataset with a sufficient variety and remove any potentially task-related samples. For this, we use a subset of 2k 100 images as validation set for the 80MTI experiments and remove the classes "bus" and "pickup-truck" as they can be confused with "car" and "truck" from CIFAR10. The second approach is to manually label a sufficient number of unlabeled samples as out-of-distribution We simulate this by using subset of 2k unseen LSUN images in the IM LSUN experiment.
The results in Table I show that our open world setting is challenging due to the low number of
task-relevant samples in the unlabeled set (lower bounded by 4.1% respectively 0.41%). MixMatch but also the open world adapted method DS3L perform worse (shown in red) than plain training on the 4k labeled samples. Our ODST and ODST+ outperform all other (open world) SSL methods in terms of test accuracy and significantly improve over the OE model that was used as base teacher (Hendrycks et al. 2019). While for 1M unlabeled data points from 80MTI LSUN, MTCF and
7

---

Under review as a conference paper at ICLR 2022
ResNet30
PyramidNet
plain
OE
NSST
NSST+
ODST
ODST+
plain
OE
NSST
NSST+
ODST+
+
Acc.
96.11
96.81
96.86
96.93
97.98
98.07
98.49
98.28
98.24
98.13
98.57
FPR
45.54
4.40
49.72
51.76
4.30
3.84
22.76
2.56
33.51
34.71
2.40
Table 2: CIFAR10-50k: We show test accuracy and FPR @9STPR as out-of-distribution detection performance. ODST+ has the best improvement (1.26%) and final test error (1.93%) for Resnet50 and is the only method which improves for the Pyramid272 architecture by 0.29% with 1.43% test error whereas NSST and NSST+ degrade in test performance.
FixMatch perform well, they show severe performance degradation for 10M data points. We believe that the non-adaptive confidence threshold of FixMatch respectively the Otsu threshold of MTCF are not sufficient for the accurate selection of task-related images. Even though the prediction perfor- mance is reasonable for 1M samples, the classifiers show a strong distribution shift as illustrated in Fig. D and Fig. 6 where we visualize unlabeled samples which are considered to be task-relevant by the different methods. As the failure of MTCF and Fix.Match is not noticeable from test accuracy, we strongly suggest that open world SSL papers should visualize high confidence samples and evaluate OOD detection performance. In Table we therefore report the average FPR over different OOD datasets. ODST and ODST+ have a 30% better FPR than any other SSL method. While ODST+ and ODST' show similar empirical performance we demonstrate potential advantages of ODST+ garding Al safety in Appendix [B.6] In Appendix [C.1 we provide details over the full run of ODST+ and ODST and report corruption robustness on CIFARIO-C.
1st Iteration (25k)
2nd Iteration (50k)
3rd Iteration (75k)
NSST
115-1A3
ODST
Figure 2: CIFAR10-50k: Plot of randomly chosen, exclusively selected samples from 80MTI for NSST (top) and ODST (bottom) over all three iterations. False positives are marked red.
4.2 CIFAR1O-50K
Up our knowledge, no SSL method could yet show performance improvements in an open world setting when using the full training set of CIFAR 10 as labeled set. We show that this indeed possible when using 80MTI unlabeled dataset. While DS3L and MTCF do not scale this setting, both noisy student variants outperform any consistency-baset (open world) method in the 10M setting and act as strong baseline for our ODST and ODST+ + models. For the in-distribution validation set, we use the recent CIFAR10.1 dataset (Recht et al.) 2018) designed to assess the generalization of classifiers trained on CIFARIO. As out-distribution validation set for ODST+, we use 2k CIFAR100 samples without conflicting classes (See Appendix B.6 for details). As threshold parameter we use 99.8% which is conservative but justified by the high accuracy of the base CIFARIO classifiers. we report the results for both architectures.
In Table 2
ResNet50; ODST+ improves test accuracy by 1.26% from 96.81% to 98.07% and outperforms NSST and NSST+ by least 1. 1.1.. We ported hefore for a ResNet50 on CIFAR IC Even though NSST and NSST+ improve slightly upon the bascline, Figure Secureehhighte highhlggs that even with the OD threshold, NSST+ suffers from a distribution
aware that such a high test accuracy has been
'Concerning other results with the same augmentation, Cubuk et al. (2019) require much larger AmoehaNet-B to achieve error rate of 1.80% and only achieve 2.6% with a WideResNet-28x10, which typically outperforms ResNet50 Zagoruyko & Komodakis (2016)
8

---

Under review as a conference paper at ICLR 2022
ResNet30
PyramidNet
plain
NSST
NSST+
ODST
ODST+
plain
OE
NSST
NSST+
ODST
Acc.
80.69
79.98
82.44
82.03
83.54
84.09
88.07
87.60
87.71
87.53
88.66
FPR
72.05
43.88
73.47
82.54
35.64
38.72
60.20
27.97
74.90
73.91
29.22
Table 3:
CIFAR100-45K: ODST (+) have the best accuracy for both model architectures. SSL-
results in red are worse than using labeled data only.
0.00
bridge
boesle 1.00wllow
1.00
luard
crob 0.90
Strow -0.96
Surfower--o 0,08
boby -0.59
woff-0.2
bee-1000 1.00
Tocooo 1.00
1oste-1..0 kargaroo -0.99
bee 1.000
keyboord 11.00
boy 0.99
NSST
ODST
Figure 3: CIFAR100-45K: Random selection of 14 samples in the third iteration of self-training for the ResNet50 architecture. False positives are marked in red.
shift and identifies completely unrelated samples as task-relevant and thus self-training degrades very early e.g. almost all images containing humans are classified "horse" "dog". ODST+ slightly outperforms ODST, most likely due to a better sample selection. But even without the addi- tional OOD-validation set ODST improves the OOD performance from 4.40% FPR to 4.30%, which further improved by ODST+ to 3.89%. We highlight that all other methods show worse FPR than the base model, which is likely caused by the distribution shift which is concerning regarding Al safety. This again highlights that just relying on test accuracy can be misleading.
PeramidNet272:: As the test error of the baseline OE is already below 2%, further improvements much harder to realize and can only be obtained by succeeding in the challenging task of selecting high-quality, task-relevant samples from the large pool of unlabeled samples. ODST+ achieves this and is able to improves test accuracy to 98.57% and FPR to 2. 2.40%. In contrast, due to their poor sample selection, NSST and NSST+ start degrade in performance after the first iteration and are thus not able to profit from unlabeled data. We had to skip ODST due to computational constraints.
4.3 CIFAR100-45K
For CIFARI00 100 we randomly select 50 out of the 500 training samples per class as in-distribution val- idation We use 2k CIFARIO samples without classes "car" and "truck", as they ambiguous wrt to "pickup- "pickuu-truck"" truck", as out-distribution set validation for the Plus methods. In- and out- -itdiitrrittionn thresholds are set to 98%, due the lower base accuracy on CIFAR100. The results are in Table where we see that on the ResNet ODST+ improves the test accuracy by 4.11% % to 84.09% compared the OE model and to 83. 54% for ODST. While NSST and NSST+ improve by 1.75% to 82.44% respectively 1.34 82.03%. we again notice a distribution shift which can be observed from ran- dom selection of samples that pass the threshold and are thus considered task-relevant (Figure) 3) This is further verified by the very high FPR of 73.47% respectively 74.19% for OOD detection for the ResNet architecture, whereas both ODST variants improve POD-performance over the OE baseline. For the PyramidNet, both NSST variants worse than the plain baseline in accuracy and FPR whereas ODST+ clearly improves test accuracy with a small degradation in FPR.
Conclusion
We show that ODST can leverage large unlabeled open-world datasets with only tiny fraction of task-related samples and consistently improves over the baseline and other SSL methods. The re- sulting classifiers Moreover, we observe that all competing methods suffer from a distribution shift, which is problem- atic regarding AI safety, whereas ODST and in particular ODST+ shows almost such degradation.
accurate, robust and show better out out-distribution detection performance.

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement
We include our entire codebase in the supplementary material of our submission which allows to easily reproduce our results. This includes indices any train/validation splits and indices of near- duplicates in 80MTI such that is possible to run the experiments on the same data that was used in the paper. Due the stochastic nature of neural network training, results can vary slightly.
Ethics Statement
There are no conflicts of interest in this work. potential danger of semi-supervised learning methods, in particular self-training, is that they suffer from distribution shift. This danger is even more severe in an open-world scenario. As we show in this paper, heavy distribution shifts happen for other SSL methods but ODST is very robust to it. Nevertheless, we have introduced ODST+ where an extra out-distribution validation set is used to prevent distribution shifts. While in the considered classification problems ODST was already very robust, this might in our opinion this a too much benchmark- systems using open-world SSL, thus encourage subsequent work check out-of-distribution detection performance, ii) check the selected unlabeled samples, and iii) consider an out- out-distribution validation set for the construction of the models.
unnecessary. But point of view. In order to build trustworthy ML
References
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raf- fel. Mixmatch: A holistic approach to semi-supervised learning, In NeurIPS, 2019.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In COLT, 1998.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative com- ponents with random forests. ECCV, 2014.
T. E. Boult, S. Cruz, A.R. Dhamija, M. Gunther, J. Henrydoss, and W.J. Scheirer. Learning and the unknown: Surveying steps toward openworld recognition. In AAAI, 2019.
S. Boyd and Vandenberghe. Corvex Optimization. Cambridge University Press, 2004.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
O. Chapelle, B. Schölkopf, and A. Zien. Semi-Supervised Learning. MIT Press, 2006.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment:
Learning augmentation strategies from data. In CVPR, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshop, 2020.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, 2005.
C. Guo, G. Pleiss, Y. Sun, and K. Weinberger. On calibration of modern neural networks. In ICML, 2017.
Lan-Zhe Guo, Zhen-Yi Yu Zhang, Yuan Jiang, Yu-Feng Li. and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICMI., 2020.
10

---

Under review as a conference paper at ICLR 2022
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
M. Hein, M. Andriushchenko, and Bitterwolf. Why ReLU networks yield high-confidence pre- dictions far away from the training data and how to mitigate the problem. CVPR, 2019.
D. Hendryeks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of--istribution examples in neural networks. In ICLR, 2017.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In CVPR, 2019.
M. Stinchcombe K. Hornik and H. White. Multilayer feedforward networks are universal approxi- mators. Neural Networks, 2:359-366, 1989.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint, 2016.
Mong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
K. Lee, H. Lee, K. Lee, and J. Shin. A simple unified framework for detecting -itor--distribution samples and adversarial attacks. In NeurIPS, 2018.
S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervupeee learning. IEEE PAMI, 41(8):1979- 1993, 2018.
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. ICVGIP, 2008
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, 2018.
AristotelisAAngelo Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang- Outlier exposure with confidence control for cat-of-distribution detection. arXiv preprint, 2019.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? In arXiv preprint, 2018.
Ellen Riloff. Automatically generating extraction patterns from untagged text. In AAAI, 1996.
11

---

Under review as a conference paper at ICLR 2022
Ellen Riloff and Janyce Wiebe. Learning extraction patterns for subjective expressions. In EMNLP, 2003.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJV, 115(3):211-252 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In NeurlPS, 2016.
Scudder. Probability of error of some adaptive pattern-rocognition machines. IEEE Transactions Information Theory, 11(3):3633377 1965.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-mat features. In ECCV, 2018.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averagee consis- tency targets improve supe-super deep learning results. In NeurIPS, 2017.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE PAMI, 30(1 1995889190 2008.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Qizhe Xie, Minh- Than-Thhngga Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, pp. 10687-10698, 2020.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint, 2019.
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning. IEEE Access, 2019.
Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint, 2015.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, 2020.
S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Giulio Zhou, Subramanya Dulloor, David G Andersen, and Michael Kaminsky. Edf: ensemble, distill, and fuse for easy video labeling, arXiv preprint, 2018.
Xiaojin Zhu and Andrew B Goldberg, Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical report, University of Misconsin-Madisse Department of Computer Sciences, 2005.
12

---

Assumptions: Let Rd be the input and y f1...., K} be the class label. We assume that labeled examples (æi, ys y=) are drawn i.i.d. from (in y) (joint distribution over inputs and labels on Rd {I..... K}) which is our in-distribution. The unlabeled data (z) is drawn i.id. from Pall (distribution on Rd) where we think of Pall in an open world setting as the marginal distribution Under review as a conference paper at ICLR 2022
First, give overview over the content of the appendix:
in Section A we give the missing proofs from the paper.
Section contains more detailed description and hyperparameters for all methods and datasets. More specifically, we discuss our duplicate removal for 80MTI and the choice of the out-distribution validation set
In Section C
we give more detailed breakthrough the main results from the paper.
Section D contains ablations studies in the large scale setting with the full 80 tiny images setting.
A Proofs for Bayesian Decision Theory of Self-Training
We provide here the missing proofs of Section 3.3 where we have analyzed our iterations of self- training in the framework of Bayesian decision theory.
We repeat the setting and assumptions so that this section is self-contained.
of a mixture of very large number of classes (much larger than K), including the in-distribution ones. We also assume that Pin(x) > 0 implies Pan (2) > O.
An open world setting: We now present particular way how such setting could arise though this is by no means the only setting under which the above assumptions are realized. How- ever, the following is good model for 80MTI the datasets CIFARIO and CIFAR 100 have been generated by subsampling from 80MTI and thus the classes of CIFARIO respectively CIFARI00 are definitely a subset of all classes present in 80MTL. We assume that every class has a spe- cific distribution p(xly) and for convenience we assume that they are ordered such that the classes y E (1...., K) are our task-relevant classes and the ones y E {K + 1,..., M) are other classes not part of the task. Then if we assume that there is no distribution shift, that is p(zly) is the same for labeled and unlabeled data, then for y (1...., K}
(in p(zly))mn
where Pmn(y) > 0 is the class probability in the in-distribution task and
K
(k) = 1.
Note that
K Pin (z, k)
K
(i)
Σ
p(m)))
A=1
k=1
The out-distribution is mixture distribution of all possible classes
M Σ p(a)k))a((k))
(an(x)
where we assume (y) 0 for all y E (1...., M) (all classes present in the unlabeled data) and it has to hold Σd' Pan = Generally, it holds Pin(y) * Pan(y) for y E (1...., K}. With this assumption we immediately see that
(m) >
Pan (x) > O,
as if Pan (z) there exists at least one class E {1,..... K} such that p(r]s) 0 which implies that
(a) 2 p(x)s)pan(s)) 0,
13

---

Under review as a conference paper at ICLR 2022
we assume that Py > 0 for all (1...., M). In closed world setting there would be no additional classes in the distribution of the unlabeled data Pall- that is M = K, whereas in our open- woendworld setting we have M > K. However, note that this model is more general than assuming that the unlabeled data is sampled from the marginal distribution Pin(2) Σx=1 p(a Pi) (k) of our joint distribution on inputs and labels ,eenn in a closed-wor worlld setting. our model would allow that the class probabilities between labeled and unlabeled data could differ which is quite realistic in practical setting.
The ODST+ base classifier, see T.. optimizes
N
M
Σ 1 1 j=1
Σ (y:.Pro +
M
which in expectation (for simplicity we omit the index 0 in f(u) yields:
((,Y))~P [(Y)),(x))] + Rzzpp I 1 11)111))]
Note that in the following we overload Pin so if we write (X,Y) ~ Pin then we mean the joint distribution over inputs times labels IR (1...., K} and if write X Pin then mean that X is distributed according to the marginal distribution.
The idea of Bayesian decision theory is characterize the optimal classifier given that one has full knowledge about the problem and one could optimize over all measurable functions. Note that feed- forward neural networks can approximate any function to arbitrary precision (K. Hornik White 1989) so that this optimal classifier is in principle achievable. The Bayes optimal prediction f over all possible measurable functions is characterized in the following lemma in Section 3.3
Lemma A.1 Let (X,Y) Dia and Z Datt. L(y, (x)) is the cross-entropy loss and let p(k)z))
Sale) Dis aftee then the Bayes optimal prediction f for the objective in (4) is given for any x with Pat(() + (I) > 0 as
(y(k)z) (m) + R Pan(x) (x) + Par(x)
t: I...,,K. K.
We can write the expected loss in 66) with the predictive distribution 0y(k(z) and the cross-entropy loss, (pp Ee Pk log(Ex),
Proof:
E B (x)) + Eznpa I 1 1.1r(2))]
K
=Ex =Exxpa Ev [L (x) + Ez Ezupul
(1,,)((1))
= =/01 Pin (ar)
Pin (k]]))) Py (x)))
Pan(z)
Mex, ((
K
k=
K
(1())) + K Pau(x))es Pel()))ex.Pp(xt) dr
3d
K
(z (A|x)) + pan(()) (log(
Σ 1=1
cd(le)) fc()))
Rd
where we have used that the cross-entropy loss linear in the first argument. We optimize pointwise, that is for every f(x), the above objective. As the cross-entropy loss is convex in f(x) in the second argument (note that the sem-summeppp function convex. see Boyd & Vandenberghe (2004)) we thus get the pointwise objective, á Rd + R, for e = f(x) E Rd if we we define
2k = (m + K Pan(2))
14

---

Under review as a conference paper at ICLR 2022
K
k
(c))
Σ
26
log
Σ
k=1
1=1
The optimality condition is then given by
0Φ
K
c
dc,
(c)
2b
Ei=1
k
2y
0.
k=1
which yields
c
K
21=1
e
1==1
and thus we get
(f)
(in)))))(((2))) + Pan(x)
E 1=1 efi(z)
(in + Pa)
where we have used that
K
Σ
(In(=))pn((22) + K
Paa(z)) = Pin(x) + Pa)(
21
1=1
1=1
Using that
efala)
[k)x)) Ex_eetce ed(())
get the final result.
The second result yields the Bayes optimal prediction for iterated training scheme where the Bayes optimal predictions P. from the previous Lemma A.1 of the Bayes optimal teacher f(1) at iteration become the soft-labels for the student model f(++1)
Then we get the total expected loss for the student model f(++1) at iteration t ++ 1:
(X,)) L(Y, ((+))) + EZzzp (Q,((2), ((+1)
Lemma A.2 The Bayes optimal prediction for
5 iteration for t 2 is given for any with
Par(2) + Pmn > and 1...., K as
pr Pin (k|a)) + ( Par(x)
(
(k)x))
(x) Parlz))
K
Proof: We prove this by induction. First we note that for t = 0 (base model) we have shown the predictive distribution (k(zz) in Lemma A.l to be
efa(x)
(n(())in(kka x)) + Pan(()))
po(k)x)
E ef((e)
Pin (x) + (x)
Noting that
(k) + & Pan( (i)( + (2)
Pan(x) Pin (k(x)) + (i)( + (an(x)
Pin (k(xz))
we observe that this agrees with the expression the lemma to prove. Thus the induction start for
t = 0 is verified and we go for the induction step. Repeating the derivation of Lemma A.I we get
Det (R(]z) Pin (in( (k(aa) + pan (1r)),
(in + (a)(
15

---

Under review as a conference paper at ICLR 2022
From plugging in the induction hypothesis we get
Pin(#) (klz) + (kk())
Pin (2) + (a)(
Pin (1) (k
Pau(
(in(k)z)) + ( Pan
1-1
(x - Pin(k(/2))
Pin(a)) + Pan
Pin(2) + (I))
(in(x) - Panl(x)
K
(in()) (k) + Pan Par (k(or)
Pall
c+2 (% Pin(k((o))
Pin (z) + (a)
(in(x) + Pan(x)
Pan (x)
e+2
1
(ki(2k (x) + (a)(
Pin(k(a)))
which finishes the proof.
In partieular, for any with (i) + Pan( we get:
Pin (k]()) if Pin (z)) >
lim (x] (-30
if Pin (2) =
K
Note that this is the perfect out-distribution aware classifier: Bayes optimal for the in-distribution and maximal uncertainty on all non-task-related regions (Pin(1:) -).
B Implementation and Dataset details
As network architecture, we use a WideRessNet2822 for CIFAR10-4K for all methods. The large scale experiments on the entire a ResNet50 respectively a PyramidNer272.
training set with 80MTI unlabeled data use
In this section, we present the hyperparameters used to train our ODST+ and the noisy student selftraining models. Note that we use the same set of hyperparameters for ODST+ and the two baselines NSST and NSST+ for all experiments.
B.1 WideResNet28x2
For the base model, the only labeled data are the 4k train images. As this yields short epochs, we train the initial model for 5000 epochs with a piecewise schedule and the learning rate decays by factor 10 at epochs 2000, 3000 and 4000. The initial learning rate is set to 0.1. Even with strong AutoAugment (Cubuk et al. 2019) augmentation (which includes Cutout Dries & Taylor 2017), due to the small dataset size, the model tends to overfit in the later stages of training. We thus use early stopping on the validation set. In practice, the model with best validation accuracy is often achieved right after the first LR drop at epoch 2000. As we add up to fives times as much unlabeled samples the "labeled" pool the epochs get significantly longer and thus we reduce to limit of 1000 of epochs with proportional rescaling of the learning rate schedule.
For NSST and NSST+ use a batchsize of 128 and for both ODST variants, use a combined batch size of 256 with 128 in-distribution samples, this includes the labeled training samples as well as the unlabeleld samples selected by the self-training method be task-relevant, and 128 samples for the second part of the loss corresponding to the non-selected unlabeled samples (which are sampled randomly from this set). We use standard SGD with Nesterov momentum 0.9 and weight decay set 0.0005, All CIFAR 10-4k experiments use 5 iterations of self training and with 2k to 10k additional samples (increasing by 2k per iteration, which corresponds to times the number of train samples which is the ratio fix for all experiments).
B.2 ResNet50
Our ResNet50 models are always trained for 250 epochs with piecewise learning rate schedule with starting learning rate of 0.1 and decay it by a factor of 10 at epochs 100, 150 and 200. The choice of the data augmentation, batchsize, the optimizer and weight decay is the same as for the WideResNet.
We evaluate validation set error throughout the last 20% of the epoch and chose the model with the best validation set performance. For the large scale experiments, we do three iterations and add up 5/10/15 times the number of labeled training samples in the first/second/third iteration.
16

---

Under review as a conference paper at ICLR 2022
B.3 Shakedrop PyramidNet272
For the Net, the base models are trained for 1000 epochs with a cosine schedule and initial learning rate of 0.05. The batch size is set to 64 for NSST and NSST+ and a total batch size of 128 for +DST+ (64 for in-distribution and 64 for the unlabeled pool). We use Nesterov SGD with a weight decay of 0.0001. For the base models we use AutoAugment as data augmentation.
As training large models for 1000 epochs and with up to 16 times the amount of labeled data is ex- pensive, we use fine-tuning train the later student models for 55 epochs. We thus always initialize the new student with the previous teacher model's weight. Note that although the student model is initialized with the teacher model that was used to label the unlabeled data, the training loss for the pseudo-labeled data is not zero due to Shakedrop Yamada et al. (2019) and data augmentation. Due to large amounts of noise from both heavy data augmentation and Shakedrop, we use a mixed aug- mentation strategy for fine-tuning. In detail we train each student for N epochs with AutoAugment and a cosine schedule with initial learning rate of 0.05 that decays to O after N epochs. We then do warm restart and train for another 55 .V epochs using a cosine schedule starting at learning rate 0.01 and only use random cropping and flipping as data augmentation. As data augmentation be- comes less useful with increasing amounts of data, we set N to 50 for the first iteration and decrease it to 40 and 30 for the second and third iteration (student). Batch size and weight decay remain at 64/128 and 0.0001 for fine-tuning. Due to computational reasons we only train the ODST+ model in this large scale scenario.
B.4 Noisy Student Self Training
To make NSST and NSST+ more comparable ODST(+), we adjusted few details about the orig- inal noisy-student self-training scheme (Xie et al.] 2020) and adapted it to our CIFAR setting, as it was orignally proposed a larger ImageNet setting with JFT-300M as unlabeled distribution. First, calibrate all models using temperature scaling (Guo et al., 2017) by minimizing the expected calibration error, whereas Xie et al. (2020) use the non-calibrated classifier outputs. Standard mod- els are typically overconfident and the calibration will make them less overconfident, which should result in better SSL. performance as the calibrated pseudo-labels are used in the next iteration.
Next, we calculate the in-distribution threshold (for NSST and the in- and out-distribution thresh- olds (for NSST+) on the validation set, whereas Xie et al. (2020) use a fixed threshold of 0.3 for ImageNet. As can be seen by the FixMatch results in Section C fixed threshold is generally not sufficient for filtering out task-related samples and it is not obvious how one would adapt the Im- ageNet threshold to CIFAR1O. Our new class-specific thresholding makes the NSST and NSST+ baselines more comparable to ODST+ and helps the two baseline methods to select fewer non- task-related samples. To make the training more comparable WedeRes we do not use any form of stochastic depth as in Xie et al. (2020), however our strong data augmentation policy still induces high amount of noise into the training scheme. On top, the PyramidNe1272 with DrakeDrop regularization is even noisier than any architecture used in Xie et al. (2020), however NSST and NSST+ are still underperforming in comparison to ODST+, implying that noise alone is not sufficient to make self- seff-training scheme robust to the open world setting. In summary, NSST and ODST only differ in two aspects, namely the base model, as ODST uses OE model in comparison to the plain teacher of NSST and in the training loss, ODST includes the second loss term from equation 33. All other details, including hyperparameters, cali- bration and threshold selection are chosen to be identical. Thus all performance improvements can be directly related to the base model and the loss function.
the other SSL methods using a
B.5 Duplicate removal
In this section, we explain our approach to duplicate removal of CIFAR test images in the 80MTT dataset. As both CIFAR 10 and CIFAR 100 are subsets of 80 million tiny images, it is important to re- move exact- and near-duplicates of test images from the unlabeled distribution to prevent them from leaking into our train set. However, after duplicate removal, the unlabeled distribution should still contain task-relevant images to use for semi- semi-supervised learning. First, we noticed that the duplicate removal from Hendrycks et al. (2019) did not remove all duplicates from 80MTI, which they use as cut-iistribttion to enforce uniform confidence (Figure 5a). While this should not improve their test accuracy, it might influence out-distribution detection when for example calculating the false posi-
17

---

Under review as a conference paper at ICLR 2022
114
32
Figure 4: Logarithmic histogram of nearest neighbour distances between CIFAR test set and 80MTI for image pairs with an lz distance below 2000
tive rate for the OOD datasets where we use the CIFARIO and CIFAR1O0 test sets. Carmon et al. (2019) follow Recht et al. (2018) and remove all 80MTI images with I2-distance smaller than 2000/255 to the nearest neighbour in the CIFARIO test set. While this is likely to remove all dupli- cates, the approach seems overly strict after this process, only 65.807 307.640 out of the 79.302.01 002.017 images remain. Thus with their definition of near-duplicate 80MTI contains nearly 14 million du- plicates of the 10.000 test images. In Figure Sb we show that almost all excluded images duplicates. While we acknowledge that is important to optimise recall instead of precision when removing duplicates, their procedure too aggressive and leads an exclusion of large set of im- ages which have low variation or close to monochrome images. In Figure 4 we show a histogram of the 12-nearest neighbor distances between the CIFAR test sets and 80MTI. One can see that the vast majority of images have a nearest neighbor distance above an lz-distance of 3.0. A visual inspection in Figure55calso confirms that most images below that threshold are duplicates, thus we first remove all images from 80MTI with an 1z- distance less than 3.0 to one of the CIFAR test images. While this removes exact duplicates, there can exist near-duplicates with larger la-distance. We thus collect all 80MTI samples with le-earest neighbour to the CIFAR test set smaller than 2000/255 as potential candidates for removal. For each candidate æ and nearest neigbhour 2 in the CIFAR test sets, then calculate the perceptual similarity metric LPIPS (Zhang et al. 2018) and SSIM (Wang) eet al. (204) and remove the image LPIPS 2) < 0.025 and 1 SSIM[r, 2) < 0.4. As both metrics closer to the visual system, we found them to be more reliable at finding near duplicates for images with larger l2 distances, see Figure 5d but they are much too expensive to use them directly for nearest neighbor search. We highlight that we do not only find exact duplicates but also degraded versions that for example contain blur, slight translations, color changes and added text or logos. Note that we still remove some non-duplicates, showing that our thresholds are still chosen rather conservatively. Overall, we remove 24k CIFAR10 test set duplicates and 60k CIR 100 test set duplicates. Additionally, we also remove all samples selected by Hendrycks et al. (2019). which in particular includes exact train set duplicates. When training CIFAR10 models, we also remove all CIFAR10.1 duplicates with the same approach.
no true
B.6 ODST+ and the OOD-validatin set
In this section, we briefly explain how the OOD-validation set for the plus variants NSST+ and ODST+ can be chosen. This method is especially valuable if the unlabeled pool contains lots of near cattof-distribution samples, for example snakes and lizards can easily be confused with frogs from CIFAR10, especially in a 32x32 resolution. We note that it is impossible to assess the confi- dence values of a model on near OOD-samples from ni-distribution samples, thus one can not easily compute a threshold that allows for a highly precise sample selection without such an OOD-dataset.
As discussed in the main paper, there are two possible ways to generate the OOD-validation set. The first one is to chose known dataset that covers a large number of classes, for example CIFAR100 or ILSVRC2012 and manually remove the classes that overlap with the train dataset. We use this approach for all experiments with 80MTI as out-distribution. As it is unrealistic to have an OOD- validation set multiple times as big as the train set, we use only 2k CIFAR100 samples in the 4k setting and only use all 50k CIFAR100 images for the PyramidNet in the large scale setting to achieve the best performance possible. In the additional experiments in Appdendix D we further demonstrate that the size of the OOD-validaton set does not have strong influence on the final
18

---

Under review as a conference paper at ICLR 2022
D:608
0.4
D:024
0.020
0:427
D:427
AM
ID
(a) 80MTI images after Hendrycks al. 2019 's duplicate removal with their nearest CIFARIO (upper) and CIFAR 100 (boltom) test neighbours sorted by lz-distance. Note that there still CIFARIO test set duplicates with distance larger than 0 and exact CIFAR100 test set duplicates.
coo
(b) Random selection of 80MTI samples with their respective CIFAR l2 neaneststt neighbour for sample pairs with distance smaller than 2000/ 0000/255 /55 = 7.84. This threshold was used by Carmon et al. (019) for duplicate removal but is aggressive at removes too many unrelated images.
0.113
0.0
0,609
000
0.00
0.1
5.10
5:10
0.107
LBI
(c) 80MTI samples with their CIFAR la-nearest neighbour for sample pairs with læ-distance S 3.0 (random selection). Most pairs are near duplicates and are thus removed in our duplicate removal. Even for this small l2 radius, we find false positives for monochrome images.
12414
(d) Random selection of samples with dz-distance in (3, 2000/225) that marked duplicates wrt both the LPIPS and SSIM threshold. We are able find mostly duplicates even in regimes where to become meaningless. In total our duplicate removal has a low number of false positives while we could not find any false negative (CIFAR test set image in 80MTI after duplicate removal).
the le distance starts
Figure 5: Visualization of various exclusion thresholds for CIFAR10 (above line) and CIFARTOM (below line). The top image shows the sample from 80MTI and the lower one the nearest neighbour in the test set.
quality. For CIFARI00, we similarly use 2k CIFARIO samples without classes "car" and "truck" and all 50k samples for the PyramidNet experiments.
19

---

Under review as a conference paper at ICLR 2022
The second approach for generating such an OOD- -ollviiiaaiion set is manually label out-of- distribution samples from the unlabeled pool. This can be done efficiently with a simple interface that sequentially presents samples from the unlabeled pool to the user to let them mark them as task-relevant or out-of-distribution. We simulate this by using 2k unseen LSUN images as OOD- validation set in the 1M LSUN experiments.
C Extended results
C.I CIFAR1O-4k
In this section, we will give some more detailed results on the small scale experiments with only 4000 training images. For MTCF and DS3L we use the official implementations and for FixMatch we use a Pytorch reimplementatiorf
Tableejoontaiis a breakdown of the 5 iterations making up an entire run for both NSST and ODST. Additionally to the results in the main paper, we also calculate corruption robustness CIFARIO- (Hendrycks & Dietterich] 2019). While NSST and NSST+ often decrease in validation accuracy after 2 iterations, ODST is able increase performance in later iterations and therefore make use of more data. This can be explained by distribution shifts that happen in the first iterations for the NSST models that lead to the addition of too many rubbish samples. In terms of corruption robustness, ODST and ODST+ are able to improve over the OE baseline. In detail, we achieve 78.80% respec- tively 77.17% for ODST and ODST+ in the 1M TI setting. beating MixMatch (3..1%)). FixMatch (77.08%). MTCF (73.08%) DS3L (59.56%), NSST .50%) and NSST+ (69.17%) Similarly, we beat the closest competitors MTCF (60.45 (60), FixMatch (66.19%) and NSST (69. (69.44)) with ODST (74.6%%) and ODST+ (74.94%) with 10M TI. With LSUN as out-distribution, we achieve 79.61% and 80.02%, strongly outperforming MixMatch (59.36%), MTCF (72.43%), FixMatch (73.18%), NSST (69.08%) and NSST+ (68.43%).
Overall, ODST and ODST+ are able to clearly outperform all other methods terms of clean accu- racy, corruption robustness and out-of-distribution detection.
We show a large random sample of unlabled images considered to be task relevant by ODST, ODST+, FixMatch and MTCF in Figures 6 and 7 for IM TI and IM LSUN images respectively. With 80 million tiny images as out-distribution, both Fix.attc and MTCF select too many images that can clearly be labeled non non-task--rlevan For FixMatch, this failure can easily be explained by the fact that they use a fixed confidence threshold of 0.95 on a model that is not trained to explicitly distinguish in- from out-distribution, which results in many OD images with very high confidence values (> 0.99). This once again highlights the fact that it is important to be out-distributtionaaaal during training as well as adapt the confidence threshold to the unlabeled distribution used, for ex- ample using an in- and out-distribution threshold like ODST+ does. MTCF is out-distribution-ane aware as they have an in-versus out-distribution binary classifier which shares the feature representation with the actual classifier. The actual threshold for the binary classifier to select the telk-relevant samples from the unlabeled dataset is then calculated via Otsu thresholding. In Figure [ we show the samples which the final MTCF model considers to be task-relevant when training has finished. In order to understand this more explicitly, we plot the confidence distribution of the CIFAR 10 clas- sifier over all calabeled-samplle (left) and of the ones above the Otsu threshold for the ID-vs-OD classifier (right) in Figure 9 It is noticeable that the two classifiers are not perfectly aligned and that relatively low confident samples are able to pass the threshold of the ID-vs-OD classifier whereas lot of highly confident examples are rejected. The generally lower confidence of MTCF compared to ODST+ and FixMatch seems to be a result of MixMatch training, which MTCF uses for the samples considered to be task-relevant (the pure MixMatch classifier has an even lower average confidence on the in-distribution).
While ODST is not perfect and still includes few unrelated images. it clearly outperforms the other methods by a margin. If an additional OOD-validation set is available, ODST+ can offer a slightly better sample selection quality. but both methods yield strong results given the low amount of labeled training samples together with the low number of task-relevant samples in the unlabeled data.
https:://githuubccmm//eee////
20

---

Under review as a conference paper at ICLR 2022
In Figure 8 for each class, plot the number of unlabeled samples considered to be task-relevant for ODST+, FixMatch and MTCF, First, we note that all different methods add very distinct class distributions. MTCF for example adds large number of frogs, however Figure verifies that most of them are indeed false positives. The dominant class for FixMatch dog. however the samples again show that the method falsely learned to label humans dogs, resulting in mostly unrelated samples. ODST+ is the most balanced method and has no strong bias towards any class. The peak cars can be explained by the large amount of car images in 80 Tiny Images and the samples again show that all car samples are indeed cars.
ODST
ODST+
Iteration
Base
1st
2nd
3rd
4th
5th
Basc
1st
2nd
3rd
4th
5th
Validation
85.22
91.44
92.52
93.74
94.12
94.62
85.22
91.50
92.70
93.62
93.68
93.96
CIFARIO
84,91
90.62
91.96
92.92
93.57
93.89
84.91
90.93
91.88
92.77
93.13
93.41
CIFARIO-C
67.19
75.43
76.32
75.47
78.38
78.80
67.19
75.29
75.48
77.65
77.05
77.17
OOD-FPR9
31.17
17.24
14.04
11.95
11.34
10.51
31.17
16.84
14.51
14.38
15.45
16.71
NSST
NSST+
Iteration
Base
1st
2nd
3rd
Base
2nd
3rd
4th
5th
Validation
86.50
88.06
89.04
89.20
87.08
87.92
86.50
88.62
88.62
88.64
88.34
88.28
CIFARIO
86.62
87.78
88.18
88.70
87.46
87.63
86.62
87.74
88.19
87.75
87.60
87.62
CIFARIO-C
69.04
69.65
68.69
67.50
65,46
65.40
69.04
69.24
69.38
69.17
66.71
66.38
OOD-FPR95
78.48
79.10
80.13
82.20
86.71
86.57
78.48
77.45
79.75
80.60
84.50
83.71
Table 4: CIFARIO-4K + 1M TI: Breakdown of the individual iterations of self-training schemes the CIFARIO-44 setting with an unlabeled pool diluted with IM samples from 80MTI. The final model is chosen based on validation accuracy and marked in black.
ODST
ODST+
Iteration
Base
1st
2nd
3rd
4th
5th
Base
1st
2nd
3rd
4th
5th
Validation
86.78
91.40
92.98
93.00
93.28
92.88
86.78
91.72
92.46
93.06
92.90
92.76
CIFARIO
86.57
91.05
92.02
92.37
92.14
91.96
86.57
90.95
92.07
92.21
92.48
92.35
CIFARIO-C
68.10
74.21
75.35
75.40
74.68
74.22
68.10
73.88
75.14
74.94
74.43
74.26
OOD-FPR95
27.13
14.29
12.68
16.97
19.33
22.52
27.13
14.06
13.19
13.78
16.98
20.52
NSST
NSST+
Iteration
Base
1st
2nd
3rd
4th
5th
Base
1st
2nd
3rd
4th
5th
Validation
86.50
87.92
88.74 74
88.16
87.74
86.78
86.50
87.52
88.64
88.62
88.16
87.56
CIFARIO
86.62
87.44
88.15
87.73
87.71
86.56
86.62
86.81
87.88
88.19
87.27
86.37
CIFARIO-C
69.04
68.65
69.44
67.91
67.30
65.20
69.04
68.50
69.08
69.25
66.33
63.80
OOD-FPR95
78.48
75.76
75.23
79.67
81.03
80.91
78.48
78.92
77.48
78.52
83.47
80.54
Table 5: CIFAR10-4k 10M TI: Breakdown of the individual iterations self self-training schemes in the CIFARI 10-4k setting with an unlabeled pool diluted with 10M samples from 80MTI. The final model chosen based on validation accuracy and marked in black.
21

---

Under review as a conference paper at ICLR 2022
ODST
ODST
Iteration
Base
1st
2nd
3rd
4th
5th
Base
1st
2nd
3rd
4th
5th
Validation
86.80
92.16
93.82
94.38
94.98
95.04
86.80
91.94
94.06
94.54
94.82
94.88
CIFARIO
86.30
91.43
93.47
93.95
94.36
94.31
86.30
91.61
93.68
94.11
94.00
94.31
CIFARIO-C
67.70
75.67
79.74
80.12
80.14
79.61
67.70
74.95
79.07
80.09
79.67
80.02
OOD-FPR95
43.98
27.98
23.42
22.83
24.55
23.26
43.98
31.49
26.17
25.16
23.03
25.61
NSST
NSST+
Iteration
Base
1st
2nd
3rd
4th
5th
Basc
1st
2nd
3rd
4th
5th
Validation
86.50
88.32
88.76
88.04
87.54
87.92
86.50
88.52
89.32
88.54
88.50
88.52
CIFARI0
86.62
87.62
88.30
87.75
86.91
86.66
86.62
87.75
87.82
87.85
87.44
87.69
CIFARIO-C
69.04
68.66
69.08
66.82
65.02
64.43
69.04
68.48
68.43
67.41
66.84
66.79
OOD-FPR95
78.48
77.61
77.47
78.73
81,58
80.43
78.48
76.47
75.73
78.22
80.36
78.95
Table 6: CIFARIO-4K + 1M LSUN: Breakdown of the individual iterations of self-training schemes
in the CIFARIO-4K setting with an unlabeled pool diluted with 1M samples from LSUN. The final model is chosen based on validation accuracy and marked in black.
22

---

Under review as a conference paper at ICLR 2022
Ued
trck-122 222
0.53
9.5
-.22
0.52
2.5
3.00
phare 3.55
bed
1.00
1,00
1.00
0:9
5.00
3,00
TOTA 0.75
DOO 690
3V2 L-2
plans 1.40
CAL 1.00
DOG -442
CA --00
dear- 0.00
bed 1.00
Trg 1.40
dog -1.0
DE-1200
EHO 1.00
422-0.99
300-550
ODST
ODST+
Noe-
Brack
Brody
Ferso
JUy
212-
Tay
DUCC
BAck 8.99
-.20
TOTT
3.50
0227
abb 1.00
021-023
1.00
tnok-110
bed 1000
-.00
3.00
NSST
NSST+
3,00
.EAT
0.55
G.2r-6.2
0.20
C.99
5:00
-.55
0.85
9
0.50
3.1
OE-O35
CH- 6.0
CO
CA CK--BAY
BTg -0.00
EIG 6.75
fog -077
493 0.99
Frag 0.97
1.00
plane 3555
TAYSO 3.00
Froy 1891
Chre- 50
ONTY
510-247
NW0
1.00
pkrne 022
FixMatch
MTCF
Figure 6: CIFAR10-4K + 1M TI: Visualisation of 42 randomly drawn samples considered to be task-relevant for various methods. For ODST and NSST those are all samples above the in- distribution threshold in the 5th training iteration and above both the in- and out-distribution thresh- old for ODST+ and NSST+ FixMatch uses a fixed confidence threshold (95%) and MTCF Otsu thresholding to determine which samples are from in- and ODST are the only methods where almost all selected samples are indeed task relevant and ODST+ achieves slightly higher precision due the additional OD-threshold. Fathh and MTCF are able to select task-relevant samples, however both show a high false-positive rate and make systematic errors, for example FixMatch and to some extent MTCF as well learns to label Humans as Dog.
Both variants of
23

---

Under review as a conference paper at ICLR 2022
TAOCE
1.00
1.00
FAOE -L.O
3.50
dag- 6-20
140-1000
Mb-000
Tog-150
hans-6a 6:20
JPp-150
2006-60 692
Tog 3.9
Bush 1
Paito
Yog
LOE
ODST
ODST+
3.00
fog
7.00
1:00
5tp-300
1,00
0.50
0055
0.90
943-191
VLOK-2.00
OIO 1-00
LED
LED-
der 100
DUt 1.00
2000-100
SR. 100
ats 1.00
atp 100
NSST
NSST+
1.-0100
150-3200
C3D-1.00
-100
Sog 5.00
Porte 5C
plaee -080
Frgg 0.2
STUC 6.00
CBT- B2-
1.00
1,00
SAE LED
SNG 222-
CUC- 2,00
GB--102
IIG 1.00
OUST- 0.22
1:90
PIETE 0.20
plane E:9
SVC- 3.55
CRT- OL22
FixMatch
MTCF
Figure 7: CIFAR10-4k + 1M LSUN: Similar to Figure we plot accepted samples for various methods. In this scenario, FixMatch, NSST and NSST+ do once again not show the desired be- haviour. On the other hand, ODST, ODST+ and MTCF show nearly perfect sample selection. While MTCF has great precision (99.70%), we note that suffers from poor recall (18.47%) and thus does not make use of majority of available in-distribution samples. ODST and ODST+ on the other hand have a comparable precision 09.26%.98.66 with a much higher recall (92.81%.92.50%).
24

---

Under review as a conference paper at ICLR 2022
Unaandes
Saypae aove
300
357
140
(a) ODST
(b) ODST+
Iecabdee
N0W
(c) FixMatch
(d) MTCF
Figure 8: CIFARIO-4K+l0 TI: Number samples per class from the unlabeled data selected as task-relevant for ODST, ODST+, FixMatch and MTCF with logarithmic y-axis scaling. The blue bar corresponds to the absolute number of added samples and the orange bar only shows the samples which come from the unlabeled train data (remember that 41k CIFARIO samples are in the unlabeled dataset). We note the very uneven selection of MTCF with almost no cars but a lot of frogs (of which most wrong, see Figure 66) while the selection of ODST+ and FixMatch more evenly distributed even though FixMatch selects far too many as task-relevant (428k out of 1041k possible). As 8OMTI contains some task relevant samples, a perfection selection will have the blue bar slightly higher than the orange one, i.e. will add the relevant samples out of the IM tiny images. However, the disparity between the two should not be too large (FixMatch) as this implies that unrelated rubbish images are added. Also, the orange bars should not be too low (MTCF) as this implies that lot of available in-distribution samples are not used.
condccenons
Selected conitdences (A31722 camplett
2006
5008
3X00
3506
2008
1000
Figure 9: CIFARI0-4K+1M TI Analysis of MTCF: We show the histogram of confidence values of the the ID-OD-classifer of MTCF above the Otsu-threshold and thus considered task-relevant by MTCF (right). Note that MTCF considers a lot of samples task-relevant for which the classifier is quite low-confident and discards the majority of samples for which the classifier is highly confident. This in strong contrast to ODST+ where only unlabeled samples are selected for which the classifier is highly confident (above the in-and out-distribution threshold).
of MTCF for all unlabeled samples (left) and of the ones which are for
25

---

Under review as a conference paper at ICLR 2022
C.2 CIFARIO-50K
In the CIFAR 10-50k setting, we use the full CIFARIO training set as labeled data and all 80 million tiny images unlabeled data. For CIFARIO, we the CIFARIO.1 set of 2k images validation set. The OOD-validation set for the Plus variants are 2k CIFAR 100 samples. In Table [7] we include CIFARIO-C accuracies on top of our standard evaluation metrics test set accuracy and OOD-FPR95 and additionally report which of the three self-training iterations is chosen. Note that our selection criterion is validation accuracy on CIFAR10.1.
On the smaller ResNet50 architecture, ODST and ODST+ can clearly improve performance over the Outlier Exposure (OE) base model in all metrics and also beat the two NSST baselines by a margin. Especially in terms of CIFARIO- accuracy. we are able to improve from 83.14% 14% achieved by the best labeled-data only model (OE) to 85.09% and 85.97% for ODST and ODST+ respectively. is noteworthy that both ODST variants are able to improve performance up until the last iteration and therefore make use of the maximum amount unlabeled data. This is only possible due to strict sample selection that prevents distribution shifts. NSST suffers from strong distribution shift in iteration and the resulting model is no longer suitable for passing knowledge to the student model, therefore validation accuracy decreases after the first iteration and we have select the first iteration model for NSST NSST+ + has the opposite problem and adds too few samples as it not trained distinguish in- from out-of-distribution samples but can improve up until the third iteration. However even with the additional threshold, NSST+ is not able to have good precision when comes to sample selection as can be seen Figure TO ODST and ODST+ are the only methods that are able to select mostly task-relevant samples. In this large scale task, ODST+ is also able to beat ODST due to even better sample selection, which is especially important when working with an unlabeled pool that contains 80 million samples.
With the larger PyramidNet architecture, models (plain and OE) that both already achieve over 98% test set accuracy. Both noisy student vari- ants fail this setting and are not able to improve performance in any metric and instead show quite significant drops in accuracy and OOD-detection performance over the plain base model. ODST+ able to improve test accuracy up to 98.57% in the last iteration (starting from 98.28% for the OE teacher model) while not only maintaining but improving FPR.
even harder increase performance over the base
Nes5e
PyramidNet NSST
plain
OF
NSST
NSST+
ODST
ODST
plain
OE
NSST+
ODST#
Acc. 1
96.11
96.81
96.86
96.93
97.98
98.07
98.49
98.28
98.24
98.13
98.57
C10-C
83.03
83.14
83.59
81.02
85.09
85.95
87.79
86.40
85.28
85.97
86.95
FPR 4
45.54
4.40
49.72
51.76
4.30
3.84
22.76
2.56
33.51
34.71
2.40
Iteration
I
3
3
Table 7: CIFAR10-50k: Additionally to the results from the main paper, we include robustness to common corruptions evaluated on CIFAR 10-C (C10-C). The last row "Iteration" refers to which of the 3 self self-training iterations is chosen based on validation set accuracy.
26

---

Under review as a conference paper at ICLR 2022
9e3-1
SAM
Siag 6.99
SPG 300
1.50
1.00
ar-L22
Ryd 1E
tv 140
NORS 1,00
pirk LED
JTS L40
Sed 100
2I-14
Dog 126
bis --
pirs-100
3FC-136
POPA 1.00
Stp- PP-100
dar 1.00
DUL 1.00
Soe 1.00
BOg 1.00
pare 10
JP 1.0
Nene 100
ted-2.00
Inso-1100
CAg C8C-166
Tog
PORE
DO
1.00
2,00
WV2-1
PACC 1.00
Bog
5og-1.08
VS--1.0
ODST
ODST+
D77
3.0
TOTE 2.0
6:7
1.00
11
1.20
0.5
CR 1,20
Sic 3355
CET-00 0,23
PACE D29
See 5.0
Dog E99
DUOCE 0.99
SAN 5.9
bN 8.99
dig 2.5
Faue 022
der 250
NSST
NSST+
Figure 10: CIFARIO-5Ok: We visualize the selected samples that the in-distribution thresh- old for ODST and NSST and above both the in- and out-distribution threshold for ODST+ in NSST+ in the third iteration of self-training. Note that both NSST variants suffer from strong distribution shift and start making systematic mistakes, like labeling humans as "dog". Both ODST variants on the other hand are able to select mostly task-relevant samples
27

---

Under review as a conference paper at ICLR 2022
C.3 CIFAR100-45k:
Similar to the CIFARIO-5Ok setting, we use the full CIFAR100 training set in this setting, however there is no validation set available we split it into 45k train and 5k validation images. We again use all 80 million tiny images as unlabeled data. The DOD-validation set for the Plus variants are 2k CIFARIO 10 samples with classes "car" and "truck" removed. In Table 8 we again report corruption robustness measured on CIFARI0O-C as well as test set accuracy and OOD-FPR95. Again we also give the chosen self -rrinirrinng iteration based on validation accuracy.
Clearly, ODST+ and ODST outperform all other methods on the smaller ResNet50 architecture. In detail we achieve up to 84.09% test accuracy and 64. 62%0 corruption robustness beating the next best performing methods by margin (82.44% and 61 61.45%). In terms of OOD-detection, our FPR values are able improve significantly over the base outlier exposure model and beat both NSST+ models by more than 30%. In Figure [1] we also show random selection of samples accepted by the various methods and demonstrate thatODST clearly outperforms the NSST baselines in terms of sample selection. The NSST models include many rubbish images that contain diagrams or computer generated graphics that are not even natural images, although the labeled distribution CIFAR 100 does not contain such synthetic images. ODST+ and ODST are not perfect, which is to be expected considering that we only see 450 labeled samples per class.
On the larger PyramidNet, ODST+ is the only method that is able increase test accuracy and achieves 88.66% whereas NSST and NSST+ decrease performance in terms of accuracy, corrup- tion robustness and OOD-detection. ODST+ is able to improve corruption robustness over it's OE base model but beaten by the plain baseline and slightly degrades in OOD detection performance. However the FPR increase from OE to ODST+ is a lot smaller than it is from plain NSST and NSST+
ResNet50
PyramidNet
plai
OE
NSST
NSST+
ODST
ODST
plain
OE
NSST
NSST+
ODST+
Acc. 1
80.69
79.98
82.44
82.03
83.54
84.09
88.07
87.60
87.71
87.53
88.66
CTOD-C 1
57.49
56.61
60.08
61.45
64.02
62.71
67.16
66.44
65.63
65.55
67.03
FPR 1
72.05
43.88
73.47
82,54
35.64
38.72
60.20
27.97
74.90
73,91
29.22
leration
2
Additionally to the results from the main paper, we include robustness to common corruptions evaluated on CIFAR 100-C (C100-C), The last row "Iteration" refers to which of the self-training iterations chosen based on validation set accuracy.
Table CIFAR100-45k:
C.4 SVHN
In addition to the previous CIFAR experiments, we present an additional experiment on the SVHN dataset. Following the CIFAR10-4k setting, we split the original SVHN train set into train/ /aiiiiaattttionuaaallldd splits We again use 4k labeled train samples, 5k validation samples and the remaining 64k samples as unlabeled data. To simulate open-world SSL setting, we add mil- lion images from 80MTI to the unlabeled set. As baseline methods, we use FixMatch, MixMatch and MTCF and compare it to our ODST. Hyerparameters are the same in the CIFAR 10-4k setting. To evaluate OOD-FPR, we use CIFARIO, CIFARION LSUN classrooms and and Flowers.
No SSL
SSL
Open World SSL
SVHN
plain
OE
MixMatch
FixMatch
MTCF
ODST
Acc. 1
94,89
94.18
93.64
96.31
96.60
96.94
FPR 1
41.64
12.20
42.31
93.89
0.1
1.1
While the task at hand is significantly easier, as distinguishing house numbers from real images is generally lot easier, ODST is still able to outperform the baseline methods. While FixMatch is able to narrow the gap between ODST and itself in this setting, we want to highlight the catastrophic OOD detection performance that results in FPR of 94.75.
28

---

Under review as a conference paper at ICLR 2022
1.02
3,00
TST L.cenccl 3.00
1.005HH+ 11.99
1000
1.0
1.0
batk: 1.00
3.00
HON 1.01
BERAY MO
DE Dep.DEL LOGESN 1/0
bents 1001
1ear-1.33
fekydene
PETpSrON Persiar 1,00 Baaand 1,00
road 3/40
TRAE 1.00
Torst -20
Tanat 1,00
Sant -12
kESk 1.00
24th Nw
DICEONCT 1.2 bet -122--2
boykk
AE EOHLE AUWW
FuTFAY 1.00 EreNKr-12 1.00
Ixx 1.00
Tak 1.00
OENE LB0
PEAIreaT LEO NOT LEE
mudremm 1.00
INTUTTWR 1.00 ol 1.00
plan 1.00
2.00
cin- Moscrpar
per-LLE
5aptb-1.00
STaN- L.C
-.0tte100cte Bar
1.0bargamo
ODST
ODST+
CrOed 1.00
DAD LEC.
butierty 1,00
DORDY 2.00
ITER 1.00
SIP 1.00
THE 1B0-1.20
OENE 0.22
cedk 1.0
pPe wre.tre-ee 1.00
plate 1.00
buscorte 1.03
boa 1,00
man: 1400
OREgI
spcer-d.
DAANST
1.00
FSESY 1.00
pllte
BusTower-
badi
BORTH
tue
bacte 1,00
TOURS 0:00
drvesar--..7
NSST
NSST+
Figure 11: CIEAR100-45K: We visualize the selected samples that are over the in-distribution threshold for ODST and NSST and above both the in- and out-distribution threshold for ODST+ AND NSST+ by the first iteration model (Top 50k).
D Ablation
In this section, we present various ablation studies to motivate our design choices. All ablation studies the ResNet50 architecture and are done CIFAR10-50k with the entire 80 million tiny images as unlabeled distribution.
D.I. OOD-validation set size
First, Table we demonstrate that relatively few OOD samples are required to achieve a good performance with ODST+. Both ODST+ variants slightly beat the ODST baseline and perform extremely similar even with a relatiely small OOD-validation set that only contains 2k images. For the rest of this section, always the full 50k samples
set.
29

---

Under review as a conference paper at ICLR 2022
D.2 Choice of peudo-labels U
While Xie et al. (202)) demonstrated that soft-labels on selected samples can improve performance. remains open question whether one should soft-labels for the remaining unlabeled samples in U \ I. In principle, one could keep enforcing uniform confidence on U \ I, that is
((2); K' 1........ K for & E U.L
like we did for training the baseline model. Thus instead of training the students with the loss presented in 3 one would minimize the loss:
Lyss Pyonn Σ [u(())
2EI
Σ [(1/K.8/0+n()) ZEUVI
UVI
The other alternative is that soft-labels on U
v(=) poto (z) for E U.
In Table TO we compare these alternative choices to the one of ODST+ (the mean of both) given in Equation which shows that our chosen pseudo-labels in ODST+ are the right compromise between these two extremes.
Using hard-labels on U decreases performance in comparison ODST+ There are two possible explanations for this. First, due to our strict thresholding, it is possible that U I contains task- relevant examples that are correctly classified but not accepted into I. In this case, the soft-label better target for the next student than strict uniform confidence. Second, even for unrelated images that contain certain features that correlate with specific class, soft-labels might be a more meaningful target. Soft-labels without label smoothing clearly outperform hard labels in terms of test accuracy but are slightly worse than ODST+.
D.3 Comparison with 500k-TI Ti
In this section, we compare our ODST+ scheme with the 500k-Ti dataset Carmon al. (019 500k-Ti was originally designed to help improve adversarial robustness on CIFARIO by leveraging data from 8OMTI. 500k-Ti generated by training an 11 class classifier and selecting the highest scoring 50k datapoints per-class from the entire 80MTT database. As training data for the 1 1th back- ground class, Carmon al, (2019) sample 1M images from 80MTI with keywords not appearing in CIFAR1O 10. Note that their training procedure therefore uses weak labels (the keywords), whereas do not use this information.
In the experiment, we train standard classifier by minimizing the cross entropy on the 50k la- beled train samples plus 500k-Ti which can thus be seen as the first iteration of a self-training procedure with their selected samples and compare it to ODST+. We use the same hyperpameters as for ODST+, in particular we train a ResNet50 with AutoAugment + Cutout. Table [T] shows that ODST+ clearly outperforms the self-training with 500k-Ti in terms of test accuracy (97.8) .9%% vs. 96.82%). corruption robustness (84.98% vs 82.65% and FPR95 for OOD detection 3.25% vs. 39.09%).
Method
ODST
ODST+ 2k
ODST+ 50k
Acc. t
97.98
98.07
98.12
FPR
4.30
3.84
3.89
Table 9: Ablation: Influence of the size of the nOD-validation set test accuracy and out-of- distribution detection performance. ODST does not use any additional OOD-dataset and for ODST+ use 2k and 50k additional OOD samples from CIFAR
30

---

Under review as a conference paper at ICLR 2022
Base
ODST+
Hard labels U
No label smoothing
CIFARIO
1st
3rd
2nd
3rd
1st
2nd
3rd
Test accuracy
96.81
97.85
97.99
98.
97.48
97.17
97.27
97.82
97.78
97.60
Validation accuracy
93.00
94.60
95.45
95.30
93.60
93.35
93.40
95.10
95.25
94.60
CIFARIO-C
83.14
85.81
84.98
84.54
84.61
85.51
86.04
85.38
84.85
84.00
NOD-FPR95
4.40
3.50
3.25
3.89
3,53
3.41
3.43
3.86
5.60
8.11
Table 10: Ablation: Comparison of different choices for the labels of non-selected images in U I. ODST+ uses soft-labels with label smoothing factor 0.5, hard labels enforces 1/ on all samples and label-smoothing is equivalent to ODST+ but without any smoothing. All models are trained using the same base model. We again highlight the model with best validation accuracy in black.
CIFARIO
ODST+
500k-Ti
Carmon et al, (2019)
Test accuracy
97.99
96.82
CIFARTO-C
84.98
82.65
OOD-FPR9S
3.25
39.09
Table 11: Ablation: We compare our ODST+ method model trained with the 500k-Ti dataset from Carmon et al. (2019). 500k-Ti was created by sampling 50k images per class from 80MTI with the goal of improving adversarial robustness. However, we show that ResNet50 classifier trained with the same parameters is not able to match the performance of ODST+ in terms of test accuracy, corruption robustness out-of-distribution detection.
31


Reviews:
[{"review_id": "tlLuUL0g20", "reviewer": "Reviewer_EKkN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on the open-set semi-supervised learning algorithm and proposes a new pseudo-label selection strategy to improve the robustness of SSL algorithms with open-set unlabeled examples.\n\nStrength:\n1) The proposal is simple and clear and the paper is well-written.\n\nWeakness:\n1) The key technique of the proposal is that the proposal adopts a new pseudo-label threshold to select pseudo labels and the threshold is calculated via in-distribution validation data. In reality, the in-distribution data set is difficult to obtain in a semi-supervised learning setting as we know SSL algorithms are designed to solve the problem with scarce labeled data. The novelty and contribution of the paper are limited.\n2) The authors analyze the proposal with bayesian decision theory. The results are mainly based on Eq.(4) and Eq.(5). However, the proposed algorithm does not optimize Eq.(4) and Eq.(5) directly, it seems that the theory can not demonstrate the effectiveness of the proposal.", "summary_of_the_paper": "This paper focuses on the open-set semi-supervised learning problem. The authors propose a modification of the classical self-learning algorithms. Compared with FixMatch, which selects pseudo-label based on a static confidence threshold. This paper proposes to select a threshold based on in-distribution validation data.", "summary_of_the_review": "This paper focuses on the open-set semi-supervised learning algorithm and proposes a new pseudo-label selection strategy to improve the robustness of SSL algorithms with open-set unlabeled examples. However, the proposed algorithms are mainly a small modification of current pseudo-label selection-based algorithms, the contribution and novelty of the proposal are limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TR8HEpXZ8Pl", "reviewer": "Reviewer_b8X7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the goal of this paper. Most SSL and self-training methods are only evaluated on a contrived dataset where the unlabelled data is actually just labelled data whose labels are removed. For example, a common setup for SSL is to use CIFAR-10, where some images are considered labelled data while rest of the data is considered unlabelled data. The obvious issue here is that the \"unlabelled\" data in this setup is actually very similar to the labelled data, even with the same distribution of classes etc. So overall I like the goal of this paper. \n\nThe method is not very novel. As far as I can tell, the method is similar to noisy student (which itself wasn't too novel or different compared to previous pseudolabelling/self-training methods), with some modifications of calibration, ranking and filtering of confidences, and then relabelling of unlabelled samples with lower confidence pseudolabels. It's worth noting that the original noisy student paper had additional experiments in appendix on filtering psuedolabelled samples by a confidence threshold, where they considered pseudolabels that are hard or soft. \n\nIt is of course perfectly fine for a method to not be too novel, as long as it leads to important improvements in empirical results or scientific understanding. I think that the presentation of ODST could be significantly more convincing if it was evaluated on a more challenging benchmark such as imagenet, where noisy student had shown a significant improvement. That said, I do think the results on cifar-4k where ODST outperforms fixmatch is impressive. I'm curious if authors could reference any results of fixmatch applied to task-unrelated unlabelled data from other published papers. Currently as a reader I'm unsure how much of the improvement over fixmatch could be due to fixmatch not being tuned as well as ODST is, since fixmatch has lots of different hyperparameters that could need retuning when unlabelled data is from a task-unrelated source. The authors could also add more information about how they tuned fixmatch and other baseline methods for this harder task where unlabelled samples come from a task-unrelated dataset.\n\nMinor: I see the phrase \"resp.\" on several pages. I wasn't sure if that's a latex bug or not.        ", "summary_of_the_paper": "This paper proposes a modified way of doing self-training that aims to improve the performance when the unlabelled data is not coming from task-related labelled data with artificially removed labels. The authors validate their approach on reduced CIFAR-10, full CIFAR-10, and full CIFAR-100.  ", "summary_of_the_review": "Since the methodological novelty is limited, the paper could benefit from more comprehensive empirical experiments. Since ODST is quite similar to Noisy-Student, it could be helpful to see experiments on Imagenet. If that is not possible, experiments on more datasets such as STL-10 or SVHN could help, as currently the only experimental evidence for ODST is on CIFAR-10/100. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JX5G5vidv2b", "reviewer": "Reviewer_3Ko3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Semi-supervised learning (SSL) has shown great potential to leverage large amounts of unlabeled data to improve model performance on downstream tasks. However, most of the recent works are hinged on a key assumption that the unlabeled data comes from the same distribution as the underlying task or from the same classes. While some recent works address this by considering the OOD setting, they assume the amount of non-task specific data to be relatively less in the mix. The authors focus on SSL for image classification and consider a more realistic setting where the ratio of non-related images in the unlabeled dataset is large.\n\nTo this end, the authors propose a self-training scheme with a careful sample selection strategy. The idea of self-training with unlabeled data has been used in several recent works including Unsupervised Data Augmentation (UDA, NeurIPS 2020) and Noisy Self-training Student (NSST, CVPR 2020) which the authors adopt as the base framework. The authors perform top-K class-based sample selection which has been shown to work well in self-training settings in prior work like Uncertainty-aware Self-training (UST, NeurIPS 2020). They further perform calibration and adapt the self-training loss function to combine (pseduo)-labeled losses from 3 different segments of the data. \n\nThe novel component in this method which sets it apart from prior self-training works with sample-selection (including NSST/UDA which uses confidence-based selection and UST which uses uncertainty-based selection) is the use of a weak form of knowledge distillation using pseudo-labels and the uniform distribution. The authors provide theoretical justification for this choice followed by ablation studies in the Appendix to evaluate of such regularization and other components of the framework on the overall model performance. \n\nFor the experiments, the authors show impressive improvements over several baselines in both standard SSL and open-world SSL settings.  Comparing NSST+ and ODST+ which is the major component that bring the most benefit for the open-world setting?\n\nOverall, the paper is well-written and easy-to-follow with significant improvements over several baselines in the open-world SSL setting.\n\n", "summary_of_the_paper": "The authors propose an out-of-distribution-aware self-learning (ODST) framework with a careful sample selection strategy that works well with unlabeled datasets containing a small portion of task-relevant data.", "summary_of_the_review": "The authors propose an out-of-distribution aware self-training method with significant improvements over several baselines in the open-world SSL setting with extensive experiments and ablation studies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FWazKsZMI0d", "reviewer": "Reviewer_xS21", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengh: The paper considers a novel mixture model for unlabeled data in an open world scenario. Each unlabeled sample is allocated with a lable with 50% possibility of an outlier, and 50% possibility as an inliner with the psuedo label assigned by a teacher model.\n\nW1: Presentation. I am afraid the presentation quality needs improvement. There are many grammar issues in the paper. Take for example, “A standard setting in SSL is to use 4k training images of CIFAR10 together with 5k as validation set”, is not correct in grammar. \n\nW2: Motivation, I doubt that the pseudo labeling strategy is in fact too noisy, owing to the misalignment of classes between in data and out data. The direct consequence is that the pseudo labeling on out of distribution classes (classes not from the same distribution as from the labeled data) might be seriously wrong. The model would then suffer from the artifact of high confidence (actually, softmax is not capable of measuring any confidence anyway) even if the confidence is already thresholded. If we further take into account the domain gap between in and out data, such plain form of pseudo labeling remains skeptical. The method therefore is sensitive to the threshold selection procedure, subject to the actual domain gap and class difference between in and out data. \n\nW3: Empirical evidence. Due to the lack of presentation clarity, I am curious how the OE model is implemented under a “labeled only” setup as in Table 1, as OE explicitly encourages uncertainty on unlabeled data. Why is OE not compared with the ODST under the same Open World SSL setup? I think OE itself is suitable for such comparisons scenarios. \n\nW4: Theoretical support. The supporting theorem does not link to the propagation error introduced by such pseudo labeling strategy. Since classes between labeled dataset and unlabeled datasets are significantly different, softmax function is prone to prediction artifact. I think what theoretically matters is how much is the error bound constrained in this case given the assumed domain gap and class discrepancy/overlapping between labeled and unlabeled data?\n\nW5: Ablations. There is no ablation study against various components in the proposed ODST method. I insist that the ablation is critical in this paper since the proposed method now is packaged with calibration method, pseudo labeling strategy, student teacher architecture, and the proposed loss Eq. (2). It is currently unclear which factor eventually contributes to the eventual performance improvement. \n", "summary_of_the_paper": "The paper assumes an open world setting for semi-supervised learning approaches. In comparison to OE method, the proposed ODST method further dampness the usage of pseudo labels. Specifically, this work further assumes that unlabeled data with lower confidence do come from a mixture model having two components with equal probabilities: one fully uncertain model with label $\\frac{1}{K}$ evenly across all classes; and one model with pseudo label determined by the teacher model. \n\n", "summary_of_the_review": "Given the above considerations, I cannot recommend acceptance at this time, unless the above issues are addressed during rebuttal. I would increase my scores if the above concerns are clarified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4X-eAcU6z8z", "reviewer": "Reviewer_kxNQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## strengths\n1. This paper studies an important problem which is underexplored in literature;\n3. This paper provides analyses on the base classifier based on Bayesian decision theory.\n4. The proposed method shows significant performance improvement against several baselines on multiple datasets.\n\n\n## weaknesses\n1. The novelty is limited. The proposed method composes of several parts, i.e., self-training, OE, classifier calibration, confidence-based sample selection, and loss functions are pretty much the same as previous works. which makes the method a combination of existing techniques;\n2. The proposed method is complex yet not well understood. More ablation studies are needed to better understand the effectiveness of each component. For example, the last term of the loss function in Eq. (3); the choice of $k$ in sample selection; the classifier calibration; the size of in-distribution validation set;\n3. The proposed method is inefficient. In particular, it needs 1000 epochs of training. Additionally, in each epoch, it involves hyper parameter selection on validation set, which adds more computational burden.", "summary_of_the_paper": "This papers studies semi-supervised learning with OOD samples appearing in unlabeled data as well as at test time, which is practical in real-world tasks. To handle OOD samples, this paper proposes a new framework based on OE and self-trainning. Extensive experiments are conducted to verify the superiority of the proposed method.", "summary_of_the_review": "As listed in **Main Review**, the novelty seems limited and the proposed method is complex yet not well explained. Therefore, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for self-training in an open-world setting where a significant portion of unlabeled data might include examples that are not task related. The proposed method (ODST) uses a more accurate OOD detection technique which allows an improved sample selection leading to higher accuracy.

Strong Points:
- This paper studies a very important and impactful problem.
- The paper is well-written.
- The empirical results show that the proposed method improves over prior work.
- To better understand the iterative scheme, authors provide theoretical analysis using Bayesian decision theory.

Weak Points:
- Novelty: Given prior work on different variants of noisy students, this work has limited novelty.
- Dataset diversity: The main results are provided for CIFAR-10 and CIFAR-100 datasets which are very similar to each other. During the discussion period, authors added results on SVHN datasets but the accuracy gap between the proposed method and FixMatch is insignificant (FPR gap is higher but since the main goal is improving performance, I think showing accuracy is a more important measure here).
- Connecting theoretical results to the rest of the paper: The paper can be improved significantly if the theoretical results are more connected to the rest of the paper and in particular with the proposed algorithm.

While 4 out of 5 reviewers are recommending rejection, I think this was a very close decision. Most reviewers were concerned with novelty which I think is a valid point. Given that and the fact that the theoretical results are very limited, showing strong empirical results are required to accept this paper. Even though the provided results on CIFAR datasets are strong, the result on SVHN does not show a significant improvement. I understand that running experiments on ImageNet might not be budget-friendly. However, it is possible to run similar experiments or other datasets to show the robustness of the proposed method to the choice of the dataset. Consequently, I recommend rejecting the paper and propose authors to resubmit after adding more datasets as part of their evaluation.
Decision: Reject