OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
Authors: Jongjin Park
Sukmin Yun
Jongheon Jeong
Jinwoo Shin
Venue:  (2021)
Keywords: semi-supervised learning
realistic semi-supervised learning
class-distribution mismatch
unsupervised learning

Abstract: Modern semi-supervised learning methods conventionally assume both labeled and unlabeled data have the same class distribution. However, unlabeled data may include out-of-class samples in practice; those that cannot have one-hot encoded labels from a closed-set of classes in label data, i.e., unlabeled data is an open-set. In this paper, we introduce OpenCoS, a method for handling this realistic semi-supervised learning scenario based on a recent framework of contrastive learning. One of our key findings is that out-of-class samples in the unlabeled dataset can be identified effectively via (unsupervised) contrastive learning. OpenCoS utilizes this information to overcome the failure modes in the existing state-of-the-art semi-supervised methods, e.g., ReMixMatch or FixMatch. In particular, we propose to assign soft-labels for out-of-class samples using the representation learned from contrastive learning. Our extensive experimental results show the effectiveness of OpenCoS, fixing the state-of-the-art semi-supervised methods to be suitable for diverse scenarios involving open-set unlabeled data.

Under review as a conference paper at ICLR 2021
OPENCOS: CONTRASTIVE SEMI-SUPERVISED LEARN- ING FOR HANDLING OPEN-SET UNLABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Modern semi-supervised learning methods conventionally assume both labeled and unlabeled data have the same class distribution. However, unlabeled data may include out-of-class samples in practice; those that cannot have one-hot encoded labels from a closed-set of classes in label data, i.e., unlabeled data is an open-set. In this paper, we introduce OpenCoS, method for handling this realistic semi- supervised learning scenario based on recent framework of contrastive learning. One of our key findings is that out-of-elass samples in the unlabeled dataset can be identified effectively via (unsupervised) contrastive learning. OpenCoS utilizes this information to overcome the failure modes in the existing state-of-the-ar semi- propose
supervised methods, e.g., ReMixxatch or FixMatch. In particular, assign soft-labels for out-of-class samples using the representation learned from contrastive learning. Our extensive experimental results show the effectiveness of OpenCoS, fixing the state-of of-te-----heerrr the-art semi- suni--upervised methods to be suitable for diverse scenarios involving open-set unlabeled data. The code will be released.
INTRODUCTION
Despite the recent success of deep neural networks with large-scale labeled data, many real-world scenarios suffer from expensive data acquisition and labeling costs. This has motivated the community to develop semi-supervised learning (SSL: Grandvalet & Bengio 2004; Chapelle et al. 2009), i.e., by further incorporating unlabeled data for training. Indeed, recent SSL works (Berthelot al., 2019; 2020; Sohn et al., 2020) demonstrate promising results on several benchmark datasets, as they could even approach the performance of fully supervised learning using only small number labels, e.g., 93.73% accuracy on CIFAR- 10 with 250 labeled data (Berthelot et al., 2020).
However, SSL methods often fail to generalize when there is mismatch between the class- distributions of labeled and unlabeled data (Oliver et al., 2018; Chen et al., 2020c; Guo et al., 2020), ie., when the unlabeled data contains out-of-class samples, whose ground-truth labels are not contained in the labeled dataset (as illustrated in Figure (a)). In this scenario, various label-guessing techniques used in the existing SSL methods may label those out-of-class samples incorrectly, which in turn significantly harms the overall training through their inner-process of entropy minimiza- tion (Grandvalet & Bengio, 2004: Lee, 2013) or consistency regularization (Xie et al., 2019; Sohn al., 2020). This problem may largely hinder the existing SSL methods from being used in practice. considering the open-set nature of unlabeled data collected in the wild (Bendale & Boult, 2016).
Contribution. In this paper, we focus on a realistic SSL scenario, where unlabeled data may contain some unknown out-of-ca oflcass samples, i.e., there is a class distribution mismatch between labeled and unlabeled data (Oliver et al., 2018). Compared to prior approaches that have bypassed this problem by simply filtering out them with some heuristic detection scores (Nair et al., 2019; Chen et al., 2020c), further leverage the information in out-of-class samples
the unique characteristic in our approach by assigning soft-labels to them: they may still contain some useful features for the in-classes.
Somewhat surprisingly. we found that a recent technique of contrastive unsupervised learning (Wu al., 2018; 8; He et al., 2020; Chen al., 2020a) can play key role for our goal. More specifically, we show that a pre-trained representation via contrastive learning, namely SimCLR (Chen et al., 2020a), on both labeled and unlabeled data enables us to design (a) an effective score for detecting oft-of-class samples in unlabeled data, and (b) a systematic way to assign soft-labell to the detected oft-of-class samples, by modeling class-conin consiconni likelihoods from labeled data. Finally, we found (c) auxiliary

---

Under review as a conference paper at ICLR 2021
Unlabeled data
ReMixbaach
ReMiiMatch-i
In-class "at or dog?
OpenCoS (ours)
90
"Cat"
"Dog"
Accuracy
80
70
Out-of-class dog?
60
20
40
60
80
Proportion of out-of-class data (%)
(a) Class-distribution mismatch
(b) Comparison of median test accuracy
Figure I: (a) Illustration of an open-set unlabeled data under cass-distribution mismatch in semi- supervised learning, i.e., unlabeled data may contain unknown out of-ssss samples. (b) Comparison of median test accuracy under varying proportions of out-of-class samples on the CIFAR-10 + TinyImageNet benchmark with 25 labels per class.
batch normalization layers (Xie et al., 2020) could further help to mitigate the dlassddistribution mismatch via decoupling batch normalization layers. We propose generic SSL framework, coined OpenCoS, based on the aforementioned techniques for handling open-set unlabeled data, which can be integrated with any existing SSL methods.
We verify the effectiivness of the proposed method on a wide range SSL benchmarks based on CIFAR-10, CIFAR- 100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) datasets, assuming datasets. the presence of various out-of-class data, e.g., SVHN (Netzer et al., 2) and sate--of-the-ar SSL Our experimental results demonstrate that OpenCoS greatly improves existing methods (Berthelot et al., 2019; 2020; Sohn et al., 2020), not only by discarding out-of-class samples, but also by further leveraging them into training. We also compare our method to other recent works (Nair et al., 2019; Chen et al., 2020c; Guo et al., 2020) addressing the same class distribution mismatch problem SSL, and again confirms the effectiveness of our framework, e.g., we achieve an accuracy of 68.37% with 40 labels (just labels per class) on CIFAR- 10 with TinylmageNet as out-of-class, compared to L (Guo et al., 2020) 56.32%.
Overall, our work highlights the benefit of unsupervised representations in (semi-) supervised learning: such label-free representation turns out to enhance model generalization due to its robustness on the novel, out-of-class samples.
PRELIMINARIES
2.1
SEMI-SUPERVISSH LEARNING
The goal of semi- superviee learning for classification is to train classifier f X + ) from labeled dataset D; {=(") M/ (6) where each label yı is from set of classes == {1,-.. ,C}.
and unlabeled dataset D. [x J==1 where each exists but is assumed to be unknown. In an attempt leverage the extra information in Du, a number of techniques have been proposed, e-g., entropy minimization (Grandvalet & Bengio, 2004; Lee, 2013) and consistency regularization (Sajjadi al., 2016). In general, recent approaches in semi-supervised learning can be distinguished by the prior they adopt for the representation of unlabeled data: for example, the consistency regularization technique (Sajjadi et al., 2016) attempt to minimize the cross-entropy loss between any two predictions of different augmentations (Ir.) and la (a(..) from given unlabeled sample vu. jointly with the standard training for a labeled sample (my, 3:):
(1)
ESsi x..) H(m f (xx)) B (.())))BBH((((tttt (()((t( (t2(2x))),
where H. is a standard cross-entropy loss for labeled data, and 3 is hyperparameter. Recently, several "holistic" approaches of various techniques (Zhang et al., 2018; Cubuk et al., 2019) have shown remarkable performance in practice, e.g., MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020), and FixMatch (Sohn et al., 2020), which we mainly consider in this paper. We note that scheme can be integrated with any recent semi-supervised learning methods.
2

---

Under review as a conference paper at ICLR 2021
Contrasivv representatiin
Detocting our af-clsss
In dloss
Semi-superv sed method e.g. ReMlicMath
"Cat"
"Dog""
Dut-f-ciass
Auxllary
Auxilary loss e.q. cross-enropy
Figure 2: Overview of our proposed framework, OpenCoS. First, our method detects out-of-class samples based on contrastive representation. The out-of-class samples detected by OpenCoS are further utilized via an auxiliary loss with soft-labels generated from the representation, while the remaining in-class samples are used for standard semi-supervised methods. Also, the out-of-class samples pass through additional batch normalization layers to handle a class-distrbuution mismatch.
2.2 CONTRASTIVE REPRESENTATION LEARNING
Contrastive learning (Oord et al., 2018; Hénaff et al., 2019; He et al., 2020; Chen et al., 2020a) defines unsupervised task for encoder fe X' - Rd from of samples {æ;]: assume that "query" sample Øg is given and there is positive "key" x+ E { that F. matches. Then the contrastive loss is defined to let f to extract the necessary information identify x. from By as follows:
exp(h(fe(xx) fc(x)))/T)
_ (fes Fgg Z4: (x+1) == log Σ. exp(h(fe(xg) fe (zi))/T)²
(2)
where .).,,) is pre-defined similarity score, and is temperature hyperparameter. In this paper, we primarily focus on SimCLR (Chen et al., 2020a), particular form of contrastive learning: for a given {z, J SimCLR first samples two separate data augmentation operations from pre-defined family T, namely f1, 12 ~ T, and matches (Ø,, @++N) = (t1 (2())) as a query-key pair interchangeably. The actual loss is then defined follows:
2
SSimLLR (fe: (x.)
Σ
(om Tq. ((++N) mod 2N: ( \ (ā₂)).
(3)
2N
g(01) g
SSiCCLR (U1, U2) == CosineSimillrityy(()))....
(4)
[lg(v) )lllll(())))
where g Rdd + Rd a 2-layer neural network called projection header. other words, the SimCLR loss defines task to identify a "semantically equivalent" sample to Fgg up the set of data augmentations T.
OPENCOS: A FRAMEWORK FOR OPEN-SET SEMI-SUPERVISED LEARNING
We consider semi semi-supervised classification problems involving C' classes. In addition to the standard assumption of cemi-supervised learning (SSL), we assume that the unlabeled dataset D., is open-set, ie.,, the hidden labels Yn, of I,, may not be in y = (1. (1..... C}. In this scenario, existing semi- supervised learning techniques may degrade the classification performance, possibly due to incorrect label-guessing procedure for those out-of-class samples. In this respect, we introduce OpenCoS, learning.
generic method for detecting and labeling oft-of-class unlabeled samples in semi-supervise Overall, key intuition to utilize the unsupervised representation from contrastive learning (Wu et al., 2018; He et al., 2020; Chen et al., 2020a) to leverage such out-of-class samples in an appropriate manner. We present brief overview of our method in Section 3.1, and describe how approach, OpenCoS, can handle out-of-class samples in Section 3.2 and 3.3.
3.1 OVERVIEW OF OPENCOS
Recall that goal is to train classifier f X - y from a labeled dataset Dj and open-set unlabeled dataset D,,- Overall, OpenCoS aims to overcome the presence out-off samples in D,, through the following procedure:
3

---

Under review as a conference paper at ICLR 2021
1. Pre-training via contrastive learning. OpenCoS first learns unsupervised representa- tion of f via SimCLR' (Chen et al., 2020a), using both Dj and Du without labels. More specifically, we learn the penultimate features of f. denoted by fer by minimizing the contrastive loss defined in (3). We also introduce a projection header g (4), which is a 2-layer MLP as per (Chen et al., 2020a).
2. Detecting out-of-class samples. From learned representation of fe and g, OpenCoS identifies an out-of-class unlabeled data D from the given data D,, Dig U D This detection process is based on the similarity score between Dj and D., in the representation space of fe and (see Section 3.2).
3. Semi-supervised learning with auxiliary loss and batch normalization. Now, one can use any semi-supervised learning scheme to train f using Dr and D e.g., ReMixMatch (Berthelot al., 2020). In addition, OpenCoS minimizes an auxiliary loss that assigns soft-label to each sample in Dite, which is also based on the representation of fe and (see Section 3.3). Furthermore, we found maintaining auxiliary batch normalization layers (Xie et al., 2020) for Dou is beneficial to our loss as they mitigate the distribution mismatch arisen from D
Putting it all together, OpenCoS provides effective and systematic way detect and utilize out- of-class data for semi-supervised learning. Due to its simplicity, our framework can incorporate the most recently proposed semi-supervised learning methods (Berthelot al., 2019; 2020; Sohn et al., 2020) and improve their performance in the presence of out-of-class samples. Figure 2 illustrates the overall training scheme of OpenCoS.
3.2 DETECTION CRITERION OF OPENCOS
For given labeled dataset D; and an open-set unlabeled dataset Du, we aim to detect a subset of the unlabeled training data D C Dr, whose elements are out-of-class, i.e., y/u $ V. A standard way to handle this task is to train confident-calibrated classifier using Dr (Hendrycks & Gimpel, 2017; Liang et al., 2018: Lee et al., 201 8a;b; Hendrycks et al., 201 9a;;; Bergman & Hoshen, 2020; Tack et al., 2020). However, such methods typically assume sufficient number of in-class samples (i.e., large D:), which does not hold in our case to the label-scarce nature of SSL. This motivates us to consider
suitable approach which leverages the open-set unlabeled dataset D, for contrastive learning. Then, OpenCoS utilizes the labeled dataset Dj to estimate the class-wise distributions of (pre-trained) embeddings, and use them define a detection score for D...
We assume that an encoder fe X' + Rdd and a projection header IRd - Rd pre-trained via SimCLR Dj UDD. Motivated by the similarity metric used in the pre-training objective of SimCLR (4), we propose simple yet effective detection score s(x,) for unlabeled input 2,, based on the cosine similarity between Fa and class-wise prototypical representations {v=)a= obtained from Dj. Namely, we first define a celas-wise similarity score? sima( for each class as follows:
1
Vo (e.)e. No
((f((z))) and o
(5)
sim (Zri Dt, g) == CosineSimilarity(d
va),
(6)
where Nf = ((x) Vi 9 yuا" = e}1 the sample size of class e Dj. Then, our detection score s(x.) is defined by the maximal similarity score between Tu and the prototypes {ve)cm
s(x.)) = max sim, (xw).
c=1,--
In practice, we use a pre-defined threshold / for detecting out-of-class samples in Du, i.e., we detect a given sample Fu as out-of-class if s(x,) < t. In our experiments, we found an empirical value of t := μ4 - 20; performs well across all the datasets tested, where ii and are mean and standard deviation computed over {s(x()) ). J respectively, although more tuning of could further improve the performance. Further analysis of our detection threshold can be found in Appendix B.4.
Nevertheless, our framework not restricted a single method of SimCIR: RI: it is easily generalizable other contrastive learning methods (Hénaff et al., 2019; He et al., 2020; Chen et al., 2020h).
In this work, we adopt the well-know cosine similarity to define our score, but there can be other designs long represents class-wise similarity (Chen al., 2020d; Vinyals et al., 2016; Snell al., 2017).
4

---

Under review as a conference paper at ICLR 2021
3.3 AUXILIARY LOSS AND BATCH NORMALIZATION OF OPENCOS
Based on the detection criterion defined in Section 3.2, the open-set unlabeled dataset D,, can be split into (a) the in-class unlabeled dataset Dir and (b) the out-of-class unlabeled dataset De The labeled dataset Dj and Di are now used to train the classifier f using any existing semi-supervised learning method (Berthelot et al., 2019; 2020; Sohn et al., 2020).
aa E Demi More specifically, for any semi-supervised learning objective Lssi aw f), we to In addition, we propose to further utilize D."". via an auxiliary loss that assigns soft-label to each
consider the following loss:
LopenCos = ESSL((r, s:: f) + A (
where H denotes the cross-entropy loss, À is hyperparameter, and g defines specific as- signment of distribution over y for a. In this paper, we propose to assign ((=..t) based on the class-wise similarity scores simo(x defined in (6), again utilizing the contrastive representation fe and g:
(x(x))) == cxp (sim fe: ()))) Σ, exp (sim, (Zui g)/T)
where 7 is a (temperature) hyperparameter.
At first glance, assigning label of Y to uut may seem counter-intuitivy as the true label of x: is not in y by definition. However, even when out-of-class samples cannot be represented as one-hot labels, one can still model their class-conditional likelihoods as linear combination (i.e., soft-label) of y: for instance, although "cat" images are out-of-class for CIFAR-100, still there are some classes in CIFAR-100 that is semantically similar to "cat", e.g.. "Icopard". "lion", "tiger", so that assigning a soft-label, e.g., 0.1 "Ieoparr" + 0.2 "lion" + 0.7 "tiger", might be beneficial. Even if out-of-classes are totally different from in-classes, one can assign the uniform labels to ignore them. We empirically found that such soft-labels based on representations learned via contrastive learning offer an effective way to utilize out-of-class samples, while they are known to significantly harm in the vanilla semi-supervised learning schemes. We present detailed discussion on our soft-label assignments in Section 4.4.
Auxiliary batch normalization. Finally, suggest to handle data dattaddittrbbution shift originated from the class-distribution mismatch (Oliver et al., 2018), i.e., Dr and D are drawn from the different underlying distribution. This may degrade the in-class classification performance as the auxiliary loss utilizes out-of-class samples. To handle the issue, we use additional batch normalization layers (BN; Sergey Ioffe 2015) for training samples in D disentangle those two distributions. In our experiments, we observe such auxiliary BNs are beneficial when using out-of-class samples via the auxiliary loss (see Section 4.4). Auxiliary BNs also have been studied in adversarial learning literature (Xie et al., 2020): decoupling BNs improves the performance of adversarial training by handling a distribution mismatch between clean and adversarial samples. In this paper, we found that a similar strategy can improve model performance in realistic semi- emn-suupee learning.
4 EXPERIMENTS
In this section, we verify the effectiveness of our method over wide range of semi-supervised learning benchmarks in the presence of various out-of-class data. The full details on experimental setups can be found in Appendix A.
Datasets. We perform experiments on image classification tasks for several benchmarks in the literature of semi-supervised learning (Berthelot et al., 2020; Sohn al., 2020): CIFAR-10. CIFAR- 100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) datasets. Specifically, we focus on settings where each dataset is extremely label-scarce: only 4 or 25 labels per class are given during training, while the rest of the training data are assumed to be unlabeled. To configure realistic comi-supervised learning scenarios, we additionally assume that unlabeled data contain samples from an external dataset: for example, in the case of CIFAR-10, we use unlabeled samples from SVHN (Netzer et al., 2011) or TinylmageNet datasets.
Baselines. We evaluate MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020). and FixMatch (Sohn et al., 2020) as baselines in our experimental setup, which are considered to be
"https://tin http://tiiny-mmmmeee.... hhwprehega...aaeeeeeeeeeeoaa..p. www.hhekkkkocooooo
5

---

Under review as a conference paper at ICLR 2021
state-of-the methods in conventional semi semi-supervised learning. We also compare our method with three prior works applicable to our setting: namely, we consider Uncertainty-Awar (UASD; Chen et al. 2020c), RealMix (Nair et al., 2019) and DS (Guo et al., 2020), which propose schemes to detect and filter out out-of-class samples in the unlabeled dataset: e.g., DS learns to re-weight unlabeled samples reduce the effect of out-of-class samples. Recall that our method uses SimCLR (Chen et al., 2020a) for pre-training. Unless otherwise noted, we also pre-train the baselines via SimCLR for fair comparison, denoting those fine-tuned models by "-ft," e.g., MixMatch-ff ft and UASD-ft. We confirm that fine-tuned models show comparable or better performance compared to those trained from scratch, as presented in Figure ((b) and Appendix A.2. Also, we report the performance purely obtainable from (unsupervised) SimCLR: namely, we additionally consider (a) SimCIR-lle a SimCLR model with linear evaluation protocol (Zhang et al., 2016; Chen et al., 2020a), i..,, it additionally learns linear layer with the labeled dataset, and (b) SimCIR--ft: the whole SimCLR model is fine-tuned with the labeled dataset. Somewhat interestingly, these models turn out to be the strongest baselines in our setups; they often outperform the ofate-of-the-ar baselines under large proportions of out-of-class samples (see Table 1). Finally, we remark that our framework can incorporate any conventional semi-supervised methods for training. We denote method built upon an existing method by "+ OpenCoS". e.g., ReMixMatch + OpenCoS.
ceff-Distillation
Training details. As suggested by Oliver et al. (2018), we have re-implemented all baseline methods considered, including SimCLR, under the same codebase and performed experiments with the same model architecture of ResNet-50 (He et al., 2016). 4 Due to the label-scarce nature of semi-supervised learning, we do not use a validation set in our setting. Instead, we checkpoint per 2 training samples and report (a) the median test accuracy of the last 5 checkpoints out of 50 checkpoints in total and (b) the best accuracy among all the checkpoints. We fix 7 1, the temperature hyperparameter in (9), and À = 0.5 in (8), in all our experiments. The full details on model architecture and hyperparameters can be found in Appendix A.2 and I, respectively.
4.1 EXPERIMENTS ON VARYING PROPORTIONS OF OUT-OF-CLASS SAMPLES
We first evaluate the effect of out-of-class unlabeled samples in semi-supervised learning, varying proportions to the total dataset. We consider CIFAR-10 10 and TinylmageNet datasets, and synthetically control the proportion between the two in 50K training samples. For example, 80% of proportion means the training dataset consists of 40K samples from TinyImageNet, and 10K samples from CIFAR-10. 10. In this experiment, we assume that 25 labels per class always given in the CIFAR- 10 side. We compare three models on varying proportions of oft-of-class: (a) a ReMixMatch model trained from scratch (ReMixMatch). (b) a SimCLR model fine-tuned by ReMixMatch (RthixMatth--f), and (c) our OpenCoS model applied to ReMixMatch-ft (+ OpenCoS).
Figure 1(b) demonstrates the results. Overall, observe that the performance of ReMixMatch rapidly degrades as the proportion of out-of-class samples increases in unlabeled data. While ReMixMatch-ft significantly mitigates this problem, however, it still fails at larger proportion: e-g., at 80% of out-of-class, the performance of ReMixMatch-fi falls into that of ReMixMatch. OpenCoS, in contrast, successfully prevents the performance degradation of ReMixMatch-fi. especially the regime that out-of-class samples dominate in-class samples.
4.2 EXPERIMENTS ON CIFAR DATASETS
In this section, we evaluate our method on several benchmarks where CIFAR datasets are assumed to be in-class: more specifically, we consider scenarios that either CIFAR-10 10 or CIFAR- 100 an in-class dataset, with an out-of-class dataset of either SVHN or Tiny TinylmageNet. Additionally, we also consider a separate benchmark called CIFAR-Animals + CIIAR- following the setup in the related work (Oliver et al., 2018): the in-class dataset consists of 6 animal classes from CIFAR- 10, while the remaining samples are considered as out-of-class. We fix every benchmark to have 50K training samples. We assume an 80% proportion of out-of-class, i.e., 10K for in-class and 40K for out-of-class samples, except for CIFAR-Animals + EIFAR-Othert which consists of 30K and 20K samples for in- and out-of-class, respectively. We report ReMMixMath--ft + OpenCoS as it tends to outperform FixMatch-ft + OpenCoS in such CIFAR- experiments, while FixMatch-ft + OpenCoS does in the large-scale ImageNet experiments in Section 4.3. Table 1 shows the results: OpenCoS consistently
'Note that this architecture is larger than 2ede- (Zagoruyko & Komodakis, 2016) used in the semi-supervised learning literature (Óliver al., 2018). We use ResNet-50 following the standard of SimCRR

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of median test accuracy on various benchmark datasets. We report the mean and standard deviation over three runs with different random seeds and splits, and also report the mean of the best accuracy in parentheses. The best scores are indicated in bold. We denote methods handling unlabeled out-of-class samples (i.e., open-set) as "Open-SSL".
In-class
CIFAR-Animals
CIFAR-10
CIFAR-100
Out-of-class
Open-SSI.
CIFAR-OIhe
SVHN
TinylmageNet
SVHN
TinylmageNet
fabels per class
SimCI.R- R-lee
65.58
56.89 19
58. 20
22.86 86
27.93
SimCLR-ft
67.29 22.70 (55.55)
12.166230000.0000
54.26 (0
18. 18.992mnrcom 11912)
29.57 (29
UASD-ft
43.92 (52.57)
42.99 (44.70)
50.38+275 (51.0)
19. 06. (1992)
25.72:40.9 (26.3))
RealMix-ft
64.42+4726
38.22+3341 (41.55)
48. 287.288
18.48 (20.4)
22.14=0.71220 (2651)
L-f
63.98 (72.20)
36.81 (47.22)
56.32
(57.58) (57
16.35
23.95 (25.6)
MixMatch-ft
44.34
23.71 1866
38, 40 (46 99)
13.45 (16
23.162185
FixMatch-ft
34.9424/1873:45
32.70 (55.98)
35. 35.99
23.56.200
(0.70.5507(2255)
RoMixMatch-ft
47.61 (64106)
24.56 (47.55)
28. 1: (85
9.36 20)
22.33
OpenCoS
77.6623477790
61.33
68.37
28.43 (2897)
(1477)
labels per class 25
SimCLR-Ie
80.03+0.75
70.31+014
7188++a.0
37.74+004
43.68-0026
SimCLR-ft
81.44::044 (1166)
64.41
73.05
39.61 2:02 (9987)
49.69
UASD-ft
82.17 -0..5 (82.50)
1.0
73.97 (74.5))
39.51 (29.5))
44.58 (44.00)
RealMix-ft
80.27 (81.04)
58.15 (67.27)
69.19+231.00
44.14 (44.9)
47.57 (4944)
L-
81.31
50.00
69. (36230077233
29. 29.00 (20117)
40. (6
MixMatch-ft
83.88 33.88821.00000 (81771
17.98123 19)
69.27
38. 20.60111 (45007)
80
FixMatch-ft
69.86 044244
68.02.4.00
(1.491113
41.73 (4223))
45.94 (09M6)
ReMixMatch-ft
8..622 (33.00)
37.98 (65.33)
67.38
32.75+00.7 (44
49.63
+ OpenCoS
86.89 =2.19
82.46=
49.02 (40.53)
54. (54.52)
improves ReMixMatch-fi, outperforming the other baselines simultancously. For example, OpenCoS improves the test accuracy of ReMixMatch-ft 28.51 - 68.37% 4 labels per class of CIFAR-10 + TinylmageNet. Also, we observe large discrepancies between the median and best accuracy of semi-supervised learning baselines, MixMatch-ft, ReMixMatch-ff. and FixMatch-ft, especially in the extreme label-scarce scenario of 4 labels per class, i.e., these methods suffer from over-fitting on out-of-class samples. One can also confirm this significant over-fitiing in state-of-the-art SSL methods by comparing other baselines with detection schemes, egg., USAD-ft, RealMix-ft, and L-ff which show less over-fitting but with lower best accuracy.
4.3 EXPERIMENTS ON IMAGENET DATASETS
We also evaluate OpenCoS on ImageNet to verify its scalability to a larger and more complex dataset. We design 9 benchmarks from ImageNet dataset, similarly to Restricted ImageNet (Tsipras et al., 2019): more specifically, we define 9 super-classes of ImageNet, each of which consists of 1 II~1 118 sub-classes. We perform our experiments on each super-class as an individual dataset. Each of the benchmarks (a super-class) contains 25 labels per sub-class, and we use the full ImageNet as an unlabeled dataset (excluding the labeled ones). In this experiment, we checkpoint per 2 training samples and report the median test accuracy of the last 3 out of 10. We present additional experimental details, e.g., configuration of the dataset, in Appendix A.3. Table 2 shows the results: OpenCoS still effectively improves the baselines. largely surpassing SimCLR-Je and SimCLR-ft well. For example, OpenCoS improves the test accuracy on Bird to 81.78% from FixMatch-FF of 78.73%, also improving SimCLR-le of 75.81% % significantly. This shows the efficacy of OpenCoS in exploiting open-set unlabeled data from unknown (but related) classes or even unseen distribution of another dataset in the real-world.
4.4 ABLATION STUDY
We perform an ablation study to understand further how OpenCoS works. Specifically, we assess the individual effects of the components in OpenCoS and show that each of them has an orthogonal contribution to the overall improvements. We also provide a detailed evaluation of our proposed detection score (7) compared to other out-of-distribution detection methods.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of median test accuracy on super-classes of ImageNet, which are obtained by grouping semantically similar classes in ImageNet; Dog, Reptile, Produce, Bird, Insect, Food, Primate, Aquatic animal, and Scenery. We report the mean and standard deviation over three runs with different random seeds and splits. The best scores are indicated in bold. We denote methods
handling unlabeled out-of-class samples (i.e., open-set) as "Open-SSL".
In-class
Dog
Reptile
Produce
Bird
Insect
Food
Primate
Aquatic
Scenery
Number of class
118
36
22
21
20
19
18
13
11
Out-of-class
Open-SSL
ImageNet
SimCLR--a
4
51.76.
64.76-04
15.81-1m
59.90.m.
56.53xxx
53.67
68.41-1v
64.73-om
SimCLR-ft
46.72.a.
51.76..e
65.21
27.37
58.93...
54.63x>
55.29.
68.82...s
62.79.uu
UASD-ft
V
45.64um
53.07
67.09.os
78.92.uu
u1.53
55.90...
56.70usas
70,31
64.36.u.
RealMix-ft
V
43.55
45.70
o6.06.
71.94
53,33
48
28:25:: x
45.89
58, 10:13
60.79.1a
MixMatch-ft
43.24
43.
71.04.ia
57.70uas
52.53
52.78.aa
62.36..n
60.06
ReMixMutch-ft
47.47
54.39.
66.88
78.95
62.30
:5.48
56.63.11
68.67
65.58...
FixMatch-ft
49.69
34
67.43_.17
62,53
54.84.1s:
57.70
69.79..s
14.12
+ OpenCoS
50.76.ass
57.19
71.82
81.78.uv
65.40 1120
60.53:am
61.37:24
73.44
66.91
Table 3: Ablation study three main components of our method: the detection criterion ("Detect"), auxiliary loss ("Aux. loss"), and auxiliary BNs ("Aux. BNs"). We report the mean and standard deviation over three runs with different random seeds and a fixed split of labeled data.
In-class + Out-of-class
CIFAR-Animals
CIFAR-10
Produce
Bird
Food
Detection
Aux. loss
Aux. BNs
+ CIFAR-Others
SVHN
ImageNet
+ ImageNet
ImageNet
79203
52.24
69.79
+9.02+0.0
54.98
79.68
55.78+0as
71.85
80.73
58.77 +au2
80.02
57.77
72.48
s2.32
60.00
Component analysis. To further analyze the individual contribution of each component of Open- CoS, we incrementally apply these components one-by-one to ReMixMatch-At (CIFAR-scale) and FixMatch-ft (ImageNet-scale) baselines. Specifically, we consider CIFAR-Animals CIFAR-10 + SVHN for CIFAR-scale, and Produce, Bird, Food + ImageNet for ImageNet-scale benchmarks. Table 3 summarizes the results, and they indeed confirm that each of what comprises OpenCoS has an orthogonal contribution to improve the accuracy of the benchmarks tested. We observe that leveraging out-of-class samples via auxiliary loss ("Aux. loss") achieves consistent improvements, and also outperforms the baselines significantly. Finally, we remark auxiliary batch normalization layers ("Aux. BNs") give a consistent improvement, and it is often significant: e.g., gives 55%78%-556 - 57.77% on CIFAR- 10 + SVHN.
CIFAR-Others
Other detection scores. In Section 3.2, we propose a detection score s(-) (7) for detecting out-of- class samples in an unlabeled dataset, based on the contrastive representation of SimCLR. This setup different to the standard out-of-distribution (OOD) detection task (Emmott et al., 2013; Liu et al., 2018): OOD detection targets unseen (i.e., "out-offdisti d-isttrbbution") samples in test time, while setup aims to detect seen out-of-class samples during training assuming few in-class labels. Due to this lack of labeled information, therefore, the standard techniques for OOD detection (Hendryeks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018b) are not guaranteed to perform still well in our setup. We examine this in Appendix B.3 by comparing detection performance of such OOD detection scores with ours (7) upon a shared SimCLR representation: in short, we indeed observe that our approach of directly leveraging the contrastive representation could perform better than simply applying OOD scores relying on few labeled samples, e.g., our score achieves an AUROC of 98, 10% on the CIFAR-Animals + CIFAR-Othhn based score (Hendryeks & Gimpel, 2017) of 80.799.9 50.79%. We present the detailed experimental setups and more results in Appendix B.3.
benchmark compared to the maximum softmax probability
Effect of soft-labeling. We emphasize that our soft-labeling scheme can be rather viewed as more reasonable way to label such out-of-class samples compared to existing state-of-the-art SSL methods, e.g., MixMatch simply assigns its sharpened predictions. On the other hand, prior work (Li & Hoiem, 2016) has a similar observation to our approach: assigning soft-labels of novel data could be beneficial for transfer learning. This motivate us to consider experiment to further support the claim that our soft-labeling gives informative signals: we train a classifier by minimizing only the cross-entropy loss with soft-labels (i.e., without in-class samples) from scratch. In Table 4, the trained
8

---

Under review as a conference paper at ICLR 2021
"Gazelle"
"Pizza"
Deer 77.55%
Truck 10.64%
Horse
16.97%
Frog 10.62%
Bird
2.11%
Ship 10.54%
Frog
Automobile 10.4%%
Automobile
Dog 10.29%
(a) Top-5 classes a soft-label "Gazelle".
(b) Top-5 classes in a soft-label "Pizza".
Figure 3: Illustration of soft-label assignments the CIFAR-10 + TinylmageNet benchmark. Un- labeled out-of-class samples from (a) "gazelle" is assigned with soft-labels of =78% confidence for "deer", and (b) "pizza" is assigned with almost uniform soft-labels (=10% of confidence). The soft-labels are scaled with the temperature = 0.1.
Table 4: Comparison of the median test accuracy of ResNet-50 trained on out-of-class samples and their soft-labels of the CIFAR-10 10 benchmarks with 4 labels per class. We denote the new setting of minimizing with the auxiliary loss as "Aux. loss only". We report the mean and standard deviation over three runs with different random seeds and splits of labeled data.
In-class
CIFAR-Animals
CIFAR-10
Out-of-class
+ CIFAR-Others
+ SVHIN
TinylmageNet
SimCLR-le
65.58+351
56.89
58.20
ReMixMatch-ft
47.61
24.56
28.51
Aux. loss only
39.76+197
25.97
21.19
classifier performs much better than (random) guessing, close baselines although this model does; this supports that generated soft-labels contain informative features of in-classes. The details on experimental setups can be found in Appendix A.2.
Examples of actual soft-labels. We also present some concrete examples of our soft-labeling scheme in Figure for better understanding, which are obtained from random unlabeled samples in the CIFAR-10 + TinylmageNet benchmark: Overall, we qualitatively observe that out-of-class samples that share some semantic features the in-classes (e.g., Figure 3(a)) have relatively high con- fidence capturing such similarity, while returning very close to uniform otherwise (e.g., Figure 3(b)).
5 CONCLUSION
In this paper, we propose a simple and general framework for handling novel unlabeled data, aiming toward
realistic assumption for semi-supervised learning. Our key idea is (intentionally) not to use label information, i.e., by relying on unsupervised representation, when handling novel data, which be naturally incorporated into semi-supervised learning with our framework: OpenCoS. In contrast to previous approaches, OpenCoS opens way to further utilize those novel data by assigning them soft-labels, which are again obtained from unsupervised learning. We hope our work would motivate researchers to extend this framework with a more realistic assumption, e.g., noisy labels (Wang et al., 2018; 8: et al., 2019), imbalanced learning (Liu et al., 2020).
REFERENCES
Abhijit Bendale and Terrance E Boult. Towards open deep networks. In CVPR, 2016.
Liron Bergman and Yedid Hoshen. Classification-bass anomaly detection for general data. In ICLR, 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR, 2020.

---

Under review as a conference paper at ICLR 2021
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. 1EEE Transactions on Neural Networks, 20(3):542-545) 2009.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 2020b.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020c.
Yun-Chun Chen, Chao-Te Chou, and Yu-Chiang Frank Wang. Learning to learn in semi-supervised fashion. In ECCV, 2020d.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint 1909, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Andrew Emmott, Shubhomoy Das, Thomas Dietterich, Alan Fern, and Weng-Keen Wong. System- atic construction of anomaly detection benchmarks from real data. In Proceedings of the ACM SIGKDD workshop on outlier detection and description, 2013.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurIPS, 2004.
Lan-Zhe Guo, Zhen-Yo Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019a
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NeurIPS, 2019b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop. 2013.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--cllibaate classifiers for detecting out-of-distributtin samples. ICLR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributtio samples and adversarial attacks. In NeurlPS, 2018b.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICMI., 2019.
10

---

Under review as a conference paper at ICLR 2021
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, 2016.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distributioo image detection in neural networks. In ICLR, 2018.
Si Liu, Risheek Garrepalli, Thomas G Dietterich, Alan Fern, and Dan Hendrycks. Open category detection with pac guarantees. arXiv preprint arXiv:1808.0055 2018.
Yunru Liu, Tingran Gao, and Haizhao Yang. Selectnet: Learning to sample from the wild for imbalanced data training. In MSMI, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and superviie learning. IEEE transactions on pattern analysis and machine intelligence, 1198):1779-199) 2018.
Varun Nair, Javier Fuentes Alonso, and Tony Beltramelli. Realmix: Towards realistic semi-supervised deep learning algorithms. arXiv preprint arXiv:1912.08766, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop, 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurlPS, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlabeled data. ICML, 2007.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In NeurIPS, 2016.
Christian Szegedy Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML., 2015.
Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In NeurIPS, 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. In NeurIPS, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NewrlPS, 2016.
Yisen Wang. Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, 2018.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. In CVPR, 2018.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adversarial examples improve image recognition. CVPR, 2020.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12844 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
11

---

Under review as a conference paper at ICLR 2021
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self Self-supervised semi- supervised learning. In ICCC,V, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In ECCV, 2016.
12

---

Under review as a conference paper at ICLR 2021
A TRAINING DETAILS
A.1 DETAILS ON THE EXPERIMENTAL SETUP
For the experiments reported in Table I, we generally follow the training details of FixMatch (Sohn et al., 2020). including optimizer, learning rate schedule, and exponential moving average. Specifi- cally, we use Nesterov SGD optimizer with momentum 0.9, cosine learning rate decay with an initial learning rate of 0.03, and a exponential moving average with decay of 0.999. The batch size is 64, which is widely adopted in semi-supervised learning (SSL) methods. We do not use weight decay for these models, as they are fine-tuned. We simple augmentation strategy, i.e., flip and crop, as default. We use the augmentation scheme of SimCLR (Chen et al., 2020a) (i.e., random crop with resize, random color distortion, and random Gaussian blur) when a SSL method requires to specify a strong augmentation strategy, e.g., for consistency regularization in the SSL literature (Berthelot et al., 2020; Sohn et al., 2020). We fix the number of augmentations as 2, following Berthelot et al. (2019): e.g.. Mix.atth--t generates two augmentations of each unlabeled sample while ReMixMatch-f generates one weak augmentation and one strong augmentation. In the case of ReMixMatch-fi we do not use the ramp-up weighting function, the pre-mixup and the rotation loss, which give marginal difference in fine-tuning, for efficient computation." $ For FixMatch-ft, we set the relative size of labeled and unlabeled batch μ 1 for a fair comparison with other baselines, and scale the learning rate linearly with μ. as suggested by Sohn et al. (2020). Following Chen et al. (2020c), UASD-ft computes the predictions by accumulative ensembling instead of using exponential moving average. OpenCoS shares all hyperparameters of the baseline SSL methods, e.g., FixMatch + OpenCoS shares hyperparameters of FixMatch-ft. For the results of ReMixMatch (from scratch) in Figure 1(b), we report the median accuracy of the last 10 checkpoints out of 200 checkpoints, where each checkpoint is saved per 2 training samples.
A.2 CIFAR EXPERIMENTS
Training from scratch. We pre-train all the baselines via SimCLR for a fair comparison, as mentioned in Section 4. In Table 5, we also report the performance of each baseline model when trained from scratch. Here, we report the median accuracy of the last 10 checkpoints out of 500 checkpoints in total. We also present the fine-tuned baselines (see Section 4.2) denoting by "-ft," e.g., MixMatch-ft. Here, we follow the training details those which originally used each baseline method. For example, ReMixMatch from scratch uses Adam optimizer with fixed learning rate of 0.002, and weight decay of 0.02. We use the simple strategy (i.e., flip and crop) and RandAugment (Cubuk et al., 2019) as weak and strong augmentation, respectively. In addition, we use the ramp-up weighting function, the pre-mixup and the rotation loss for ReMixMatch. We consider the CIFAR- R-00 + TinylmageNet benchmark assuming 80% proportion of out-of-class, i.e., 10K samples for in-class and 40K samples for out-of-class.
Training without in-class samples from scratch. For the experiments reported in Table 4, we train a classifier from scratch only with the unlabeled out-of-class samples and their soft-labels, on the CIFAR-10 benchmarks with 4 labels per class. We use ResNet-50, and SGD with momentum 0.9, weight decay 0,0001, and an initial learning rate of 0. The learning rate is divided by 10 after epochs 100 and 150, and total epochs are 200. We set batch size as 128, and use a simple data augmentation strategy, i.e., flip and crop. We minimize the cross- cross-entropy loss between the soft-labels q(-) and model predictions f(-), i.e., L H(g(2 "Mtt ff (1 )). We use the temperature scaling on the both sides of soft-labels and model predictions for a stable training, specifically 0.1 and 4, respectively.
Analysis of model architectures. For all our experiments, we use Re following the standard of SimCLR (Chen et al., 2020a). This architecture is larger than Wide-ResNett22-2 (Zagoruyko & Komodakis, 2016), a more widely adopted architecture in the semi-supervised learning litera- ture (Oliver et al., 2018). We have found that using larger network, i.e., 55, is necessary to leverage the pre-trained features of SimCLR: In Table 5, we provide an evaluation on another choice of model architecture, i.e., Wide-ResNett28-22 The hyperparameters are the same as the experiments on ResNer-50. Here, one can observe that OpenCoS trained on Wide-ResNet-28-2 still improves Rathx.at outperforming the other baselines. More importantly, however, observe that pre-training Wide-ResNet-Pe via SimCLR does not significantly improve the baselines trained from
Forr a fair comparison, ReMixMatch-ftt OpenCoS shares these settings.
13

---

Under review as a conference paper at ICLR 2021
Table 5: Comparison of the median accuracy of Widde-ResNet-28- and ResNet-50 on CIFAR-I00 + TinyImageNet benchmark over baseline methods. The best scores are indicated in bold. We denote methods handling unlabeled out-of-class samples (i.e., open-set) as "Open-SSL".
In-class
CIFAR-100
Out-of-class
TinylmageNet
Model architecture
Wide- Res----2
ResNet-50
Labels per class
Open-SSi RSLL
4
25
25
SimCLR-le
18.14
33.48
27.93
43.68
SimCLR-ft
17.55
36.94
29.57
49.69
UASD
8.76
27.62
7.80
19.21
UASD-ft
11.48
27.65
25.72
44.58
RealMix
13.15
36.97
12.15
28.46
RealMix-ft
13.56
33.31
22.14
47.57
MixMatch
14.83
37.94
14.19
32.49
MixMatch-ft
13.11
39.66
23.16
50.23
FixMatch
20.68
44.33
27.75
44.78
FixMatch-ft
16.59
36.84
30.70
45.94
ReMixMatch
16.06
40.45
17.87
39.86
ReMixMatch-ft
16.94
45.21
22.33
49.63
+ OpenCoS (ours)
27.45
46.95
36.51
54.09
seratch, contrary the results of ResNet-50. As also explored by Chen et al. (2020a), suspect this is due to that pre-training via SimCLR requires a larger model in practice, and suggest future SSL research explore larger architectures incorporate more rich features into their methods, e.g., features learned via unsupervised learning (Hénaff et al., 2019; Chen et al., 2020a:b).
Experiments on more labeled data. We have performed additional experiments on the CIFAR-10 10 + SVHN benchmark with 400 labels per class, and the results are given in Table 6. One can still observe that OpenCoS consistently outperforms other methods when more labeled data are available.
Table 6: Comparison of the median test accuracy on the CIFAR- 10 + SVHN benchmark with 400 labels per class over baseline methods. The best scores are indicated in bold.
SimCLR le
SimCLR-R ft
UASD-n n
RealMix n
MixMatch n
FixMatch ft
ReMixMatchh
OOpenCS (ours)
Accuracy
78.13
82.44
82.48
85.76
77.10
83.60
A.3 IMAGENET EXPERIMENTS
Benchmarks of ImageNet dataset. Section 4.3, we introduce benchmarks from ImageNet dataset, similar to Restricted ImageNet (Tsipras et al., 2019). In detail, we group together subsets of semantically similar classes into different super-classes, as shown Table
Details of IeageNet experiments. For the experiments reported in Table 2, we use pre-trained ResNet-50 model of Chen et al. (2020a) and fine-tune the projection header for epochs on ImageNet. We follow the optimization details of the fine-tuning experiments of SimCLR (Chen et al., 2020a): specifically, we use Nesterov SGD optimizer with momentum 0.9, and learning rate of 0.00625 (following LearningRate 0.05 BatchSize (ttch),ze/256,. We set the batch size to 32, and report the median accuracy of the last checkpoints out regularization techniques, and other hyperparameters are the case of FixMatch-ft OpenCoS, we empirically observe that it more beneficial not to discard the detected out-of-class samples in FixMatch training, it performs better than using in-class samples
checkpoints in total. Data augmentation,
as CIFAR experiments. In the
"https:/// :/// thub. combb..co//oooo coo/goog1 e resea rch malr
14

---

Under review as a conference paper at ICLR 2021
Table Super-classes used in benchmarks from ImageNet dataset. The class ranges are inclusive.
Super-class
Corresponding ImageNet Classes
"Dog"
151 to 268
"Reptile"
33 to
Producee
936 to 957
"Bird"
80 to 100
"Insect"
300 to 319
"Food"
928 to 935 & 959 to 969
"Primate"
365 to 382
"qquatic
118 to 121 & 389 to 397
"Scenery"
970 to 980
only: the auxiliary loss still use only out-of-class samples. Since FixMatch filters low-confidence unlabeled samples out, it is possibly due decrease in the number of training data.
B ABLATION STUDY
B. EFFECTS OF THE TEMPERATURE AND THE LOSS WEIGITT
and loss weight À 0.5. To examine the effect of hyperparameters т and X, we additionally test the hyperparameters across an array of т E {0.1, 0.5. 4} and À {0.1, 1, 2, 4} on the CIFAR- Tiny ImtgeNet benchmark with ResNet-50. The results are presented in Table 8. Overall, we found our method is fairly robust on r and
In Section 4, we perform the experiments with the fixed temperature
Table 8: Comparison of median test accuracy on the CIFAR- + TinyImageNe benchmark with (a) and (b) 25 labels per class, over various hyperparameters t and X.
(a) 4 labels per class
(b) 25 labels per class
0.1
0.5
1
4
0.1
0.5
2
4
0.1
37.07
37.62
36.89
37.63
37.76
0.1
55.86
55.71
55.99
55.64
55.44
0.5
38.22
38.02
36.48
37.51
38.18
0.5
54.70
56.14
56.30
55.62
56.09
37.76
37.46
37.54
37.45
37.31
1
55.66
56.01
56.12
55.45
55.99
37.06
37.86
36.92
37.15
37.12
2
55.67
55.79
55.95
55.96
56.04
37.14
37.41
37.94
36.94
37.43
4
55.75
55.58
56.23
56.56
55.31
B.2 EFFECTS OF OUT-OF-CLASS SAMPLES IN CONTRASTIVE LEARNING
To clarify how the improvements of OpenCoS comes from out-of-class samples, we have considered additional CIFAR-scale experiments with 4 labels per class. We newly pre-train and fine-tune SimCLR models using in-class samples only, i.e.. 30,000 for CIFAR-Animals, 10,000 for CIFAR-I0, CIFAR-100 100 benchmarks, and compare two baselines: SimCLR-le and ReMixMatch-ffi Interestingly. we found that just merging out-of-class samples to the training dataset improves the performance of SimCLR models in several cases (see Table 9), e.g., SimCI.R-le of CIFAR- 10 enhances from 55.27% 58.20% with TinyImageNer Also, OpenCoS significantly outperforms overall baselines, when out-of-class samples hurt the performance of SimCLR-le RathxMatch-f. We confirm that the proposed method effectively utilizes contrastive representations of out-of-class samples beneficially, compared other SSL baselines.
Robustness to incorrect detection. We observe our method is quite robust on incorrectly detected out-of-class samples, i.e., those samples still leveraged via auxiliary loss instead of SSL algorithm. We have considered an additional experiment on CIFAR-10 with 250 labels (out of 50,000 samples), that assumes (i) all the unlabeled samples in-class, and (ii) 80% of those in-class samples
15

---

Under review as a conference paper at ICLR 2021
Table Comparison of the median test accuracy for the use of out-of-class samples on the CIFAR- scale benchmarks with 4 labels per class. We denote whether using out-of-class samples for training as "w/ out-of-class". We report the mean and standard deviation over three runs with different random seeds and splits. The best scores are indicated in bold.
In-class Out-of-class
CIFAR-Animals
CIFAR-10
CIFAR-100
Method
w/ out-offeelass
CIFAR-Others
SVHN
TinyImageNet
SVHN
TinylmageNet
SimCLR-Ic
64.30
55.27
25.17
SimCLR-le
V
65.58
56.89
18.20
22.86_
27.93
RcMixMatch--f
54,04+77s
18.15
16.55
RcMixMatch-ft
V
47.611iaa
24.56
28.51 L
9.36
22.33
-OpenCoS (ours)
77.661M7
61.33
68.37 JAM
28.43
36.51 11.4
incorrectly detected as out-of-class in OpenCoS, Here, we compare OpenCoS with a baseline which only uses the correctly-dee coreectl-detected (in-class) samples without auxiliary loss, i.e., the baseline is trained on 10,000 samples while OpenCoS on 50,000 in total. In this scenario, OpenCoS achieves 89.54% in the median test accuracy, while the baseline does 89.27%: this shows that our auxiliary loss does harm the training even when it is incorrectly applied to in-class samples.
B.3 EVALUATIONS OF OUR DETECTION SCORE
Baselines. We consider maximum softmax probability (MSP; Hendrycks & Gimpel 2017), ODIN (Liang et al., 2018), and Mahalanobis distance-based score (Lee et al., 2018b) base- line detection methods. As MSP and ODIN require a classifier to obtain their scores, we employ SimCLR-le: a SimCLR model which additionally learns linear layer with the labeled dataset, for both baselines.
ODIN performs an input pre-processing by adding small perturbations with temperature scaling as follows:
Ply = c/x;T) P(y=c/x;T) = exp(fc(x)/TT (x)) x' = - e sign( V log P
= c)x;
(10)
where f = (ff,..,, is the logit vector of deep neural network, T is temperature scaling pa- rameter, and is a magnitude of noise. ODIN calculates the pre-processed data x' and feeds it into the classifier compute the confidence score, i.e., max, P(yl='; T), and identifies it as in-class if the con- fidence score is higher than some threshold 8. We choose the temperature T and the noise magnitude from {1, 10, 100, 1000} and {0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.2). respectively, by using 2,000 validation data.
Mahalanobis distance- based score (Mahalanobis) assumes the features of the neural network follows the clas-conniiional Gaussian distribution. Then, it computes Mahalanobis distance between input & and the closest class-conditional Gaussian distribution, i.e..
(xx) max (s(()) - -.) -- pc),
(11)
where ue is the class mean and Σ is the covariance of the labeled data. We fix the covariance matrix as the identity because the number of labeled samples is insufficient to compute it: the feature dimension of SimCLR encoder fe and projection header g are 2048. Moreover, Mahalanobis has the noise magnitude parameter e for input pre- pro-prccssinng like ODIN, and use feature ensemble method of Lee et al. (2018b). We choose from {0, 0.0005, 0.001, 0.0014, 4, 0.002, 0.005, 0.01). }. and perform the feature ensemble of intermediate features including fa 's and g 's by using 2,000 validation data.
Metrics. We follow the threshold-fres detection metrics used in Lee et al. (2018b) to measure the effectiveness of detection scores in identifying out-of-class samples.
True negative rate (TNR) at 95% true positive rate (TPR). We denote true positive, true negative, false positive, and false negative as TP, TN, FP, and FN, respectively. We measure TNR TN/ (FP+TN) TPR TP/ (TP+FN) is 95%.
16

---

Under review as a conference paper at ICLR 2021
Detection accuracy. For unlabeled data (u(= D U Di"",, this metric corresponds to the maximum classification probability over all possible thresholds
min (FNR. P(x E D + FPR P(a e )
where false negative rate FNR FN (FN+TP), and false positive rate FPR FP (FP+TN).
Area under the receiver operating characteristic curve (AUROC). The receiver operat- ing characteristic (ROC) curve a graph of the true positive rate (TPR) against the false positive rate (FPR) by varying threshold, and we measure its area.
Area under the precision-recall curve (AUPR). The precision-recall (PR) curve is graph of the precision TP (TP+FP) against recall TP (TP+FN) by varying a threshold. AUPR-in (or -out) is AUPR where in- (or out-of-) class samples are specified positive.
Table 10: Comparison of detection methods on the CIFAR-Animal + CIFAR-Others benchmark with labels per class under various evaluation metrics. We denote our detection method without the projection header as "Ours w/o header". The best scores indicated in bold.
Detection method
TNR at TPR 95% f Detection Accuracy
AUROC
AUPR-in
AUPR-out f
MSP
18.20
72.81
80.79
88.72
66.38
ODIN
24.10
79.41
86.55
92.55
72.75
Mahalanobis
90.90
93.57
97.48
98.60
94.36
Ours w/o header Ours
40.44
80.11
88.80
93.46
79.35
91.20
93.49
98.10
98.79
96.86
this section, we present evaluations our detection score s(.) (7) under various detection metrics on the CIFAR- CIFAR-Others with 4 labels per class. Table 10 shows the results: interestingly, our score outperforms MSP and ODIN and also performs comparable to Mahalanobis, even these baselines require more computational costs, e.g., input pre- pre-processing We confirm that the design of our score is an effective and efficient way to detect out-of-class samples based on the representation of SimCLR. In Figure 4, we provide the receiver operating characteristic (ROC) curves that support the above results. We remark that the projection header (4) is crucial for the detection, e.g., 9 enhances AUROC of our score from 88. 88. to 98. 10%. According the definition of our score (7), can be viewed as simpler version of Mahalanobis without its input pre-processing and feature ensembles under an assumption of identity covariance, which may explain their comparable performances.
1.0
True Positive Rate
0.8
0.6
0.4
MSP
ODIN
0.2
Mahalanobis w/o header Ours
0.2
0.6 False Positive Rate
0.4
0.8
1.0
Figure 4: Receiver operating characteristic (ROC) curves of detection methods on CInAR-Animals + CIFAR-Others benchmark with labels per class.
We additionally provide the performance OpenCoS among various detection methods, including the above baselines and two artificial methods: consider (a) Random: random detection with probability of 0.5, and (b) Oracle: a perfect detection. For MSP, ODIN, and Mahalanobis, we choose their detection thresholds at TPR 95%. Table 11 shows the results: observe that the classification accuracy is proportional to the detection performance. Remarkably, our detection method achieves comparable accuracy Oracle, which is the optimal performance of OpenCoS.
17

---

Under review as a conference paper at ICLR 2021
Table 1I: Comparison of the median test accuracy on the CIFAR-Animals CIFAR-Other benchmark with 4 labels per class among various detection methods. We denote our detection method without the projection header as "Ours w/o header". We report mean and standard deviation over three runs with different random seeds and fixed split of labeled data. The best scores are indicated in bold.
Detection method
Random
MSP
ODIN
Mahalanohis
Ours w/o header
Oracle
Accuracy
g9.7
61.78
64.08 12m
78.03
75.27
80.02
81.10.14
B.4 EVALUATIONS OF THE DETECTION THRESHOLD
We additionally provide the detection performance on various proportions of out-of-class samples, i.., 50% and 67%, on this benchmark. For each setting, the number of out-of-class samples is fixed 20K, while in-class samples are controlled to 20K and 10K, respectively. We choose the same detection threshold f := μ 20, throughout all experiments: it is reasonable choice, giving z 95% confidence if the score follows Gaussian distribution. Table 12(a) shows the detection performance of our threshold and its applicability over various proportions of out-of-class samples. Although tuning gives further improvements though (see Table 12(b)), we fix the threshold without any tuning.
Table 12:
The detection performance across different (a) proportions of out-of-class and (b) detection thresholds in CIFAR-Animals CIFAR-Othe Otthers benchmark with labels per class.
(a) The detection performance of the proposed
(b) The detection performance and median test accuracy across different thresholds t, i.e., k 1, 3, of t ji KI.
threshold L = fr 20; on varying proportions of out-of-class.
Proportion of out-of-class
40%
50%
67%
k
1
2
3
4
True Positive Rale (TPR)
63.61
72.19
81.10
True Positive Rate (TPR)
37.48
63.61
85.55
98.62
True Negative Rate (TNR)
99.76
99.66
99.55
True Negative Rate (TNR)
99.94
99.76
97.22
74.48
AUROC
98.10
98.50
98.90
Accuracy
75.47
80.67
81.12
78.37
C ALGORITHM
The full training procedure of OpenCoS is summarized in Algorithm 1.
18

---

Under review as a conference paper at ICLR 2021
Algorithm OpenCoS: A general framework for open-set semi-supervised learning (SSL).
1: Input: The classifier f, the encoder fe (i.e., penultimate features of f), and the projection header
g. labeled dataset D; and an open-set unlabeled data D...
2: Pre-train fe and g via SimCLR.
3: S = Φ
4: for each labeled sample æq E DD do
5: S = Sj {s(
> The similarity score (7).
6: end for
7: = [S] 2 Var Var
[
> Compute the threshold t.
8:
9: D.r - Φ. D o
for each unlabeled sample Fa D,, do
11:
if 8(F.; Se. g) < then
D D U {xu}
Detect out-of-class unlabeled samples.
13: else
Din Di (xu}
15: end if
16:
end for
17:
18:
for each sample 2ι E Dt, I E Din and aa E Dout do
19:
Using contrastive representations (9). D SSL with auxiliary loss (8). Update parameters of by computing the gradients of the proposed loss LopenCoS.
q(x:((t)) +- Compute soft-label of zn
20:
CopenCos = Cssi, xi f) + X H((() f(r
21:
22:
end for
19


Reviews:
[{"review_id": "wRe855ial8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n\nThe paper proposes a new approach for open set semi-supervised learning, where there are unlabeled data from classes not in the labeled data. The paper uses a contrastive representation learning paradigm to learn a feature encoder and a similarity measurement. Then the paper filters outlier samples by the similarity measurement and further utilizes outlier samples with soft labels. The separate BN layers address the distribution shift between in-class and out-class data.\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for accepting. I like the idea of learning representation in an unsupervised way for both labeled and unlabeled data. My major concern is about how to select a reasonable threshold and why the threshold works well (see cons below). Hopefully the authors can address my concern in the rebuttal period. \n\n########################################################################## \n\nPros:\n\nThe paper addresses a very interesting and practical problem in semi-supervised learning, where the unlabeled samples may include out-class samples. This problem promotes the practical use of semi-supervised learning in real world applications.\n\nThe paper employs contrastive representation learning to learn the encoder and similarity measurement, which preserves the similarity between features of a sample transformed by different transformations. The learned encoder can encode the semantic information into the feature for both labeled and unlabeled data in an unsupervised way. \n\nThe paper filters the out-class samples by the learned similarity measure based on a threshold, which can better filter out out-class samples.\n\nThe paper employs different batch normalization layers for the in-class and out-class samples, which avoids the influence of the distribution shift.\n\nThe paper is well-written and the claims are clearly clarified.\n\nExperimental results on three semi-supervised learning benchmarks: CIFAR-10, CIFAR-100 and ImageNet, show that the proposed method can detect open-class samples and achieves higher accuracy the previous semi-supervised learning methods. Qualitative results show that the method is not very sensitive to hyper-parameters.\n\n##########################################################################\n\nCons:\n\nThe performance gain of the proposed method is only over closed-set semi-supervised learning methods. The paper does not compare with the state-of-the-art open set semi-supervised learning method (Guo et al., 2020).\n\nThe threshold hyper-parameter is the key to filtering out-class samples. The paper decides the threshold by mean minus 2 standard deviation. Could the authors explain why choosing this value? Is there any insight in this threshold or why does this threshold fitful for any dataset?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "taEABg8678a", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of semi-supervised learning, where the unlabeled data may include out-of-class samples. To address this task, the paper proposes a method consisting of three steps: (1) detecting out-of-class samples in the unlabeled set, (2) assigning soft-labels to the detected out-of-class samples using class-conditional likelihoods from labeled data, and (3) using auxiliary batch normalization layers  to help mitigate the class distribution mismatch problem. Experiments are conducted on CIFAR-10, CIFAR-100, ImageNet datasets. Results show improvements over competing methods.\n\nQuality: This paper is well written and well organized. I find this paper easy to follow.\n\nClarify: The preliminaries section clearly describes the setting of semi-supervised learning concerned in this paper and also clearly describes the contrastive representation learning. I like the way that the authors include a preliminaries section before the proposed method section. This allows me to understand which parts of the method are built upon existing approaches and helps me better identify the contributions and new things of this paper.\n\nOriginality: While I think the idea of this paper makes sense, I think there are several parts that are highly similar from a recent paper published in ECCV 2020 [a]. While the setting considered in [a] (for metric learning problems) is a bit different from that concerned in this submission (for image classification tasks), the high-level idea (learning representations that can be used to describe unlabeled images with labels different from those in the training set) is very similar. For example, the idea of Equation 5 and Equation 6 in this submission is almost exactly the same as that in Equation 2 of [a]. In addition, the idea of Equation 9 of this submission is very similar to that of Equation 3 of [a]. However, the authors did not acknowledge the similarity with [a] in the submission. This will make readers feel like the idea of Equation 5 and Equation 6 is original in this paper.\n\n[a] Chen et al. Learning to learn in a semi-supervised fashion. In ECCV, 2020. https://arxiv.org/pdf/2008.11203.pdf\n\nSignificance: Given that some parts of this paper are highly similar to [a], the significance of this paper is downplayed.\n\nRequest for author response: I would like to see how the authors compare their paper with [a] in terms of problem setting, idea of class-wise similarity and representation learning. In particular, I would like to know the pros and cons of this paper compared to [a]. It would be great if the authors can talk about whether the semantics-oriented similarity representation in [a] could be used (and how to use it) to help improve the performance of the proposed method in the setting concerned by the paper.\n\nRating: Given that there are many parts similar to [a] and they are not acknowledged in the paper, I can only rate 4 for this submission at this point. I will reevaluate this paper after seeing the reviews from the other reviews as well as the author response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "otBlKaAUQw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: \nThe quality of the paper is below average. The loss function that integrates out-of-class samples is counter intuitive and seems to be chosen based on improved empirical evidence.\n\nClarity:\nThe paper reads well.\n\nOriginality/Significance:\nThe originality of the approach is limited. Main ideas are borrowed from SimCLR and applied to SSL. \n\nDetailed Comments:\n\nThe paper uses contrastive learning idea proposed in SimCLR (Chen et al. 2020) to detect out-of-class samples and treat them in a different way than in-class unlabeled samples during semi-supervised learning. It also explores the idea of auxiliary batch normalization (from Xie et al. 2020) in the open-set SSL setting but the results of the ablation study suggest the level of improvement achieved by this normalization is negligible and the most of the improvement comes from more accurate detection of out-of-class samples through using the projection header function introduced in SimCLR paper. Although results across multiple benchmark datasets report significant improvement over other SSL techniques these improvements could be artificial as other techniques have no way of handling out-of-class samples. Integrating these other out-of-class detection techniques in ReMixMatch and comparing results with the proposed technique would offer a more compelling argument. Overall, the paper proposes a nice practical idea but for publication in ICLR one would like to see more theoretical insight along with empirical evidence. \n\nClass conditional likelihoods has been shown to be not very useful in detecting out-of-distribution samples in cross-entropy loss learning. What aspect of contrastive learning makes it more useful for open-set classification?\n\nCross-entropy term in equation 8 does not make much sense. It is simply computing the loss with respect to an incorrect label. If a sample belongs to an unknown class it is not clear why this would help SSL. If the purpose here is to capture shared characteristics of the samples then SimCLR trained with both labeled and unlabeled data already takes care of it. The authors try to justify this by considering that some seen classes would be similar to the unseen one but for a fine-grained classification task such samples may still hurt predictive performance. If there are no similar classes among labeled classes the authors argue that this loss will have a uniform affect for all classes. Again, this is not a compelling argument. No matter how dissimilar the unseen sample to labeled classes are it would be more similar to some of the classes than some others. Due to the normalization affect the weight distributions will significantly deviate from a uniform distribution. \n\nBaseline techniques are all SSL techniques. These are guaranteed to perform worse than the proposed technique because they have no way of handling out-of-class samples. What about other baselines? From Table 3 one can see that the main contribution comes from the detection of out-of-class samples. The authors compare the detection performance of their technique against other standard outlier detection techniques and show that the proposed detection outperforms all of them to achieve the highest AUC in the \"isolated\" detection task. However, it is still not clear why the projection header g achieves something that softmax probabilities cannot do. In other words why do class-conditional probabilities obtained from g are useful for out-of-class detection but the probabilities that one would obtain from the softmax layer of an architecture (such resnet) trained by cross-entropy loss is not much useful for the same task. Interesting  that not much insight has been provided in the SimCLR paper either. Along the same lines, as a contrastive loss function, triplet loss also seems to do well in open-world settings. \n\nMinor:\n\nPlease correct the following:\n\npage1 Compared to prior approaches approaches have that bypassed ...", "extra_scores": {}, "extra_text": {}}, {"review_id": "LKP5-OFrf-B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed to address the task of semi-supervised learning (SSL) by contrastive learning techniques, and the proposed techniques can be applied to handle open-set unlabeled data (i.e., the label spaces between label and unlabeled data are partially disjoint). I found the paper clearly written and easy to follow. The review comments are discussed below.\n\nThe proposed learning scheme is extended from SimCLR by Chen et al. More precisely, the authors perform unsupervised learning using D_l (w/o observing labels) and D_u using SimCLR techniques. In order to handle out-of-class samples in D_u, the authors present the idea of learning class-wise prototypical representation based on the above contrastive features. Detection of such samples is performed by a simple cosine similarity comparison between each instance in D_u and the prototypes of D_l. The main contribution lies in the last stage, i.e., SSL with auxiliary loss, which trains classifiers for recognizing in-class samples while assigning soft label scores for out-of-sample ones. The use of such soft labels allows the training of such unlabeled and out-of-class samples, which would be the major novelty of this work. (I do not feel that the use of batch norm would be viewed as technical contributions.)\n\nOverall, I feel that most of the technical components come from existing works (e.g., SimCLR from Chen et al. 2020, auxiliary batch norm from Xie et al. 2020.). The use of soft-label assignments has also been proposed by existing works, which makes the overall technical contributions to be marginal. The lack of verification on the soft label assignment would be the concern as well. For example, is \"cat\" assigned with 0.1*leopard + 0.2*lion + 0.7*tiger? Most importantly, are out-of-class samples assigned with uniformly soft labels (i.e., 1/C)? Out-of-class samples might still exhibit similarity with selected in-class categories, and thus forcing their soft labels to be a uniform distribution might not seem to be practical (if that's the case). Existing vision and learning models have been considering class-similarity based representation, etc. techniques for handling zero-shot or open-dataset learning problems (e.g., Xian et al. PAMI'18 and Scheirer et al. PAMI'12). Based on the above remarks, I feel that the paper is yet above the ICLR standard for acceptance.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes OpenCos for semi-supervised learning that can leverage unsupervised information in open-set scenarios where samples can be out-of-class.  They first pre-train by learning an unsupervised representation using SimCLR on both the labeled and unlabeled data.  Then, they detect out-of-class samples in the unlabeled set based on similarity measures on the representation learned in the previous step.  The unlabeled data can now be split into in-class and out-of-class.  OpenCos optimizes (8) which combines a semi-supervised loss for in-class unlabeled data and an auxiliary cross-entropy loss with soft-labels for the out-of-class samples.  Finally, they perform an auxiliary batch normalization.

The paper is easy to read and clearly structured.  It also places the work well with respect to related work.

The proposed approach makes sense; however, as pointed out by the reviewers, the novelty is marginal.  The technical innovation seems to be an extension of SimCLR and the auxiliary batch norm of Xie et al.

Decision: Reject