OPENCOS: CONTRASTIVE SEMI-SUPERVISED LEARN-
Year: 2020

Abstract:
Modern semi-supervised learning methods conventionally assume both labeled and unlabeled data have the same class distribution. However, unlabeled data may include out-of-class samples in practice; those that cannot have one-hot encoded labels from a closed-set of classes in label data, i.e., unlabeled data is an open-set. In this paper, we introduce OpenCoS, a method for handling this realistic semisupervised learning scenario based on a recent framework of contrastive learning. One of our key findings is that out-of-class samples in the unlabeled dataset can be identified effectively via (unsupervised) contrastive learning. OpenCoS utilizes this information to overcome the failure modes in the existing state-of-the-art semisupervised methods, e.g., ReMixMatch or FixMatch. In particular, we propose to assign soft-labels for out-of-class samples using the representation learned from contrastive learning. Our extensive experimental results show the effectiveness of OpenCoS, fixing the state-of-the-art semi-supervised methods to be suitable for diverse scenarios involving open-set unlabeled data. The code will be released.

heading: 1 INTRODUCTION
text: Despite the recent success of deep neural networks with large-scale labeled data, many real-world scenarios suffer from expensive data acquisition and labeling costs. This has motivated the community to develop semi-supervised learning (SSL; Grandvalet & Bengio 2004; Chapelle et al. 2009), i.e., by further incorporating unlabeled data for training. Indeed, recent SSL works (Berthelot et al., 2019; 2020; Sohn et al., 2020) demonstrate promising results on several benchmark datasets, as they could even approach the performance of fully supervised learning using only a small number of labels, e.g., 93.73% accuracy on CIFAR-10 with 250 labeled data (Berthelot et al., 2020).
However, SSL methods often fail to generalize when there is a mismatch between the classdistributions of labeled and unlabeled data (Oliver et al., 2018; Chen et al., 2020c; Guo et al., 2020), i.e., when the unlabeled data contains out-of-class samples, whose ground-truth labels are not contained in the labeled dataset (as illustrated in Figure 1(a)). In this scenario, various label-guessing techniques used in the existing SSL methods may label those out-of-class samples incorrectly, which in turn significantly harms the overall training through their inner-process of entropy minimization (Grandvalet & Bengio, 2004; Lee, 2013) or consistency regularization (Xie et al., 2019; Sohn et al., 2020). This problem may largely hinder the existing SSL methods from being used in practice, considering the open-set nature of unlabeled data collected in the wild (Bendale & Boult, 2016).
Contribution. In this paper, we focus on a realistic SSL scenario, where unlabeled data may contain some unknown out-of-class samples, i.e., there is a class distribution mismatch between labeled and unlabeled data (Oliver et al., 2018). Compared to prior approaches that have bypassed this problem by simply filtering out them with some heuristic detection scores (Nair et al., 2019; Chen et al., 2020c), the unique characteristic in our approach is to further leverage the information in out-of-class samples by assigning soft-labels to them: they may still contain some useful features for the in-classes.
Somewhat surprisingly, we found that a recent technique of contrastive unsupervised learning (Wu et al., 2018; He et al., 2020; Chen et al., 2020a) can play a key role for our goal. More specifically, we show that a pre-trained representation via contrastive learning, namely SimCLR (Chen et al., 2020a), on both labeled and unlabeled data enables us to design (a) an effective score for detecting out-of-class samples in unlabeled data, and (b) a systematic way to assign soft-labels to the detected out-of-class samples, by modeling class-conditional likelihoods from labeled data. Finally, we found (c) auxiliary
batch normalization layers (Xie et al., 2020) could further help to mitigate the class-distribution mismatch via decoupling batch normalization layers. We propose a generic SSL framework, coined OpenCoS, based on the aforementioned techniques for handling open-set unlabeled data, which can be integrated with any existing SSL methods.
We verify the effectiveness of the proposed method on a wide range of SSL benchmarks based on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) datasets, assuming the presence of various out-of-class data, e.g., SVHN (Netzer et al., 2011) and TinyImageNet datasets. Our experimental results demonstrate that OpenCoS greatly improves existing state-of-the-art SSL methods (Berthelot et al., 2019; 2020; Sohn et al., 2020), not only by discarding out-of-class samples, but also by further leveraging them into training. We also compare our method to other recent works (Nair et al., 2019; Chen et al., 2020c; Guo et al., 2020) addressing the same class distribution mismatch problem in SSL, and again confirms the effectiveness of our framework, e.g., we achieve an accuracy of 68.37% with 40 labels (just 4 labels per class) on CIFAR-10 with TinyImageNet as out-of-class, compared to DS3L (Guo et al., 2020) of 56.32%.
Overall, our work highlights the benefit of unsupervised representations in (semi-) supervised learning: such a label-free representation turns out to enhance model generalization due to its robustness on the novel, out-of-class samples.

heading: 2 PRELIMINARIES

heading: 2.1 SEMI-SUPERVISED LEARNING
text: The goal of semi-supervised learning for classification is to train a classifier f : X → Y from a labeled dataset Dl = {x(i)l , y (i) l } Nl i=1 where each label yl is from a set of classes Y := {1, · · · , C}, and an unlabeled dataset Du = {x(i)u }Nui=1 where each yu exists but is assumed to be unknown. In an attempt to leverage the extra information in Du, a number of techniques have been proposed, e.g., entropy minimization (Grandvalet & Bengio, 2004; Lee, 2013) and consistency regularization (Sajjadi et al., 2016). In general, recent approaches in semi-supervised learning can be distinguished by the prior they adopt for the representation of unlabeled data: for example, the consistency regularization technique (Sajjadi et al., 2016) attempt to minimize the cross-entropy loss between any two predictions of different augmentations t1(xu) and t2(xu) from a given unlabeled sample xu, jointly with the standard training for a labeled sample (xl, yl):
LSSL(xl, xu) := H(yl, f(xl)) + β ·H(f(t1(xu)), f(t2(xu))), (1) where H is a standard cross-entropy loss for labeled data, and β is a hyperparameter. Recently, several “holistic” approaches of various techniques (Zhang et al., 2018; Cubuk et al., 2019) have shown remarkable performance in practice, e.g., MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020), and FixMatch (Sohn et al., 2020), which we mainly consider in this paper. We note that our scheme can be integrated with any recent semi-supervised learning methods.

heading: 2.2 CONTRASTIVE REPRESENTATION LEARNING
text: Contrastive learning (Oord et al., 2018; Hénaff et al., 2019; He et al., 2020; Chen et al., 2020a) defines an unsupervised task for an encoder fe : X → Rde from a set of samples {xi}: assume that a “query” sample xq is given and there is a positive “key” x+ ∈ {xi} that xq matches. Then the contrastive loss is defined to let f to extract the necessary information to identify x+ from xq as follows:
Lcon(fe, xq, x+; {xi}) := − log exp(h(fe(xq), fe(x+))/τ)∑ i exp(h(fe(xq), fe(xi))/τ) , (2)
where h(·, ·) is a pre-defined similarity score, and τ is a temperature hyperparameter. In this paper, we primarily focus on SimCLR (Chen et al., 2020a), a particular form of contrastive learning: for a given {xi}Ni=1, SimCLR first samples two separate data augmentation operations from a pre-defined family T , namely t1, t2 ∼ T , and matches (x̃i, x̃i+N ) := (t1(xi), t2(xi)) as a query-key pair interchangeably. The actual loss is then defined as follows:
LSimCLR(fe; {xi}Ni=1) := 1
2N 2N∑ q=1 Lcon(fe, x̃q, x̃(q+N) mod 2N ; {x̃i}2Ni=1 \ {x̃q}), (3)
hSimCLR(v1, v2) := CosineSimilarity(g(v1), g(v2)) = g(v1) · g(v2)
||g(v1)||2||g(v2)||2 , (4)
where g : Rde → Rdp is a 2-layer neural network called projection header. In other words, the SimCLR loss defines a task to identify a “semantically equivalent” sample to xq up to the set of data augmentations T .

heading: 3 OPENCOS: A FRAMEWORK FOR OPEN-SET SEMI-SUPERVISED LEARNING
text: We consider semi-supervised classification problems involving C classes. In addition to the standard assumption of semi-supervised learning (SSL), we assume that the unlabeled dataset Du is open-set, i.e., the hidden labels yu of xu may not be in Y := {1, · · · , C}. In this scenario, existing semisupervised learning techniques may degrade the classification performance, possibly due to incorrect label-guessing procedure for those out-of-class samples. In this respect, we introduce OpenCoS, a generic method for detecting and labeling out-of-class unlabeled samples in semi-supervised learning. Overall, our key intuition is to utilize the unsupervised representation from contrastive learning (Wu et al., 2018; He et al., 2020; Chen et al., 2020a) to leverage such out-of-class samples in an appropriate manner. We present a brief overview of our method in Section 3.1, and describe how our approach, OpenCoS, can handle out-of-class samples in Section 3.2 and 3.3.

heading: 3.1 OVERVIEW OF OPENCOS
text: Recall that our goal is to train a classifier f : X → Y from a labeled dataset Dl and an open-set unlabeled dataset Du. Overall, OpenCoS aims to overcome the presence of out-of-class samples in Du through the following procedure:
1. Pre-training via contrastive learning. OpenCoS first learns an unsupervised representation of f via SimCLR1 (Chen et al., 2020a), using both Dl and Du without labels. More specifically, we learn the penultimate features of f , denoted by fe, by minimizing the contrastive loss defined in (3). We also introduce a projection header g (4), which is a 2-layer MLP as per (Chen et al., 2020a).
2. Detecting out-of-class samples. From a learned representation of fe and g, OpenCoS identifies an out-of-class unlabeled data Doutu from the given data Du = Dinu ∪ Doutu . This detection process is based on the similarity score between Dl and Du in the representation space of fe and g (see Section 3.2).
3. Semi-supervised learning with auxiliary loss and batch normalization. Now, one can use any semi-supervised learning scheme to train f using Dl and Dinu , e.g., ReMixMatch (Berthelot et al., 2020). In addition, OpenCoS minimizes an auxiliary loss that assigns a soft-label to each sample in Doutu , which is also based on the representation of fe and g (see Section 3.3). Furthermore, we found maintaining auxiliary batch normalization layers (Xie et al., 2020) for Doutu is beneficial to our loss as they mitigate the distribution mismatch arisen from Doutu .
Putting it all together, OpenCoS provides an effective and systematic way to detect and utilize outof-class data for semi-supervised learning. Due to its simplicity, our framework can incorporate the most recently proposed semi-supervised learning methods (Berthelot et al., 2019; 2020; Sohn et al., 2020) and improve their performance in the presence of out-of-class samples. Figure 2 illustrates the overall training scheme of OpenCoS.

heading: 3.2 DETECTION CRITERION OF OPENCOS
text: For a given labeled dataset Dl and an open-set unlabeled dataset Du, we aim to detect a subset of the unlabeled training data Doutu ⊆ Du whose elements are out-of-class, i.e., yu /∈ Y . A standard way to handle this task is to train a confident-calibrated classifier using Dl (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018a;b; Hendrycks et al., 2019a;b; Bergman & Hoshen, 2020; Tack et al., 2020). However, such methods typically assume a sufficient number of in-class samples (i.e., large Dl), which does not hold in our case to the label-scarce nature of SSL. This motivates us to consider a more suitable approach which leverages the open-set unlabeled dataset Du for contrastive learning. Then, OpenCoS utilizes the labeled dataset Dl to estimate the class-wise distributions of (pre-trained) embeddings, and use them to define a detection score for Du. We assume that an encoder fe : X → Rde and a projection header g : Rde → Rdp pre-trained via SimCLR on Dl ∪ Du. Motivated by the similarity metric used in the pre-training objective of SimCLR (4), we propose a simple yet effective detection score s(xu) for unlabeled input xu based on the cosine similarity between xu and class-wise prototypical representations {vc}Cc=1 obtained from Dl. Namely, we first define a class-wise similarity score2 simc(xu) for each class c as follows:
vc (Dl; fe, g) := 1
N cl ∑ i 1 y (i) l =c · g(fe(x(i)l )), and (5)
simc (xu;Dl, fe, g) := CosineSimilarity(g(fe(xu)), vc), (6)
where N cl := |{(x (i) l , y (i) l )|y (i) l = c}| is the sample size of class c in Dl. Then, our detection score s(xu) is defined by the maximal similarity score between xu and the prototypes {vc}Cc=1: s(xu) := max
c=1,··· ,C simc (xu) . (7)
In practice, we use a pre-defined threshold t for detecting out-of-class samples in Du, i.e., we detect a given sample xu as out-of-class if s(xu) < t. In our experiments, we found an empirical value of t := µl − 2σl performs well across all the datasets tested, where µl and σl are mean and standard deviation computed over {s(x(i)l )} Nl i=1, respectively, although more tuning of t could further improve the performance. Further analysis of our detection threshold can be found in Appendix B.4. 1Nevertheless, our framework is not restricted to a single method of SimCLR; it is easily generalizable to other contrastive learning methods (Hénaff et al., 2019; He et al., 2020; Chen et al., 2020b). 2In this work, we adopt the well-known cosine similarity to define our score, but there can be other designs as long as it represents class-wise similarity (Chen et al., 2020d; Vinyals et al., 2016; Snell et al., 2017).

heading: 3.3 AUXILIARY LOSS AND BATCH NORMALIZATION OF OPENCOS
text: Based on the detection criterion defined in Section 3.2, the open-set unlabeled dataset Du can be split into (a) the in-class unlabeled dataset Dinu and (b) the out-of-class unlabeled dataset Doutu . The labeled dataset Dl and Dinu are now used to train the classifier f using any existing semi-supervised learning method (Berthelot et al., 2019; 2020; Sohn et al., 2020).
In addition, we propose to further utilize Doutu via an auxiliary loss that assigns a soft-label to each xoutu ∈ Doutu . More specifically, for any semi-supervised learning objective LSSL(xl, xinu ; f), we consider the following loss:
LOpenCoS = LSSL(xl, xinu ; f) + λ ·H(q(xoutu ), f(xoutu )), (8) where H denotes the cross-entropy loss, λ is a hyperparameter, and q(xoutu ) defines a specific assignment of distribution over Y for xoutu . In this paper, we propose to assign q(xoutu ) based on the class-wise similarity scores simc(xoutu ) defined in (6), again utilizing the contrastive representation fe and g:
qc(xu) := exp (simc(xu; fe, g)/τ)∑ i exp (simi(xu; fe, g)/τ) , (9)
where τ is a (temperature) hyperparameter.
At first glance, assigning a label of Y to xoutu may seem counter-intuitive, as the true label of xoutu is not in Y by definition. However, even when out-of-class samples cannot be represented as one-hot labels, one can still model their class-conditional likelihoods as a linear combination (i.e., soft-label) of Y: for instance, although “cat” images are out-of-class for CIFAR-100, still there are some classes in CIFAR-100 that is semantically similar to “cat”, e.g., “leopard”, “lion”, or “tiger”, so that assigning a soft-label, e.g., 0.1 · “leopard” + 0.2 · “lion” + 0.7 · “tiger”, might be beneficial. Even if out-of-classes are totally different from in-classes, one can assign the uniform labels to ignore them. We empirically found that such soft-labels based on representations learned via contrastive learning offer an effective way to utilize out-of-class samples, while they are known to significantly harm in the vanilla semi-supervised learning schemes. We present detailed discussion on our soft-label assignments in Section 4.4.
Auxiliary batch normalization. Finally, we suggest to handle a data-distribution shift originated from the class-distribution mismatch (Oliver et al., 2018), i.e., Dl and Doutu are drawn from the different underlying distribution. This may degrade the in-class classification performance as the auxiliary loss utilizes out-of-class samples. To handle the issue, we use additional batch normalization layers (BN; Sergey Ioffe 2015) for training samples in Doutu to disentangle those two distributions. In our experiments, we observe such auxiliary BNs are beneficial when using out-of-class samples via the auxiliary loss (see Section 4.4). Auxiliary BNs also have been studied in adversarial learning literature (Xie et al., 2020): decoupling BNs improves the performance of adversarial training by handling a distribution mismatch between clean and adversarial samples. In this paper, we found that a similar strategy can improve model performance in realistic semi-supervised learning.

heading: 4 EXPERIMENTS
text: In this section, we verify the effectiveness of our method over a wide range of semi-supervised learning benchmarks in the presence of various out-of-class data. The full details on experimental setups can be found in Appendix A.
Datasets. We perform experiments on image classification tasks for several benchmarks in the literature of semi-supervised learning (Berthelot et al., 2020; Sohn et al., 2020): CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) datasets. Specifically, we focus on settings where each dataset is extremely label-scarce: only 4 or 25 labels per class are given during training, while the rest of the training data are assumed to be unlabeled. To configure realistic semi-supervised learning scenarios, we additionally assume that unlabeled data contain samples from an external dataset: for example, in the case of CIFAR-10, we use unlabeled samples from SVHN (Netzer et al., 2011) or TinyImageNet3 datasets.
Baselines. We evaluate MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020), and FixMatch (Sohn et al., 2020) as baselines in our experimental setup, which are considered to be
3https://tiny-imagenet.herokuapp.com/
state-of-the-art methods in conventional semi-supervised learning. We also compare our method with three prior works applicable to our setting: namely, we consider Uncertainty-Aware Self-Distillation (UASD; Chen et al. 2020c), RealMix (Nair et al., 2019) and DS3L (Guo et al., 2020), which propose schemes to detect and filter out out-of-class samples in the unlabeled dataset: e.g., DS3L learns to re-weight unlabeled samples to reduce the effect of out-of-class samples. Recall that our method uses SimCLR (Chen et al., 2020a) for pre-training. Unless otherwise noted, we also pre-train the baselines via SimCLR for a fair comparison, denoting those fine-tuned models by “-ft,” e.g., MixMatch-ft and UASD-ft. We confirm that fine-tuned models show comparable or better performance compared to those trained from scratch, as presented in Figure 1(b) and Appendix A.2. Also, we report the performance purely obtainable from (unsupervised) SimCLR: namely, we additionally consider (a) SimCLR-le: a SimCLR model with linear evaluation protocol (Zhang et al., 2016; Chen et al., 2020a), i.e., it additionally learns a linear layer with the labeled dataset, and (b) SimCLR-ft: the whole SimCLR model is fine-tuned with the labeled dataset. Somewhat interestingly, these models turn out to be the strongest baselines in our setups; they often outperform the state-of-the-art semi-supervised baselines under large proportions of out-of-class samples (see Table 1). Finally, we remark that our framework can incorporate any conventional semi-supervised methods for training. We denote our method built upon an existing method by “+ OpenCoS”, e.g., ReMixMatch + OpenCoS.
Training details. As suggested by Oliver et al. (2018), we have re-implemented all baseline methods considered, including SimCLR, under the same codebase and performed experiments with the same model architecture of ResNet-50 (He et al., 2016).4 Due to the label-scarce nature of semi-supervised learning, we do not use a validation set in our setting. Instead, we checkpoint per 216 training samples and report (a) the median test accuracy of the last 5 checkpoints out of 50 checkpoints in total and (b) the best accuracy among all the checkpoints. We fix τ = 1, the temperature hyperparameter in (9), and λ = 0.5 in (8), in all our experiments. The full details on model architecture and hyperparameters can be found in Appendix A.2 and B.1, respectively.

heading: 4.1 EXPERIMENTS ON VARYING PROPORTIONS OF OUT-OF-CLASS SAMPLES
text: We first evaluate the effect of out-of-class unlabeled samples in semi-supervised learning, on varying proportions to the total dataset. We consider CIFAR-10 and TinyImageNet datasets, and synthetically control the proportion between the two in 50K training samples. For example, 80% of proportion means the training dataset consists of 40K samples from TinyImageNet, and 10K samples from CIFAR-10. In this experiment, we assume that 25 labels per class are always given in the CIFAR10 side. We compare three models on varying proportions of out-of-class: (a) a ReMixMatch model trained from scratch (ReMixMatch), (b) a SimCLR model fine-tuned by ReMixMatch (ReMixMatch-ft), and (c) our OpenCoS model applied to ReMixMatch-ft (+ OpenCoS).
Figure 1(b) demonstrates the results. Overall, we observe that the performance of ReMixMatch rapidly degrades as the proportion of out-of-class samples increases in unlabeled data. While ReMixMatch-ft significantly mitigates this problem, however, it still fails at a larger proportion: e.g., at 80% of out-of-class, the performance of ReMixMatch-ft falls into that of ReMixMatch. OpenCoS, in contrast, successfully prevents the performance degradation of ReMixMatch-ft, especially at the regime that out-of-class samples dominate in-class samples.

heading: 4.2 EXPERIMENTS ON CIFAR DATASETS
text: In this section, we evaluate our method on several benchmarks where CIFAR datasets are assumed to be in-class: more specifically, we consider scenarios that either CIFAR-10 or CIFAR-100 is an in-class dataset, with an out-of-class dataset of either SVHN or TinyImageNet. Additionally, we also consider a separate benchmark called CIFAR-Animals + CIFAR-Others following the setup in the related work (Oliver et al., 2018): the in-class dataset consists of 6 animal classes from CIFAR-10, while the remaining samples are considered as out-of-class. We fix every benchmark to have 50K training samples. We assume an 80% proportion of out-of-class, i.e., 10K for in-class and 40K for out-of-class samples, except for CIFAR-Animals + CIFAR-Others, which consists of 30K and 20K samples for in- and out-of-class, respectively. We report ReMixMatch-ft + OpenCoS as it tends to outperform FixMatch-ft + OpenCoS in such CIFAR-scale experiments, while FixMatch-ft + OpenCoS does in the large-scale ImageNet experiments in Section 4.3. Table 1 shows the results: OpenCoS consistently
4Note that this architecture is larger than Wide-ResNet-28-2 (Zagoruyko & Komodakis, 2016) used in the semi-supervised learning literature (Oliver et al., 2018). We use ResNet-50 following the standard of SimCLR.
improves ReMixMatch-ft, outperforming the other baselines simultaneously. For example, OpenCoS improves the test accuracy of ReMixMatch-ft 28.51%→ 68.37% on 4 labels per class of CIFAR-10 + TinyImageNet. Also, we observe large discrepancies between the median and best accuracy of semi-supervised learning baselines, MixMatch-ft, ReMixMatch-ft, and FixMatch-ft, especially in the extreme label-scarce scenario of 4 labels per class, i.e., these methods suffer from over-fitting on out-of-class samples. One can also confirm this significant over-fitting in state-of-the-art SSL methods by comparing other baselines with detection schemes, e.g., USAD-ft, RealMix-ft, and DS3L-ft, which show less over-fitting but with lower best accuracy.

heading: 4.3 EXPERIMENTS ON IMAGENET DATASETS
text: We also evaluate OpenCoS on ImageNet to verify its scalability to a larger and more complex dataset. We design 9 benchmarks from ImageNet dataset, similarly to Restricted ImageNet (Tsipras et al., 2019): more specifically, we define 9 super-classes of ImageNet, each of which consists of 11∼118 sub-classes. We perform our experiments on each super-class as an individual dataset. Each of the benchmarks (a super-class) contains 25 labels per sub-class, and we use the full ImageNet as an unlabeled dataset (excluding the labeled ones). In this experiment, we checkpoint per 215 training samples and report the median test accuracy of the last 3 out of 10. We present additional experimental details, e.g., configuration of the dataset, in Appendix A.3. Table 2 shows the results: OpenCoS still effectively improves the baselines, largely surpassing SimCLR-le and SimCLR-ft as well. For example, OpenCoS improves the test accuracy on Bird to 81.78% from FixMatch-ft of 78.73%, also improving SimCLR-le of 75.81% significantly. This shows the efficacy of OpenCoS in exploiting open-set unlabeled data from unknown (but related) classes or even unseen distribution of another dataset in the real-world.

heading: 4.4 ABLATION STUDY
text: We perform an ablation study to understand further how OpenCoS works. Specifically, we assess the individual effects of the components in OpenCoS and show that each of them has an orthogonal contribution to the overall improvements. We also provide a detailed evaluation of our proposed detection score (7) compared to other out-of-distribution detection methods.
Component analysis. To further analyze the individual contribution of each component of OpenCoS, we incrementally apply these components one-by-one to ReMixMatch-ft (CIFAR-scale) and FixMatch-ft (ImageNet-scale) baselines. Specifically, we consider CIFAR-Animals + CIFAR-Others, CIFAR-10 + SVHN for CIFAR-scale, and Produce, Bird, Food + ImageNet for ImageNet-scale benchmarks. Table 3 summarizes the results, and they indeed confirm that each of what comprises OpenCoS has an orthogonal contribution to improve the accuracy of the benchmarks tested. We observe that leveraging out-of-class samples via auxiliary loss (“Aux. loss”) achieves consistent improvements, and also outperforms the baselines significantly. Finally, we remark auxiliary batch normalization layers (“Aux. BNs”) give a consistent improvement, and it is often significant: e.g., it gives 55.78%→ 57.77% on CIFAR-10 + SVHN. Other detection scores. In Section 3.2, we propose a detection score s(·) (7) for detecting out-ofclass samples in an unlabeled dataset, based on the contrastive representation of SimCLR. This setup is different to the standard out-of-distribution (OOD) detection task (Emmott et al., 2013; Liu et al., 2018): OOD detection targets unseen (i.e., “out-of-distribution”) samples in test time, while our setup aims to detect seen out-of-class samples during training assuming few in-class labels. Due to this lack of labeled information, therefore, the standard techniques for OOD detection (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018b) are not guaranteed to perform still well in our setup. We examine this in Appendix B.3 by comparing detection performance of such OOD detection scores with ours (7) upon a shared SimCLR representation: in short, we indeed observe that our approach of directly leveraging the contrastive representation could perform better than simply applying OOD scores relying on few labeled samples, e.g., our score achieves an AUROC of 98.10% on the CIFAR-Animals + CIFAR-Others benchmark compared to the maximum softmax probability based score (Hendrycks & Gimpel, 2017) of 80.79%. We present the detailed experimental setups and more results in Appendix B.3.
Effect of soft-labeling. We emphasize that our soft-labeling scheme can be rather viewed as a more reasonable way to label such out-of-class samples compared to existing state-of-the-art SSL methods, e.g., MixMatch simply assigns its sharpened predictions. On the other hand, a prior work (Li & Hoiem, 2016) has a similar observation to our approach: assigning soft-labels of novel data could be beneficial for transfer learning. This motivate us to consider an experiment to further support the claim that our soft-labeling gives informative signals: we train a classifier by minimizing only the cross-entropy loss with soft-labels (i.e., without in-class samples) from scratch. In Table 4, the trained
classifier performs much better than (random) guessing, even close to some baselines although this model does; this supports that generated soft-labels contain informative features of in-classes. The details on experimental setups can be found in Appendix A.2.
Examples of actual soft-labels. We also present some concrete examples of our soft-labeling scheme in Figure 3 for a better understanding, which are obtained from random unlabeled samples in the CIFAR-10 + TinyImageNet benchmark: Overall, we qualitatively observe that out-of-class samples that share some semantic features to the in-classes (e.g., Figure 3(a)) have relatively high confidence capturing such similarity, while returning very close to uniform otherwise (e.g., Figure 3(b)).

heading: 5 CONCLUSION
text: In this paper, we propose a simple and general framework for handling novel unlabeled data, aiming toward a more realistic assumption for semi-supervised learning. Our key idea is (intentionally) not to use label information, i.e., by relying on unsupervised representation, when handling novel data, which can be naturally incorporated into semi-supervised learning with our framework: OpenCoS. In contrast to previous approaches, OpenCoS opens a way to further utilize those novel data by assigning them soft-labels, which are again obtained from unsupervised learning. We hope our work would motivate researchers to extend this framework with a more realistic assumption, e.g., noisy labels (Wang et al., 2018; Lee et al., 2019), imbalanced learning (Liu et al., 2020).

heading: A TRAINING DETAILS
text: A.1 DETAILS ON THE EXPERIMENTAL SETUP
For the experiments reported in Table 1, we generally follow the training details of FixMatch (Sohn et al., 2020), including optimizer, learning rate schedule, and an exponential moving average. Specifically, we use Nesterov SGD optimizer with momentum 0.9, a cosine learning rate decay with an initial learning rate of 0.03, and an exponential moving average with a decay of 0.999. The batch size is 64, which is widely adopted in semi-supervised learning (SSL) methods. We do not use weight decay for these models, as they are fine-tuned. We use a simple augmentation strategy, i.e., flip and crop, as a default. We use the augmentation scheme of SimCLR (Chen et al., 2020a) (i.e., random crop with resize, random color distortion, and random Gaussian blur) when a SSL method requires to specify a strong augmentation strategy, e.g., for consistency regularization in the SSL literature (Berthelot et al., 2020; Sohn et al., 2020). We fix the number of augmentations as 2, following Berthelot et al. (2019): e.g., MixMatch-ft generates two augmentations of each unlabeled sample while ReMixMatch-ft generates one weak augmentation and one strong augmentation. In the case of ReMixMatch-ft, we do not use the ramp-up weighting function, the pre-mixup and the rotation loss, which give a marginal difference in fine-tuning, for efficient computation.5 For FixMatch-ft, we set the relative size of labeled and unlabeled batch µ = 1 for a fair comparison with other baselines, and scale the learning rate linearly with µ, as suggested by Sohn et al. (2020). Following Chen et al. (2020c), UASD-ft computes the predictions by accumulative ensembling instead of using an exponential moving average. OpenCoS shares all hyperparameters of the baseline SSL methods, e.g., FixMatch + OpenCoS shares hyperparameters of FixMatch-ft. For the results of ReMixMatch (from scratch) in Figure 1(b), we report the median accuracy of the last 10 checkpoints out of 200 checkpoints, where each checkpoint is saved per 216 training samples.
A.2 CIFAR EXPERIMENTS
Training from scratch. We pre-train all the baselines via SimCLR for a fair comparison, as mentioned in Section 4. In Table 5, we also report the performance of each baseline model when trained from scratch. Here, we report the median accuracy of the last 10 checkpoints out of 500 checkpoints in total. We also present the fine-tuned baselines (see Section 4.2) denoting by “-ft,” e.g., MixMatch-ft. Here, we follow the training details those which originally used in each baseline method. For example, ReMixMatch from scratch uses Adam optimizer with a fixed learning rate of 0.002, and weight decay of 0.02. We use the simple strategy (i.e., flip and crop) and RandAugment (Cubuk et al., 2019) as a weak and strong augmentation, respectively. In addition, we use the ramp-up weighting function, the pre-mixup and the rotation loss for ReMixMatch. We consider the CIFAR-100 + TinyImageNet benchmark assuming 80% proportion of out-of-class, i.e., 10K samples for in-class and 40K samples for out-of-class.
Training without in-class samples from scratch. For the experiments reported in Table 4, we train a classifier from scratch only with the unlabeled out-of-class samples and their soft-labels, on the CIFAR-10 benchmarks with 4 labels per class. We use ResNet-50, and SGD with momentum 0.9, weight decay 0.0001, and an initial learning rate of 0.1. The learning rate is divided by 10 after epochs 100 and 150, and total epochs are 200. We set batch size as 128, and use a simple data augmentation strategy, i.e., flip and crop. We minimize the cross-entropy loss between the soft-labels q(·) and model predictions f(·), i.e., L = H(q(xoutu ), f(xoutu )). We use the temperature scaling on the both sides of soft-labels and model predictions for a stable training, specifically 0.1 and 4, respectively.
Analysis of model architectures. For all our experiments, we use ResNet-50 following the standard of SimCLR (Chen et al., 2020a). This architecture is larger than Wide-ResNet-28-2 (Zagoruyko & Komodakis, 2016), a more widely adopted architecture in the semi-supervised learning literature (Oliver et al., 2018). We have found that using a larger network, i.e., ResNet-50, is necessary to leverage the pre-trained features of SimCLR: In Table 5, we provide an evaluation on another choice of model architecture, i.e., Wide-ResNet-28-2. The hyperparameters are the same as the experiments on ResNet-50. Here, one can observe that OpenCoS trained on Wide-ResNet-28-2 still improves ReMixMatch-ft, outperforming the other baselines. More importantly, however, we observe that pre-training Wide-ResNet-28-2 via SimCLR does not significantly improve the baselines trained from
5For a fair comparison, ReMixMatch-ft + OpenCoS shares these settings.
scratch, contrary to the results of ResNet-50. As also explored by Chen et al. (2020a), we suspect this is due to that pre-training via SimCLR requires a larger model in practice, and suggest future SSL research to explore larger architectures to incorporate more rich features into their methods, e.g., features learned via unsupervised learning (Hénaff et al., 2019; Chen et al., 2020a;b).
Experiments on more labeled data. We have performed additional experiments on the CIFAR-10 + SVHN benchmark with 400 labels per class, and the results are given in Table 6. One can still observe that OpenCoS consistently outperforms other methods when more labeled data are available.
A.3 IMAGENET EXPERIMENTS
Benchmarks of ImageNet dataset. In Section 4.3, we introduce 9 benchmarks from ImageNet dataset, similar to Restricted ImageNet (Tsipras et al., 2019). In detail, we group together subsets of semantically similar classes into 9 different super-classes, as shown in Table 7.
Details of ImageNet experiments. For the experiments reported in Table 2, we use a pre-trained ResNet-50 model6 of Chen et al. (2020a) and fine-tune the projection header for 5 epochs on ImageNet. We follow the optimization details of the fine-tuning experiments of SimCLR (Chen et al., 2020a): specifically, we use Nesterov SGD optimizer with momentum 0.9, and a learning rate of 0.00625 (following LearningRate = 0.05 · BatchSize/256). We set the batch size to 32, and report the median accuracy of the last 3 checkpoints out of 10 checkpoints in total. Data augmentation, regularization techniques, and other hyperparameters are the same as CIFAR experiments. In the case of FixMatch-ft + OpenCoS, we empirically observe that it is more beneficial not to discard the detected out-of-class samples in FixMatch training, as it performs better than using in-class samples
6https://github.com/google-research/simclr
only: the auxiliary loss still use only out-of-class samples. Since FixMatch filters low-confidence unlabeled samples out, it is possibly due to a decrease in the number of training data.

heading: B ABLATION STUDY
text: B.1 EFFECTS OF THE TEMPERATURE AND THE LOSS WEIGHT
In Section 4, we perform all the experiments with the fixed temperature τ = 1 and loss weight λ = 0.5. To examine the effect of hyperparameters τ and λ, we additionally test the hyperparameters across an array of τ ∈ {0.1, 0.5, 1, 2, 4} and λ ∈ {0.1, 0.5, 1, 2, 4} on the CIFAR-100 + TinyImageNet benchmark with ResNet-50. The results are presented in Table 8. Overall, we found our method is fairly robust on τ and λ.
B.2 EFFECTS OF OUT-OF-CLASS SAMPLES IN CONTRASTIVE LEARNING
To clarify how the improvements of OpenCoS comes from out-of-class samples, we have considered additional CIFAR-scale experiments with 4 labels per class. We newly pre-train and fine-tune SimCLR models using in-class samples only, i.e., 30,000 for CIFAR-Animals, 10,000 for CIFAR-10, CIFAR-100 benchmarks, and compare two baselines: SimCLR-le and ReMixMatch-ft. Interestingly, we found that just merging out-of-class samples to the training dataset improves the performance of SimCLR models in several cases (see Table 9), e.g., SimCLR-le of CIFAR-10 enhances from 55.27% to 58.20% with TinyImageNet. Also, OpenCoS significantly outperforms overall baselines, even when out-of-class samples hurt the performance of SimCLR-le or ReMixMatch-ft. We confirm that the proposed method effectively utilizes contrastive representations of out-of-class samples beneficially, compared to other SSL baselines.
Robustness to incorrect detection. We observe our method is quite robust on incorrectly detected out-of-class samples, i.e., those samples are still leveraged via auxiliary loss instead of SSL algorithm. We have considered an additional experiment on CIFAR-10 with 250 labels (out of 50,000 samples), that assumes (i) all the unlabeled samples are in-class, and (ii) 80% of those in-class samples are
incorrectly detected as out-of-class in OpenCoS. Here, we compare OpenCoS with a baseline which only uses the correctly-detected (in-class) samples without auxiliary loss, i.e., the baseline is trained on 10,000 samples while OpenCoS on 50,000 in total. In this scenario, OpenCoS achieves 89.54% in the median test accuracy, while the baseline does 89.27%: this shows that our auxiliary loss does not harm the training even when it is incorrectly applied to in-class samples.
B.3 EVALUATIONS OF OUR DETECTION SCORE
Baselines. We consider maximum softmax probability (MSP; Hendrycks & Gimpel 2017), ODIN (Liang et al., 2018), and Mahalanobis distance-based score (Lee et al., 2018b) as baseline detection methods. As MSP and ODIN require a classifier to obtain their scores, we employ SimCLR-le: a SimCLR model which additionally learns a linear layer with the labeled dataset, for both baselines.
ODIN performs an input pre-processing by adding small perturbations with a temperature scaling as follows:
P (y = c|x;T ) = exp(fc(x)/T )∑ y exp(fy(x)/T ) , x′ = x− · sign(−∇x logP (y = c|x;T )), (10)
where f = (f1, ..., fC) is the logit vector of deep neural network, T > 0 is a temperature scaling parameter, and is a magnitude of noise. ODIN calculates the pre-processed data x′ and feeds it into the classifier to compute the confidence score, i.e., maxy P (y|x′;T ), and identifies it as in-class if the confidence score is higher than some threshold δ. We choose the temperature T and the noise magnitude from {1, 10, 100, 1000} and {0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.2}, respectively, by using 2,000 validation data.
Mahalanobis distance-based score (Mahalanobis) assumes the features of the neural network f follows the class-conditional Gaussian distribution. Then, it computes Mahalanobis distance between input x and the closest class-conditional Gaussian distribution, i.e.,
M(x) = max c −(f(x)− µc)>Σ−1(f(x)− µc), (11)
where µc is the class mean and Σ is the covariance of the labeled data. We fix the covariance matrix as the identity because the number of labeled samples is insufficient to compute it: the feature dimension of SimCLR encoder fe and projection header g are 2048. Moreover, Mahalanobis has the noise magnitude parameter for input pre-processing like ODIN, and use a feature ensemble method of Lee et al. (2018b). We choose from {0, 0.0005, 0.001, 0.0014, 0.002, 0.005, 0.01}, and perform the feature ensemble of intermediate features including fe’s and g’s by using 2,000 validation data.
Metrics. We follow the threshold-free detection metrics used in Lee et al. (2018b) to measure the effectiveness of detection scores in identifying out-of-class samples.
• True negative rate (TNR) at 95% true positive rate (TPR). We denote true positive, true negative, false positive, and false negative as TP, TN, FP, and FN, respectively. We measure TNR = TN / (FP+TN) at TPR = TP / (TP+FN) is 95%.
Detection method TNR at TPR 95% ↑ Detection Accuracy ↑ AUROC ↑ AUPR-in ↑ AUPR-out ↑ MSP 18.20 72.81 80.79 88.72 66.38 ODIN 24.10 79.41 86.55 92.55 72.75 Mahalanobis 90.90 93.57 97.48 98.60 94.36 Ours w/o header 40.44 80.11 88.80 93.46 79.35 Ours 91.20 93.49 98.10 98.79 96.86
In this section, we present evaluations of our detection score s(·) (7) under various detection metrics on the CIFAR-Animals + CIFAR-Others with 4 labels per class. Table 10 shows the results: interestingly, our score outperforms MSP and ODIN and also performs comparable to Mahalanobis, even these baselines require more computational costs, e.g., input pre-processing. We confirm that the design of our score is an effective and efficient way to detect out-of-class samples based on the representation of SimCLR. In Figure 4, we provide the receiver operating characteristic (ROC) curves that support the above results. We remark that the projection header g (4) is crucial for the detection, e.g., g enhances AUROC of our score from 88.80% to 98.10%. According to the definition of our score (7), it can be viewed as a simpler version of Mahalanobis without its input pre-processing and feature ensembles under an assumption of identity covariance, which may explain their comparable performances.
We additionally provide the performance of OpenCoS among various detection methods, including the above baselines and two artificial methods: we consider (a) Random: a random detection with a probability of 0.5, and (b) Oracle: a perfect detection. For MSP, ODIN, and Mahalanobis, we choose their detection thresholds at TPR 95%. Table 11 shows the results: we observe that the classification accuracy is proportional to the detection performance. Remarkably, our detection method achieves comparable accuracy to Oracle, which is the optimal performance of OpenCoS.
B.4 EVALUATIONS OF THE DETECTION THRESHOLD
We additionally provide the detection performance on various proportions of out-of-class samples, i.e., 50% and 67%, on this benchmark. For each setting, the number of out-of-class samples is fixed at 20K, while in-class samples are controlled to 20K and 10K, respectively. We choose the same detection threshold t := µl − 2σl throughout all experiments: it is a reasonable choice, giving ≈ 95% confidence if the score follows Gaussian distribution. Table 12(a) shows the detection performance of our threshold and its applicability over various proportions of out-of-class samples. Although tuning t gives further improvements though (see Table 12(b)), we fix the threshold without any tuning.
Table 12: The detection performance across different (a) proportions of out-of-class and (b) detection thresholds in CIFAR-Animals + CIFAR-Others benchmark with 4 labels per class.
(a) The detection performance of the proposed threshold t := µl − 2σl on varying proportions of out-of-class.
Proportion of out-of-class 40% 50% 67%
True Positive Rate (TPR) 63.61 72.19 81.10 True Negative Rate (TNR) 99.76 99.66 99.55
AUROC 98.10 98.50 98.90
(b) The detection performance and median test accuracy across different thresholds t, i.e., k = 1, 2, 3, 4 of t = µl − k · σl. k 1 2 3 4
True Positive Rate (TPR) 37.48 63.61 85.55 98.62 True Negative Rate (TNR) 99.94 99.76 97.22 74.48
Accuracy 75.47 80.67 81.12 78.37

heading: C ALGORITHM
text: The full training procedure of OpenCoS is summarized in Algorithm 1.
Algorithm 1 OpenCoS: A general framework for open-set semi-supervised learning (SSL). 1: Input: The classifier f , the encoder fe (i.e., penultimate features of f ), and the projection header g. A labeled dataset Dl and an open-set unlabeled data Du.
2: Pre-train fe and g via SimCLR. 3: Sl = φ 4: for each labeled sample xl ∈ Dl do 5: Sl = Sl ∪ {s(xl; fe, g)} . The similarity score (7). 6: end for 7: t := E [Sl]− 2 √ Var [Sl] . Compute the threshold t. 8: 9: Dinu = φ, Doutu = φ 10: for each unlabeled sample xu ∈ Du do 11: if s(xu; fe, g) < t then 12: Doutu = Doutu ∪ {xu} . Detect out-of-class unlabeled samples. 13: else 14: Dinu = Dinu ∪ {xu} 15: end if 16: end for 17: 18: for each sample xl ∈ Dl, xinu ∈ Dinu , and xoutu ∈ Doutu do 19: q(xoutu )← Compute a soft-label of xoutu . Using contrastive representations (9). 20: LOpenCoS = LSSL(xl, xinu ; f) + λ ·H(q(xoutu ), f(xoutu )) . SSL with auxiliary loss (8). 21: Update parameters of f by computing the gradients of the proposed loss LOpenCoS. 22: end for

Review summaries:
The paper proposes a new approach for open set semi-supervised learning, where there are unlabeled data from classes not in the labeled data. The paper uses a contrastive representation learning paradigm to learn a feature encoder and a similarity measurement. Then the paper filters outlier samples by the similarity measurement and further utilizes outlier samples with soft labels. The separate BN layers address the distribution shift between in-class and out-class data.

This paper considers the problem of semi-supervised learning, where the unlabeled data may include out-of-class samples. To address this task, the paper proposes a method consisting of three steps: (1) detecting out-of-class samples in the unlabeled set, (2) assigning soft-labels to the detected out-of-class samples using class-conditional likelihoods from labeled data, and (3) using auxiliary batch normalization layers  to help mitigate the class distribution mismatch problem. Experiments are conducted on CIFAR-10, CIFAR-100, ImageNet datasets. Results show improvements over competing methods.