Year: 2021

Abstract:
In open-set recognition (OSR), classifiers should be able to reject unknown-class samples while maintaining robust closed-set classification performance. To solve the OSR problem based on pre-trained Softmax classifiers, previous studies investigated offline analyses, e.g., distance-based sample rejection, which can limit the feature space of known-class data items. Since such classifiers are trained solely based on known-class samples, one can use background class regularization (BCR), which employs background-class data as surrogates of unknown-class ones during training phase, to enhance OSR performance. However, previous regularization methods have limited OSR performance, since they categorized known-class data into a single group and then aimed to distinguish them from anomalies. In this paper, we propose a novel distance-based BCR method suitable for OSR, which limits the feature space of known-class data in a class-wise manner and then makes background-class samples located far away from the limited feature space. Instead of conventional Softmax classifiers, we use distance-based classifiers, which utilize the principle of linear discriminant analysis. Based on the distance measure used for classification, we design a novel regularization loss function that can contrast known-class and background-class samples while keeping robust closed-set classification performance. Through our extensive experiments, we show that the proposed method provides robust OSR results with a simple inference process.

heading: 1 INTRODUCTION
text: In machine learning (ML), classification algorithms have achieved great success. Through recent advances in convolutional neural networks, their classification performance already surpassed the human-level performance in image classification (He et al., 2015). However, such algorithms are usually developed under a closed-set assumption, i.e., the class of each test sample is assumed to always belong to one of the pre-defined set of classes. Although this conventional assumption can be easily violated in real-world applications (classifiers can face unknown-class data), traditional algorithms are likely to force unknown-class samples to be classified into one of the known classes. To tackle this issue, the open-set recognition (OSR) problem (Scheirer et al., 2013) aims to properly classify unknown-class samples as “unknown” and known-class samples as one of the known classes.
According to the definition of OSR (Scheirer et al., 2013), it is required to properly limit the feature space of known-class data. To satisfy the requirement, various OSR methods were developed based on traditional ML models. Previously, Scheirer et al. (2014) calibrated the decision scores of support vector machines (SVMs). Based on the intuition that a large set of data samples of unknown classes can be rejected if those of known classes are accurately modeled, Jain et al. (2014) proposed PI -SVM, which utilized the statistical modeling of known-class samples located near the decision boundary of SVMs. Afterward, Júnior et al. (2016) attempted to solve the OSR problem based on the principle of the nearest neighbors. Taking distribution information of data into account, Rudd et al. (2018) proposed the extreme value machine by utilizing the concept of margin distributions.
Since deep neural networks (DNNs) have robust classification performance by learning high-level representations of data, OSR methods for DNNs have received great attention. Based on the theoretical foundations used in traditional ML-based OSR methods, Bendale & Boult (2016) proposed the first OSR strategy for DNNs called Openmax, which calibrates the output logits of pre-trained Softmax classifiers. To improve Openmax, Yoshihashi et al. (2019) proposed the classification-reconstruction
learning to make robust latent feature vectors. Afterward, Oza & Patel (2019) proposed to exploit a class-conditioned autoencoder and use its reconstruction error to assess each input sample. Sun et al. (2020) employed several class-conditioned variational auto-encoders for generative modeling.
Although previous OSR methods using discriminative models applied offline analyses to pre-trained Softmax classifiers or employed complicated DNN models, they have limited performance since the classifiers were trained solely based on known-class samples. To mitigate the problem, one can use background class regularization (BCR) to achieve robust empirical results. However, previous BCR methods (Dhamija et al., 2018; Hendrycks et al., 2019; Liu et al., 2020) are insufficient to properly solve the OSR problem. To design an effective open-set classifier that can overcome the previous limitations, we propose a novel BCR method suitable for OSR, which uses a distance-based classifier and a novel loss function for regularization. We provide detailed description in the next section.

heading: 2 PRELIMINARIES AND OUR CONTRIBUTIONS

heading: 2.1 THE OPEN-SET RECOGNITION PROBLEM
text: The OSR problem addresses the classification setting that can face test samples from classes unseen during training. In this setting, open-set classifiers aim to properly classify known-class samples while rejecting unknown-class samples simultaneously. A similar problem to OSR is out-of-distribution (OOD) detection (Hendrycks & Gimpel, 2017), which typically aims to reject data items drawn far away from the training data distribution. In previous OOD detection studies (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018a;b), OOD samples tend to be drawn from other datasets or even images of noise. In this paper, we aim to reject test data whose classes are unknown but related to the training data, which narrows down the scope of conventional OOD detection tasks.
Previously, Scheirer et al. (2013) introduced a formal definition of the OSR problem based on the notion of open-space risk RO, which is a relative measure of a positively labeled union of balls SV and open space O located far from SV . Since labeling any data item in O incurs open-space risk, it is straightforward that a classifier cannot be a solution for the OSR problem if the classifier accepts data in infinitely wide regions, i.e., its open-space risk is unbounded (RO = ∞). The definition implies that essential requirements to solve the OSR problem are 1) bounding open-space risk and 2) ideally balancing it with empirical risk to maintain a low classification error rate.
Unlike traditional closed-set classifiers, open-set classifiers are required to limit the space of knownclass data to bound their open-space risk. To ensure bounded open-space risk, Scheirer et al. (2014) proposed to formulate compact abating probability (CAP) models. The principle of CAP models is that if the support region of a classifier decays in all directions from the training data, thresholding the region will bound the classifier’s open-space risk (Boult et al., 2019). As depicted in Figure 1, which compares traditional closed-set and open-set classification problems, properly building class-wise CAP models is an effective strategy to satisfy the two essential requirements of OSR.

heading: 2.2 POST-CLASSIFICATION ANALYSIS FOR PRE-TRAINED SOFTMAX CLASSIFIER
text: In this paper, we aim to solve the OSR problem by using a standard DNN-based classifier architecture f as a latent feature extractor. Applying a fully-connected layer to f , a conventional Softmax classifier
computes the posterior probability of an input x belonging to the c-th known class by
Ps(y = c|x) = exp(wTc f(x) + bc)∑C i=1 exp(w T i f(x) + bi) , (1)
where c ∈ {1, · · · , C}, f(x) ∈ Rn is the latent feature vector of x, and wc and bc are the weight and the bias for the c-th class, respectively. For pre-trained Softmax classifiers, Hendrycks & Gimpel (2017) proposed a baseline to detect anomalous samples, which imposes a threshold on the predictive confidence of Eq. (1). When using the baseline method to solve the OSR problem, one can estimate the class of each known-class sample and recognize unknown-class data by
ŷ = { argmaxc∈{1,··· ,C} Ps(y = c|x), if maxc∈{1,··· ,C} Ps(y = c|x) ≥ τ, C + 1 (unknown class), otherwise.
(2)
However, Eq. (2) cannot formally bound open-space risk since it only rejects test samples near the decision boundary of classifiers, thereby having infinitely wide regions of acceptance (Boult et al., 2019). Therefore, additional post-classification analysis methods using an auxiliary measure other than the Softmax probability are necessary to bound open-space risk, where distance measures have been widely used to build auxiliary CAP models in the latent feature space of f .
To construct class-wise CAP models, Openmax (Bendale & Boult, 2016) defined radial-basis decaying functions {s(x, i)}Ci=1, each of which measures the class-belongingness of x for the c-th class, in the latent feature space of f . For each s(x, c), the authors employed distance measures between f(x) and an empirical class mean vector µc, e.g., s(x, c) = D2E(f(x),µc) = (f(x)− µc)T (f(x)− µc). To formulate effective CAP models based on s(x, c), they statistically modeled the distribution of s(x, c) for known-class data based on the extreme value theory (EVT) (Scheirer, 2017), which provides a theoretical foundation that the Weibull distribution is suitable for modeling known-class samples located far from the class mean vectors (extreme samples). To be specific, Openmax fits a Weibull distribution on extreme samples of the c-th class having the highest DE(f(x),µc) values, where its cumulative distribution function (CDF) formulates the probability of inclusion PI(x, c) (Jain et al., 2014; Rudd et al., 2018), i.e., PI(x, c) = 1 − WeibullCDF, which rapidly decays near extreme samples. Based on PI(x, c), the decision rule of Eq. (2) can be calibrated to conduct robust OSR.

heading: 2.3 GENERALIZED OPEN-SET RECOGNITION AND BACKGROUND CLASS REGULARIZATION
text: Although they require additional inference procedures (e.g., EVT modeling) to bound the open-space risk of pre-trained Softmax classifiers, previous post-classification analyses may have limited OSR performance since the classifiers are trained solely based on known-class samples. To empirically obtain robust OSR results without complicated post-classification analyses, one can use the strategy of BCR at the training phase, which exploits background-class samples as surrogates of unknownclass samples. Geng et al. (2020) argued that the generalized OSR setting that utilizes background samples is still less-explored and an important research direction to improve OSR performance. In the generalized OSR setting, classifiers should consider the following data classes among the infinite label space of all classes Y (Dhamija et al., 2018; Geng et al., 2020).
• Known known classes (KKCs; K = {1, · · · , C} ⊂ Y) include distinctly labeled positive classes, which also have side information, where U = Y \ K is the entire unknown classes. • Known unknown classes (KUCs; B ⊂ U ) include background classes, e.g., labeled negative classes which are not necessarily grouped into a pre-defined set of KKCs K. • Unknown unknown classes (UUCs; A = U \B) represent the rest of U , where UUC samples are not available at training time and have no side information, but occur at inference time.
Throughout this paper, we denote the corresponding datasets for the data classes as follows. Dt is a training set consisting of multiple pairs of a KKC data sample and the corresponding class label y ∈ {1, · · · , C}. Dktest and Dutest are test sets of KKCs and UUCs, respectively. Db is a background dataset of KUCs. Then, a loss function for training classifiers with BCR can be formulated by
L = Lcf + λLbg = E(xk,y)∼Dt [ − logPs(y|xk) + λExb∼Db [ freg ( xk, y,xb )]] , (3)
where Lcf and Lbg are the losses for classification and BCR, respectively, and λ is a hyperparameter that balances Lcf and Lbg. For Lbg, previous studies designed their own freg, where Dhamija et al.
(2018) proposed the objectosphere loss for OSR, and Hendrycks et al. (2019) and Liu et al. (2020) employed the uniformity and the energy losses for OOD detection, respectively.
In this paper, we tackle the following limitations of the previous BCR methods.
• In the previous BCR methods, Lbg were designed to make normal data and anomalies more distinguishable in terms of the corresponding anomaly scores. Since they categorized normal samples into a single group (do not consider the classes) in Lbg , the previous methods cannot be suitable for rejecting UUC samples located relatively close to KKC data and maintaining robust closed-set classification results. • The previous methods using the decision rule of Eq. (2) (e.g., objectosphere (Dhamija et al., 2018) and uniformity (Hendrycks et al., 2019)) cannot bound open-space risk. Although one can use post-classification analyses to bound open-space risk, trained latent feature spaces are likely to be inappropriate for using another metric such as distance measures. • To increase the gap between KKC and KUC samples in terms of latent feature magnitude and energy value in the objectosphere (Dhamija et al., 2018) and the energy (Liu et al., 2020) losses, respectively, it is necessary to find proper margin parameters for each dataset.

heading: 2.4 OUR CONTRIBUTIONS
text: Based on a standard DNN-based classifier architecture f , we aim to design open-set classifiers having a simple yet effective inference process by using the principle of BCR. To overcome the limitations described in Section 2.3, we use distance-based classifiers and propose a novel BCR strategy based on the framework of Eq. (3). In the following, we summarize our proposed strategy.
• Instead of applying fully-connected layers to feature extractors f , we use the principle of linear discriminant analysis (LDA) (Murphy, 2012) to classify each input based on a distance measure. By imposing a threshold on the distance measure as in Eq. (2), such distance-based classifiers can bound their open-space risk without additional offline analyses. • To enhance the OSR performance of distance-based classifiers, we propose a novel distancebased BCR strategy. For Lbg , we design a loss function that does not require data-dependent margin parameters. Our method first limits the feature space of KKC data by formulating class-wise boundaries based on the concept of the probability of inclusion, and then forces KUC data to be located outside the entire boundaries. By imposing additional regularization on each KKC data item to be located inside the corresponding class-wise boundary, our proposed method can also maintain robust closed-set classification performance.

heading: 3 PROPOSED METHOD

heading: 3.1 DISTANCE-BASED CLASSIFICATION MODELS
text: To train a robust open-set classifier as we described in Section 2.4, we use a distance-based classifier
Pd(y = c|x) = P (y = c) · N (f(x)|µc, I)∑C i=1 P (y = i) · N (f(x)|µi, I) = P (y = c) · exp
( −D2E(f(x),µc) )∑C i=1 P (y = i) · exp (−D2E(f(x),µi)) (4)
as an alternative of Eq. (1), where Eq. (4) uses the principle of LDA in f(x). In Eq. (4), we exploit an identity covariance matrix I and P (y = c) = C−1 for all c for KKC data. Then, Eq. (4) classifies each x by computing D2E(f(x),µc) = (f(x)− µc)T (f(x)− µc), the Euclidean distance between f(x) and µc ∈ Rn, where we call µc a class-wise anchor. Instead of updating or computing empirical µc, we randomly sample each µc from the standard Gaussian distribution for all c and then fix it as an anchor during the training process. Such strategy also showed successful results in Izmailov et al. (2020), which formulated generative classifiers based on the principle of Gaussian mixture models.
At inference time, each KKC sample x can be classified by ŷ = argminc∈{1,··· ,C} D 2 E(f(x),µc). It is noteworthy that thresholding D2E(f(x),µc), which is the same metric used for classification, can bound the open-space risk of a distance-based classifier by formulating class-wise CAP models. Therefore, one can properly conduct OSR without additional post-classification analyses as follows:
ŷ =
{ argminc∈{1,··· ,C} D 2 E(f(x),µc), if maxc∈{1,··· ,C} −D2E(f(x),µc) ≥ τ,
C + 1 (unknown class), otherwise. (5)
Distance-based classification methods were also employed in prototypical networks (Snell et al., 2017), nearest class mean classifiers (Mensink et al., 2012), and the previous studies of the center loss function (Wen et al., 2019) and convolutional prototype classifiers (Yang et al., 2020). Furthermore, polyhedral conic classifiers (Cevikalp et al., 2021) used the idea of returning compact class regions for KKC samples based on distance-based feature analyses. It is noteworthy that our main contribution is a novel BCR method that can effectively utilize KUC samples in a distance-based classification scheme (described in Sections 3.2 and 3.3), not the distance-based classifier method itself. To the best of our knowledge, we are the first to discuss the necessity of distance-based BCR methods for OSR and propose a reasonable regularization method for distance-based classifiers.

heading: 3.2 BACKGROUND CLASS REGULARIZATION FOR DISTANCE-BASED CLASSIFIERS
text: To obtain robust OSR performance based on distance-based classifiers using the simple inference of Eq. (5), we aim to design a BCR method, which uses Dt and Db as surrogates of Dktest and Dutest at training time, respectively. Although it cannot provide any information of Dutest, Db can be effective to limit the latent feature space of KKC data, while reserving space for UUC samples. Note that the decision rule of Eq. (5) conducts both closed-set classification and unknown-class rejection based on a single distance measure D2E(f(x),µc). Thus, it is intuitive that the primary objective of our BCR method is to make KUC samples located far away from µi for all i ∈ {1, · · · , C}. Before we illustrate our BCR method, we first introduce hypersphere classifiers (HSCs) (Ruff et al., 2020), whose concept was also used in (Liznerski et al., 2021). An HSC conducts anomaly detection by using a DNN-based feature extractor g, where its anomaly score is the Euclidean distance between a center vector µ and the latent feature vector g(x) of each input x. When training the HSC model, the authors used normal and background data, Dt and Db, respectively, and a loss function
Lhsc = Exk∼Dt [ h ( D2E ( g(xk),µ ))] − Exb∼Db [ log ( 1− exp ( −h ( D2E ( g(xb),µ ))))] , (6)
which aims to decrease the Euclidean distances between normal samples xk and µ while increasing the distances for background samples xb. In Eq. (6), h(x) = √ x+ 1 − 1, which implies that the Euclidean distance D2E(g(x),µ) is scaled into the range of (0, 1] via exp(−h(D2E(g(x),µ))). It is straightforward that the decision rule of Eq. (5) for distance-based classifiers uses the principle of HSCs in a class-wise manner, where the original HSC formulates its decision boundary for anomaly detection as a hypersphere having a constant radius. In other words, the class-wise HSC for the c-th class determines whether a test input belongs to the c-th class by computing D2E(f(x),µc), where the input is determined as an unknown-class sample if the entire class-wise HSCs reject the data item. Thus, a proper BCR strategy for distance-based classifiers should force each KUC sample xb to be rejected by the entire class-wise HSCs (increase D2E(f(x
b),µi) for all i). Since it is inefficient to consider the entire class-wise HSCs to regularize xb at each iteration, we approximate the process by only taking the closest class-wise HSC into account (increase mini∈{1,··· ,C} D2E(f(x b),µi)).
Although one can adopt Eq. (6) to formulate Lbg for distance-based classifiers, scaling D2E(f(x),µc) into (0, 1] via exp(−h(D2E(f(x),µc))) can be inappropriate to guarantee sufficient spaces for KKC data and simultaneously move KUC samples far away from the limited space of KKC data, since exp(−h(D2E(f(x),µc))) rapidly decays near µc. To overcome the issue, we propose to design Lbg based on the principle of the probability of inclusion (Jain et al., 2014; Bendale & Boult, 2016).

heading: 3.3 PROBABILITY OF INCLUSION AND CLASS-INCLUSION LOSS
text: As we described in Section 2.2, the probability of inclusion builds effective CAP models, since it is designed to rapidly decay near extreme data, i.e., PI(x, c) ≈ 1 in the region that a majority of class-c KKC samples are located. In the following, we describe how to utilize the principle of the probability of inclusion when training distance-based classifiers, and then formulate a loss function for BCR.
Based on pre-trained Softmax classifiers, Openmax (Bendale & Boult, 2016) formulated the probability of inclusion via EVT modeling of latent features at inference time. However, it is intractable to use such EVT-based analyses during the training phase of f , since it is inappropriate to limit the space of KKC data by analyzing extreme samples at each training iteration. Thus, we use the underlying assumption of LDA to formulate PI(x, c), since we define the training and the inference processes of distance-based classifiers based on the principle of LDA. Under the assumption that each class-c
latent feature vector is drawn from a unimodal Gaussian distribution N (f(x)|µc, I), the Euclidean distance D2E(f(x),µc), a simplified version of the Mahalanobis distance, can be assumed to follow the Chi-square distribution having the degree of freedom n. Then, we have
P ( D2E(f(x),µc) = t ) =
t n 2 −1
2 n 2 · Γ(n/2) · exp ( − t 2 ) , (7)
where t ≥ 0, Γ(·) is the Gamma function, and n is equivalent to the dimension of f(x). As previous studies (Jain et al., 2014; Bendale & Boult, 2016; Rudd et al., 2018) formulated the probability of inclusion by computing the CDF of the Weibull distribution, we define our PI(x, c) as
PI(x, c) = 1− ∫ D2E(x,c)/2 0 t n 2 −1 Γ(n/2) · exp (−t) dt = Γ(n/2, D 2 E(f(x),µc)/2) Γ(n/2) (8)
by using the CDF of Eq. (7), where Γ(·, ·) is the upper incomplete Gamma function. It is noteworthy that Eq. (8) can be easily computable via igammac function in PyTorch (Paszke et al., 2019).
Based on Dt, Db, and PI(x, c) of Eq. (8), the primary objective of our BCR, which is to make each KUC sample located far away from the closest class-wise HSC, can be achieved by using a loss function Lbg,u = Exb∼Db [− log(1−maxi∈{1,··· ,C} PI(xb, i))]. To compare PI(x, c) and PH(x, c) = exp(−h(D2E(f(x),µc))), which was used in Eq. (6), we plot PI(x, c) and PH(x, c) in Figure 2 with respect to ||f(x)− µc|| by assuming n = 128. The figure implies that unlike PH(x, c), our PI(x, c) can effectively assign space for KKC data and force KUC samples to be located outside the space.
At training time, note that each PI(x, c) = 0.5 builds an auxiliary decision boundary between the c-th class KKC sample and the others, where Lbg,u ensures a majority of KUC samples located outside the entire class-wise boundaries. However, Lbg,u can be insufficient to achieve robust UUC rejection and closed-set classification results, since it does not control correctly classified KKC samples to be located inside the corresponding class-wise boundaries. Thus, in addition to Lcf , we impose a regularization Lbg,k on KKC data to maintain closed-set classification performance and enhance the gap between KKC and KUC data in terms of the Euclidean distance. By formulating Lbg,k = E(xk,y)∼Dt [−1(y = ĉ) log(PI(xk, ĉ))], where ĉ = argmaxi∈{1,··· ,C} PI(xk, i), we define our BCR loss as Lbg = Lbg,k + Lbg,u and call Lbg the class-inclusion loss. In summary, we use L = Lcf + λLbg = Lcf + λ(Lbg,k + Lbg,u) = E(xk,y)∼Dt [ − logPd(y|xk)− λExb∼Db [ 1(y = ĉ) log(PI(x k, ĉ)) + log(1− P ∗I (xb)) ]] (9) as our total loss, where ĉ = argmaxi∈{1,··· ,C} PI(x k, i) and P ∗I (x b) = maxi∈{1,··· ,C} PI(x b, i).

heading: 4 EXPERIMENTS
text: In this section, we compared our class-inclusion loss for distance-based classifiers to the objectosphere (Dhamija et al., 2018), the uniformity (also widely known as OE) (Hendrycks et al., 2019), and the energy (Liu et al., 2020) losses for Softmax classifiers. Through our extensive experiments, we aim to show that whether our proposed framework provides competitive UUC rejection results in comparison with the previous BCR methods, while keeping robust closed-set classification results.

heading: 4.1 EXPERIMENTAL SETTINGS
text: To assess the OSR performance of each method, we used two experimental settings, which correspond to Sections 4.1.1 and 4.1.2, respectively. When evaluating the OSR performance, we first measured the closed-set classification accuracy. To quantify the performance of UUC data rejection, we also measured the area under the receiver operating characteristic curve (AUROC). Furthermore, we used the open-set classification rate (OSCR) measure by quantifying the correct classification rate at the false positive rate of 10−1. For in-depth explanation of the OSCR measure, readers are referred to (Dhamija et al., 2018). As Db, we used ImageNet (Russakovsky et al., 2015), which was also employed in (Li & Vasconcelos, 2020). To ensure that the classes of Db and our test sets are disjoint, we used only the remaining classes of the ImageNet dataset, which are not included in the test sets.

heading: 4.1.1 OPEN-SET RECOGNITION - SETTING 1
text: In Setting 1, we split a single dataset into KKCs and UUCs, where we used the KKCs in the training set as Dt, and the KKCs and UUCs in the test set as Dktest and Dutest, respectively. Following the experiment protocol in (Neal et al., 2018), which were also employed in (Oza & Patel, 2019; Sun et al., 2020), we conducted experiments by using the following standard image datasets: SVHN (Netzer et al., 2011), CIFAR10 & CIFAR100 (Krizhevsky, 2009), and TinyImageNet (Le & Yang, 2015).
SVHN, CIFAR10 For the SVHN and CIFAR10 datasets, each of which consists of images of 10 classes, each dataset was randomly partitioned into 6 KKCs and 4 UUCs.
CIFAR+10, CIFAR+50 For the CIFAR+M experiments, we used 4 randomly selected classes of CIFAR10 as KKCs and M randomly selected classes of CIFAR100 as UUCs.
TinyImageNet For experiments with a large number of classes, we randomly selected 20 classes of TinyImageNet as KKCs and then used the remaining 180 classes as UUCs.

heading: 4.1.2 OPEN-SET RECOGNITION - SETTING 2
text: By using the training and the test sets of a single dataset as Dt and Dktest, respectively, we employed the test set of another dataset relatively close to Dt as Dutest in Setting 2. Adopting the experiment settings in (Yoshihashi et al., 2019) and (Liang et al., 2018), we used the entire classes of a dataset as KKCs for CIFAR10 & CIFAR100. For UUC dataset, TinyImageNet, LSUN (Yu et al., 2015), and iSUN (Xu et al., 2015) were selected. TinyImageNet and LSUN consists of 10,000 test samples each, where the samples in each dataset were resized (RE) or cropped (CR) into the size 32 × 32. The iSUN dataset has 8,925 test samples and they were also resized into the size of 32× 32. The modified datasets can be obtained in the Github repository of (Liang et al., 2018).

heading: 4.2 TRAINING DETAILS
text: For f , we employed the Wide-ResNet (WRN) (Zagoruyko & Komodakis, 2016) and then used its penultimate layer f(x) ∈ Rn for the latent feature vector of each input sample x. For CIFAR10 and TinyImageNet, we used WRN 40-2 with a dropout rate of 0.3, where WRN 28-10 was employed for CIFAR100 with the same dropout rate. For SVHN, we used WRN 16-4 with a dropout rate of 0.4.
For the entire BCR methods, we set the mini-batch sizes of KKC training samples and KUC samples to 128. We kept λ as a constant during training, i.e., each f was trained with the BCR method from scratch. To select hyperparameters and margin parameters of the previous regularization methods, we followed the official implementations 1,2,3. For SVHN, CIFAR10, CIFAR100, and TinyImageNet, we trained the corresponding classifiers for 80, 100, 200, and 200 epochs, respectively, where we used the stochastic gradient descent for optimization. For SVHN and the other datasets, we used initial learning rates of 0.01 and 0.1, respectively, and a cosine learning rate decay (Loshchilov & Hutter, 2016). We also used the learning rate warm-up strategy for the first 5 epochs of each training process.
Average runtime We conducted all the experiments with PyTorch and a single GeForce RTX 3090 GPU. At each trial in the CIFAR10 experiment of Setting 1, the running time of each training epoch
1https://github.com/Vastlab/Reducing-Network-Agnostophobia 2https://github.com/hendrycks/outlier-exposure 3https://github.com/wetliu/energy_ood
took 28 seconds for our method, where its OSR evaluation required approximately 6.5 seconds. We observed that the other methods take similar running time at their training and inference phases.

heading: 4.3 EXPERIMENT RESULTS
text: The OSR results of our framework and the previous methods are reported in Tables 1 and 2. All the reported values were averaged over five randomized trials, by randomly sampling seed, split of KKCs and UUCs, and class-wise anchors. In the tables, ↑ indicates that it is better to have larger values for the corresponding measure. Also, an underlined value is marginally worse than the best score (bold).
Setting 1 For the first setting, Table 1 compares our proposed BCR methods for distance-based classifiers with the previous methods designed for Softmax classifiers. The results demonstrate that our proposed method obtained robust UUC rejection results, which were superior to the results of the previous approaches. It is noteworthy that our method achieved higher classification accuracy values, which were critical in acquiring better OSR results in terms of OSCR, than the previous methods. Such results imply that the proposed framework effectively satisfies the two essential requirements described in Section 2.1, bounding open-space risk and ideally balancing it with empirical risk.
Setting 2 In Table 2, we present our experiment results of the second setting. When using CIFAR10 and CIFAR100 as KKC datasets, the table shows that our model can achieve the highest closed-set classification accuracy, which is consistent with the experiment results of Setting 1. Furthermore, by averaging the AUROC and the OSCR values over the various UUC datasets, the table shows that our model outperformed the previous methods in UUC sample rejection and OSR.
In summary, the experiment results show that our proposed method can successfully train a robust open-set classifier. To further investigate the effectiveness of our framework in another domain, we compared our class-inclusion loss to the uniformity loss in text classification applications.
Text classification For text classification, we used 20 Newsgroups and WikiText103 for KKCs and KUCs, respectively, and trained a simple GRU model (Cho et al., 2014) for f as in (Hendrycks et al., 2019). For UUCs, we used Multi30K, WMT16, and IMDB. Since the margin parameters of the objectosphere and the energy losses selected for image classification were not suitable for the text classification, we tested the uniformity loss for comparison. In Table 3, we present the results, where we additionally reported the area under the precision-recall curve (AUPR) and the false-positive rate at 95% true-positive rate (FPR95). As it outperformed the uniformity loss in image classification tasks, our method also showed significantly better performance in text classification experiments.

heading: 4.4 ABLATION STUDY AND DISCUSSIONS
text: For ablation study, we further analyzed our BCR method by using various λ and class-wise anchor initialization & updating methods in our loss function. Also, we compared the proposed method to the triplet loss (Schroff et al., 2015) and analyzed each component of our class-inclusion loss. Using the
CIFAR10 and TinyImageNet experiments in Setting 1, we present quantitative results in Appendix.
Selecting λ Conducting OSR experiments with λ ∈ {0.1, 0.5, 1, 5, 10}, we observed that λ = 5 and λ = 1 showed the best OSR results in the CIFAR10 and TinyImageNet experiments of Setting 1, respectively, where the results are presented in Section A.1. In our additional experiments, λ = 5 yielded the best results in the SVHN and CIFAR + M experiments of Setting 1 and the CIFAR10 experiments of Setting 2, where λ = 0.5 showed the best results in the CIFAR100 experiments. Since the other λ values did not significantly degrade OSR performance, one can flexibly select λ, where we empirically observed that a lower λ value is better when handling more KKCs.
Class-wise anchors In Section 3.1, we mentioned that each class-wise anchor was sampled from the standard Gaussian distribution. Although one can use data-dependent initialization approaches, which are 1) train a Softmax classifier and then use its empirical class mean vectors as class-wise anchors of a distance-based classifier and 2) compute class mean vectors from the initial feature representations of a distance-based classifier and then use the vectors as class-wise anchors, such approaches showed worse results than the random initialization method. In addition, we observed that the strategy of updating or training class-wise anchors was not effective to achieve robust OSR performance. We present the corresponding experiment results and discussions in Section A.2.
Triplet loss To the best of our knowledge, we propose the first distance-based BCR method for the OSR problem. Since the triplet loss (Schroff et al., 2015) has been widely employed to control the distances between latent feature vectors effectively, we formulated another distance-based BCR strategy with the triplet loss for comparison. We present in-depth details and experiment results for the regularization method using the triplet loss in Section A.3, where the experiment results demonstrate that our class-inclusion loss is more effective than the triplet loss-based regularization.
Loss function Note that our loss function consists of three components Lcf , Lbg,k, and Lbg,u. By using various compositions of the components, we verified in Section A.4 that the three components are necessary to obtain robust closed-classification and UUC rejection results in our BCR method.
Vanilla distance-based classifiers To show the effectiveness of our method, we assessed the OSR performance of vanilla distance-based classifiers (trained solely based on Lcf ), where we present the results in the form of (Accuracy / AUROC / OSCR). For the CIFAR10 and TinyImageNet experiments of Setting 1, we obtained the results of (0.962 / 0.757 / 0.470) and (0.785 / 0.629 / 0.315), respectively. In addition, for the CIFAR10 and CIFAR100 experiments of Setting 2, the OSR results averaged over the five UUC test sets in vanilla distance-based classifiers were (0.936 / 0.838 / 0.709) and (0.766 / 0.807 / 0.549), respectively. Comparing these results to the results in Tables 1 and 2, we show that our regularization strategy can significantly improve the OSR performance of distance-based classifiers.

heading: 5 CONCLUDING REMARKS
text: In this paper, we propose a novel BCR method to train open-set classifiers that can provide robust OSR results with a simple inference step. By employing distance-based classifiers with the principle of LDA, we designed a novel class-inclusion loss based on the principle of probability of inclusion, which effectively limits the feature space of KKC data in a class-wise manner and then regularizes UUC samples to be located far away from the limited space. Through our extensive experiments and ablation study, we present that our method can achieve robust UUC rejection performance, while maintaining high closed-set classification accuracy. As this paper aims to improve the reliability of modern DNN-based classifiers, we hope our work to enhance reliability and robustness in various classification applications by providing a novel methodology of handling UUC samples.

heading: 6 ETHICS STATEMENT
text: Although there are many beneficial applications of classifiers and anomaly detectors, they may lead to inadvertent discrimination and encoding societal biases in decision-making systems. This is one of the cases where ethical considerations depend strongly on the specific applications.
We use several standard datasets including the ImageNet dataset. Although they have been widely used in research, some of the datasets, created from images available on the web, include images of people, raising ethical questions related to human-derived data, such as encoding of biases and underrepresentation issues. The creators of ImageNet are also aware of some of these concerns, and have been attempted to address them for the past few years (https://image-net.org/update-sep-17-2019.php).

heading: 7 REPRODUCIBILITY STATEMENT
text: We describe details about distance-based classifiers, our loss function, and the corresponding training process in Sections 3.1, 3.3, and 4.2. Also, Section 4.1 provide the list of the datasets, which are used in this paper. Our source code and the corresponding instructions will be released.

heading: A EXPERIMENT RESULTS OF ABLATION STUDIES
text: To supplement our discussions in Section 4.4, this section provides quantitative experiment results. Based on the CIFAR10 and TinyImageNet experiments of Setting 1, we present OSR results with various selections of λ (Section A.1). Also, we report OSR results by using various class-wise anchor initialization & updating methods (Section A.2). For the settings of fixed and trainable anchors in Section A.2, we compare our class-inclusion loss to the triplet loss (Section A.3). Furthermore, we provide additional OSR results by varying the components of our class-inclusion loss (Sections A.4).
A.1 HYPERPARAMETER λ
This subsection presents OSR results by employing various selections of the hyperparameter λ in our loss function L = Lcf + λLbg . In Table 4, we report the OSR experiment results of our method with λ ∈ {0.1, 0.5, 1, 5, 10}. Since the other λ values did not significantly degrade OSR performance, one can flexibly select the parameter λ to balance empirical and open-space risks. The experiment results in Sections A.2 and A.3 are also provided by using various λ ∈ {0.1, 0.5, 1, 5, 10}.
A.2 CLASS-WISE ANCHORS
This subsection presents OSR results by using various initialization and updating methods of anchors in our approach. In Table 5, we first present OSR results based on trainable class-wise anchors. In this setting, we set each entry of µc as a trainable parameter and then optimized the anchors along with our model parameters via the same training objective L = Lcf + λLbg and optimization processes.
Comparing the CIFAR10 experiment results in Tables 4 and 5, we observed that there exist only a negligible difference of OSR performance between our fixed and the trainable anchor strategies. However, the TinyImageNet experiment results in the tables show that the fixed anchor strategy with random initialization can yield more accurate UUC rejection and OSR results.
To further investigate various anchor initialization and updating methods, we additionally assess the OSR performance of our method by employing the following three initialization methods:
Initial Compute anchors from the initial feature representations of a distance-based classifier. Transfer Train a Softmax classifier and then use its empirical means for a distance-based classifier. Random (Ours) Randomly sample each class-wise anchor from a standard Gaussian distribution.
For each initialization method, we tested two anchor updating strategies: 1) fix the anchors during our training phase, 2) update the anchors via moving average, which was also used in (Wen et al., 2019).
The results of Tables 5 and 6 present that the OSR results of our approach, which randomly sampled class-wise anchors and then fixed them, are superior to those of the other initialization and updating methods. In the following, we discuss the advantages of random sampling and fixed anchor strategies.
• In contrast to classical closed-set classification, our BCR method for OSR should ensure sufficient distance gaps between inter-class latent features (inter-class separability) and force class-wise features to have compact feature space (intra-class compactness) by contrasting KKC samples and KUC data located near the KKC data. As discussed in (Izmailov et al., 2020), the Euclidean distances between anchors drawn from a standard Gaussian distribution are sufficiently large in expectation, which is suitable for our BCR method. • Our loss is based on the probability of inclusion of Eq. (8), where PI = 0.5 builds a hypersphere decision boundary in a class-wise manner at training time. The class-inclusion loss makes repulsion forces between KUC samples and the corresponding closest anchors, while simultaneously making KKC samples to be located inside the corresponding hypersphere boundary. In this regularization method, it is desirable to fix class-wise anchors to control each KKC or KUC sample in a consistent direction. In other words, the strategy of updating or training class-wise anchors, which also changes the location of class-wise hypersphere boundaries, is likely to obstruct the process of contrasting KKC and KUC samples.
A.3 TRIPLET LOSS-BASED REGULARIZATION
This section compares our BCR method to a regularization method based on the triplet loss Ltri, in order to show that our class-inclusion loss is an effective BCR method for distance-based classifiers. Following the conventional definition of the triplet loss, we simply formulated Ltri by setting classwise anchors, KKC training data, and KUC data as anchors, positive samples, and negative samples, respectively. Then, we compared our loss to Ltri in fixed and trainable anchor settings. In Tables 4 and 5, which presented the OSR results of distance-based classifiers with fixed class means and trainable class means, respectively, we reported experiment results by using the triplet loss as Lbg. Since we observed that training classifiers solely based on the triplet loss L = Ltri yields significantly worse OSR results in comparison with the regularization method L = Lcf + λLtri, we employed Ltri as a regularization loss function for BCR. The results show that our proposed method (class-inclusion loss) outperforms the regularization method using the triplet loss. Also, the discussions in Sections A.1 and A.2 can also be applicable to the triplet loss-based regularization.
A.4 LOSS FUNCTION FORMULATION
In our class-inclusion loss L = Lcf +λ(Lbg,k+Lbg,u), Lcf is a conventional negative log-likelihood loss for the distance-based classifier of Eq. (4). Based on our probability of inclusion of Eq. (8), we formulated Lbg,k and Lbg,u for regularization. This subsection investigates the necessity of the three terms Lcf , Lbg,k, and Lbg,u, since both of the Lcf and Lbg,k terms aim to force KKC training samples located close to the corresponding class-wise anchors. We employed various compositions of the three terms to formulate loss functions and then reported their OSR results in Table 7.
As our original Lbg,k term only controls KKC training samples having correct class predictions with Lbg,k = E(xk,y)∼Dt [−1(y = ĉ) log(PI(xk, ĉ))] and ĉ = argmaxi∈{1,··· ,C} PI(xk, i), we define a modified version L∗bg,k = E(xk,y)∼Dt [− log(PI(xk, y))] to correctly remove Lcf from L.
In Table 7, it is shown that our loss function (Lcf +λ(Lbg,k+Lbg,u)) can achieve robust OSR results for various selections of λ. As L∗bg,k (or Lbg,k) only forces KKC samples to be located inside the class-wise hypersphere boundaries of the probability of inclusion, which rapidly decays from 1 to 0, the loss term is not sufficient to achieve high closed-set classification accuracy. On the contrary, KKC samples cannot be forced to be located inside the boundaries when using Lcf without Lbg,k, thereby making inaccurate OSR results. In summary, both Lcf and Lbg,k are necessary to conduct accurate classification and increase the distance gap between KKC and KUC data, respectively. It is noteworthy that our formulation (using both Lcf and Lbg,k) is beneficial in maintaining robust classification accuracy. Also, our method, which only regularizes KKC samples having correct class prediction, can mitigate overfitting issues in comparison with the approach of (Lcf + λ(L∗bg,k + Lbg,u)).

heading: B ADDITIONAL EXPERIMENTS
text: In additional experiments, we employed CIFAR10 for Dt. For Setting 1 of our OSR experiments, we split the 10 classes of CIFAR10 into 6 known known classes (KKCs) and 4 unknown unknown classes (UUCs). In Setting 2, we used CIFAR10 and another dataset as KKCs and as UUCs, respectively. For the first setting of our experiments, we report quantitative results in the form of (Accuracy / AUROC / OSCR). In this section, we denote Settings 1 and 2 by S1 and S2 for simplicity.
B.1 QUANTITATIVE RESULTS WITH RESNET-18 ARCHITECTURE
In Section 4, we used the Wide-ResNet (WRN) architectures as feature extractors for the image datasets. To further investigate the effectiveness of our method, we used another standard classifier architecture, ResNet-18 (He et al., 2016). In S1, we obtained quantitative results of (0.963 / 0.945 /
0.844), (0.966 / 0.950 / 0.845), (0.967 / 0.945 / 0.848), and (0.971 / 0.947 / 0.850) for the regularization methods using the objectosphere (Dhamija et al., 2018), the uniformity (Hendrycks et al., 2019), the energy (Liu et al., 2020), and our class-inclusion losses, respectively. Such quantitative OSR results and the results in Table 8 imply that our method can outperform the previous BCR methods with ResNet-18, as we observed in the experiments using the WRN architectures.
B.2 LATENT FEATURE SPACE VISUALIZATION
By visualizing feature spaces via t-SNE (Van der Maaten & Hinton, 2008), we compared Softmax classifiers and distance-based classifiers, and their regularized versions in S1 and S2. We selected ImageNet-RE for UUC data in S2, since the results of Table 2 imply that the dataset is more difficult to be recognized by open-set classifiers than the others. In each t-SNE result, black dots indicate UUC samples. For the other colors, each distinct color represents the corresponding class of KKCs.
B.2.1 VANILLA CLASSIFIERS (NO BACKGROUND-CLASS REGULARIZATION)
Figure 3 depicts the t-SNE results of vanilla Softmax and distance-based classifiers, which are trained only with L = Lcf , in S1 and S2. The visualization results imply that in comparison with the UUC data used in S2, it would be much more difficult to distinguish UUC samples in S1 from KKCs.
B.2.2 BACKGROUND CLASS REGULARIZED CLASSIFIERS
Figure 4 presents the t-SNE results of Softmax and distance-based classifiers trained with BCR (L = Lcf + λLbg) in S1 and S2. For the Softmax and distance-based classifiers, we employed the uniformity loss (Hendrycks et al., 2019) and our class-inclusion loss as Lbg, respectively. In comparison with Figure 3, Figure 4 shows that such regularization techniques assist classifiers to learn robust latent feature representations in distinguishing UUCs from KKCs, especially in S1. Also, as mentioned in Section 2.3, the figure shows that the latent feature space regularized by the uniformity loss can yield inaccurate results in distance-based post-classification analysis. By applying the Openmax approach (Bendale & Boult, 2016) to the model regularized by the uniformity loss in S1 (Figure 4. (a)), we obtained the OSR performance of (0.923 / 0.886 / 0.752), which is worse than the original result (0.964 / 0.923 / 0.814) and the result of our framework (0.973 / 0.946 / 0.869).

Review summaries:
In this paper, the authors propose a new method for open set recognition. To this end, the authors use distance based classifiers and they minimize the distance between the class samples and their corresponding means for the labeled data samples and try to enforce the background data samples to be outside the class acceptance regions. Then the samples coming from the unknown classes can be rejected based on the distances to the centers of the known classes. The authors use ImageNet data samples are used as background class samples. The proposed method is compared to related open set recognition methods and better accuracies are reported.

For open set recognition (OSR), the authors propose using background data to represent instances from unknown classes.  The background data are from data outside the known classes.  Based on LDA, they proposed using the distance from an instance to the class mean to estimate the posterior class probability via softmax.  The class means are randomly chosen beforehand.  Their loss functions includes cross entropy, L_cf, and background class regularization (BCR), which is denoted as L_bg.   For L_bg, they introduce probability of inclusion, P_I, which is estimated by the CDF for Weibull distribution.  With the background data, they calculate L_bg,u  based on 1-P_I.  With the known data, they calculate L_bg,k based on P_I.  Empirical evaluations have two settings.  The unknown classes from the first setting are from the same dataset as the known classes.  In the second setting the unknown classes are from another dataset.  Background data are from another dataset.  In the both settings, the proposed method compares favorably against a few existing algorithms on a few datasets.  

This paper proposes a new approach for generalized open-set recognition (OSR) in classification setting where the idea is to build a model which could simultaneously detect unknown classes and maintain the accuracy of traditional closed-set classification of known classes. The proposed OSR is based on background class regularization (BCR) where data of known unknown classes are utilized in the training phase of the classifiers based on typical deep neural network (DNN) architectures.  Previous works applying BCR have treated known-classes as one group and unknowns as other with SoftMax classification layer, leading to limited OSR performance, especially near the known classes, whereas proposed work regularizes and limits feature space in class-wise manner, making the data of unknown classes to be located far away from the known classes. Distance-based classifier applying linear discriminative analysis is used to model the latent features from DNN and to design novel loss function which can regulate the solution between known classes and unknown background classes in OSR setting, simultaneously keeping the robust performance of closed-set classification.  Proposed approach is experimentally compared to three previous works applying BCR-based OSR with SoftMax classifiers. Proposed method outperforms those three related methods in well-known public benchmarks datasets of image and text classification tasks. Different performance metrics of classification accuracy, area under the receiver operating characteristic curve (AUROC), and the open-set classification rate (OSCR) are utilized in comparison.  To summarize the contribution, paper presents LDA principled distance-based classifier (applied to output of neural network latent feature layer) and new loss function to train OSR-sensitive classifiers, developed and formulated based on the limitations of previous work and experimentally validated against previous BCR-based approaches.