Distance-Based Background Class Regularization for Open-Set Recognition
Authors: Wonwoo Cho
Jaegul Choo
Venue: ICLR 2022 Submitted (2022)

Abstract: In open-set recognition (OSR), classifiers should be able to reject unknown-class samples while maintaining robust closed-set classification performance. To solve the OSR problem based on pre-trained Softmax classifiers, previous studies investigated offline analyses, e.g., distance-based sample rejection, which can limit the feature space of known-class data items. Since such classifiers are trained solely based on known-class samples, one can use background class regularization (BCR), which employs background-class data as surrogates of unknown-class data during training phase, to enhance OSR performance. However, previous regularization methods have limited OSR performance, since they categorized known-class data into a single group and then aimed to distinguish them from anomalies. In this paper, we propose a novel distance-based BCR method suitable for OSR, which limits the feature space of known-class data in a class-wise manner and then makes background-class samples located far away from the limited feature space. Instead of conventional Softmax classifiers, we use distance-based classifiers, which utilize the principle of linear discriminant analysis. Based on the distance measure used for classification, we design a novel regularization loss function that can contrast known-class and background-class samples while maintaining robust closed-set classification performance. Through our extensive experiments, we show that the proposed method provides robust OSR results with a simple inference process.

Under review as a conference paper at ICLR 2022
DISTANCE-BASED BACKGROUND CLASS REGULARIZATION FOR OPEN-SET RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
In open-set recognition (OSR), classifiers should be able to reject unknown-class samples while maintaining robust closed-sed set classification performance. To solve the OSR problem based on pre-trained Softmax classifiers, previous studies investi- gated offline analyses, e.g., distance-based sample rejection, which can limit the feature space of known-class data items. Since such classifiers are trained solely based on known-class samples, one can use background class regularization (BCR), which employs background-clasy data surrogates of unknown-class ones during training phase, to enhance OSR performance. However, previous regularization methods have limited OSR performance, since they categorized known-class data into a single group and then aimed to distinguish them from anomalies. In this paper, we propose novel distance-based BCR method suitable for OSR, which limits the feature space of known-class data in a class-wiss manner and then makes background-class samples located far away from the limited feature space. Instead of conventional Softmax classifiers, we use distance-based classifiers, which utilize the principle of linear discriminant analysis. Based on the distance measure used for classification, we design novel regularization loss function that can contrast known-class and background-class samples while keeping robust closed-set classifi- cation performance. Through our extensive experiments, we show that the proposed method provides robust OSR results with a simple inference process.
INTRODUCTION
In machine learning (MI.), classification algorithms have achieved great success. Through recent advances in convolutional neural networks, their classification performance already surpassed the human-level performance image classification (He et al., 2015). However, such algorithms are usually developed under closed-set assumption, i.e., the class of each test sample is assumed to always belong to one of the pre-defined set of classes. Although this conventional assumption can be easily violated in algorithms are likely to force unknown-class samples to be classified into one of the known classes. To tackle this issue, the open-sel recognition (OSR) problem (Scheirer al. 2013) aims properly classify unknown-clan samples as "unknown" and known-class samples as one of the known classes.
applications (classifiers
face aakknonn-class data), traditional
According to the definition of OSR (Scheirer et al. 2013). required to properly limit the feature space of known-class data. To satisfy the requirement, various OSR methods were developed based on traditional ML models. Previously, /Schiirer et al. (2014) calibrated the decision scores of support vector machines (SVMs) Based on the intuition that large set of data samples of unknown classes can be rejected if those of known classes are accurately modeled, Dain al. (2014) proposed PV-SV which utilized the statistical modeling of known-class samples located near the decision boundary of SVMs. Afterward, Júnior et al. (2016) attempted to solve the OSR problem based on the principle of the nearest neighbors. Taking distribution information of data into account, Rudd et al. (2018) proposed the extreme value machine by utilizing the concept of margin distributions.
Since deep neural networks (DNNs) have robust classification performance by learning high-level representations of data, OSR methods for DNNs have received great attention. Based on the theoretical foundations used in traditional ML-based OSR methods, Bendale & Boult (2016) proposed the first OSR strategy for DNNs called Openmax, which calibrates the output logits of pre- classifiers. To improve Openmax, Yoshihashi et al. (019) proposed the pre-trainne trained Softmax

---

Under review as a conference paper at ICLR 2022
X 1.
KICG
3 S JUCT
KWC
Aight UUC
JUC
UUC
LUC
Open space
OF
KXC 2
KXCC
KXC 2
KXC 4
KRC
KXD
(a) Latent feature space
(b) Closed-sel set classification
(c) Open-set classification
Figure I: Given (a) a latent feature space, we demonstrate (b) closed-set and (c) open-set classification examples, where KKCs and UUCs are known and unknown classes, respectively.
learning to make robust latent feature vectors. Afterward, Oza & Patel (2019) proposed to exploit class-conditioned autoencoder and use its reconstruction error to assess each input sample. /Sun et al. (2020) employed several class-conditioned variational auto-encoders for generative modeling.
Although previous OSR methods using discriminative models applied offline analyses to pre-trained Softmax classifiers employed complicated DNN models, they have limited performance since the classifiers were trained solely based on known-clas samples. To mitigate the problem, one can use background class regularization (BCR) achieve robust empirical results. However, previous BCR methods (Dhamija et al. 2018 Hendrycks et al.] 2019 Liu et al., 2020) are insufficient to properly solve the OSR problem. To design an effective open-set classifier that can overcome the previous limitations, we propose novel BCR method suitable for OSR, which uses a distance-based classifier and novel loss function for regularization. We provide detailed description in the next section.
2 PRELIMINARIES AND OUR CONTRIBUTIONS
2.1 THE OPEN-SET RECOGNITION PROBLEM
The OSR problem addresses the classification setting that can face test samples from classes unseen during training. In this setting, open-set classifiers aim properly classify known-class samples while rejecting unknown-claa
samples simultaneously. A similar problem to OSR is out-of--istribute (OOD) detection (Hendrycks & Gimpel| 2017), which typically aims to reject data items drawn far away from the training data distribution. In previous OOD detection studies (Hendrycks & Gimpel| 2017: Liang et al. 2018 8 Lee al. 2018a b). OOD samples tend to be drawn from other datasets or even images of noise. In this paper, we aim to reject test data whose classes are unknown but related the training data, which narrows down the scope of conventional OOD detection tasks.
Previously, Scheirer et al. (2013) introduced a formal definition of the OSR problem based on the notion of open-space risk Ro. which is a relative measure of positively labeled union of balls Sy and open space O located far from Sy. Since labeling any data item in O incurs open-space risk, it is straightforward that a classifier cannot be a solution for the OSR problem if the classifier accepts data in infinitely wide regions, i.e., its open-space risk is unbounded (Ro oo). The definition implies that essential requirements solve the OSR problem are 1) bounding open-space risk and 2) ideally balancing it with empirical risk to maintain a low classification error rate.
Unlike traditional closed-set classifiers, open-set classifiers are required to limit the space of known- class data to bound their open-space risk. To ensure bounded open-space risk, Scheirer et al. (2014) proposed to formulate compact abating probability (CAP) models. The principle of CAP models is that if the support region of classifier decays in all directions from the training data, thresholding the region will bound the classifier's 's open-space risk (Boult et al.) 2019). As depicted in Figure I which compares traditional closed-set and open-set classification problems, properly building class-wise CAP models is an effective strategy to satisfy the two essential requirements of OSR.
2.2
ANALYSIS FOR PRE-TRAINED SOFTMAX CLASSIFIER
In this paper, we aim to solve the OSR problem by using a standard DNN- based classifier architecture a latent feature extractor. Applying fully-connected layer f. conventional Softmax classifier
2

---

Under review as a conference paper at ICLR 2022
computes the posterior probability of imput belonging to the e-th known class by
exp(w" + ba)
P. = cix) ΣS exp(w] [() + 6i)
where c (I,...,C),.. f(x) E R" is the latent feature vector of x, and w. and b. are the weight and the bias for the e-th class, respectively, For pre-trained Softmax classifiers, Hendryeks & Gimpel (2017) proposed a baseline to detect anomalous samples, which imposes a threshold on the predictive confidence of Eq. (T). When using the baseline method to solve the OSR problem, one can estimate the class of each known-class sample and recognize unknown-class data by
{ arg max .c) P.(y clx), if ma.. c) (y clx) 2 T otherwise.
1 =
C + (unknown class),
However, Eq. cannot formally bound open-space risk since it only rejects test samples near the decision boundary of classifiers, thereby having infinitely wide regions of acceptance (Boult et al.) 2019). Therefore, additional post-classification analysis methods using an auxiliary measure other than the Softmax probability are necessary to bound open-space risk, where distance measures have been widely used to build auxiliary CAP models in the latent feature space of f.
To construct class-wise CAP models, Openmax (Bendale & Boult 2016) defined radial-basis decaying functions {s(x, } each of which measures the class-belongingrer of x for the e-th class, in the latent feature space of f. For each s(x, c). the authors employed distance measures between f(x and an empirical class mean vector Me. e.g., s(x, c) (f (xx), me) (f(x) p
Hc).. To
formulate effective CAP models based on 8(x, c), they statistically modeled the distribution of s e)
for known-class data based the extreme value theory (EVT) (Scheirer (017)) which provides theoretical foundation that the Weibull distribution is suitable for modeling known-class samples located far from the class mean vectors (extreme samples). To be specific, Openmax fits Weibull distribution on extreme samples of the e-th class having the highest (x) Mc) values, where its cumulative distribution function (CDF) formulates the probability of inclusion Pr Pxx e) (Jain et al., 2014; Rudd et al., 2018), i.e., Pr c) = - We Weibulle ICDF, which rapidly decays near extreme samples. Based on Px(x; c), the decision rule of Eq. 2 can calibrated to conduct robust OSR.
2.3 GENERALIZED OPEN-SET RECOGNITION AND BACKGROUND CLASS REGULARIZATION
Although they require additional inference procedures (c.g., EVT modeling) to bound the open-space risk of pre-trained Softmax classifiers, previous post-classification analyses may have limited OSR performance since the classifiers are trained solely based on known-class samples. To empirically obtain robust OSR results without complicated post-classification analyses, one use the strategy of BCR at the training phase, which exploits background- samples as surrogates of unknown- class samples. Geng et al. (2020) argued that the generalized OSR setting that utilizes background samples is still less-explored and an important research direction to improve OSR performance. In the generalized OSR setting, classifiers should consider the following data classes among the infinite label space of all classes y (Dhamija et al. 2018 Geng et al. (2200).
Known known classes (KKCs; = {1, .} y) include distinctly labeled positive classes, which also have side information, where L = I. is the entire unknown classes. Known unknown classes (KUCs; B c 2f) include background classes, e.g., labeled negative classes which are not necessarily grouped into pre-defined set of KKCs K.
Unknown unknown classes (UUCs; A = U / B) represent the rest U, where UUC samples are not available at training time and have side information, but occur inference time.
Throughout this paper, we denote the corresponding datasets for the data classes as follows. D, is training set consisting of multiple pairs of a KKC data sample and the corresponding class label y E [1,.... ".. D and Diest are test sets of KKCs and UUCs, respectively. D, is a background dataset of KUCs. Then, loss function for training classifiers with BCR can be formulated by
C = Lof + ALbg = Eixt w)>D, |- log P. (ylx*) + AE [Freg (x* S.,yyx3 []]]]
(3)
where Cef and Log are the losses for classification and BCR, respectively, and À is a hyperparameter that balances Lef and Lbg- For Log previous studies designed their own freg. where Dhamija et al.]
3

---

Under review as a conference paper at ICLR 2022
(0018) proposed the objectosphere loss for OSR, and Hendrycks al. (2019) and Liu et al. (2020) employed the uniformity and the energy losses for OOD detection, respectively.
In this paper, we tackle the following limitations of the previous BCR methods.
In the previous BCR methods, Lig were designed to make normal data and anomalies more distinguishable in terms of the corresponding anomaly scores. Since they categorized normal samples into a single group (do not consider the classes) in Lbg. the previous methods cannot be suitable for rejecting UUC samples located relatively close to K.C data and maintaining robust closed-set classification results.
The previous methods using the decision rule of Eq. 2 objectossjer (Dhamija et al. 2018) and uniformity (Hendrycks et al. 2019)) cannot bound open-space risk. Although one can use postcclassfiication analyses bound open-space risk, trained latent feature spaces are likely to be inappropriate for using another metric such as distance measures.
To increase the gap between KKC and KUC samples in terms of latent feature magnitude and energy value in the objectosphere (Dhamija et al. (2018) and the energy (Liu et al. (020)
losses, respectively, it is necessary to find proper margin parameters for each dataset.
2.4 OUR CONTRIBUTIONS
Based standard DNN-based classifier architecture f. we aim to design open-set classifiers having a simple yet effective inference process by using the principle of BCR. To overcome the limitations deseribed in Section 2.3 we use distance- based classifiers and propose novel BCR strategy based on the framework of Eq. 3h. In the following, we summarize our proposed strategy.
Instead of applying fully-connected layers feature extractors linear discriminant analysis (L.DA) (Murphy) (012) to classify each input based on a distance measure. By imposing a threshold on the distance measure as in Eq. (2). such distance-based classifiers can bound their open-space risk without additional offline analyses.
the principle of
To enhance the OSR performance of distance-based classifiers, we propose novel distance- based BCR strategy. For Lóg, we design a loss function that does not require data-dependent margin parameters. Our method first limits the feature space of KKC data by formulating class-wise boundaries based on the concept of the probability of inclusion, and then forces KUC data to be located outside the entire boundaries. By imposing additional regularization each KKC data item to be located inside the corresponding class-wise boundary, proposed method can also maintain robust closed-set classification performance.
PROPOSED METHOD
3.1 DISTANCE- BASED CLASSIFICATION MODELS
To train a robust open-set classifier as we described in Section 2.4 we use distance-hased classifier
Ply = (V((x) (e,
P(y c) exp (- (-D%(((xx, (x).
Pa(y = cxx Σ P(y
(4)
DE P(y = 1) exp (-D3 (fx)(((
as an alternative of Eq. D where Eq. (4) uses the principle of LDA in fxx) In Eq. 44. we exploit an identity covariance matrix I and P(y c) C-1 for all e for KKC data. Then, Eq. (4) classifies each x by computing D (x(),, Max)
(J(() (f(()) (c), the Euclidean distance between (x) and f.. "R" where we call μa, a class-wise anchor. Instead of updating or computing empirical He, we randomly sample each fe, from the standard Gaussian distribution for all e and then fix it as an anchor during the training process. Such strategy also showed successful results in Izmailov et al.) (002)), which formulated generative classifiers based on the principle of Gaussian mixture models.
At inference time, each KKC sample x can be classified by z = arg min .cy (x),, pc). is noteworthy that thresholding (f pe). which is the same metric used for classification, can bound the open-space risk of distance-based classifier by formulating class-wise CAP models. Therefore, one can properly conduct OSR without additional post- clsssflaasii analyses as follows:
arg min, (1,.,, .c) D (x),, pc), if max.e(1,--.cC) -D% M.) T,
1 C 1 (unknown class),
(5)
otherwise.
4

---

Under review as a conference paper at ICLR 2022
Distance-based based classification methods were also employed in prototypical networks (Snell et al. 2017), nearest class mean classifiers (Mensink et al. 2012), and the previous studies of the center loss function (Wen et al., 2019) and convolutional prototype classifiers (Yang et al.] 2020). Furthermore, polyhedral conic classifiers (Cevikalp et al. 2021) used the idea of returning compact class regions for KKC samples based on distance- based feature analyses. It is noteworthy that main contribution is a novel BCR method that can effectively utilize KUC samples in distance- based classification scheme (described in Sections 3.2 and (.3). not the distance-based classifier method itself. To the best of our knowledge, we are the first to discuss the necessity of distance- -baeed BCR methods for OSR and propose a reasonable regularization method for distance-base based classifiers.
3.2 BACKGROUND CLASS REGULARIZATION FOR DISTANCE- DISTACC-BBSSED CLASSIFIERS
To obtain robust OSR performance based on distance-basee classifiers using the simple inference of Eq. (5). we aim to design BCR method, which uses D. and Dj as surrogates of Di Diest and Diest at training time, respectively. Although it cannot provide any information of D instt Dd can be effective to limit the latent feature space of KKC data, while reserving space for UUC samples. Note that the decision rule of Eq. (5) conducts both closed-set classification and unknown-class rejection based on a single distance measure D (x)),, Pc). Thus, it is intuitive that the primary objective of our BCR method is to make KUC samples located far away from μii for all i {L..... C).
Before we illustrate our BCR method, we first introduce hypersphere classifiers (HSCs) (Ruff' et al. 2020), whose concept was also used in (Liznerski al. 202T). An HSC conducts anomaly detection by using a DNN- based feature extractor g. where its anomaly score is the Euclidean distance between a center vector ii and the latent feature vector g(x) of each input x. When training the HSC model, the authors used normal and background data, D, and D,, respectively, and a loss function
Cr = ExtnD, [h (D' (g(x*), [4)] - Exa~D, [log (1 - exp (-h (Dg ()))), (6)
which aims to decrease the Euclidean distances between normal samples x* and while increasing the distances for background samples x². In Eq. 6.. h(x) = Vx + I - 1, which implics that the Euclidean distance D (xx),, 1t) is scaled into the range of (0, 1] via (xp(=M(D((g(x)),))))
is straightforward that the decision rule Eq. (5) for distance-based classifiers uses the principle of HSCs in class-wise manner, where the original HSC formulates its decision boundary for anomaly detection as a hypersphere having a constant radius. In other words, the class-wise HSC for the c-th class determines whether test imput belongs to the e-th class by computing Of (xx), Ml.,, where the input is determined as an unknown-class sample if the entire class-wise HSCs reject the data item. Thus, a proper BCR strategy for distance-based classifiers should force each KUC sample x" to be rejected by the entire class-wise HSCs (increase D3 (f(x), p4) for all i). Since it is inefficient to consider the entire class-wise HSCs to regularize x² at each iteration, we approximate the process by only taking the closest class-wise HSC into account (increase min,e .cy D pc)).
Although one can adopt Eq. (6 to formulate Log for distance-based classifiers, scaling DE (f(x,, Mc) into (0, 1] via exp(-h((2 D (f(),, (e)))) can be inappropriate guarantee sufficient spaces for KKC data and simultaneously move KUC samples far away from the limited space of KKC data, since exp(-)( (D) (ee)) rapidly decays near pec To overcome the issue, we propose design Log
based on the principle of the probability of inclusion (Jain et al. 2014 Bendale & Boult 2016).
3.3 PROBABILITY OF INCLUSION AND CLASS-INCLUSION LOSS
As we described in Section 2.2] the probability of inclusion builds effective CAP models, since is designed to rapidly decay near extreme data, i.e., Pr(x, c) R 1 in the region that majority of class-c KKC samples are located. the following, we describe how to utilize the principle of the probability of inclusion when training distance-based classifiers, and then formulate a loss function for BCR.
Based on pre-trained Softmax classifiers, Openmax (Bendale & Boult, 2016) formulated the probabil- ity of inclusion via EVT modeling of latent features at inference time. However, it is intractable to use such EVT-based analyses during the training phase of f, since it is inappropriate to limit the space of KKC data by analyzing extreme samples at each training iteration. Thus, assumption of LDA to formulate (x c). since we define the training and the inference processes of distance-based elassifiers based on the principle of LDA. Under the assumption that each class-c
the underlying
5

---

Under review as a conference paper at ICLR 2022
latent feature vector is drawn from unimodal Gaussian distribution (()(x)) I), the Euclidean distance (f),,, Me), a simplified version of the Mahalanobis distance, can be assumed to follow the Chi-square distribution having the degree of freedom Then, we have
t
P ( (2) = 2 [(n/2)
exp
where 2 O, r(-) is the Gamma function, and is equivalent to the dimension of fxx)
As previous studies (Jain et al., 2014 [Beddale & Boult, 2016 Rudd et al., 2018 formulated the probability of inclusion by computing the CDF of the Weibull distribution, we define our Pr Pr c) as
D (x.c)/2
r(n/2) exp (-t) dt (m//2, (f(x), Mz) F(n/2)
1
(xx = -1-1111
(8)
by using the CDF of Eq. (7). where ():-) is the upper incomplete Gamma function. It is noteworthy that Eq. (8 can be easily computable via igammac function in PyTorch (Paszke et al. 2019).
Based on Dr, Db, and (x(x,c) c) of Eq. (8). the primary objective of our BCR, which is to make each KUC sample located far away from the closest class-wise HSC,
Pr
be achieved by using a loss
Probability
P
function L.g.u = Eg Ex*DD, log(1 max (1. ..) (x(x, i))l To compare Pr(x, c) and Pn (x, c)
0.5
() pc))) which was used in Eq. 6 we plot Pr c) and Pn (x, e) in Figure with Mell by assuming n = 128. The figure implies that unlike PH (x, c), our (x e) can effectively assign space for KKC data and force KUC samples to be located outside the space.
0.0
respect to [
5
10
15
Distance [[fxx) - Mell
Figure 2: PH and P, (Ours)
At training time, note that each (x c) = 0.5 builds an auxiliary decision boundary between the e-th class KKC sample and the others, where Lag,r. ensures majority of KUC samples located outside the entire class-wise boundaries. However, Lbg.u can be insufficient to achieve robust UUC rejection and closed-set classification results, since it does not control correctly classified KKC samples to be located inside the corresponding class-wise boundaries. Thus, in addition to Lef, we impose regularization Lbg.k k KKC data to maintain closed-set classification performance and enhance the gap between KKC and KUC data in terms of the Euclidean distance. By formulating Lbg.k ixt w)~D, 1nw.D,-1(1p) (y ē) log (x<*,, ,.)]] where è arg max .C Pr Pr i)), we define our BCR loss as Lbg = Lbg.k + Lbg,u and call Lig the class-inclusion loss. In summary, we use
L -Lff + Ahng- Laf + X X(Dng, + Lr
-B(x* D) [- log Pa(y)x*)) - AIx DD [1ly = 8) log(Pr () log(1 (x))]]] (9)
as our total loss, where é = arg max; E (1.... .c PI (x i) and (x(") = maxie(1,"... .C) P1 (x
EXPERIMENTS
In this section, we compared our class-inclusion loss for distance-based classifiers to the objecto- sphere (Dhamija et al. (018)), the uniformity (also widely known as OF) (Hendrycks et al. (2019). and the energy (Liu et al. 20200) losses for Softmax classifiers. Through our extensive experiments, we aim to show that whether our proposed framework provides competitive UUC rejection results in comparison with the previous BCR methods, while keeping robust closed-set classification results.
4.1 EXPERIMENTAL SETTINGS
To assess the OSR performance of each method, we used two experimental settings, which correspond to Sections 1.1. and 4.2. respectively. When evaluating the OSR performance, we first measured the closed-set classification accuracy. To quantify the performance of UUC data rejection, we also measured the area under the receiver operating characteristic curve (AUROC), Furthermore, we used the open-set classification rate (OSCR) measure by quantifying the correct classification rate at the false positive rate of 10-! For in-depth explanation of the OSCR measure, readers are referred to (Dhamija al. 2018). As D,, used ImageNet (Russakovsky et al. 2015), which employed in (Li & Vasconcelos 2020). To ensure that the classes of D, and our test sets are disjoint, used only the remaining classes of the ImageNet dataset, which are not included in the test sets.
also

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison with the previous BCR methods in the first setting of our OSR experiments.
Accuracy (t)
AUROC (t)
OSCR (1)
Experiments
Objectosphere/ / Uniformity / Energy / Class-inclusion (Ours)
SVHN CIFARIO CIFAR+10 CIFAR+50
0.968 00996/ / 0,972 / 0.972
0.914 /0.908 / 00,94/ /0.919
0.809 / /0.785 0.766/ 00,288
0.964 / 0.964 /0.96/ 0.973
0.942 / /0.923 00.93
00946
0.851 /0.814 / 0.807 00869
0.945 / 0.950 / 0.936 00056/0..00 0.960
0.839/ / 0.867 0.808/ 0.880
0,958 0.969 0.949
0.976
0.944 / 0..42/ / 0.933 00,955
0.837 / 00887 / 0.808/ 0.865
TinyImageNet
0.778 / 0.779 / 0.715/ 0.801
0.755 / 0.771 / 0.722 00.88
0.484 / 0.488 / 0.357 0,492
4.1.1 OPEN-SEE SET RECOGNITION SETTING
In Setting 1, we split single dataset into KKCs and UUCs, where we used the KKCs in the training set as Dj, and the KKCs and UUCs in the test set as Diest and Di.st= respectively, Following the experiment protocol in (Neal et al. 2018), which were also employed in (Oza & Patel 2019 Sun et al., 2020), we conducted experiments by using the following standard image datasets: SVHN (Netzer al. 2011), CIFARIO & CIFAR100 (Krizhevsky 2009). and TinyImageNet (Le & Yang 2015).
SVHN CIFAR10 For the SVHN and CIFAR1O datasets, each of which consists of images of 10 classes, each dataset was randomly partitioned into 6 KKCs and UUCs.
CIFAR+10, CIFAR+50 For the CIFAR+M experiments, we used 4 randomly selected classes of CIFAR1O as KKCs and M randomly selected classes of CIFAR 100 UUCs.
TinyImageNet For experiments with large number of classes, randomly selected 20 classes of TinylmageNet as KKCs and then used the remaining 180 classes as UUCs.
4.1.2 OPEN-SET RECOGNITION SETTING 2
By using the training and the test sets of a single dataset as Dr and Dies., respectively, we employed the test set of another dataset relatively close to Dj as Di in Setting 2. Adopting the experiment settings in Yoshihashi et al. 2019) and (Liang et al. 2018), we used the entire classes of a dataset as KKCs for CIFARIO & CIFARI00. For UUC dataset, TinylmageNet, LSUN (Yu al. 2015). and iSUN (Xu et al. 2015) were selected. TinylmageNet and LSUN consists of 10,000 test samples each, where the samples in each dataset were resized (RF) cropped (CR) into the size 32 x 32. The ISUN dataset has 8,925 test samples and they were also resized into the size of 32 x 32. The modified datasets can be obtained in the Github repository of (Liang et 2018).
4.2 TRAINING DETAILS
For f. we employed the Wide-ResNet (WRN) ZZgoruyko & Komodakis 2016) and then used its penultimate layer (x) E Rn for the latent feature vector of each input sample x. For CIFARI 10 and TinylmageNet, we used WRN 40-2 with dropout rate of 0.3, where WRN 28- 10 was employed for CIFAR100 with the same dropout rate. For SVHN, we used WRN 16-4 with a dropout rate of 0.4.
For the entire BCR methods, set the mini-batch sizes of KKC training samples and KUC samples to 128. We kept À as a constant during training, i.c., each ff was trained with the BCR method from scratch. To select hyperparameters and margin parameters of the previous regularization methods, followed the official implementations! F For SVHN CIFAR 10, CIFARIOO, and TinylmageNet, we trained the corresponding classifiers for 80, 100, 200, and 200 epochs, respectively, where we used the stochastic gradient descent for optimization. For SVHN and the other datasets, we used initial learning rates of 0.01 and 0.1, respectively, and cosine learning rate decay (Loshchilov & Hutter 2016). We also used the learning rate warm-up strategy for the first 5 epochs of each training process.
Average runtime We conducted all the experiments with PyTorch and a single GeForce RTX 3090 GPU. At each trial in the CIFARIO experiment of Setting 1, the running time of each training epoch
7

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison with the previous methods in the second setting of our OSR experiments. The corresponding classification accuracy values are reported in the first column.
AUROC (t)
OSCR (1)
Di/'iees
D
Leat
Objectosphere / Uniformity/ Energy Class-inclusion (Ours)
ImageNet-CH
0.988 0.986/ 0,981 0.987
0.929/ 0.928 / 0.894 0.931
ImageNet-RF
0.979/ / 0.984 / 0.992 0..94
0.923 / 0.926 / 0,886 / 0.927
CIFARIO
LSUN-CR
0.994 / 0.990 / 0.989 / 0.993
0.938 / 0,931 / 0.904 / 0.940
0.940 0.939 0,925 0.947
LSUN-RE
0.985 0,988/ 00984/ 0.990
0.928 0.931 / 0.899 0.935
iSUN
0.985 0.989 0.984 0.991
0.928 0.932 0.896 0.936
Average
0,986 0886/0.30 0.987 0,982/ 0.989
0.929 0.930 / 0.895 / 0.934
ImageNet-CR
0.886 / 0,929/ 0.925 / 0.922
0.641 0.686 / 0.652 / 0.693
ImageNet-RE:
0.815 0.910/ / 0.934 00.902
0.572 0.674 0.658 0.683
CIFARI00 0.727 / 0..735 / 007705 0.779
LSUN-CR
0.967 / 0.931 0,901 0.965
0.685 / 0.680 / 0.643 0.751
LSUN-RE
0,844 / 0.930/ 0.959 / 0.945
0.608 / 0.695 / 0.684 0.731
iSUN
0.842 / 0.923 0.954 0.943
0.603 0.689 0.680 0.729
Average
0,871 0.925 0.935 0.935
0.621 / 0.685 0.663/ 0.717
took 28 seconds for our method, where its OSR evaluation required approximately 6.5 seconds. We observed that the other methods take similar running time at their training and inference phases.
4.3 EXPERIMENT RESULTS
The OSR results of our framework and the previous methods are reported in Tables I and and reported values were averaged over five randomized trials, by randomly sampling seed, split of KKCs and UUCs, and class-wiss anchors. In the tables, indicates that it is better to have larger values for the corresponding measure. Also, an underlined value is marginally worse than the best score (bold).
2
All the
Setting 1 For the first setting, Table [ our proposed BCR methods for distance- based classifiers with the previous methods designed for Softmax classifiers. The results demonstrate that our proposed method obtained robust UUC rejection results, which were superior the results of the previous approaches. It is noteworthy that our method achieved higher classification accuracy values, which were critical in acquiring better OSR results in terms of OSCR, than the previous methods. Such results imply that the proposed framework effectively satisfies the two essential requirements described in Section
2.1] bounding open-space risk and ideally balancing it with empirical risk.
Setting In Table 2 we present our experiment results of the second setting. When using CIFAR10 and CIFAR 100 as KKC datasets, the table shows that our model can achieve the highest closed-set set classification accuracy, which is consistent with the experiment results of Setting 1. Furthermore, by averaging the AUROC and the OSCR values over the various UUC datasets, the table shows that our model outperformed the previous methods in UUC sample rejection and OSR.
In summary, the experiment results show that our proposed method can successfully train a robust open-set classifier. To further investigate the effectiveness of our framework in another domain, compared our class-inclusion loss to the uniformity loss in text classification applications.
Text classification For text classification, we used 20 Newsgroups and WikiText103 for KKCs and KUCs, respectively, and trained simple GRU model (Cho et al. 2014) for f as in (Hendrycks) ct al. 2019). For UUCs, we used Multi3K,, WMT16, and IMDB. Since the margin parameters of the objectosphere and the energy losses selected for image classification were not suitable for the text classification, we tested the uniformity loss for comparison. In Table 3 we present the results, where we additionally reported the area under the precision-recall curve (AUPR) and the false-positive rate at 95% true-positive rate (FPR95). As it outperformed the uniformity loss in image classification tasks, our method also showed significantly better performance in text classification experiments.
4.4 ABLATION STUDY AND DISCUSSIONS
For ablation study, further analyzed our BCR method by using various À and class wise anchor initialization & updating methods in our loss function. Also, we compared the proposed method to the triplet loss (Schroff et al. 2015) and analyzed each component of our class-inclusion loss. Using the
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison with the previous BCR method in text classification experiments.
AUROC (1)
AUPR (t)
FPR95 (1)
OSCR (t)
Dzf Dr/Diest test
Desst
Uniformity Class-inclusion (Ours)
Multi30k
0.997 00.997
0.998 / 0.997
0.002 / /0.010
0.715 / 0.745
20 Newsgroups
WMT16
0.997 0.999
0.997 / 0..95
0.010 / 0.016
0.715 0.742
IMDB
0.805 0.999
0.692/ 0.999
0.367 / 0.003
0.585 0.747
0.719/ /0.749
Average
0.933 0.997
0.896 0.997
0.126 / 0.010
0.672 0,745
CIFARIO 10 and TinylmageNet experiments in Setting 1, we present quantitative results in Appendix.
Selecting À Conducting OSR experiments with À E {0.1, 0.5. 1,5, 10), we observed that À = 5 and À = 1 showed the best OSR results in the CIFAR1O and TinylmageNet experiments of Setting 1, respectively, where the results are presented in Section A.1 In our additional experiments. À = 5 yielded the best results in the SVHN and CIFAR + M experiments of Setting and the CIFAR1O experiments of Setting 2, where À = 0.5 showed the best results in the CIFAR 100 experiments. Since the other À values did not significantly degrade OSR performance, one can flexibly select X, where we empirically observed that a lower À value is better when handling more KKCs.
Class-wise anchors In Section 3.1 we mentioned that each class-wise anchor was sampled from the standard Gaussian distribution. Although one can use data-dependent initialization approaches, which are 1) train Softmax classifier and then use its empirical class mean vectors as class-wise anchors of a distance-basee
classifier and 2) compute class mean vectors from the initial feature representations of distance- based classifier and then use the vectors as class- class-wise anchors, such approaches showed worse results than the random initialization method. In addition, we observed that the strategy of updating or training class-wise anchors was not effective to achieve robust OSR performance. We present the corresponding experiment results and discussions in Section [A.2]
Triplet loss To the best of our knowledge, we propose the first distance-based BCR method for the OSR problem. Since the triplet loss (Schroff et al.) 2015) has been widely employed to control the distances between latent feature vectors effectively, we formulated another distance-based BCR strategy with the triplet loss for comparison. We present in-depth details and experiment results for the regularization method using the triplet loss in Section A.3 where the experiment results demonstrate that our class-inclusion loss is more effective than the triplet loss-based regularization.
Loss function Note that our loss function consists of three components Lef, Log.k: and Cog. By using various compositions of the components, we verified in Section (A.4 that the three components are necessary to obtain robust clasffcclassiiicationn and UUC rejection results in our BCR method.
Vanilla distance-based classifiers To show the effectiveness of our method, we assessed the OSR performance of vanilla eestance-based classifiers (trained solely based on Lef). where we present the results in the form of (Accuracy/ AUROC/ OSCR). For the CIFARIO and TinyImageNet experiments of Setting 1, we obtained the results of (0.962 / 0.757 / 0.470) and (0.785/ / 0.629/ 0.315), respectively. In addition, for the CIFAR 10 and CIFAR 100 experiments of Setting 2, the OSR results averaged over the five UUC test sets in vanilla distance- based classifiers were (0.936 / 0.838 0.709) and (0.766 / 0.807 / 0.549), respectively. Comparing these results to the results in Tables [ [and 2 regularization strategy can significantly improve the OSR performance of distance-based classifiers.
we show that our
CONCLUDING REMARKS
In this paper, propose novel BCR method to train open-set classifiers that can provide robust OSR results with a simple inference step. By employing distance-hased classifiers with the principle of LDA, we designed novel class-inclusion loss based on the principle of probability of inclusion, which effectively limits the feature space of KKC data in a class-wise manner and then regularizes UUC samples be located far away from the limited space. Through our extensive experiments and ablation study, we present that our method can achieve robust UUC rejection performance, while maintaining high closed-set set classification accuracy, As this paper aims to improve the reliability of modern DNN-based classifiers, we hope our work to enhance reliability and robustness in various classification applications by providing novel methodology of handling UUC samples.

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Although there are many beneficial applications of classifiers and anomaly detectors, they may lead inadvertent discrimination and encoding societal biases recision-making systems. This one of the cases where ethical considerations depend strongly on the specific applications.
We use several standard datasets including the ImageNet dataset. Although they have been widely used in research, some of the datasets, created from images available on the web, include images of people, raising ethical questions related human-derived data, such as encoding of biases and under- representation issues. The creators of ImageNet are also aware of some of these concerns, and have been attempted address them for the past few years https//mmaageeeettooguuudatteess 11-1111..2pppp
REPRODUCIBILITY STATEMENT
We describe details about distance-based classifiers, our loss function, and the corresponding training and 4.2 4.2 Also, Section 4.1 provide the list of the datasets, which are used
process in Sections 3.1 3.3
this paper. Our source code and the corresponding instructions will be released.
REFERENCES
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1563-1572, 2016.
the
Terrance E Boult, Steve Cruz, Akshay Raj Dhamija, M Gunther, James Henrydoss, and Walter Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9801-9807, 2019.
Hakan Cevikalp, Bedirhan Uzun, Okan Köpüklüü and Gurkan Ozturk. Deep compact polyhedral conic classifier for open and closed set recognition. Puttern Recognition, 1 19: 100080, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In The 2014 Conference on Empirical Methods in Natural Language Processing, 2014.
Akshay Raj Dhamija, Manuel Güinter, and Terrance Boult. Reducing network agnostophobia. Advances in Neural Information Processing Systems, 31:9157-9116 2018.
Chuanxing Geng, Sheng-jun Huang, and Songean Chen. Recent advances in open recognition: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision, pp.
2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribttion examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazcika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2019.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. cemi-supervised learning with normalizing flows. In International Conference on Machine Learning, pp. 4615-4630, 2020.
Lalit P Jain, Walter J Scheirer, and Terrance E Boult. Multi-class open set recognition using probability of inclusion. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 39333939999 409. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2022
Pedro Ribeiro Mendes Júnior, Terrance E Boult, Jacques Wainer, and Anderson Rocha. Specialized support vector machines for open-set recognition. arXiv preprint arXiv:1606.038800 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrateed classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributtin samples and adversarial attacks. Advances in Neural Information Processing Systems, 31:7167-71777, 2018b.
Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13218-13227, 2020.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-disttibution image detection in neural networks. In International Conference Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based nut-of-distribution detection. Advances in Neural Information Processing Systems, 33, 2020.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus- Robert Müller. Explainable deep one-class classification. In International Conference on Learning Representations, 2021.
Nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Metric learning for large scale image classification: Generalizing to new classes at near-zero cost. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 488-501. Springer, 2012.
Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference Computer Vision (ECCV), pp. 613-628, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2307-2316, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 0024-8033 Curran Associates, Inc., 2019.
Ethan M Rudd, Lalit P Jain, Walter J Scheirer, and Terrance E Boult. The extreme value machine. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):762-768, 2018.
Lukas Ruff. Robert Vandermeulen, Billy Joe Franks, Robus-Robett Müller, and Marius Kloft. Rethinking assumptions in deep anomaly detection. arXiv preprint arXiv:2006.03339 2020.
11

---

Under review as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 1-252, 2015. doi: 10. 1007/s1 1263-015-0816-y.
Walter J Scheirer. Extreme value theory-based methods for visual recognition. Synthesis Lectures on Computer Vision, 7(1):1-131, 2017.
Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance Boult. Toward open set recognition. IEEE Transactions Puttern Analysis and Machine Intelligence, 35(7); 1757-1772, 2013.
Walter Scheirer, Lalit Jain, and Terrance E Boult. Probability models for open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(1 (6(11):23177233 2014.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Advances in Neural Information Processing Systems, volume 30, pp. 4080-4090, 2017. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In
Xin Sun, Zhenning Yang. Chi Zhang, Keck- Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13480-13489, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A comprehensive study center loss for deep face recognition. International Journal of Computer Vision, 127(6))668-683, 2019.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
Hong-Ming Yang, Xu-Ya Yao Zhang, Fei Yin, Qing Yang, and Cheng-Lin Liu. Convolutional prototype network for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. cuassffiationn-rcoonstucuiii learning for open-set recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4016-4025, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 160.00066.. 2016.
12

---

Under review as a conference paper at ICLR 2022
A EXPERIMENT RESULTS OF ABLATION STUDIES
To supplement our discussions in Section 4.4, this section provides quantitative experiment results. Based on the CIFARIO and Tiny ImageNet experiments of Setting 1, we present OSR results with various selections of X (Section A.I Also, we report OSR results by using various class-wise anchor initialization & updating methods [S.ction[A.2] For the settings of fixed and trainable anchors in Section A.2 we compare our class-inclusion loss to the triplet loss (Section A.3). Furthermore, we provide additional OSR results by varying the components of our class-inclusion loss (Sections (A.4).
A.1 HYPERPARAMETER À
This subsection presents OSR results by employing various selections of the hyperparameter in our loss function L -= Lef + XLng. In Table we report the OSR experiment results of our method with E {0.1, 0.5, 1, 5, 10). Since the other À values did not significantly degrade OSR performance, one can flexibly select the parameter À to balance empirical and open-space risks. The experiment results in Sections A.2 and A.3 are also provided by using various À E {0.1, 0.5, 1, 5, 10).
Table 4: OSR experiment results by using various À in our daass-ncclusion and the triplet losses. In each cell, the results are presented in the form of (Accuracy (t) / AUROC (t) / OSCR (t)).
CIFARIO
TinylmageNet
Parameter À
Class inclusion
Triplet
Class inclusion
Triplet
0 (Vanilla)
0.962 / 0.757 0.470
0.785 / 0.629 00336
0.1
0.966 / 0.894 0.740
0.963 / 0.820/ / 0.537
0.789 / 0,766 0.442
0.787
0.746 00,31
0.5
0,968 0,927 0.810
0.968 0.844 0.572
0.793 / 0.775 0.463
0.785
0.729 /
0.973 / 0.934 0.839
0.966 0.860 0.628
0,801 / 0.784 0.492
0.793
5
0.973 / 0.946 0.869
0.965 / 0.872 0.628
0.795 / 0.784 / 0.479
0.785
0.707 0.360
10
0.967 / 0.946 0.862
0.958 0.856 0.597
0.785 / 0.698 0.358
0.738 / 0.637 0.220
A.2 CLASS- WISE ANCHORS
This subsection presents OSR results by using various initialization and updating methods of anchors in our approach. In Table 3 we first present OSR results based on trainable class-wise anchors. In this setting, we set each entry of fa, as a trainable parameter and then optimized the anchors along with our model parameters via the same training objective == =Lff + Xhng and optimization processes.
Table 5: OSR experiment results using trainable class-wise anchors in our loss and the triplet loss. In each cell, the results are presented in the form of (Accuracy (t) / AUROC (t) OSCR (f)).
CIFARIO
Tiny Imggeet
Parameter X
Class inclusion
Triplet
Class inclusion
Triplet
0.1
0.966 / 0.894 0.740
0.964 0.767 / 0.318
0.795 0.443 0.038
0.798 0.383 0.052
0.5
0.968 / 0.927 0.810
0.967 0.829 0.550 0.540
0.782 / 0.487 0.050
0.795 / 0.380 / 0.043
1
0.966 / 0.937 / 0.833
0.968 / 0.840 0.577
0.765 / 0.503 0.061
0.798 / 0.383 / 0.046
5
0.971 / 0.946 0.860
0.967 0.872 0.667
0.768 / 0.457 0.041
0.769 / 0.420 / 0.055
0.967 / 0.940 0.827
0.956 0.845 0.580
0.588 / 0.516/ 0.050
0.771 0.426 / 0.051
we observed that there exist only negligible difference of OSR performance between our fixed and the trainable anchor strategies. However, the EnnymmaeNeet experiment results in the tables show that the fixed anchor strategy with random initialization can yield more accurate UUC rejection and OSR results.
Comparing the CIFARIO experiment results in Tables and 5
To further investigate various anchor initialization and updating methods, we additionally assess the OSR performance of our method by employing the following three initialization methods:
Initial Compute anchors from the initial feature representations of distance- classifier.
Transfer Train a Softmax classifier and then use its empirical means for a distance- based classifier.
Random (Ours) Randomly sample each class-wise anchor from standard Gaussian distribution.
13

---

Under review as a conference paper at ICLR 2022
For each initialization method, tested two anchor updating strategies: 1) fix the anchors during training phase, 2) update the anchors via moving average, which was also used in (Wen et al. 2211))
Table 6: OSR experiment results by using various initialization and updating methods for class-wise anchors. In each cell, the results are shown in the form of (Accuracy (t) / AUROC (1) / OSCR (t)).
CIFARIO
TinyImageNet
Anchor
A
Fix
Update
Fix
Update
0.1
0.956 0.801 /0.498
0,962 0.852/ 0.490
0.576 0.378 00011
0.775 /0.415 0.049
0.5
0.964 0.897 / 0.638
0.967 / 0.915 / 0.768
0.576 / 0.596 / 0.042
0.734 0734/0,446 0,446 / 0.055
Initial
0.957 / 0.893 / 00756
0.967 / 0.919 / 008011
0.536 / 0,530 / 0.017
0.740 / 0.429/ 0.042
5
0.958 0.913 /..727
0.961 0,918 0.772
0.506 / 0.426 / 0.007
0.726 / 0.377 / 0.027
10
0.901 0.917 / 0.760
0.957 0.894/ 0.650
0.228 0.481 / 0.052
0.632 0.438 0.036
0.1
0,969 / 0.890 / 0.719
0.965 / 0.863 0.715
0.792 0.553/ / 0.092
0,765 0.390 / 0.035
0.5
0.972 / 0.923 / 0.803
0,971 0,909 0.765
0.802 0.554/ / 0.111
0.777 0.425 0.049
Transfer
0.972 / 0,938/ / 0.838
0,971 / 0.924/ 0.808
0.796 / 0. 5.544/ / 0.093
0.780 0.392 / 0.054
5
0.970 / 0.937 / 00815
0.972 / 0.926 / 0.790
0.779 / 0,551 / 0200
0.738 0.421 / 0.033
10
0.963 / 0.941 / 0.828
0.964 / 0.928 0.776
0.757 / O 448 0.078
0.761 0.377
0.1
0,966 / 0.894 0.740
0.959 0.782 0.26
0.789 / 0666 / 0.395
0.784/ / 0,436/ 0.041
0,5
0,968 / 0.927 / 0.810
0.960 / 0,862/ 0.604
0.793 / 0.775 / 0.463
0.777 0.412 0.049
Random
0,973 0.934/ /.889 0.973 0,946 / 0.869
0.960 / 0.660 / 0.558
0.801 0.784 0.492
0.767 0,438 0.044
5
0 0.262
0.795 / 0. 784 / 0.479
0.738 0,409/
/ 0.042
10
0.967
/ 0,966/
0.947 / 0.740 / 091
0.785 / 0.698 / 0.358
0.683 / 0.440/ 0.046
The results of Tables and 6 present that the OSR results of our approach, which randomly sampled class-wise anchors and then fixed them, are superior those of the other initialization and updating methods. In the following, we discuss the advantages of random sampling and fixed anchor strategies.
In contrast to classical closed-set classification, our BCR method for OSR should ensure sufficient distance gaps between inter-class latent features (inter-class separability) and force class-wise features to have compact feature space (intra-class compactness) by contrasting KKC samples and KUC data located near the KKC data. As discussed in (Izmailov et al.. 2020). the Euclidean distances between anchors drawn from a standard Gaussian distribution are sufficiently large in expectation, which is suitable for our BCR method.
Our loss is based on the probability of inclusion of Eq. (8) where P1 = 0.3 builds a hyper- sphere decision boundary in a class-win manner at training time. The class-inclusion loss makes repulsion forces between KUC samples and the corresponding closest anchors, while simultaneously making KKC samples be located inside the corresponding hypersphere boundary. In this regularization method, it is desirable to fix class-wise anchors to control each KKC or KUC sample in a consistent direction. In other words, the strategy of updating or training class-wise anchors, which also changes the location of class-wise hypersphere boundaries, is likely to obstruct the process of contrasting KKC and KUC samples.
A.3 TRIPLET LOSS-BASED REGULARIZATION
This section compares our BCR method to a regularization method based on the triplet loss Lirs, in order to show that our class-inclusion loss is an effective BCR method for distance-based classifiers Following the conventional definition of the triplet loss, we simply formulated Ciri by setting class- wise anchors, KKC training data, and KUC data as anchors, positive samples, and negative samples, respectively. Then, we compared our loss to Ler: in fixed and trainable anchor settings.
In Tables 4 and 5 which presented the OSR results of distance-based classifiers with fixed class means and trainable class means, respectively, we reported experiment results by using the triplet loss as Lbg. Since we observed that training classifiers solely based on the triplet loss L = Levi yields significantly worse OSR results in comparison with the regularization method L = Lef + ALirs- we employed Lirs as a regularization loss function for BCR. The results show that method
proposed
loss) outperforms the regularization method using the triplet loss. Also, the discussions in Sections A. and A.2 can also be applicable to the triplet base- regularization.
14

---

Under review as a conference paper at ICLR 2022
A.4 LOSS FUNCTION FORMULATION
In our inasusncl loss C = Las + A Abg & + Lig.u), Les is a conventional negative log-likelihood loss for the distance-based classifier of Eq. 44). Based on our probability of inclusion of Eq. ( we formulated Lig.k and L.g.a. for regularization. This subsection investigates the necessity of the three terms Lef. Lhg.k. and Lig.u, since both of the Lef and Lbg.k terms aim to force KKC training samples located close to the corresponding class-wise anchors. We employed various compositions of the three terms to formulate loss functions and then reported their OSR results in Table 7
As our original L term only controls KKC training samples having correct class predictions with b = (y)D,,))D, [- 11 = 2) log( Py (x*, .))] and è = arg (1.., ..) Px i), we define modified version Lig. (xx w))w, log(P (x* y)) to correctly remove Lef from L.
Table 7: OSR experiment results based on various settings of loss functions.
CIFAR10
TinylmageNet
Loss function C
Parameter
Accuracy (t) / AUROC OSCR (1)
Cbg.k + Lbg.u
0.937 /..932 / 0.799
0.629 /0.688 0.334
0.1
0,963 / 0.821 / 0.509
0.793 / 0..33/ / 00.408
0.5
0,961 00800 0.488
0.779 /0.732 00.24
Lef ALog.u
0.958 0.806 0,485
0.787 / 0.726 0.392
0.963 / 0.812 / 0..00
0.767 / 0.733 0.397
10
0,960 / 0.798 0.459
0.779 / 0.732 / 0.424
0.1
0.965 0.868/ 0.674
0.795 / 0.781/ / 0,443
0.5
0.972 / 0.930 / 0.822
0.800 / 0.763 00.66
Lez M(Cigg + Cog.u)
0,973 0,933 0.835
0.785 / 0.720/ 0.378
0.966 0.931 0.815
0.639 / 0.630 0.278
10
0.956 0.919 0.771
0.458 / 0.629 0.178
0,966 8.4 0,740
0.789 0.766/ 0.442
0.5
0.968 0,927 / 0.810
0.793 / 0.775 0. 0.463
Les + AChnggk + Lagga)
0.973 / 0.934/ / 0.839
0.801 / 0.784/ / 0..92
0.973 / 0,946/ / 0.869
0.795 / 0.784/ 0.479
10
0.967 0.946 0.862
0.785 / 0.698 0.358
In Table it is shown that our loss function (Cof + A(Cog, + Lbg.u)) can achieve robust OSR results for various selections of X. As fig.k (or Ltg.k) only forces KKC samples to be located inside the class-wise hypersphere boundaries of the probability of inclusion, which rapidly decays from to O, the loss term is not sufficient to achieve high closed-set classification accuracy. On the contrary, KKC samples cannot be forced to be located inside the boundaries when using Lej without Log,k, thereby making inaccurate OSR results. In summary, both Lef and Log.k are necessary to conduct accurate classification and increase the distance gap between KKC and KUC data, respectively. It noteworthy that our formulation (using both Lef and Log,k) is beneficial in maintaining robust classification accuracy. Also, our method, which only regularizes KKC samples having correct class prediction, can mitigate overfitting issues in comparison with the approach of (Cef + X + Lhg.a))..
B ADDITIONAL EXPERIMENTS
In additional experiments. we employed CIFARIO for D,, For Setting of our OSR experiments, we split the 10 classes of CIFARI0 into 6 known known classes (KKCs) and 4 unknown unknown classes (UUCs). In Setting 2, we used CIFARIO 10 and another dataset as KKCs and as UUCs, respectively. For the first setting of our experiments, we report quantitative results in the form of (Accuracy / AUROC / OSCR). In this section, we denote Settings and 2 by S1 and s2 for simplicity.
B. QUANTITATIVE RESULTS WITH RESNET-18 18 ARCHITECTURE
In Section 4, used the Wide ReseeResNet (WRN) architectures as feature extractors for the image datasets. To further investigate the effectiveness of our method, we used another standard classifier architecture, R8 (He et al. 2016). In s1, obtained quantitative results of (0.963 0.945 /
15

---

Under review as a conference paper at ICLR 2022
0.844), (0.966/ 0.950 0.845), (0.967/0.945 0..48).888..... and (0.971 / /0947/ / 0.850) for the regularization
methods using the objectosphere (Dhamija et al., 2018). the uniformity (Hendrycks et al., 2019). the energy (Liu et al., 2020), and our dass-inclusion losses, respectively. Such quantitative OSR results and the results in Table g imply that our method can outperform the previous BCR methods with ResNet- 18, as we observed in the experiments using the WRN architectures.
Table 8: Comparison with the previous methods in the second setting of our OSR experiments using ResNet- 18. The corresponding classification accuracy values are reported in the first column.
AUROC (t)
OSCR (1)
Di/Diest
D
text
Objectosphere / Uniformity Energy Class-inclusion (Ours)
ImageNet-CR
0.982 0,999 0,983 0.987
0,932 0.928 / 0.917 0.941
ImageNet-RR Net-RE
0.977 0.982 0,975 0.988
0.918 0,934 / 0,909 / 0.945
CIFARIO 0.937 0.949/ /0993/ 0.951
LSUN-CR
0.991 0.984 0,987 0.993
0,934/ / 0.935 / 0,919 / 0.946
LSUN-RE
0,987 0,987 0,986 0.990
0.932 0.938 / 0.918 / 0,942
iSUN
0.987 0.986 0.987 0.994
0,932 0.938 0,919/ / 0.946
Average
0.985 / 0.55/0094//.... 0.984 / 0.991
0.930 0.935 / 0,916 0,944
B.2 LATENT FEATURE SPACE VISUALIZATION
By visualizing feature spaces via I-SNE Vaan der Maaten & Hinton 2008), we compared Softmax classifiers and distance-based classifiers, and their regularized versions in S1 and s2. We selected ImageNet-RE RE for UUC data in S2, since the results of Sableiimply that the dataset is more difficult to be recognized by open-set classifiers than the others. In each t-SNE result, black dots indicate UUC samples. For the other colors, each distinct color represents the corresponding class of KKCs.
B.2.1 VANILLA CLASSIFIERS (NO BACKGROUND-CLASS REGULARIZATION)
(a) S1 Softmax
(b) S1 Distance
(c) S2 Softmax
(d) S2 - Distance
Figure 3: t-SNE results of vanilla Softmax classifiers and distance-based classifiers.
Figure 3 depicts the t-S results of vanilla Softmax and distance- based classifiers, which are trained only with L Left in S1 and S2. The visualization results imply that in comparison with the UUC data used in s2, it would be much more difficult to distinguish UUC samples in S1 from KKCs.
B.2.2 BACKGROUND CLASS REGULARIZED CLASSIFIERS
(a) S1 - Uniformity
(b) S1- Ours
(c) S2 Uniformity
(d) 32 Ours
Figure 4: t-SNE results of regularized Softmax classifiers and distance-based classifiers.
16

---

Under review as a conference paper at ICLR 2022
Figure 4 presents the t-SNE results of Softmax and distance-based classifiers trained with BCR (C= = Lef + >Log) in S1 and S2. For the Softmax and distance-based classifiers, we employed the uniformity loss (Hendrycks et al. 2019) and our class-inclusion loss Lag. respectively, In comparison with Figure 3 Figure 4 shows that such regularization techniques assist classifiers to learn robust latent feature representations in distinguishing UUCs from KKCs, especially in $1. Also, as mentioned in Section 2.3 the figure shows that the latent feature space regularized by the uniformity loss can yield inaccurate results in distance-based post-classification analysis. By applying the Openmax approach (Bendale & Boult 2016) to the model regularized by the uniformity loss in (Figure 4. (a)), we obtained the OSR performance of (0.923 0.886 / 0.752), which worse than the original result (0.964/ / 0.923 / 0.814) and the result of our framework (0.973/ / 0.946 0.869).
17


Reviews:
[{"review_id": "se7Ap1HJGnj", "reviewer": "Reviewer_1wRX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main goal of the work is to define a novel open-set classifier on the top of DNN which can improve the detection of unknown classes while keeping the accuracy of original closed-set recognition. This is a well-studied problem and typical applications of image and text classification are considered. A group of different techniques from prior work are combined in a novel way and each part of the model is justified and the whole model is experimentally verified against state-of-the-art.\n\nStrong points\n \nThe paper is presenting an interesting approach to open-set recognition in the case of background class information (i.e., data of known-unknown classes) is available. Although, proposed method combines several different techniques (i.e., LDA classifier, inclusion probability modelling, and loss function extended with class-inclusion loss), the model choices are clearly and well justified. Furthermore, the limitations, issues, and differences of the most similar previous works are described in detail to support and motivate the proposed methodologies and model. The claims of model properties are experimentally validated against three previous approaches, and it is shown that open-set recognition rate is improved, and the similar or better close-set accuracy is achieved simultaneously. It is also stated that code and experiments will be available to reproduce the results.\n\nSummary of pros\n- Clearly written\n- Clear and theoretically sound techniques\n- Comprehensive experiments on benchmark datasets\n- Reproducibility based on detailed equations and code/experiments to be released\n\nWeak points\n\nAlthough the whole classifier is well-validated, there is room for improvement in studying the behavior of different parts and parameters of the model. Especially, in ablation study section, to get better view how different part of the model and parameters are behaving, it would be useful to give more detailed description and illustrations: 1) When experimenting with different lambda parameter, it would be useful to show graphically how the change of lambda affects the classification accuracy and OSCR in different datasets. 2) When experimenting different class mean vectors initialization, it would be useful to show the detailed results and collect them to a single table, to get better view how robust they are and how significant the differences are.\n\nAs claimed, a simple inference process, loss function with fewer data-dependent parameters, and ability to bound open-space risk without additional post-analysis, are proposed. To strengthen the presentation and claims, it would be good to show how computational complexity and efficiency of the proposed classifier in relation to training and inference times compares to previous methods utilizing SoftMax classifiers.\n\nSummary of cons\n- Ablation study is somewhat vague; more detailed and illustrative experiments would strengthen the analysis of model parameters and choices (e.g., selecting lambda, detailed performance of different initialization of class mean vectors)\n- Missing computation complexity and efficiency analysis of proposed method in relation to training and inference times (if there are any significant differences compared to previous methods?)\n\nOther (minor) comments and questions\n- Eq. 9: It would be useful to write out the final loss equation with classification and class-inclusion parts included. Now they are given in the middle of the text only\n- In 4.1: use similar term S1/S2 or setting 1 / setting 2 in each following subsection when referring to different experiments\n- Background dataset (ImageNet) could be introduced already in 4.1 (because it is used in both S1 and S2)\n- Result presentation (Table 1 and Table 2): Equally good results of previous methods should be also bolded (Table 1: SVHN accuracy of Energy model, Table 2: CIFAR100 average AUROC of Energy model), or are these rounded to third decimal and class-inclusion model is marginally better?\n- In ablation studies related to classification accuracy (with conventional loss function), it would be useful to collect these results to a new table for comparison\n- How is the best/optimal lambda selected in experiments? How much does it affect the classification accuracy and OSCR numerically?\n- How much does different class mean vector initialization strategies (empirical mean, feature mean, random sampling, re-computed empirical mean) affect the model performance? and are there any explanations of why they behave/perform differently?\n- Are there some good practices of how “known unknown classes” (KUC) data should be chosen or generated? Could this be automated? How robust will the proposed model be given the different choices of KUC data?", "summary_of_the_paper": "This paper proposes a new approach for generalized open-set recognition (OSR) in classification setting where the idea is to build a model which could simultaneously detect unknown classes and maintain the accuracy of traditional closed-set classification of known classes. The proposed OSR is based on background class regularization (BCR) where data of known unknown classes are utilized in the training phase of the classifiers based on typical deep neural network (DNN) architectures.\n\nPrevious works applying BCR have treated known-classes as one group and unknowns as other with SoftMax classification layer, leading to limited OSR performance, especially near the known classes, whereas proposed work regularizes and limits feature space in class-wise manner, making the data of unknown classes to be located far away from the known classes. Distance-based classifier applying linear discriminative analysis is used to model the latent features from DNN and to design novel loss function which can regulate the solution between known classes and unknown background classes in OSR setting, simultaneously keeping the robust performance of closed-set classification.\n\nProposed approach is experimentally compared to three previous works applying BCR-based OSR with SoftMax classifiers. Proposed method outperforms those three related methods in well-known public benchmarks datasets of image and text classification tasks. Different performance metrics of classification accuracy, area under the receiver operating characteristic curve (AUROC), and the open-set classification rate (OSCR) are utilized in comparison.\n\nTo summarize the contribution, paper presents LDA principled distance-based classifier (applied to output of neural network latent feature layer) and new loss function to train OSR-sensitive classifiers, developed and formulated based on the limitations of previous work and experimentally validated against previous BCR-based approaches.", "summary_of_the_review": "The paper presents an interesting and novel approach combining several techniques to improve DNN-based open-set recognition with distance-based classifier. Paper is well-written and the methodological choices are properly justified. More detailed ablation study and the analysis of the model efficiency would still improve the presentation and verify the claims better. This is an important and practical problem and can bring new insight for the field. I recommend the paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ORd1Vgkb7sF", "reviewer": "Reviewer_Mkdh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper:\n+ The paper is well-written and easy to read.\n+ A distance-based background class regularization (BCR) method using the principle of linear discriminant analysis is proposed to address the problem of open-set recognition.\n+ Ablation study is well designed to show the performance of the proposed distance-based method.\n\nThe weaknesses of the paper:\n- The novelty of this work is limited. The distance-based classifier that uses the principle of linear discriminant analysis has been studied in previous method, e.g., [1] Thomas Mensink, Jakob Verbeek, Florent Perronnin, Gabriela Csurka, Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.\n- The experimental evaluation of this paper is not sufficient. The current version lacks comparisons with existing Softmax based methods, e.g.,  [2] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song, Sphereface: Deep hypersphere embedding for face recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2017; [3] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu, Additive margin softmax for face verification, IEEE Signal Processing Letters, 2018.", "summary_of_the_paper": "This paper introduces a distance-based background class regularization (BCR) method for open-set recognition (OSR), in which the distance-based classifier that uses the principle of linear discriminant analysis is utilized to limit the feature space of known-class data in a class-wise manner and make background-class samples far away from the limited feature space. Experiments show the robust OSR performance of the proposed method.", "summary_of_the_review": "The paper is well-written and easy to read. However, the novelty is limited and the experiments are not sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7oJL7fbK9p", "reviewer": "Reviewer_rop6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors propose a new method for open set recognition. To this end, the authors use distance based classifiers and they minimize the distance between the class samples and their corresponding means for the labeled data samples and try to enforce the background data samples to be outside the class acceptance regions. \nThe strengths of the paper can be summarized as follows:\n1) The paper is written well and the motivation and the proposed methodology are described well.\n2) The authors aim to propose compact acceptance regions for the known classes and they enforce the background samples to lie outside of these regions. The background samples help to return more compact and correct class regions. This idea makes sense and there are recent studies applying the same idea.\n3) Experimental results are good in the sense that the proposed method outperforms the competing methods.\nThe weaknesses of the paper can be summarized as follows:\n1) The idea of using distance-based classifiers for returning compact class regions where the known class samples lie inside these regions and negative (background) samples lie far from these regions have been used before. The authors cited some of these studies, but some important references are missing. Among these, the method using center loss function [R1], polyhedral conic classifiers [R2] and Convolutional prototype networks [R3] should be mentioned. Especially, the deep polyhedral conic classifier [R2] implements exactly the same idea. It returns bounded convex acceptance regions for the class samples and enforces the negative samples to lie outside of these regions. \nThe authors use a similar idea, and they enforce to minimize the distances between the known class samples and their corresponding class centers. They also use background class samples to lie outside the class acceptance regions. To this end, they adopt the idea from Explainable Deep One-Class classification method and they introduce a slightly different loss term that utilized background samples. Therefore, the overall novelty is limited in my opinion.\n2) I strongly believe that setting class specific means in the proposed methodology is problematic. The authors randomly select the class means from the standard Gaussian distribution and fix them. This is unacceptable. It should be noted that the features change in each epoch in deep neural network classifiers, and the class means also change in the process. Therefore, the class means must be also updated based on the changed features. This way, local similarities between the semantic classes can be preserved (For example, in the feature space we expect that the class samples belonging to the cat and dog classes lie in close regions and they are probably far from other classes such as aeroplanes, cars, etc.). But when the class samples are chosen randomly and the class samples are enforced to lie in the vicinity of these randomly chosen centers the semantic similarities are completely destroyed. Note that Izmailiov et al. use this setting in the generative models and this is not appropriate for the classification setting. Furthermore, they also recommend the updating class centers. \nThe authors report that updating class mean vectors yield worse results than fixed class mean vectors. In contrast, both [R1] and [R2] also update the class mean vectors and report good accuracies. The authors must look into this issue in more details (change parameter settings or update centers using the formulations given in [R1] with appropriate weights).\n3) The authors use ImageNet dataset for background classes. I was wondering if the overlapping classes between tested datasets and this one are removed. Also, the authors use a completely different setting (using a background class) in their experiments and therefore it is hard to compare the results to the some published related methods. Of course, it is hard to implement every method, but the authors must definitely give results for a distance metric learning method using triplet loss function. Anchors can be set class means and constraints can be easily formed by using known class labels and background class labels. Lastly, an ablation study showing the importance of the utilized background class is necessary in my opinion to judge the effect of background class samples for returning more compact class acceptance regions.\nReferences:\n[R1] Wen et al., A comprehensive study on center loss for deep face recognition, Int. Journal of Computer Vision, 127, 2019.\n[R2] Cevikalp et al., Deep compact polyhedral conic classifier for open and closed set recognition, Pattern Recognition, 2021.\n[R3] Yang et al., Convolutional prototype network for open set recognition IEEE Trans. on Pattern Analyis Machine Intelligence, 2020.\n", "summary_of_the_paper": "In this paper, the authors propose a new method for open set recognition. To this end, the authors use distance based classifiers and they minimize the distance between the class samples and their corresponding means for the labeled data samples and try to enforce the background data samples to be outside the class acceptance regions. Then the samples coming from the unknown classes can be rejected based on the distances to the centers of the known classes. The authors use ImageNet data samples are used as background class samples. The proposed method is compared to related open set recognition methods and better accuracies are reported.", "summary_of_the_review": "In general, the novelty is kind of limited but the experimental results are quite good. There are some issues that must be addressed regarding the setting class centers. Also, more comparison to the related methods and some ablations studies are needed in experiments. A borderline paper closer to the acceptance region in my opinion. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4YfKKXDbFhF", "reviewer": "Reviewer_ujMG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe main contribution is a loss function that includes background class regularization (BCR) based on probability of inclusion estimated by the CDF of Weibull distribution, though BCR and CDF of Weibull distribution are not new. Empirical results indicate that the proposed method compares favorably against a few existing algorithms on a few datasets.  The paper is generally well written.\n\nWeakness:\n\nSince both L_cf and L_bg,k try to get known instances close to their class means,  an ablation study on excluding either one of them would be interesting.  Also, a more detailed analysis and discussion of why L_bg,k is needed would be helpful. \n\nMinor:\n\nParagraph next to Figure 2: Eq 7 -> Eq 8 for P_I", "summary_of_the_paper": "For open set recognition (OSR), the authors propose using background data to represent instances from unknown classes.  The background data are from data outside the known classes.  Based on LDA, they proposed using the distance from an instance to the class mean to estimate the posterior class probability via softmax.  The class means are randomly chosen beforehand.  Their loss functions includes cross entropy, L_cf, and background class regularization (BCR), which is denoted as L_bg.   For L_bg, they introduce probability of inclusion, P_I, which is estimated by the CDF for Weibull distribution.  With the background data, they calculate L_bg,u  based on 1-P_I.  With the known data, they calculate L_bg,k based on P_I.\n\nEmpirical evaluations have two settings.  The unknown classes from the first setting are from the same dataset as the known classes.  In the second setting the unknown classes are from another dataset.  Background data are from another dataset.  In the both settings, the proposed method compares favorably against a few existing algorithms on a few datasets.\n\n", "summary_of_the_review": "While the empirical results are favorable, justification and analysis of including L_bg,k could be more detailed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the open-set recognition problem, specifically the subset that looks at rejecting test data that with unknown classes that are related to the training data. The proposed approach uses an existing distance-based classifier (based on LDA) combined with a new background class regularizer. Results, comparing to a few prior OSR methods, are shown across image/text datasets. 

  The reviewers gave a mixed set of scores, with concerns about visualization/ablation studies and the lambda parameter with affect on classification accuracy (1wRX, ujMG, rop6), computation complexity and efficiency (1wRX), limited novelty and discussion of relationship to prior works (Mkdh, rop6), and limited comparison to state of art as only a few algorithms are compared to the proposed approach (Mkdh), and initialization method. Notably, the authors make a strong claim for the latter point that the method should only be compared to previous BCR methods (as opposed to softmax-based classifiers, for example); this seems to ignore whole classes of different methods that can approach the OSR problem. While it is true that comparing to previous BCR methods can directly show your approach is superior to them under similar class of algorithms (thereby showing that it is an improvement), putting the method within the context of the entire literature is absolutely necessary to discuss relative impact to the field. For example, the improvements in AUROC are not that great (and in some cases worse) than even the methods you compare to, while OSCR is improved significantly, so it is not clear how it stacks up with respect to the current state of art. Even if it doesn't beat it, you could argue your contribution, but not presenting it all prevents a holistic perspective that is necessary. 

  The authors provided thorough rebuttals, including additional ablations and experiments. However, after the review period the scores remain mixed (5,5,6,8) and the reviewers expressed remaining concerns about novelty and comparison to the current state of art (not just BCR-based methods). As a result of these remaining concerns, I recommend rejection at this time.
Decision: Reject