Distance-Based Background Class Regularization for Open-Set Recognition

1 INTRODUCTION . In machine learning ( ML ) , classification algorithms have achieved great success . Through recent advances in convolutional neural networks , their classification performance already surpassed the human-level performance in image classification ( He et al. , 2015 ) . However , such algorithms are usually developed under a closed-set assumption , i.e. , the class of each test sample is assumed to always belong to one of the pre-defined set of classes . Although this conventional assumption can be easily violated in real-world applications ( classifiers can face unknown-class data ) , traditional algorithms are likely to force unknown-class samples to be classified into one of the known classes . To tackle this issue , the open-set recognition ( OSR ) problem ( Scheirer et al. , 2013 ) aims to properly classify unknown-class samples as “ unknown ” and known-class samples as one of the known classes . According to the definition of OSR ( Scheirer et al. , 2013 ) , it is required to properly limit the feature space of known-class data . To satisfy the requirement , various OSR methods were developed based on traditional ML models . Previously , Scheirer et al . ( 2014 ) calibrated the decision scores of support vector machines ( SVMs ) . Based on the intuition that a large set of data samples of unknown classes can be rejected if those of known classes are accurately modeled , Jain et al . ( 2014 ) proposed PI -SVM , which utilized the statistical modeling of known-class samples located near the decision boundary of SVMs . Afterward , Júnior et al . ( 2016 ) attempted to solve the OSR problem based on the principle of the nearest neighbors . Taking distribution information of data into account , Rudd et al . ( 2018 ) proposed the extreme value machine by utilizing the concept of margin distributions . Since deep neural networks ( DNNs ) have robust classification performance by learning high-level representations of data , OSR methods for DNNs have received great attention . Based on the theoretical foundations used in traditional ML-based OSR methods , Bendale & Boult ( 2016 ) proposed the first OSR strategy for DNNs called Openmax , which calibrates the output logits of pre-trained Softmax classifiers . To improve Openmax , Yoshihashi et al . ( 2019 ) proposed the classification-reconstruction learning to make robust latent feature vectors . Afterward , Oza & Patel ( 2019 ) proposed to exploit a class-conditioned autoencoder and use its reconstruction error to assess each input sample . Sun et al . ( 2020 ) employed several class-conditioned variational auto-encoders for generative modeling . Although previous OSR methods using discriminative models applied offline analyses to pre-trained Softmax classifiers or employed complicated DNN models , they have limited performance since the classifiers were trained solely based on known-class samples . To mitigate the problem , one can use background class regularization ( BCR ) to achieve robust empirical results . However , previous BCR methods ( Dhamija et al. , 2018 ; Hendrycks et al. , 2019 ; Liu et al. , 2020 ) are insufficient to properly solve the OSR problem . To design an effective open-set classifier that can overcome the previous limitations , we propose a novel BCR method suitable for OSR , which uses a distance-based classifier and a novel loss function for regularization . We provide detailed description in the next section . 2 PRELIMINARIES AND OUR CONTRIBUTIONS . 2.1 THE OPEN-SET RECOGNITION PROBLEM . The OSR problem addresses the classification setting that can face test samples from classes unseen during training . In this setting , open-set classifiers aim to properly classify known-class samples while rejecting unknown-class samples simultaneously . A similar problem to OSR is out-of-distribution ( OOD ) detection ( Hendrycks & Gimpel , 2017 ) , which typically aims to reject data items drawn far away from the training data distribution . In previous OOD detection studies ( Hendrycks & Gimpel , 2017 ; Liang et al. , 2018 ; Lee et al. , 2018a ; b ) , OOD samples tend to be drawn from other datasets or even images of noise . In this paper , we aim to reject test data whose classes are unknown but related to the training data , which narrows down the scope of conventional OOD detection tasks . Previously , Scheirer et al . ( 2013 ) introduced a formal definition of the OSR problem based on the notion of open-space risk RO , which is a relative measure of a positively labeled union of balls SV and open space O located far from SV . Since labeling any data item in O incurs open-space risk , it is straightforward that a classifier can not be a solution for the OSR problem if the classifier accepts data in infinitely wide regions , i.e. , its open-space risk is unbounded ( RO = ∞ ) . The definition implies that essential requirements to solve the OSR problem are 1 ) bounding open-space risk and 2 ) ideally balancing it with empirical risk to maintain a low classification error rate . Unlike traditional closed-set classifiers , open-set classifiers are required to limit the space of knownclass data to bound their open-space risk . To ensure bounded open-space risk , Scheirer et al . ( 2014 ) proposed to formulate compact abating probability ( CAP ) models . The principle of CAP models is that if the support region of a classifier decays in all directions from the training data , thresholding the region will bound the classifier ’ s open-space risk ( Boult et al. , 2019 ) . As depicted in Figure 1 , which compares traditional closed-set and open-set classification problems , properly building class-wise CAP models is an effective strategy to satisfy the two essential requirements of OSR . 2.2 POST-CLASSIFICATION ANALYSIS FOR PRE-TRAINED SOFTMAX CLASSIFIER . In this paper , we aim to solve the OSR problem by using a standard DNN-based classifier architecture f as a latent feature extractor . Applying a fully-connected layer to f , a conventional Softmax classifier computes the posterior probability of an input x belonging to the c-th known class by Ps ( y = c|x ) = exp ( wTc f ( x ) + bc ) ∑C i=1 exp ( w T i f ( x ) + bi ) , ( 1 ) where c ∈ { 1 , · · · , C } , f ( x ) ∈ Rn is the latent feature vector of x , and wc and bc are the weight and the bias for the c-th class , respectively . For pre-trained Softmax classifiers , Hendrycks & Gimpel ( 2017 ) proposed a baseline to detect anomalous samples , which imposes a threshold on the predictive confidence of Eq . ( 1 ) . When using the baseline method to solve the OSR problem , one can estimate the class of each known-class sample and recognize unknown-class data by ŷ = { argmaxc∈ { 1 , ··· , C } Ps ( y = c|x ) , if maxc∈ { 1 , ··· , C } Ps ( y = c|x ) ≥ τ , C + 1 ( unknown class ) , otherwise . ( 2 ) However , Eq . ( 2 ) can not formally bound open-space risk since it only rejects test samples near the decision boundary of classifiers , thereby having infinitely wide regions of acceptance ( Boult et al. , 2019 ) . Therefore , additional post-classification analysis methods using an auxiliary measure other than the Softmax probability are necessary to bound open-space risk , where distance measures have been widely used to build auxiliary CAP models in the latent feature space of f . To construct class-wise CAP models , Openmax ( Bendale & Boult , 2016 ) defined radial-basis decaying functions { s ( x , i ) } Ci=1 , each of which measures the class-belongingness of x for the c-th class , in the latent feature space of f . For each s ( x , c ) , the authors employed distance measures between f ( x ) and an empirical class mean vector µc , e.g. , s ( x , c ) = D2E ( f ( x ) , µc ) = ( f ( x ) − µc ) T ( f ( x ) − µc ) . To formulate effective CAP models based on s ( x , c ) , they statistically modeled the distribution of s ( x , c ) for known-class data based on the extreme value theory ( EVT ) ( Scheirer , 2017 ) , which provides a theoretical foundation that the Weibull distribution is suitable for modeling known-class samples located far from the class mean vectors ( extreme samples ) . To be specific , Openmax fits a Weibull distribution on extreme samples of the c-th class having the highest DE ( f ( x ) , µc ) values , where its cumulative distribution function ( CDF ) formulates the probability of inclusion PI ( x , c ) ( Jain et al. , 2014 ; Rudd et al. , 2018 ) , i.e. , PI ( x , c ) = 1 − WeibullCDF , which rapidly decays near extreme samples . Based on PI ( x , c ) , the decision rule of Eq . ( 2 ) can be calibrated to conduct robust OSR . 2.3 GENERALIZED OPEN-SET RECOGNITION AND BACKGROUND CLASS REGULARIZATION . Although they require additional inference procedures ( e.g. , EVT modeling ) to bound the open-space risk of pre-trained Softmax classifiers , previous post-classification analyses may have limited OSR performance since the classifiers are trained solely based on known-class samples . To empirically obtain robust OSR results without complicated post-classification analyses , one can use the strategy of BCR at the training phase , which exploits background-class samples as surrogates of unknownclass samples . Geng et al . ( 2020 ) argued that the generalized OSR setting that utilizes background samples is still less-explored and an important research direction to improve OSR performance . In the generalized OSR setting , classifiers should consider the following data classes among the infinite label space of all classes Y ( Dhamija et al. , 2018 ; Geng et al. , 2020 ) . • Known known classes ( KKCs ; K = { 1 , · · · , C } ⊂ Y ) include distinctly labeled positive classes , which also have side information , where U = Y \ K is the entire unknown classes . • Known unknown classes ( KUCs ; B ⊂ U ) include background classes , e.g. , labeled negative classes which are not necessarily grouped into a pre-defined set of KKCs K. • Unknown unknown classes ( UUCs ; A = U \B ) represent the rest of U , where UUC samples are not available at training time and have no side information , but occur at inference time . Throughout this paper , we denote the corresponding datasets for the data classes as follows . Dt is a training set consisting of multiple pairs of a KKC data sample and the corresponding class label y ∈ { 1 , · · · , C } . Dktest and Dutest are test sets of KKCs and UUCs , respectively . Db is a background dataset of KUCs . Then , a loss function for training classifiers with BCR can be formulated by L = Lcf + λLbg = E ( xk , y ) ∼Dt [ − logPs ( y|xk ) + λExb∼Db [ freg ( xk , y , xb ) ] ] , ( 3 ) where Lcf and Lbg are the losses for classification and BCR , respectively , and λ is a hyperparameter that balances Lcf and Lbg . For Lbg , previous studies designed their own freg , where Dhamija et al . ( 2018 ) proposed the objectosphere loss for OSR , and Hendrycks et al . ( 2019 ) and Liu et al . ( 2020 ) employed the uniformity and the energy losses for OOD detection , respectively . In this paper , we tackle the following limitations of the previous BCR methods . • In the previous BCR methods , Lbg were designed to make normal data and anomalies more distinguishable in terms of the corresponding anomaly scores . Since they categorized normal samples into a single group ( do not consider the classes ) in Lbg , the previous methods can not be suitable for rejecting UUC samples located relatively close to KKC data and maintaining robust closed-set classification results . • The previous methods using the decision rule of Eq . ( 2 ) ( e.g. , objectosphere ( Dhamija et al. , 2018 ) and uniformity ( Hendrycks et al. , 2019 ) ) can not bound open-space risk . Although one can use post-classification analyses to bound open-space risk , trained latent feature spaces are likely to be inappropriate for using another metric such as distance measures . • To increase the gap between KKC and KUC samples in terms of latent feature magnitude and energy value in the objectosphere ( Dhamija et al. , 2018 ) and the energy ( Liu et al. , 2020 ) losses , respectively , it is necessary to find proper margin parameters for each dataset .

Summary: This paper proposes a new approach for generalized open-set recognition (OSR) in classification setting where the idea is to build a model which could simultaneously detect unknown classes and maintain the accuracy of traditional closed-set classification of known classes. The proposed OSR is based on background class regularization (BCR) where data of known unknown classes are utilized in the training phase of the classifiers based on typical deep neural network (DNN) architectures.  Previous works applying BCR have treated known-classes as one group and unknowns as other with SoftMax classification layer, leading to limited OSR performance, especially near the known classes, whereas proposed work regularizes and limits feature space in class-wise manner, making the data of unknown classes to be located far away from the known classes. Distance-based classifier applying linear discriminative analysis is used to model the latent features from DNN and to design novel loss function which can regulate the solution between known classes and unknown background classes in OSR setting, simultaneously keeping the robust performance of closed-set classification.  Proposed approach is experimentally compared to three previous works applying BCR-based OSR with SoftMax classifiers. Proposed method outperforms those three related methods in well-known public benchmarks datasets of image and text classification tasks. Different performance metrics of classification accuracy, area under the receiver operating characteristic curve (AUROC), and the open-set classification rate (OSCR) are utilized in comparison.  To summarize the contribution, paper presents LDA principled distance-based classifier (applied to output of neural network latent feature layer) and new loss function to train OSR-sensitive classifiers, developed and formulated based on the limitations of previous work and experimentally validated against previous BCR-based approaches.