Can multi-label classification networks know what they don’t know?

1 Introduction . Despite many breakthroughs in machine learning , formidable obstacles obstruct its deployment in the real world , where a model can encounter unknown out-of-distribution ( OOD ) samples . The problem of OOD detection has gained significant research attention lately [ 4 , 18 , 21 , 25 , 27 , 28 , 34 , 30 , 44 ] . OOD detection aims to identify test-time inputs that have no label intersection with training classes , thus should not be predicted by the model . Previous studies have primarily focused on detecting OOD examples in multi-class classification , where each sample is assigned to a single label . Unfortunately , this can be unrealistic in many real-world applications where images often have multiple labels of interest . For example , in medical imaging , multiple abnormalities may be present in a medical scan [ 51 ] . Currently , a critical research gap exists in developing and evaluating OOD detection algorithms for multi-label classification tasks that are more applicable to the real world . While one may expect solutions for multi-class setting should transfer to the multi-label setting , we show that this is far from the truth . The main challenges posed in multi-label setting stem from the need to estimate uncertainty by jointly leveraging the information across different labels , as opposed to relying on one ⇤Equal contribution . Work done while H.W was working at UW-Madison as an undergraduate researcher . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dominant label . Our analysis reveals that simply using the largest model output ( i.e. , MaxLogit ) can be limiting . As a simple illustration , we contrast in Figure 1 of estimating OOD uncertainty using joint vs. single label information . MaxLogit can only capture the difference between the dominant outputs for dog ( in-distribution ) and car ( OOD ) , while positive information from another dominant label cat ( in-distribution ) is dismissed . In this paper , we address the important problem of OOD uncertainty estimation in the multi-label classification setting , and propose a simple and surprisingly effective method that jointly characterizes uncertainty from multiple labels . As a major advantage , our method circumvents the challenge to directly estimate the joint likelihood using generative models , which can be can computationally intractable to train and optimize , especially on multi-label datasets [ 17 ] . Formally , our proposed method , JointEnergy , derives a novel OOD score by combining label-wise energies over all labels . Despite its simplicity , we show that the JointEnergy can be theoretically interpreted from a joint likelihood perspective . The joint likelihood allows separability between in-distribution vs. OOD data , since OOD data is expected to have lower joint likelihood ( i.e. , not associated with any of the labels ) . In contrast , having multiple dominant labels is indicative of an in-distribution input , which is the key aspect that JointEnergy captures . As shown in Figure 1 , JointEnergy effectively amplifies the difference in scores between in-distribution and OOD inputs , compared to MaxLogit . Extensive experiments show that JointEnergy outperforms existing methods on three common multilabel classification tasks , establishing state-of-the-art performance . For example , on a DenseNet trained with MS-COCO [ 29 ] , our method reduces the false positive rate ( at 95 % TPR ) by 10.05 % when evaluated against OOD data from ImageNet [ 9 ] , compared to the best performing baselines . Consistent performance improvement is observed on other multi-label tasks including PASCALVOC [ 11 ] and NUS-WIDE [ 6 ] , as well as alternative network architecture . Importantly , our analysis demonstrates a strong compatibility between the label-wise energy function and aggregation function , supported by both mathematical interpretation and empirical results . As an ablation , we explore the effectiveness of applying summation to popular OOD scoring functions [ 15 , 16 , 28 , 27 ] . We find that summing labels ’ scores using previous methods is inferior to summing labels ’ energies , emphasizing the need for JointEnergy . For example , simply summing over the logits across labels results in up to 51.93 % degradation in FPR95 on MS-COCO . Our study therefore underlines the importance of properly choosing both the label-wise scoring function and the aggregation method . Below we summarize our key results and contributions : • We propose a novel method JointEnergy—addressing an important yet underexplored problem—OOD detection for multi-label classification networks . Our method establishes state-of-the-art performance , reducing the average FPR95 by up to 10.05 % . We show theoretical interpretation , underpinning our method from a joint likelihood perspective . • We conduct extensive ablations which reveals important insights for multi-label OOD uncertainty estimation under ( 1 ) different aggregation functions , ( 2 ) different label-wise OOD scoring functions , and ( 3 ) the compatibility thereof . • We curate three evaluation tasks in the multi-label setting from three real-world highresolution image databases , which enables future research to evaluate OOD detection in a multi-label setting . Our code and dataset is released for reproducible research2 . 2 Background . Multi-label Classification Multi-label classification is the supervised learning problem where an instance may be associated with multiple labels . Let X ( resp . Y ) be the input ( resp . output ) space and let P be a distribution over X ⇥Y , and let f : X ! R|Y| be a neural network trained on samples drawn from P . An input can be associated with a subset of labels in Y = { 1 , 2 , ... , K } . This set is represented by a vector y = [ y1 , y2 , ... , yK ] , where yi = 1 if and only if label i is associated with instance x , and 0 otherwise . We use a convolutional neural network with shared feature space and derive the multi-label output prediction . In contrast to learning completely disjoint classifiers [ 47 ] , the end-to-end training with a shared feature space is computationally more efficient than training K completely independent models . This has become a de facto training mechanism for multi-label classification , with various domain applications [ 32 , 33 , 46 , 51 ] . Out-of-distribution Detection The problem of OOD detection for multi-label classification is defined as follows . Denote by Din the marginal distribution of P over X , which represents the distribution of in-distribution data . At test time , the environment can incur an out-of-distribution Dout over X . The goal of OOD detection is to define a decision function G such that : G ( x ; f ) = ⇢ 0 if x ⇠ Dout , 1 if x ⇠ Din . An input is considered an OOD if it does not contain any label in the in-distribution data . In practice , Dout is often defined by a distribution that simulates anomalies encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . Energy Function Liu et al . [ 34 ] first propose using free energy as a scoring function for OOD uncertainty estimation in the multi-class setting . Given a neural classifier f ( x ) : X ! RK that maps an input x 2 X to K real-valued numbers as logits , a softmax function is used to derive a categorical distribution , p ( yi = 1 | x ) = efyi ( x ) PK j=1 e fyj ( x ) . ( 1 ) The energy model defines the probability distribution through the logits . The transformation from logits to probability distribution is by the Boltzmann distribution : p ( yi = 1 | x ) = e E ( x , yi ) R y0 e E ( x , y0 ) = e E ( x , yi ) e E ( x ) . Therefore , a multi-class classifier can be interpreted from an energy-based perspective by viewing the logit fyi ( x ) of class yi as an energy function E ( x , yi ) = fyi ( x ) . By equalizing the two denominators above , the free energy function E ( x ) for any given input x is : E ( x ) = log KX i=1 efyi ( x ) . ( 2 ) 2Code and data is available : https : //github.com/deeplearning-wisc/multi-label-ood 3 Method . In this work , we propose a novel method for OOD detection in multi-label classification networks , where an input can have several labels ( see Figure 1 ) . In what follows , we first introduce a label-wise energy function , and then propose JointEnergy that can leverage the joint information across labels for OOD uncertainty estimation . Label-wise Free Energy We consider a standard pre-trained multi-label neural classifier , with a shared parameter space ✓ up to the penultimate feature layer . During inference time , for a given input x , the ( logit ) output for the i-th class is : fyi ( x ) = h ( x ; ✓ ) ·wicls , ( 3 ) where wicls is the weight vector corresponding to class i , and h ( x ; ✓ ) is the feature vector in the penultimate layer . The predictive probability for each binary label yi is made by a binary logistic classifier : p ( yi = 1 | x ) = efyi ( x ) 1 + efyi ( x ) , where i 2 { 1 , 2 , ... , K } . The logistic classifier output can be viewed as the softmax with two logits—0 and fyi ( x ) , respectively . For each class yi , we define label-wise free energy as follows : Eyi ( x ) = log ( 1 + efyi ( x ) ) , ( 4 ) which can be viewed as a special case of free energy in [ 34 ] . For illustration , we show the label-wise energy distribution for a subset of PASCAL-VOC classes in Figure 2 ( green color ) . Label-wise energy captures the OOD uncertainty for a single label , but unfortunately does not capture uncertainty jointly across labels . JointEnergy We propose a novel scoring function that takes into account the joint uncertainty across labels , and provide mathematical justification from a joint likelihood perspective . In particular , our method is the first to consider the joint estimation of OOD uncertainty across labels : Ejoint ( x ) = KX i=1 Eyi ( x ) ( 5 ) In particular , JointEnergy takes the summation of label-wise energy scores across all labels . Note that in the above equations , label-wise energy Eyi ( x ) by definition is a negative value , and the aggregation methods output a positive value by negation . This aligns with the convention that a larger score indicates in-distribution and vice versa . Mathematical Interpretation We provide mathematical interpretation for JointEnergy . To interpret JointEnergy , we first resort to the energy-based model [ 26 ] , where the conditional likelihood p ( x | yi = 1 ) is given by : p ( x | yi = 1 ) = e Eyi ( x ) R x|yi e Eyi ( x ) , ( 6 ) and Zyi = R x|yi e Eyi ( x ) is the normalized density . We now show that JointEnergy can be interpreted from the joint likelihood perspective : Ejoint ( x ) = KX i=1 log ✓ p ( x | yi = 1 ) · Zyi ◆ ( 7 ) = KX i=1 log p ( x | yi = 1 ) + KX i=1 logZyi | { z } Z ( 8 ) By applying Bayesian rule for each term log p ( x | yi = 1 ) in Equation 8 , we have Ejoint ( x ) = log KY i=1 p ( yi = 1 | x ) · p ( x ) p ( yi = 1 ) + Z ( 9 ) = log KY i=1 p ( yi = 1 | x ) +K · log p ( x ) + ( Z log KY i=1 p ( yi = 1 ) ) | { z } C ( 10 ) Given all labels yi are conditionally independent3 , we have QK i=1 p ( yi = 1 | x ) = p ( y1 = 1 , y2 = 1 , ... , yK = 1 | x ) . Therefore , Equation 10 is equivalent to : Ejoint ( x ) = log p ( y1 = 1 , y2 = 1 , . . . , yK = 1 | x ) +K · log p ( x ) + C ( 11 ) = log p ( x | y1 = 1 , y2 = 1 , ... , yK = 1 ) · QK i=1 p ( yi = 1 ) p ( x ) +K · log p ( x ) + C ( 12 ) = log p ( x | y1 = 1 , y2 = 1 , ... , yK = 1 ) | { z } joint conditional log likelihood , `` for in-distribution + ( K 1 ) · log p ( x ) | { z } log data density , `` for in-distribution + Z ( 13 ) Rationale of Equation 13 The equation above suggests that Ejoint ( x ) can be interpreted from the joint conditional log likelihood and log data density perspective . The second term is desirable for OOD detection since it reflects the underlying data density , which is higher for in-distribution data 3Note that this is sufficient but not necessary for our results to hold . Our theoretical assumption is made to ease and facilitate the interpretation from a joint likelihood perspective . Importantly , our experiments in Section 4 hold without imposing any condition and demonstrate superior performance . x . The first term takes into account joint estimation across labels , which is new to our multi-label setting and was not previously considered in multi-class setting [ 34 ] . The first term allows even further discriminativity between in- vs. OOD data , since OOD data is expected to have lower joint conditional likelihood ( i.e. , not associated with any of the labels ) . In contrast , having multiple dominant labels is indicative of an in-distribution input , which is a characteristic that JointEnergy captures . As a major advantage , our method circumvents the challenge to directly estimate the joint likelihood using generative models , which can be computationally intractable to train and optimize on multi-label datasets [ 17 ] .

Summary: This paper studies a new task that detects out-of-distribution (OOD) data in the multi-label setting. Inspired by recent progress in multi-class classification problems, the authors proposed a joint energy score to achieve their goal. Specifically, the proposed method sums up the label-wise energy score. Then, it is used as a criterion to determine which samples are out-of-distribution by thresholding. Empirical results demonstrate that the proposed method achieved promising results on COCO, NUSWIDE, and VOC benchmark datasets.