Can multi-label classification networks know what they don’t know?

1 Introduction . Despite many breakthroughs in machine learning , formidable obstacles obstruct its deployment in the real world , where a model can encounter unknown out-of-distribution ( OOD ) samples . The problem of OOD detection has gained significant research attention lately [ 4 , 18 , 21 , 25 , 27 , 28 , 34 , 30 , 44 ] . OOD detection aims to identify test-time inputs that have no label intersection with training classes , thus should not be predicted by the model . Previous studies have primarily focused on detecting OOD examples in multi-class classification , where each sample is assigned to a single label . Unfortunately , this can be unrealistic in many real-world applications where images often have multiple labels of interest . For example , in medical imaging , multiple abnormalities may be present in a medical scan [ 51 ] . Currently , a critical research gap exists in developing and evaluating OOD detection algorithms for multi-label classification tasks that are more applicable to the real world . While one may expect solutions for multi-class setting should transfer to the multi-label setting , we show that this is far from the truth . The main challenges posed in multi-label setting stem from the need to estimate uncertainty by jointly leveraging the information across different labels , as opposed to relying on one ⇤Equal contribution . Work done while H.W was working at UW-Madison as an undergraduate researcher . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dominant label . Our analysis reveals that simply using the largest model output ( i.e. , MaxLogit ) can be limiting . As a simple illustration , we contrast in Figure 1 of estimating OOD uncertainty using joint vs. single label information . MaxLogit can only capture the difference between the dominant outputs for dog ( in-distribution ) and car ( OOD ) , while positive information from another dominant label cat ( in-distribution ) is dismissed . In this paper , we address the important problem of OOD uncertainty estimation in the multi-label classification setting , and propose a simple and surprisingly effective method that jointly characterizes uncertainty from multiple labels . As a major advantage , our method circumvents the challenge to directly estimate the joint likelihood using generative models , which can be can computationally intractable to train and optimize , especially on multi-label datasets [ 17 ] . Formally , our proposed method , JointEnergy , derives a novel OOD score by combining label-wise energies over all labels . Despite its simplicity , we show that the JointEnergy can be theoretically interpreted from a joint likelihood perspective . The joint likelihood allows separability between in-distribution vs. OOD data , since OOD data is expected to have lower joint likelihood ( i.e. , not associated with any of the labels ) . In contrast , having multiple dominant labels is indicative of an in-distribution input , which is the key aspect that JointEnergy captures . As shown in Figure 1 , JointEnergy effectively amplifies the difference in scores between in-distribution and OOD inputs , compared to MaxLogit . Extensive experiments show that JointEnergy outperforms existing methods on three common multilabel classification tasks , establishing state-of-the-art performance . For example , on a DenseNet trained with MS-COCO [ 29 ] , our method reduces the false positive rate ( at 95 % TPR ) by 10.05 % when evaluated against OOD data from ImageNet [ 9 ] , compared to the best performing baselines . Consistent performance improvement is observed on other multi-label tasks including PASCALVOC [ 11 ] and NUS-WIDE [ 6 ] , as well as alternative network architecture . Importantly , our analysis demonstrates a strong compatibility between the label-wise energy function and aggregation function , supported by both mathematical interpretation and empirical results . As an ablation , we explore the effectiveness of applying summation to popular OOD scoring functions [ 15 , 16 , 28 , 27 ] . We find that summing labels ’ scores using previous methods is inferior to summing labels ’ energies , emphasizing the need for JointEnergy . For example , simply summing over the logits across labels results in up to 51.93 % degradation in FPR95 on MS-COCO . Our study therefore underlines the importance of properly choosing both the label-wise scoring function and the aggregation method . Below we summarize our key results and contributions : • We propose a novel method JointEnergy—addressing an important yet underexplored problem—OOD detection for multi-label classification networks . Our method establishes state-of-the-art performance , reducing the average FPR95 by up to 10.05 % . We show theoretical interpretation , underpinning our method from a joint likelihood perspective . • We conduct extensive ablations which reveals important insights for multi-label OOD uncertainty estimation under ( 1 ) different aggregation functions , ( 2 ) different label-wise OOD scoring functions , and ( 3 ) the compatibility thereof . • We curate three evaluation tasks in the multi-label setting from three real-world highresolution image databases , which enables future research to evaluate OOD detection in a multi-label setting . Our code and dataset is released for reproducible research2 . 2 Background . Multi-label Classification Multi-label classification is the supervised learning problem where an instance may be associated with multiple labels . Let X ( resp . Y ) be the input ( resp . output ) space and let P be a distribution over X ⇥Y , and let f : X ! R|Y| be a neural network trained on samples drawn from P . An input can be associated with a subset of labels in Y = { 1 , 2 , ... , K } . This set is represented by a vector y = [ y1 , y2 , ... , yK ] , where yi = 1 if and only if label i is associated with instance x , and 0 otherwise . We use a convolutional neural network with shared feature space and derive the multi-label output prediction . In contrast to learning completely disjoint classifiers [ 47 ] , the end-to-end training with a shared feature space is computationally more efficient than training K completely independent models . This has become a de facto training mechanism for multi-label classification , with various domain applications [ 32 , 33 , 46 , 51 ] . Out-of-distribution Detection The problem of OOD detection for multi-label classification is defined as follows . Denote by Din the marginal distribution of P over X , which represents the distribution of in-distribution data . At test time , the environment can incur an out-of-distribution Dout over X . The goal of OOD detection is to define a decision function G such that : G ( x ; f ) = ⇢ 0 if x ⇠ Dout , 1 if x ⇠ Din . An input is considered an OOD if it does not contain any label in the in-distribution data . In practice , Dout is often defined by a distribution that simulates anomalies encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . Energy Function Liu et al . [ 34 ] first propose using free energy as a scoring function for OOD uncertainty estimation in the multi-class setting . Given a neural classifier f ( x ) : X ! RK that maps an input x 2 X to K real-valued numbers as logits , a softmax function is used to derive a categorical distribution , p ( yi = 1 | x ) = efyi ( x ) PK j=1 e fyj ( x ) . ( 1 ) The energy model defines the probability distribution through the logits . The transformation from logits to probability distribution is by the Boltzmann distribution : p ( yi = 1 | x ) = e E ( x , yi ) R y0 e E ( x , y0 ) = e E ( x , yi ) e E ( x ) . Therefore , a multi-class classifier can be interpreted from an energy-based perspective by viewing the logit fyi ( x ) of class yi as an energy function E ( x , yi ) = fyi ( x ) . By equalizing the two denominators above , the free energy function E ( x ) for any given input x is : E ( x ) = log KX i=1 efyi ( x ) . ( 2 ) 2Code and data is available : https : //github.com/deeplearning-wisc/multi-label-ood 3 Method . In this work , we propose a novel method for OOD detection in multi-label classification networks , where an input can have several labels ( see Figure 1 ) . In what follows , we first introduce a label-wise energy function , and then propose JointEnergy that can leverage the joint information across labels for OOD uncertainty estimation . Label-wise Free Energy We consider a standard pre-trained multi-label neural classifier , with a shared parameter space ✓ up to the penultimate feature layer . During inference time , for a given input x , the ( logit ) output for the i-th class is : fyi ( x ) = h ( x ; ✓ ) ·wicls , ( 3 ) where wicls is the weight vector corresponding to class i , and h ( x ; ✓ ) is the feature vector in the penultimate layer . The predictive probability for each binary label yi is made by a binary logistic classifier : p ( yi = 1 | x ) = efyi ( x ) 1 + efyi ( x ) , where i 2 { 1 , 2 , ... , K } . The logistic classifier output can be viewed as the softmax with two logits—0 and fyi ( x ) , respectively . For each class yi , we define label-wise free energy as follows : Eyi ( x ) = log ( 1 + efyi ( x ) ) , ( 4 ) which can be viewed as a special case of free energy in [ 34 ] . For illustration , we show the label-wise energy distribution for a subset of PASCAL-VOC classes in Figure 2 ( green color ) . Label-wise energy captures the OOD uncertainty for a single label , but unfortunately does not capture uncertainty jointly across labels . JointEnergy We propose a novel scoring function that takes into account the joint uncertainty across labels , and provide mathematical justification from a joint likelihood perspective . In particular , our method is the first to consider the joint estimation of OOD uncertainty across labels : Ejoint ( x ) = KX i=1 Eyi ( x ) ( 5 ) In particular , JointEnergy takes the summation of label-wise energy scores across all labels . Note that in the above equations , label-wise energy Eyi ( x ) by definition is a negative value , and the aggregation methods output a positive value by negation . This aligns with the convention that a larger score indicates in-distribution and vice versa . Mathematical Interpretation We provide mathematical interpretation for JointEnergy . To interpret JointEnergy , we first resort to the energy-based model [ 26 ] , where the conditional likelihood p ( x | yi = 1 ) is given by : p ( x | yi = 1 ) = e Eyi ( x ) R x|yi e Eyi ( x ) , ( 6 ) and Zyi = R x|yi e Eyi ( x ) is the normalized density . We now show that JointEnergy can be interpreted from the joint likelihood perspective : Ejoint ( x ) = KX i=1 log ✓ p ( x | yi = 1 ) · Zyi ◆ ( 7 ) = KX i=1 log p ( x | yi = 1 ) + KX i=1 logZyi | { z } Z ( 8 ) By applying Bayesian rule for each term log p ( x | yi = 1 ) in Equation 8 , we have Ejoint ( x ) = log KY i=1 p ( yi = 1 | x ) · p ( x ) p ( yi = 1 ) + Z ( 9 ) = log KY i=1 p ( yi = 1 | x ) +K · log p ( x ) + ( Z log KY i=1 p ( yi = 1 ) ) | { z } C ( 10 ) Given all labels yi are conditionally independent3 , we have QK i=1 p ( yi = 1 | x ) = p ( y1 = 1 , y2 = 1 , ... , yK = 1 | x ) . Therefore , Equation 10 is equivalent to : Ejoint ( x ) = log p ( y1 = 1 , y2 = 1 , . . . , yK = 1 | x ) +K · log p ( x ) + C ( 11 ) = log p ( x | y1 = 1 , y2 = 1 , ... , yK = 1 ) · QK i=1 p ( yi = 1 ) p ( x ) +K · log p ( x ) + C ( 12 ) = log p ( x | y1 = 1 , y2 = 1 , ... , yK = 1 ) | { z } joint conditional log likelihood , `` for in-distribution + ( K 1 ) · log p ( x ) | { z } log data density , `` for in-distribution + Z ( 13 ) Rationale of Equation 13 The equation above suggests that Ejoint ( x ) can be interpreted from the joint conditional log likelihood and log data density perspective . The second term is desirable for OOD detection since it reflects the underlying data density , which is higher for in-distribution data 3Note that this is sufficient but not necessary for our results to hold . Our theoretical assumption is made to ease and facilitate the interpretation from a joint likelihood perspective . Importantly , our experiments in Section 4 hold without imposing any condition and demonstrate superior performance . x . The first term takes into account joint estimation across labels , which is new to our multi-label setting and was not previously considered in multi-class setting [ 34 ] . The first term allows even further discriminativity between in- vs. OOD data , since OOD data is expected to have lower joint conditional likelihood ( i.e. , not associated with any of the labels ) . In contrast , having multiple dominant labels is indicative of an in-distribution input , which is a characteristic that JointEnergy captures . As a major advantage , our method circumvents the challenge to directly estimate the joint likelihood using generative models , which can be computationally intractable to train and optimize on multi-label datasets [ 17 ] .

Summary: The paper presents a new score - JointEnergy, as a sum of free energy across labels to detect out of distribution inputs in a multi-label classification setting. It presents the definition of this new score, motivates it theoretically by connecting it to joint likelihood of data, and demonstrates the superior performance of this score over existing alternatives on 3 different datasets. The results are interesting and significant.