Can multi-label classification networks know what they don’t know?
Year: 2021

Abstract:
Estimating out-of-distribution (OOD) uncertainty is a major challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating label-wise energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MSCOCO, PASCAL-VOC, and NUS-WIDE. We show that JointEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance.

heading: 1 Introduction
text: Despite many breakthroughs in machine learning, formidable obstacles obstruct its deployment in the real world, where a model can encounter unknown out-of-distribution (OOD) samples. The problem of OOD detection has gained significant research attention lately [4, 18, 21, 25, 27, 28, 34, 30, 44]. OOD detection aims to identify test-time inputs that have no label intersection with training classes, thus should not be predicted by the model. Previous studies have primarily focused on detecting OOD examples in multi-class classification, where each sample is assigned to a single label. Unfortunately, this can be unrealistic in many real-world applications where images often have multiple labels of interest. For example, in medical imaging, multiple abnormalities may be present in a medical scan [51].
Currently, a critical research gap exists in developing and evaluating OOD detection algorithms for multi-label classification tasks that are more applicable to the real world. While one may expect solutions for multi-class setting should transfer to the multi-label setting, we show that this is far from the truth. The main challenges posed in multi-label setting stem from the need to estimate uncertainty by jointly leveraging the information across different labels, as opposed to relying on one
⇤Equal contribution. Work done while H.W was working at UW-Madison as an undergraduate researcher.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
dominant label. Our analysis reveals that simply using the largest model output (i.e., MaxLogit) can be limiting. As a simple illustration, we contrast in Figure 1 of estimating OOD uncertainty using joint vs. single label information. MaxLogit can only capture the difference between the dominant outputs for dog (in-distribution) and car (OOD), while positive information from another dominant label cat (in-distribution) is dismissed.
In this paper, we address the important problem of OOD uncertainty estimation in the multi-label classification setting, and propose a simple and surprisingly effective method that jointly characterizes uncertainty from multiple labels. As a major advantage, our method circumvents the challenge to directly estimate the joint likelihood using generative models, which can be can computationally intractable to train and optimize, especially on multi-label datasets [17]. Formally, our proposed method, JointEnergy, derives a novel OOD score by combining label-wise energies over all labels. Despite its simplicity, we show that the JointEnergy can be theoretically interpreted from a joint likelihood perspective. The joint likelihood allows separability between in-distribution vs. OOD data, since OOD data is expected to have lower joint likelihood (i.e., not associated with any of the labels). In contrast, having multiple dominant labels is indicative of an in-distribution input, which is the key aspect that JointEnergy captures. As shown in Figure 1, JointEnergy effectively amplifies the difference in scores between in-distribution and OOD inputs, compared to MaxLogit.
Extensive experiments show that JointEnergy outperforms existing methods on three common multilabel classification tasks, establishing state-of-the-art performance. For example, on a DenseNet trained with MS-COCO [29], our method reduces the false positive rate (at 95% TPR) by 10.05% when evaluated against OOD data from ImageNet [9], compared to the best performing baselines. Consistent performance improvement is observed on other multi-label tasks including PASCALVOC [11] and NUS-WIDE [6], as well as alternative network architecture.
Importantly, our analysis demonstrates a strong compatibility between the label-wise energy function and aggregation function, supported by both mathematical interpretation and empirical results. As an ablation, we explore the effectiveness of applying summation to popular OOD scoring functions [15, 16, 28, 27]. We find that summing labels’ scores using previous methods is inferior to summing labels’ energies, emphasizing the need for JointEnergy. For example, simply summing over the logits across labels results in up to 51.93% degradation in FPR95 on MS-COCO. Our study therefore underlines the importance of properly choosing both the label-wise scoring function and the aggregation method. Below we summarize our key results and contributions:
• We propose a novel method JointEnergy—addressing an important yet underexplored problem—OOD detection for multi-label classification networks. Our method establishes
state-of-the-art performance, reducing the average FPR95 by up to 10.05%. We show theoretical interpretation, underpinning our method from a joint likelihood perspective.
• We conduct extensive ablations which reveals important insights for multi-label OOD uncertainty estimation under (1) different aggregation functions, (2) different label-wise OOD scoring functions, and (3) the compatibility thereof.
• We curate three evaluation tasks in the multi-label setting from three real-world highresolution image databases, which enables future research to evaluate OOD detection in a multi-label setting. Our code and dataset is released for reproducible research2.

heading: 2 Background
text: Multi-label Classification Multi-label classification is the supervised learning problem where an instance may be associated with multiple labels. Let X (resp. Y) be the input (resp. output) space and let P be a distribution over X ⇥Y , and let f : X ! R|Y| be a neural network trained on samples drawn from P . An input can be associated with a subset of labels in Y = {1, 2, ...,K}. This set is represented by a vector y = [y1, y2, ..., yK ], where yi = 1 if and only if label i is associated with instance x, and 0 otherwise. We use a convolutional neural network with shared feature space and derive the multi-label output prediction. In contrast to learning completely disjoint classifiers [47], the end-to-end training with a shared feature space is computationally more efficient than training K completely independent models. This has become a de facto training mechanism for multi-label classification, with various domain applications [32, 33, 46, 51].
Out-of-distribution Detection The problem of OOD detection for multi-label classification is defined as follows. Denote by Din the marginal distribution of P over X , which represents the distribution of in-distribution data. At test time, the environment can incur an out-of-distribution Dout over X . The goal of OOD detection is to define a decision function G such that:
G(x; f) = ⇢ 0 if x ⇠ Dout, 1 if x ⇠ Din.
An input is considered an OOD if it does not contain any label in the in-distribution data. In practice, Dout is often defined by a distribution that simulates anomalies encountered during deployment time, such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model.
Energy Function Liu et al. [34] first propose using free energy as a scoring function for OOD uncertainty estimation in the multi-class setting. Given a neural classifier f(x) : X ! RK that maps an input x 2 X to K real-valued numbers as logits, a softmax function is used to derive a categorical distribution,
p(yi = 1 | x) = efyi (x)
PK j=1 e fyj (x) . (1)
The energy model defines the probability distribution through the logits. The transformation from logits to probability distribution is by the Boltzmann distribution:
p(yi = 1 | x) = e E(x,yi)R y0 e E(x,y0) = e E(x,yi) e E(x) .
Therefore, a multi-class classifier can be interpreted from an energy-based perspective by viewing the logit fyi(x) of class yi as an energy function E(x, yi) = fyi(x). By equalizing the two denominators above, the free energy function E(x) for any given input x is:
E(x) = log KX
i=1
efyi (x). (2)
2Code and data is available: https://github.com/deeplearning-wisc/multi-label-ood

heading: 3 Method
text: In this work, we propose a novel method for OOD detection in multi-label classification networks, where an input can have several labels (see Figure 1). In what follows, we first introduce a label-wise energy function, and then propose JointEnergy that can leverage the joint information across labels for OOD uncertainty estimation.
Label-wise Free Energy We consider a standard pre-trained multi-label neural classifier, with a shared parameter space ✓ up to the penultimate feature layer. During inference time, for a given input x, the (logit) output for the i-th class is:
fyi(x) = h(x; ✓) ·wicls, (3)
where wicls is the weight vector corresponding to class i, and h(x; ✓) is the feature vector in the penultimate layer. The predictive probability for each binary label yi is made by a binary logistic classifier:
p(yi = 1 | x) = efyi (x)
1 + efyi (x) ,
where i 2 {1, 2, ...,K}. The logistic classifier output can be viewed as the softmax with two logits—0 and fyi(x), respectively. For each class yi, we define label-wise free energy as follows:
Eyi(x) = log(1 + efyi (x)), (4)
which can be viewed as a special case of free energy in [34]. For illustration, we show the label-wise energy distribution for a subset of PASCAL-VOC classes in Figure 2 (green color). Label-wise energy captures the OOD uncertainty for a single label, but unfortunately does not capture uncertainty jointly across labels.
JointEnergy We propose a novel scoring function that takes into account the joint uncertainty across labels, and provide mathematical justification from a joint likelihood perspective. In particular, our method is the first to consider the joint estimation of OOD uncertainty across labels:
Ejoint(x) = KX
i=1
Eyi(x) (5)
In particular, JointEnergy takes the summation of label-wise energy scores across all labels. Note that in the above equations, label-wise energy Eyi(x) by definition is a negative value, and the aggregation methods output a positive value by negation. This aligns with the convention that a larger score indicates in-distribution and vice versa.
Mathematical Interpretation We provide mathematical interpretation for JointEnergy. To interpret JointEnergy, we first resort to the energy-based model [26], where the conditional likelihood p(x | yi = 1) is given by:
p(x | yi = 1) = e Eyi (x)R
x|yi e Eyi (x)
, (6)
and Zyi = R x|yi e Eyi (x) is the normalized density.
We now show that JointEnergy can be interpreted from the joint likelihood perspective:
Ejoint(x) = KX
i=1
log ✓ p(x | yi = 1) · Zyi ◆ (7)
= KX
i=1
log p(x | yi = 1) + KX
i=1
logZyi
| {z } Z
(8)
By applying Bayesian rule for each term log p(x | yi = 1) in Equation 8, we have
Ejoint(x) = log KY
i=1
p(yi = 1 | x) · p(x) p(yi = 1) + Z (9)
= log KY
i=1
p(yi = 1 | x) +K · log p(x)
+ (Z log KY
i=1
p(yi = 1))
| {z } C
(10)
Given all labels yi are conditionally independent3, we have QK
i=1 p(yi = 1 | x) = p(y1 = 1, y2 = 1, ..., yK = 1 | x). Therefore, Equation 10 is equivalent to:
Ejoint(x) = log p(y1 = 1, y2 = 1, . . . , yK = 1 | x) +K · log p(x) + C (11)
= log p(x | y1 = 1, y2 = 1, ..., yK = 1) ·
QK i=1 p(yi = 1)
p(x)
+K · log p(x) + C (12) = log p(x | y1 = 1, y2 = 1, ..., yK = 1)| {z }
joint conditional log likelihood, " for in-distribution
+ (K 1) · log p(x)| {z } log data density, " for in-distribution + Z (13)
Rationale of Equation 13 The equation above suggests that Ejoint(x) can be interpreted from the joint conditional log likelihood and log data density perspective. The second term is desirable for OOD detection since it reflects the underlying data density, which is higher for in-distribution data
3Note that this is sufficient but not necessary for our results to hold. Our theoretical assumption is made to ease and facilitate the interpretation from a joint likelihood perspective. Importantly, our experiments in Section 4 hold without imposing any condition and demonstrate superior performance.
x. The first term takes into account joint estimation across labels, which is new to our multi-label setting and was not previously considered in multi-class setting [34]. The first term allows even further discriminativity between in- vs. OOD data, since OOD data is expected to have lower joint conditional likelihood (i.e., not associated with any of the labels). In contrast, having multiple dominant labels is indicative of an in-distribution input, which is a characteristic that JointEnergy captures. As a major advantage, our method circumvents the challenge to directly estimate the joint likelihood using generative models, which can be computationally intractable to train and optimize on multi-label datasets [17].

heading: 3.1 JointEnergy for Multi-Label OOD Detection
text: We propose using the JointEnergy function Ejoint(x) defined in Section 3 for OOD detection:
G(x; ⌧) = ⇢ out if Ejoint(x)  ⌧, in if Ejoint(x) > ⌧,
(14)
where ⌧ is the energy threshold, and can be chosen so that a high fraction (e.g., 95%) of in-distribution data is correctly classified by G(x; ⌧). The sensitivity analysis on ⌧ is provided in Figure 3. A data point with higher JointEnergy Ejoint(x) is considered as in-distribution, and vice versa (see Fig. 1).

heading: 4 Experiments
text: In this section, we describe our experimental setup (Section 4.1) and demonstrate the effectiveness of our method on several OOD evaluation tasks (Section 4.2). We also conduct extensive ablation studies and comparative analysis that lead to an improved understanding of different methods.

heading: 4.1 Setup
text: In-distribution Datasets We consider three multi-label datasets: MS-COCO [29], PASCAL-VOC [11], and NUS-WIDE [6]. MS-COCO consists of 82,783 training, 40,504 validation, and 40,775 testing images with 80 common object categories. PASCAL-VOC contains 22,531 images across 20 classes. NUS-WIDE includes 269,648 images across 81 concept labels. Since NUS-WIDE has invalid and untagged images, we follow [56] and use 119,986 training images and 80,283 test images.
Training Details We train three multi-label classifiers, one for each dataset above. The classifiers have a DenseNet-121 backbone architecture, with a final layer that is replaced by 2 fully connected layers. Each classifier is pre-trained on ImageNet-1K and then fine-tuned with the logistic sigmoid function to its corresponding multi-label dataset. We use the Adam optimizer [23] with standard parameters ( 1 = 0.9, 2 = 0.999). The initial learning rate is 10 4 for the fully connected layers and 10 5 for convolutional layers. We also augmented the data with random crops and random flips to obtain color images of size 256 ⇥ 256. After training, the mAP is 87.51% for PASCAL-VOC, 73.83% for MS-COCO, and 60.22% for NUS-WIDE. All experiments are conducted on NVIDIA GeForce RTX 2080Ti.
Out-of-distribution Datasets To evaluate the models trained on the in-distribution datasets above, we follow the same set up as in [15] and use ImageNet [9] for its generality. Besides, we evaluate against the Textures dataset [8] as OOD. For ImageNet, we use the same set of 20 classes chosen from ImageNet-22K as in [15]. These classes are chosen not to overlap with ImageNet-1k since the multi-label classifiers are pre-trained on ImageNet-1K. Specifically, we use the following classes for evaluating the MS-COCO and PASCAL-VOC pre-trained models: dolphin, deer, bat, rhino, raccoon, octopus, giant clam, leech, venus flytrap, cherry tree, Japanese cherry blossoms, redwood, sunflower, croissant, stick cinnamon, cotton, rice, sugar cane, bamboo, and turmeric. Since NUS-WIDE contains high-level concepts like animal, plants and flowers, we use a different set of classes that are distinct from NUS-WIDE: asterism, battery, cave, cylinder, delta, fabric, filament, fire bell, hornet nest, kazoo, lichen, naval equipment, newspaper, paperclip, pythium, satellite, thumb, x-ray tube, yeast, zither.
Evaluation Metrics We measure the following metrics that are commonly used for OOD detection: (1) the false positive rate (FPR95) of OOD examples when the true positive rate of in-distribution examples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR).
4.2 Results
How does JointEnergy compare to common OOD detection methods? In Table 1, we compare energybased approaches against competitive OOD detection methods in literature, where JointEnergy demonstrates state-of-the-art performance. For fair comparisons, we consider approaches that rely on pretrained models (without performing retraining or finetuning). Following the setup in [15], all the numbers reported are evaluated on ImageNet OOD test data, as described in Section 4.1. We provide additional evaluation results for the Texture OOD test dataset in the supplementary. Most baselines such as MaxLogit [15], Maximum Softmax Probability (MSP) [16], ODIN [28] and Mahalanobis [27] derive OOD indicator scores based on the maximumvalued statistics among all labels. Local Outlier Factor (LOF) [3] uses K-nearest neighbors (KNN) to
estimate the local density, where OOD examples are detected from having lower density compared to their neighbors. Isolation forest [31] is a tree-based approach, which detects anomaly based on the path length from the root node to the terminating node.
Among different approaches, JointEnergy outperforms the best-performing baseline across all three multi-label classifiers considered. In particular, on a network trained with the MS-COCO dataset, JointEnergy reduces FPR95 by 10.05%, compared to MaxLogit. We provide the AUROC curves for our method JointEnergy in Figure 3, for all three in-distribution datasets considered. The y-axis
is the true positive rate (TPR), whereas the x-axis is the FPR. The curves indicate how the OOD detection performance changes as we vary the threshold ⌧ in Equation 14. We additionally evaluate on a different architecture, ResNet [14], for which we observe consistent improvement and provide details in the supplementary.
We also note here that existing approaches (such as Mahalanobis distance [27]) requires training a separate classifier for OOD detection. In contrast, JointEnergy is hyperparameter-free and easy to use in practice. In particular, the Mahalanobis approach is based on the assumption that feature representation forms class-conditional Gaussian distributions, and hence may not be well suited for the multi-label setting (which requires joint distribution to be learned).
How do different aggregation methods affect OOD detection performance? In Table 3, we also perform a comparative analysis of the effect of different aggregation functions that combine label-wise energy scores. As an alternative, we consider
Emax(x) = max i Eyi(x), (15)
which finds the largest label-wise energy score among all labels. We observe that MaxEnergy does not outperform JointEnergy, which utilizes information jointly from all the labels. The performance of MaxEnergy is on par with MaxLogit since MaxEnergy, given by maxi log(1 + efyi (x)), is approximately close to the MaxLogit when fyi(x) is large. The results underline the importance of taking into account information from multiple labels, not just the maximum-valued label. This is because, in multi-label classification, the model may assign high probabilities to several classes. Theoretically, JointEnergy is also more meaningful, and can be interpreted from a joint likelihood perspective as shown in Section 3.
What is the effect of applying the aggregation method to prior methods? As an extension, we explore the effectiveness of applying the aggregation method to previous scoring functions. The results are summarized in Table 2. We calculate scores based on the logit fyi(x), sigmoid of the logit 1
1+e fyi (x) , ODIN score, and Mahalanobis distance score Myi(x) for each label independently.
We then perform summation across the label-wise scores as the overall OOD score. This ablation essentially replaces the Max aggregation with Sum, which helps understand the extent to which previous approaches are amenable in the multi-label setting. Note that the summation aggregation method does not apply to tree-based or KNN-based approaches such as LOF and Isolation Forest.
We found that applying summation over individual logit/MSP/ODIN/Mahalanobis scores from each label does not enhance but sometimes worsen the performance. For example, simply summing over
the logits across the labels leads to severe degradation in performance since the outputs are mixed with positive and negative numbers. On MS-COCO, the FPR degrades from 43.53% using MaxLogit to 95.46% (using SumLogit). In contrast, JointEnergy does not suffer from this issue. This underlines the importance of choosing proper label-wise scoring function to be compatible with the aggregation method.
JointEnergy vs. SumProb We highlight the advantage of JointEnergy over SumProb both empirically and theoretically. As seen in Table 2, the performance difference between JointEnergy and SumProb is substantial. In particular, on MS-COCO, our method outperforms SumProb by 11.56% (FPR95). For threshold independent metric AUROC, JointEnergy consistently outperforms SumProb by 3.38% (MS-COCO), 4.57% (PASCAL), and 4.48% (NUS-WIDE).
Qualitative Case Study Lastly, to provide further insights on our method, we qualitatively examine examples from the multi-label classification dataset PASCAL-VOC (in-dist.) and OOD input from the ImageNet that are correctly classified by JointEnergy but not MaxLogit. In Figure 4 (left), we see an in-distribution example is labeled as dog, car, chair and person, with MaxLogit score 1.63 and JointEnergy score 3.23. We also show an OOD input (Figure 4, right) with a single dominant activation on the bird class, with MaxLogit score 2.14 and JointEnergy score 2.19. In this example, taking the sum appropriately results in a higher score for the in-distribution image than the OOD image. Contrarily, MaxLogit score for the in-distribution image is lower than that of the OOD image, which results in ineffective detection.

heading: 5 Related Work
text: Multi-label Classification The task of identifying multiple classes within an input example is of significant interest in many applications [47] where deep neural networks are commonly used as the classifiers. Natural images usually contain several objects and may have many associated tags [50]. Chen et al. [12] used convolutional neural networks (CNN) to annotate images with 3 or 5 tags on the NUS-WIDE dataset. [5] used CNNs to tag images of road scenes from 52 possible labels. In the medical domain, Wang et al. [51] presented a chest X-ray dataset in which one image may contain multiple abnormalities. Multi-label classification is also prominent in natural language processing [36]. Our proposed method is therefore relevant to a wide range of applications in the real world.
Out-of-distribution Uncertainty Estimation Detecting and rejecting unknowns has a long history in machine learning; see [54] for a comprehensive survey of the main ideas. We highlight a few representative works in the context of deep learning. The phenomenon of neural networks’ overconfidence to out-of-distribution data is revealed by Nguyen et al. [37]. Previous works attempt to improve the OOD uncertainty estimation by proposing the ODIN score [18, 28], Mahalanobis distance-based confidence score [27], and gradient-based GradNorm score [20]. Recent work by Liu et al. [34] proposed using an energy score for OOD detection, which demonstrated advantages over the softmax confidence score both empirically and theoretically. Huang and Li [21] proposed a group-based OOD detection method that scales effectively to large-scale dataset ImageNet. However, previous methods primarily focused on multi-class classification networks. In contrast, we propose a hyperparameterfree measurement that allows effective OOD detection in the underexplored multi-label setting, where the information from various labels is combined in a theoretically interpretable manner.
Generative-based Out-of-distribution Detection Generative models [10, 22, 24, 41, 45, 49] can be alternative approaches for detecting OOD examples, as they directly estimate the in-distribution density and can declare a test sample to be out-of-distribution if it lies in the low-density regions. However, as shown by Nalisnick et al. [35], deep generative models can assign a high likelihood to out-of-distribution data. Deep generative models can be more effective for out-of-distribution detection using likelihood ratio test [40, 43] and likelihood regret [52]. Though our work is based on discriminative classification models, we show that label-wise energy scores can be theoretically interpreted from a data density perspective. More importantly, generative based models [17] can be prohibitively challenging to train and optimize, especially on large and complex multi-label datasets that we considered (e.g., MS-COCO, NUS-WIDE etc.). In contrast, our method relies on a discriminative multi-label classifier, which can be easily optimized using standard SGD.
Energy-based Learning Energy-based machine learning models date back to Boltzmann machines [1, 42]. Energy-based learning [26, 38, 39] provides a unified framework for many probabilistic
and non-probabilistic approaches to learning. Recent work [55] also demonstrated using energy functions to train GANs [13], where the discriminator uses energy values to differentiate between real and generated images. Xie et al. [53] showed that a discriminative classifier can be interpreted from an energy-based perspective. Energy-based methods are also used in structure prediction [2, 48]. Liu et al. [34] first proposed using energy score for OOD uncertainty estimation, which demonstrated superior performance for multi-class classification networks. In contrast, our work focuses on a multi-label setting, where we contribute both empirical and theoretical insights and demonstrate the effectiveness of utilizing information jointly from across all labels.

heading: 6 Conclusion and Outlook
text: In this work, we propose energy scores for OOD uncertainty estimation in the multi-label classification setting. We show that aggregating energies over all labels into JointEnergy results in better separation between in-distribution and OOD inputs compared to using information from only one label’s information. Additionally, we justify the mathematical interpretation of JointEnergy from a joint likelihood perspective. JointEnergy obtains better OOD detection performance compared to competitive baseline methods, establishing new state-of-the-art on this task. Applications of multi-label classification can benefit from our methods, and we anticipate further research in OOD detection to extend this work to tasks beyond image recognition. We hope our work will increase the attention toward a broader view of OOD uncertainty estimation for multi-label classification.

heading: 7 Societal Impact
text: Our project aims to improve the dependability and trustworthiness of modern machine learning models for multi-label classification. This stands to benefit a wide range of fields and societal activities. We believe out-of-distribution uncertainty estimation is an increasingly critical component of systems that range from consumer and business applications (e.g., digital content understanding) to transportation (e.g., driver assistance systems and autonomous vehicles), and to health care (e.g., unseen disease identification). Many of these applications require multi-label classification models in operation. Through this work and by releasing our code, we hope to provide machine learning researchers a new methodological perspective and offer machine learning practitioners an easy-to-use tool that renders safety against OOD data in the real world. While we do not anticipate any negative consequences to our work, we hope to continue to build on our framework in future work.

heading: Acknowledgement
text: Research is supported by the Office of the Vice Chancellor for Research and Graduate Education (OVCRGE) with funding from the Wisconsin Alumni Research Foundation (WARF).

Review summaries:
This paper studies the out-of-distribution detection problem in the multi-label classification setting. The authors propose a scoring function named JointEnergy to measure the prediction uncertainty of a given instance. In particular, the JointEnergy aggregates the label wise free-engery by summation and a hard thresholding on the value of JointEnergy is used to determine whether a given input is OOD or not. It is shown that the propose scoring function can be decomposed into three terms: joint conditional log-likelihood on all classes, un-conditional log data density and a constant term. Empirical results on three multi-label classification datasets show that the proposed method can achieve lower false positive rate (FPR95) while having higher area under curve scores (AUROC, AUPR).

This paper studies a new task that detects out-of-distribution (OOD) data in the multi-label setting. Inspired by recent progress in multi-class classification problems, the authors proposed a joint energy score to achieve their goal. Specifically, the proposed method sums up the label-wise energy score. Then, it is used as a criterion to determine which samples are out-of-distribution by thresholding. Empirical results demonstrate that the proposed method achieved promising results on COCO, NUSWIDE, and VOC benchmark datasets. 

The paper presents a new score - JointEnergy, as a sum of free energy across labels to detect out of distribution inputs in a multi-label classification setting. It presents the definition of this new score, motivates it theoretically by connecting it to joint likelihood of data, and demonstrates the superior performance of this score over existing alternatives on 3 different datasets. The results are interesting and significant.