Can multi-label classification networks know what they don’t know?
Authors: Haoran Wang
Weitang Liu
Alex Bocchieri
Yixuan Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Out-of-distribution detection
multi-label classification

Abstract: Estimating out-of-distribution (OOD) uncertainty is a major challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating label-wise energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that JointEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance. 


Can multi-label classification networks know what they don't know?
Haoran Wang"
Weitang Liu
Information Networking Institute Carnegie Mellon University haoranwa@andrew. cmu.edu
Department of Computer Science and Eng. University of California, San Diego wel rel002@ucsd.eem
Alex Bocchieri
Yixuan Li
Department of Computer Sciences University of Wisconsin-Madison
Department of Computer Sciences University of Wisconsin-Madise sharonl i@css wisc.
eerechieeriiee cencciieriiiic.... @wwsc.edu. edu
Abstract
Estimating outtof-distribution (OOD) uncertainty a major challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEneryy, a simple and effective method, which estimates the OOD indicator scores by aggregating label-wise energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS- COCO, PASCAL-VOC, and NUS-WIDE. We show that JointEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the performance.
Introduction
Despite many breakthroughs machine learning, formidable obstacles obstruct its deployment the real world, where model can encounter unknown of OOD detection has gained significant research attention lately 4 I8 21
(OOD) samples. The problem
25
27
28
34
30
44]
OOD detection aims to identify test-time inputs that have label intersection with training classes, thus should not be predicted by the model. Previous studies have primarily focused on detecting OOD examples in multi-class classification, where each sample is assigned to single label. Unfortunately, this can be unrealistic in many real-world applications where images often have multiple labels of interest. For example, in medical imaging, multiple abnormalities may be present medical scan 511
Currently, a critical research gap exists in developing and evaluating OOD detection algorithms for multi-label classification tasks that are more applicable to the real world. While one may expect solutions for multi-class setting should transfer the multi-label setting, show that this is far from the truth. The main challenges posed in multi-label setting stem from the need to estimate uncertainty by jointly leveraging the information across different labels, opposed to relying one
Equaa contribution. Work done while H.W was working UW- UW-Madison undergraduate researcher.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

nn-Ditribution
fx:00)
Plane
Car
Bird
Cat
Daer
Dog
From Horse Ship
Truck
Max
Sum
aggragating over many
Out Oubtat--istrbuttion
In-Distribution
Out-of-Distribution
Misllnggt
JuinnEnergy
(ours)
detection for multi-label classification networks. During inference time, input x is passed through classifier f. and label-wise scores are computed for each label. OOD indicator scores are either the cenimmmm-valued score (denoted by green outlines) or the sum of all scores. Taking the sum results in larger difference in scores and more separation between in-distribution and OOD inputs (denoted by red lincs), resulting better OOD detection. Plots the bottom right depict probability densities of MaxLogit [TS] versus Jneryy (ours).
Figure I:
dominant label. Our analysis reveals that simply using the largest model output (i.e., MaxLogit) can be limiting. As simple illustration, contrast in Figure of estimating OOD uncertainty using joint vs. single label information. MaxLogit can only capture the difference between the dominant outputs for dog in-distribution) and car (OOD), while positive information from another dominant label cat fin-distibbution) is dismissed.
In this paper, we address the important problem of OOD uncertainty estimation in the multi-label classification setting, and propose simple and surprisingly effective method that jointly characterizes uncertainty from multiple labels. As a major advantage, our method circumvents the challenge to directly estimate the joint likelihood using generative models, which can be can computationally intractable train and optimize, especially method, JointEnergy, derives a novel OOD score by combining label-wise energies over all labels. Despite its simplicity, likelihood perspective. The joint likelihood allows separability between in-distribution vs. OOD data, since OOD data is expected to have lower joint likelihood (i.e., not associated with any of the labels). In contrast, having multiple dominant labels is indicative of an in-distribution input, which is the key aspect that JointEnergy captures. As shown in Figure T JointEnergy effectively amplifies the difference scores between in-distribution and OOD inputs, compared to MaxLogiit.
multi-label datasets [17]. Formally, our proposed
show that the JointEnergy
be theoretically interpreted from joint
Extensive experiments show that JointEnergy outperforms existing methods three common multi- label classification tasks, establishing state-of-the-a performance. For example, on a DenseNet trained with MS-COCO (29] our method reduces the false positive rate (at 95% TPR) by 10.05% when evaluated against OOD data from ImageNet [ compared to the best performing baselines. Consistent performance improvement is observed other multi-label tasks including PASCAL- VOC M and NUS- WIDE K. as well as alternative network architecture.
Importantly, our analysis demonstrates a strong compatibility between the label-wise energy function and aggregation function, supported by both mathematical interpretation and empirical results. As an ablation, we explore the effectiveness of applying summation to popular OOD scoring functions [15] We find that summing labels' scores using previous methods is inferior to summing labels' energies, emphasizing the need for JointEnergy. For example, simply summing over the logits across labels results in up to 51.93% degradation FPR95 on MS-COCO. Our study therefore underlines the importance of properly choosing both the label-wise scoring function and the aggregation method. Below we summarize our key results and contributions:
16 28 27
We propose a novel method JointEnergy--ddeee problem-O00 -OD detection for multi-label classification networks. Our method establishes
an important yet underexplored

---

sfae-of-the-arr performance, reducing the average FPR95 by up to 10.05%. We show theoretical interpretation, underpinning our method from a joint likelihood perspective.
We conduct extensive ablations which reveals important insights for multi-label OOD uncertainty estimation under (1) different aggregation functions, (2) different label-wise OOD scoring functions, and (3) the compatibility thereof.
We curate three evaluation tasks in the multi-label setting from three real-world high- resolution image databases, which enables future research evaluate OOD detection in multi-label setting. Our code and dataset is released for reproducible research
2 Background
Multi-label Classification Multi-label classification is the supervised learning problem where an instance may be associated with multiple labels. Let X' (resp. J) be the input (resp. output) space and let P distribution over X' x y, and let f X + R/ be a neural network trained on samples drawn from P. An input be associated with subset of labels in J 2,, K}. This set is represented by vector y [y), 1/2 ..., yk]. where = 1 if and only if label is associated with instance x, and 0 otherwise. We convolutional neural network with shared feature space and derive the multi-label output prediction. In contrast to learning completely disjoint classifiers [77). the end-to-end training with shared feature space is computationally more efficient than training K completely independent models. This has become de facto training mechanism for multi-label classification, with various domain applications
51
Outtof-distribution Detection The problem of OOD detection for multi-label classification is de- fined follows. Denote by Din the marginal distribution of P over X, which represents the distribution of in-distribution data. At test time, the environment can incur an out-of-distribution Dout over X. The goal of OOD detection is to define decision function G such that:
if Dout: G(x: f) = if Din-
So
An input is considered an OOD it does not contain any label in the in-distribution data. In practice, Dout is often defined by distribution that simulates anomalies encountered during deployment time, such as samples from an irrelevant distribution whose label set has no intersection with y and therefore should not be predicted by the model.
Liu et al. [54] first propose using free energy as a scoring function for OOD uncertainty estimation in the multi-class setting. Given neural classifier f(x) X - IR that maps an input x X to real-valued numbers as logits, a softmax function is used to derive a categorical distribution,
Energy Function
efv (x)
plu 1 x
Σド= clyy (x)
The energy model defines the probability distribution through the logits. The transformation from logits probability distribution is by the Boltzmann distribution:
E(x.v))
p(y-
x
I F(x.8))
c F((x)
Therefore, multi-class classifier can be interpreted from energy-based perspective by viewing
the logit fy. (x) of class yi as an energy function F(x, vt) = -fy, (x). By equalizing the two denominators above, the free energy function E(x) for any given input is:
E(x)
log (x)
(2)
"Code and data is available:

---

3 Method
In this work, we propose novel method for OOD detection in multi-label classification networks, where an input can have several labels (see Figure I. In what follows, we first introduce a label-wise energy function, and then propose JointEnergy that can leverage the joint information across labels for OOD uncertainty estimation.
Label-wise Free Energy We consider a standard pre-trained multi-lahel neural classifier, with a shared parameter space up the penultimate feature layer. During inference time, for given input x, the (logit) output for the i-th class is:
y (x) 1(x: 0) w
where Was is the weight vector corresponding class i, and h(x; 9) is the feature vector in the penultimate layer. The predictive probability for each binary label Yi is made by a binary logistic classifier:
e (x)
p(u = |x)
+ e
where E (1,2, K. K.. The logistic classifier output can viewed as the softmax with two logits and fv. (x), respectively. For each class yı, we define label-wise free energy as follows:
g. (x) = log([1 e.v, ex,x())). (x)),
which can be viewed as a special case of free energy in For illustration, we show the label-wise energy distribution for a subset PASCAL-VOC classes in Figure 2 (green color). Libel-wise energy captures the OOD uncertainty for a single label, but unfortunately does not capture uncertainty jointly across labels.
JointEnergy We propose novel scoring function that takes into account the joint uncertainty across labels, and provide mathematical justification from a joint likelihood perspective. In particular, our method is the first to consider the joint estimation of OOD uncertainty across labels:
K
Ejoint (x)
Σ
-E (x)
(5)
In particular, JointEnergy takes the summation of label-wise energy scores across all labels. Note that in the above equations, label-wise energy Ey, (x) by definition a negative value, and the aggregation methods output a positive value by negation. This aligns with the convention that a larger score indicates in-distribution and vice versa.
Mathematical Interpretation We provide mathematical interpretation for JointEnergy. To interpret JointEnergy, we first resort to the energy-based based model (26), where the conditional likelihood p(x V: 1) is given by:
Fv,
p(x 81 ===]] 1)
(6)
Sxlvs
Ey, (x)
and Zy, = -Ixly. -Ey, (x) is the normalized density.
We now show that JointEnergy can be interpreted from the joint likelihood perspective:
K
Ejoint (x)
log
p(x y/s = 1) Zyn
log p(x 1/: 11) +
Σ log Zeg
(8)
Z

---

0OD
In-distribution
aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
Cow
Label-wise Energy
Figure 2: Label-wise energy scores Frg. (x) distribution. The in-distribution classes (each row)
subset from PASCAL-VOC (green). OOD test data from ImageNet (gray), which the same for all labels. x-axis in log scale visibility.
By applying Bayesian rule for each term log p(x X y/ 1) in Equation g
we have
p(y: 1 xx) p(() 7
Ejuini (x) log
II
(9)
p(y: = 1)
i=1
log II p(ys = x) K logg((x
K
+ (Z log
p(u 1))
(10)
Given all labels ys conditionally independen have I P(yi x) p(y1
1,82
1,.., yK = 1 x). Therefore, Equation 10 is equivalent to:
(x) = log = 1, 1/2 I,.... UK =1
+ K log +C
(11)
p(x f1 1,3/2 1,............... yK = 1) II 1= p(u = 1)
= log
p(x)
+ logp(x)) +
(12)
log y1 32 1, UK =
conditional log likelihood, for in-distribation
(K 1) log p(x) log data density, for in-distribution
+ Z
(13)
The equation above suggests that Fjjin(()) (x) can be interpreted from the joint conditional log likelihood and log data density perspective. The second term is desirable for OOD detection since reflects the underlying data density, which is higher for in-distribution data
Rationale of Equation 13
*Note that this sufficient but not necessary for our results to hold. Our theoretical assumption is made to ease and facilitate the interpretation from joint likelihood perspective. Importantly, our experiments in Section hold without imposing any condition and demonstrate superior performance.

---

Table 1: OOD detection performance comparison using JointEnergy vs, competitive baselines. We use DenseNct 19] to train on the in-distribution datascts. We use a subset of ImageNct classes as OOD test data, as deseribed in Section 4.1 All values pereentages. 1 indicates larger values are better, and indicates smaller values better. Bold numbers are superior results. Description of baseline methods, additional evaluation results on different OOD test data, and different architecture (e.g., ResNet [4]
can be found in the Appendix.
Din
MS-COCO
PASCAL VOSCAL-VOC FPR95 AUROC/ AUPR
MUS-WIDE
OOD Score
MaxLogit [15
43.53 / 89. 99.74
45.06 / 89.22 83.14
56.46 / 83.58 94,32
MSP [66]
79.90 / 73.70/ 85.37
74.05 / 79.32 72.54
88.50 / 60.81 / 87.00
ODIN 2N
43.53 / 19.11 / /93.74
45.06 / 89.22 83.16
56.46 / 83 94.32
Mahalanobis 247
46.86 / 88.59 / 193885
41.74/ 88.65 81.12
62.67 / 84.02/ 95.25
LOF g
80.44 / 73.95 / 86.01
86.34 / 69.21 58.93
85.21 67.75 89.61
Isolation Forest 31
94.39 / 49.04 / 66.87
93.22 / 50.67 / 33.78
95.69 / 53.12/ 83.32
JointEnergy
33.48 / 92.70 / 96.25
41.01 91.10 86.33
48.98 / 88.30 96.40
x. The first term takes into account joint estimation across labels, which is new to our multi-label setting and was not previously considered in multi-class setting [4. The first term allows even further discriminativity between in- vs. OOD data, since OOD data is expected to have lower joint conditional likelihood (i.e., not associated with any of the labels). In contrast, having multiple dominant labels is indicative of an in-distribution input, which is a characteristic that Jnirl captures. As a major advantage, our method circumvents the challenge to directly estimate the joint likelihood using generative models, which can be computationally intractable to train and optimize on multi-label datasets [i7
3.1 JointEnergy for Multi-Label OOD Detection
We propose using the JointEnergy function (x) defined in Section for OOD detection:
out if Ejoint (x) S T if Eegiin (x) > T;
G(x: r)
(14)
in
where is the energy threshold, and can be chosen so that high fraction (e.g., 95%) of in-distribution data is correctly classified by G(x; T). The sensitivity analysis on T is provided in Figure 3 A data point with higher JointEnergy Ejoim (x) is considered as in-distribution, and vice versa (see Fig.
4 Experiments
In this section, we describe our experimental setup (Section 4.1 and demonstrate the effectiveness of our method on several OOD evaluation tasks (Section 4.2) We also conduct extensive ablation studies and comparative analysis that lead to an improved understanding of different methods.
4.1 Setup
VOC In-distribution Datasets We consider three multi-label datasets: MS-COCO 2
[II] and NUS-WIDS 6 MS-COCO consists of 82,783 training, 40,504 validation, and 40,775
contains 22,531 images across 20 classes. NUS-WIDR includes 269,648 images across 81 concept labels. Since NUS- JUS-WIDE has invalid and untagged images, we follow [56] and use 1 19,986 training images and 80,283 test images.
testing images with common object categories. PASCAL-
Training Details We train three multi-label classifiers, one for each dataset above. The classifiers have a DenseNet-12) 121 backbone architecture, with final layer that is replaced by 2 fully connected layers. Each classifier is pre- rreitrained on ImageNet- IK and then fine-tuned with the logistic sigmoid function its corresponding multi-labed dataset. We use the Adam optimizer 23 with standard parameters (31 = 0.9, 32 0.999). The initial learning rate is 10 for the fully connected layers and 10-5 for convolutional layers. We also augmented the data with random crops and random flips to obtain color images of size 256 x 256. After training, the MAP is 87.51 % for PASCAL-VOC. 73.83% for MS-COCO, and 60. 20% for GeForce RTX 2080Ti.
All experiments are conducted on NVIDIA

---

Table 2: Ablation study on the effect summation for prior approaches. We use DenseNet [9] to train on the in-distribution datascts. We use ImageNet as OOD test data as described in Section 4.1 Note that Sum does not apply to tree-based KNN-bbsed approaches (c.g., LOF and Isolation Forest).
Dm
MS MS-COCO
PASCAL
NUS WIDE
FPR95/ AUROC AUPR
OOD Score
Aggregation
Logit
Sum
95,46 / 61.81 80.39
87.18 72,68 61.24
96.53
51.75/ 82.55
Prob
Sum
45.04 89.32 99.44
38.57
86.53 79.10
50.84
83.82 195.15
ODIN
Sum
56,56 / 84.62 92.24
50.35
/ 79.45
70.19
56.26
81.04 94.34
Mahalanobis
Sum
53.43 / 87.52 / 93.35
44.43 / 87.76 79.86
69.05 / 80.46/
94.09
LOF
Sum
N/A
N/A
N/A
Isolation Forest
Sum
N/A
N/A
N/A
JointEnergy (ours)
Sum
33.48 / 92.70 / 96.25
41.01 /9110/ 86.33
48.98/ 88.30 / 96.40
Datasets To evaluate the models trained on the in-distribution datasets above, we follow the same set up as in [15 and use ImageNet [9] for its generality. Besides, we evaluate against the Textures dataset [ OOD, For ImageNet, we use the same set of 20 classes chosen from ImageNet-22K as in 15 These classes are chosen not to overlap with ImageNet- k since the multi-label classifiers are pre-trained on ImageNet-1K Specifically, we use the following classes for evaluating the MS-COCO and PASCAL-VOC pre-trained models: dolphin, deer, bat, rhino, raccoon, octopus, giant clam, leech, venus flytrap, cherry tree, Japanese cherry blossoms, redwood, sunflower, croissant, stick cinnamon, cotton, rice, sugar cane, bamboo, and turmeric. Since NUS-WIDD contains high-level concepts like animal, plants and flowers, we use a different set of classes that are distinct from NUS-WIDE: asterism, butlery, cave, cylinder, delta, fabric, filament, fire bell, hornet nest, kazoo, lichen, naval equipment, newspaper. paperclip, pythium, satellite, thumb, x-ray tube, yeast, zither.
Evaluation Metries We measure the following metries that are commonly used for OOD detection: (1) the false positive rate (FPR95) of OOD examples when the true positive rate of in-distribution examples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR).
4.2 Results
How does JointEnergy compare to common OOD
1.00
detection methods? In Table D we compare energy- based approaches against competitive OOD detec-
0.90
0.80
tion methods in literature, where JointEnergy demon- strates state-of-the-e parisons, we consider approaches that rely on pre- trained models (without performing retraining or fine- tuning). Following the setup in [15], all the num- bers reported are evaluated on ImageNet OOD test
0.70
performance. For fair com-
True Positi
0.50
0.40
PASCAL-VOC
0.30
COCO
0.20
NUS-WIDE
0.10
data, as described in Section 4.1 We provide ad- ditional evaluation results for the Texture OOD test dataset in the supplementary. Most baselines such as MaxLogit (15), Maximum Softmax Probability (MSP) [66. ODIN 28 and Mahalanobis 27 de-
0.4
0.6
0.8
1.0
False Positive Rate
Figure 3: AUROC curves for OOD detec- tor obtained from three in-distribution multi- label classification datasets.
rive OOD indicator scores based on the maximum- valued statistics among all labels. Local Outlier Fac-
tor (LOF) B uses K-nearest neighbors (KNN) to estimate the local density, where OOD examples are detected from having lower density compared to their neighbors. Isolation forest [31 a brsed approach, which detects anomaly based on the path length from the root node to the terminating node.
Among different approaches, JointEnergy outperforms the best-performing baseline across three multi-label classifiers considered. In particular, on network trained with the MS-COCO dataset, Joint reduces FPR95 by 10.05%, compared to MaxL.ogit. We provide the AUROC curves for our method JointEnergy in Figure for all three in-distribution datasets considered. The y-axis
7

---

-200
-5,00
16
2.00
1:75
12
1.50
1.25
Distribution
1.00
Dut-of-Distriutbon
0.6
0.75
0.50
0.5
0.0
0.00
200
bon
Figure 4: Label-wise energy scores - Ey, (x) for in-distribution example from PASCAL-VOC (left), and OOD input from ImageNet (right). The OOD input is misclassified using MaxLogit score since the dominant output has high activation, making it indistinguishable from in o--iitribution data's MaxLogit score. In contrast, JointEnergy correctly classifies both images since results in larger differences scores between in-distribution and OOD inputs.
the true positive rate (TPR), whereas the x-axis is the FPR. The curves indicate how the OOD detection performance changes we vary the threshold r in Equation [4] on a different architecture, ResNet [14], for which we observe consistent improvement and provide details in the supplementary.
We additionally evaluate
We also note here that existing approaches (such Mahalanobis distance 27 requires training a separate classifier for OOD detection. In contrast, JointEnergy is hyperparameter-fres and easy to use in practice. In particular, the Mahalanobis approach is based on the assumption that feature representation forms class-conditional Gaussian distributions, and hence may not be well suited for the multi-label setting (which requires joint distribution to be learned).
How do different aggregation methods affect OOD detection performance? Table 5 we also perform a comparative analysis of the effect of different aggregation functions that combine label-wise energy scores. As an alternative, we consider
Emax (x) = max EE,, (x),
(15)
which finds the largest label-wise energy score among all labels. We observe that MaxEnergy does not outperform JointEnergy, which utilizes information jointly from all the labels. The performance of MaxEnergy on par with MaxLogit since MaxEnergy, given by max; log(1 + efn, ), is approx- imately close to the Maxl.ogit when fu. (x) is large. The results underline the importance of taking label. This because, in
into account information from multiple labels, not just the maximum-valus multi-label classification, the model may assign high probabilities to several classes. Theoretically, JointEnergy is also more meaningful, and can be interpreted from a joint likelihood perspective as shown in Section K
What is the effect of applying the aggregation method to prior methods? As an extension, we explore the effectiveness of applying the aggregation method to previous scoring functions. The results are summarized in Table 2
We calculate scores based on the logit fu. (x), sigmoid of the logit I+e 00) ODIN score, and Mahalanobis distance score Mg. (x) for each label independently.
1
We then perform summation across the label-wise scores as the overall OOD score. This ablation essentially replaces the Max aggregation with Sum, which helps understand the extent to which previous approaches are amenable in the multi-label setting. Note that the summation aggregation method does not apply to tree-based or KNN-based approaches such as LOF and Isolation Forest.
We found that applying summation over individual logi/MMSPODDI//Maaaammhh scores from each label does not enhance but sometimes worsen the performance. For example, simply summing over Table 3: Ablation study on the effect aggregation methods: max vs summation, Values are AUROC
Din
MaxEnergy
JointEnergy
MS-COCO
89.11
92.70
PASCAL- VOC
89.22
91.10
NUS-WIDE
83.58
88.30
8

---

the logits across the labels leads to severe degradation in performance since the outputs are mixed with positive and negative numbers. On MS-COCO, the FPR degrades from 43.53% using MaxLogit to 95.46% (using Suml lggit). In contrast, Joint does not suffer from this issue. This underlines the importance of choosing proper label-wise scoring function to be compatible with the aggregation method.
JointEnergy vs. SumProb We highlight the advantage of JointEnergy over SumProb both empir- ically and theoretically. As seen in Table 2 the performance difference between JointEnergy and SumProb is substantial. In particular, on MS-COCO, our method outperforms SumProb by 11.56% (FPR95). For threshold independent metric AUROC, JointEnergy consistently outperforms SumProb by 3.38% (MS-COCO), 4.57% (PASCAL), and 4.48% (NUS-WiL) WIDE).
Qualitative Case Study Lastly, to provide further insights on our method, we qualitatively examine examples from the multi-label classification dataset PASCAL-VOC (in-dist.) and OOD input from the ImageNet that are correctly classified by JointEnergy but not MaxLoggi.. In Figure (left), we see an in-distribution example labeled as dog, car, chair and person, with MaxLogit score 1.63 and JointEnergy score 3.23. We also show an OOD input (Figure 3 activation on the bird class, with MaxLogit score 2.14 and JointEnergy score 2.19. In this example, taking the sum appropriately results in a higher score for the in-distribution image than the OOD image. Contrarily, MaxLogit score for the in-distribution image is lower than that of the OOD image, which results in ineffective detection.
right) with a single dominant
Related Work
Multi-label Classification The task of identifying multiple classes within an input example is of significant interest in many applications [47] where deep neural networks are commonly used as the classifiers. Natural images usually contain several objects and may have many associated tags [50] Chen et al. [12] used convolutional neural networks (CNN) to annotate images with 3 or 5 tags on the NUS-WIDI dataset. (5) used CNNs to tag images of road scenes from 52 possible labels. In the medical domain, Wang et al. [51] presented a chest X-ray dataset in which one image may contain multiple abnormalities. Multi-label classification is also prominent in natural language processing [36]. Our proposed method is therefore relevant to a wide range of applications in the real world.
Out-of- outtof-distribution Uncertainty Estimation Detecting and rejecting unknowns has long history in machine learning; see [54] for a comprehensive survey of the main ideas. We highlight few repre- sentative works in the context of deep learning. The phenomenon of neural networks' overconfidence to out-of-distribution data revealed by Nguyen el al. (37] Previous works attempt to improve the Mahalanobis distance-based GradNorm score 20 Recent work by Liu et al. KB
OOD uncertainty estimation by proposing the ODIN score [18 28 confidence score [27]. and gradient- proposed using an energy score for OOD detection, which demonstrated advantages over the softmax confidence score both empirically and theoretically. Huang and Li 21 proposed a group-based OOD detection method that scales effectively to large-scale dataset ImageNet. However, previous methods primarily focused on multi-class classification networks. In contrast, we propose a hyperparameter- free measurement that allows effective OOD detection in the underexplored multi-label setting, where the information from various labels is combined in a theoretically interpretable manner.
Generative-based -fit----iittiittionn Detection Generative models [10 22
24
41
45 49 can
be alternative approaches for detecting OOD examples, as they directly estimate the in-distribution density and can declare a test sample to be out-of-distribution if it lies in the low-density regions. However, as shown by Nalisnick et al. (35), deep generative models can assign a high likelihood to distri-iittiittiionn data. Deep generative models can be more effective for out-of-distribution detection using likelihood ratio test 40 4
and likelihood regret (52) Though our work is based on discriminative classification models, we show that label-wise energy scores can be theoretically interpreted from a data density perspective. More importantly, generative based models [17] can be prohibitively challenging to train and optimize, especially on large and complex multi-label datasets that we considered (e.g., MS-COCÓ, NUS-WIDE etc.). In contrast, our method relies on a discriminative multi-label classifier, which can be easily optimized using standard SGD.
Energy-based Learning Energy-based machine learning models date back to Boltzmann ma- chines 42. Energy-basse based learning 26 38 39 provides unified framework for many probabilistic

---

and pro-oabbabiiiiticc approaches to learning. Recent work [55 also demonstrated using energy functions to train GANs [3]. where the discriminator uses energy values differentiate between real and generated images. Xie et al. 53 showed that a discriminative classifier can be interpreted from an energy-based perspective. Energy-based methods are also used structure prediction 2 48 Liu et al. [4 first proposed using energy score for OOD uncertainty estimation, which demonstrated superior performance for multi-class classification networks. In contrast, our work focuses on a multi-label setting, where we contribute both empirical and theoretical insights and demonstrate the effectiveness of utilizing information jointly from across labels.
Conclusion and Outlook
In this work, we propose energy scores for OOD uncertainty estimation in the multi-label classifi- cation setting. We show that aggregating energies over all labels into JointEnergy results in better separation between in-distribution and OOD inputs compared using information from only one label's information. Additionally, we justify the mathematical interpretation of Join(Energy from joint likelihood perspective. JointEnergy obtains better OOD detection performance compared to competitive baseline methods, establishing new state-of state-of-the-ar on this task. Applications of multi-label classification can benefit from our methods, and we anticipate further research in OOD detection to extend this work to tasks beyond image recognition. We hope our work will increase the attention toward a broader view of OOD uncertainty estimation for multi-label classification.
Societal Impact
Our project aims improve the dependability and trustworthiness of modern machine learning models for multi-label classification. This stands to benefit wide range of fields and societal activities. We believe out-of-distribution uncertainty estimation is an increasingly critical component of systems that range from consumer and business applications (e.g.. digital content understanding) to transportation (e.g., driver assistance systems and autonomous vehicles), and to health care (e.g., unseen disease identification). Many of these applications require multi-label classification models in operation. Through this work and by releasing researchers a new methodological perspective and offer machine learning practitioners an easy-to-use tool that renders safety against OOD data in the real world. While do not anticipate any negative consequences to our work, we hope to continue to build on our framework in future work.
code, hope provide machine learning
Acknowledgement
Research supported by the Office of the Vice Chancellor for Research and Graduate Education (OVCRGE) with funding from the Wisconsin Alumni Research Foundation (WARF).
10

---

References
[1] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9()):147-169, 1985.
[2] David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference Machine Learning, pages 983-992, 2016.
[3] Markus M Breunig, Hans-Petee Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[4] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2021.
[5] Long Chen, Wujing Zhan, Wei Tian, Yuhang He, and Qin Zou. Deep integration: A multi-label architecture for road scene recognition. IEEE Transactions on Image Processing, 28(10):4883- 4898, 2019.
[6] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. Nus-wide: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR 009), Santorini, Greece., 2009.
[7] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[8] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-20482 255.255. leee, 2009.
[10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605.08005 2016.
[11] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 1 111(1):98-136 January 2015.
[12] Yunchao Gong, Yangging Jia, Thomas Leung, Alexander Toshev, and Sergey loffe. Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4889, 2013.
[13] Ian Goodfellow, Jean Pouget-Abadia Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-645. Springer, 2016.
[15] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. arXiv preprint arXiv: 1al:::911111 1112, 2019.
[16] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and cit-of-distribution examples neural networks. Proceedings of International Conference on Learning Represen- tations, 2017.
[17] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. arXiv preprint 1XXv:1901.00686 2019.

---

[18] Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from -utoo--distribution data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10951-10960, 2020.
[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceddings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[20] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
[21] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8710-87111 June 2021.
[22] Xun Huang, Yixuan Li, Omid Poursaced, John E Hopcroft, and Serge J Belongie. Stacked generative adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, volume 2, page 3, 2017.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698, 2014.
[24] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.6114, 2013.
[25] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pages 6402-6413, 2017.
[26] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[28] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018
[29] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[30] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15313-15323, June 2021.
[31] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pages 413 422. IEEE, 2008
[32] Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi-label text classification. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 115-124, 2017.
[33] Shuhua Monica Liu and Jiun-Hung Chen. A multi-label classification based approach for sentiment classification. Expert Systems with Applications, 42(3): 2015.
[34] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -uitt--disttibttion detection. Advances in Neural Information Processing Systems, 2020.
[35] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint 1rXivv1810.099 2018.
12

---

[36] Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, and Johannes Fümkkanz. Large- scale mulli-label text classification -eviiitttion-rviitiiiig neural networks. In Joint european conference on machine learning and knowledge discovery in databases, pages 437-452. Springer, 2014.
revisiting
[37] Anh Nguyen, Jason Yosinski. and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[38] Marc' Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann I. Cun. Efficient learning of sparse representations with an energy-based model. Advances in neural information processing systems, pages 1137-1144, 2007.
[39] Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based framework for unsupervised learning. In Artificial Intelligence and Statistics, pages 371-379, 2007.
[40] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshninarayanan. Likelihood ratios for out-of-distribution detection, In Advances in Neural Information Processing Systems, pages 14680-14691, 2019.
[41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint vrXiv:1401.4082 2014.
[42] Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 693-700, 2010.
[43] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and out-of- of-distribution detection with likelihood- basdd generative models. arXiv preprint arXiv: I 1909. 1 1480, 2019.
[44] Yiyou Sun, Chuan Guo, and Yixuan Li. React: fut-of-disttibution detection with rectified activations. Advances in Neural Information Processing Systems, 2021.
[45] Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algo- rithms. Communications on Pure and Applied Mathematics, 66(2):145-164 2013.
[46] Konstantinos Trohidis, Grigorios Tsoumakas, George Kalliris, and Ioannis P Vlahavas. Multi- label classification of music into emotions. In ISMIR, volume 8, pages 325-330, 2008.
[47] Grigorios Tsoumakas and loannis Katakis. Multi-label classification: An overview. Interna- tional Journal of Data Warehousing and Mining (IJDWM),, 3(3):1113, 2007.
[48] Lifu Tu and Kevin Gimpel. Learning approximate inference networks for structured prediction. In International Conference on Learning Representations, 2018.
[49] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances in neural information processing systems, pages 4790-4798, 2016.
[50] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2285-2294, 2016.
[51] Xiaosong Wang, Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[52] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-dist detection score for variational auto-encoder. Advances Neural Information Processing Systems, 33, 2020.
13

---

[53] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning. pages 2635-2644, 2016.
[54] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.1133 2021.
[55] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, 2019.
[56] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial regularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5513-5522, 2017.
14


Reviews:
[{"review_id": "cVQA1q7Xmo", "reviewer": "Reviewer_p44K", "summary": "The paper presents a new score - JointEnergy, as a sum of free energy across labels to detect out of distribution inputs in a multi-label classification setting. It presents the definition of this new score, motivates it theoretically by connecting it to joint likelihood of data, and demonstrates the superior performance of this score over existing alternatives on 3 different datasets. The results are interesting and significant.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The problem of OOD detection in multi-label classification setting is under studied. This paper extends existing ideas in a novel way to achieve superior detection performance. \n\nQuality: While the proposed new OOD  detection score is simple and straightforward extension of the free energy interpretation of logit scores, the new score is better motivated theoretically, and shows superior performance to existing alternatives. It uses more information in the scores of all labels instead of just the dominant label and hence is better able to detect OOD inputs. Despite its simplicity, the new score has better performance. It is also free of hyper parameters, which makes it easier to implement. The method applies to general multi-label classification and hence should be extensible to tasks beyond vision (although the paper does not demonstrate it). \n\nClarity: The paper is written well. The new score is motivated well, is theoretically justified, and experiments are simple and conclusive\n\nSignificance: Despite its simplicity, the new score seems to be quite powerful in detecting OOD inputs in multi-label classification, and can become a significant evaluation metric for this purpose in the future.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not clear how the new score performs against baselines in detecting OOD inputs in other tasks (other than vision - like language, speech/audio, structured datasets etc.). It is also not clear how it performs when training a different model for each label (i.e without sharing a pre-trained representation). The authors can better address these 2 aspects through both theoretical analysis and experiments. \n\nI do not anticipate any significant negative societal impact of this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not anticipate any significant ethical concerns of this work. ", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "Skhk4tqQ8Wb", "reviewer": "Reviewer_ggDB", "summary": "The paper proposes a new method for OOD detection in the multi-label setting, called JointEnergy. \nThe authors claim that JointEnergy can be mathematically interpreted from a joint likelihood perspective, and that their model achieves sota results on a variety of benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance**: high - the problem of OOD detection is well studied for multi-class classification problems, but there is little work on OOD detection for multi-label classification problems. \n\n**Clarity**: low - the authors do not provide any background on energy-based models, which is necessary to follow the theoretical analysis.\n\n**Quality**: The experimental analysis shows that the proposed solution is very effective, however I have some concerns with the theoretical analysis.\n\n1. the authors define the label-wise free energy as: $E_{y_i}({x}) = -\u0000\\log(1 + \\exp^{f_{y_i (x)}})$. However, they do not explain neither  why they defined the energy function in that way nor which properties the function has that make it an energy function. \nThis does not allow the reviewers to check whether such definition (and hence all the subsequent calculations) are correct. \nI would advise the authors to look into how [1] defined the energy function in the case considered, as it is very clear and easy to follow. \n\n2. the authors make the strong assumption of label independence. However, if we assume the label independence (which is almost never verified in practice), then we are basically dealing with a series of binary classification problems. The work would be more complete if this assumption was dropped.\n\n**Novelty**: medium - the paper presents a novel and simple idea that is able to beat sota models on the task of OOD detection. \nHowever, the strong label independence assumption hinders its novelty. \n\n[1] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.\n\n**After author response**: the authors have addressed my concerns, thus I change my score from 6 to 7. Congrats!\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8O5-L8b9pFB", "reviewer": "Reviewer_QXpR", "summary": "This paper studies the out-of-distribution detection problem in the multi-label classification setting. The authors propose a scoring function named JointEnergy to measure the prediction uncertainty of a given instance. In particular, the JointEnergy aggregates the label wise free-engery by summation and a hard thresholding on the value of JointEnergy is used to determine whether a given input is OOD or not. It is shown that the propose scoring function can be decomposed into three terms: joint conditional log-likelihood on all classes, un-conditional log data density and a constant term. Empirical results on three multi-label classification datasets show that the proposed method can achieve lower false positive rate (FPR95) while having higher area under curve scores (AUROC, AUPR).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The scoring function takes a seemingly trivial format which is the simple summation of the per-label free energy for a given input. However, the authors justify the choice by associate the JointEnergy with the joint conditional log-likelihood and the log data density, both are expected to be high for in-distribution samples. The experiments also shows that the choice is valid. In general, I think this paper proposes a simple but effective method to handle an often overlooked problem and vote for accept. Several minor comments:\n\n(1)  The label-wise free energy form is based on that the predictive probability is given by sigmoid(f(x)). For the experiments, models are fine-tuned with the logistic sigmoid functions. What's the connection between how the model is trained and how the JointEnergy estimator would work? Can the JointEnergy estimator work on models trained with other logit smoothing functions?\n\n(2) The two terms, log data density and joint conditional log-likelihood, in the JointEnergy seems to be scaling linearly with the number of labels K. Therefore the contributions of the two terms will remain roughly the same even with very large K. In the case where K is very large, i.e. extreme multi-label classification, only top-k predictions are given. I wonder if the JointEnergy can be generalized to that case where the logit from k+1 to K are truncated.\n\n(3) It would be helpful to do some ablation study on how the log data density and log-likelihood would perform on their own.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper studies the out-of-distribution detection problem in the multi-label classification setting. To my understanding, the work poses no negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "-D6zuEmHG6-", "reviewer": "Reviewer_GeqC", "summary": "This paper studies a new task that detects out-of-distribution (OOD) data in the multi-label setting. Inspired by recent progress in multi-class classification problems, the authors proposed a joint energy score to achieve their goal. Specifically, the proposed method sums up the label-wise energy score. Then, it is used as a criterion to determine which samples are out-of-distribution by thresholding. Empirical results demonstrate that the proposed method achieved promising results on COCO, NUSWIDE, and VOC benchmark datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. While the out-of-distribution detection task has been widely studied in the multi-class classification context, it could be very tricky in multi-label learning, because there can be combinational objects and some combinations may never occur. \n2. The experimental results are thorough to verify the effectiveness of the proposed method, which I believe will be beneficial to the community.\n\nCons:\n1. The definition of energy score in [1] is an unconditional version. However, the authors explain their criterion through the line of the conditional version of energy. They should refer their definition to other works that give the conditional form of energy, e.g. https://arxiv.org/pdf/2002.11537.pdf.\n2. The authors claim that in the case of OOD, each of the conditional likelihoods should be small, hence the product. Actually, we are considering a conditional probability $P(X=x|Y_1=1,Y_2=1,...,Y_k=1)$. I suppose the only property of OOD data is: given an OOD data, $P(y_1=0,...,y_K=0|X=x)$ is far larger than other data. It is not an obvious induction to conclude $P(X=x_1|Y_1=1,Y_2=1,...,Y_k=1)<P(X=x_2|Y_1=1,Y_2=1,...,Y_k=1)$, s.t. $x_1$ is OOD while $x_2$ is in-distribution. Actually, we can find the key in Eq. (8) where the production of per-label posterior is low for in-distribution data. Thus, we needn't further explore the conditional density.\n\n[1] LeCun Y, Chopra S, Hadsell R, et al. A tutorial on energy-based learning[J]. Predicting structured data, 2006, 1(0).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have mixed thoughts on this paper. On one hand, the proposed joint density score is a simple generalization from the previous work. On the other hand, I believe this work can inspire a series of works and appreciate its contribution to well-defining new research topics and establishing a simple yet effective benchmark.\n\nMoreover, I recommend the authors further polished the mathematical interpretation section. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend to accept this paper

In this paper, the authors proposed a novel method to address an important yet under explored problem: out-of-distribution detection for multi-label classification models. This paper is well-written, addressing important problem, proposing simple-but-effective method. All the reviewers are inclined to accept this paper. I would suggest the authors to add additional experimental results done in the rebuttal phase and take the reviewers' suggestions into the camera-ready version.
Decision: Accept (Poster)