Energy-based Out-of-distribution Detection for Multi-label Classification
Authors: Haoran Wang
Weitang Liu
Alex Bocchieri
Yixuan Li
Venue:  (2021)

Abstract: Out-of-distribution (OOD) detection is essential to prevent anomalous inputs from causing a model to fail during deployment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose SumEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that SumEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that SumEnergy reduces the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance. 

Under review as a conference paper at ICLR 2021
ENERGY-BASED OUT-OF-DISTRIBUTION DETECTION FOR MULTI-LABEL CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Out-of oifto---ittiiutiinn (OOD) detection is essential to prevent anomalous inputs from causing model to fail during deployment. Improved methods for OOD detec- tion in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose SumEnergy, simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that SumEnergy can be mathematically interpreted from joint likelihood perspective. Our results show consistent improvement over previous methods that are based the maximum-valuee scores, which fail to capture joint information from mul- tiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that SumEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing of-te-of-hhearttt performance.
INTRODUCTION
Out-of-distrbuttion (OOD) detection is central for reliably deploying machine learning models in open-worid environments, where new forms of test-time data may appear that were nonexistent during the training time. The problem of OOD detection has gained significant research attention lately, given its importance for safety-critical applications such as unseen disease identification (Cao
al. (2020). However, recent studies have primarily focused on detecting OOD examples in multi-
class classification, where each sample is assigned to one and only one label (Bevandić et al. 2018
Hein et al.
2019 Hendrycks
Gimpell 2016,] Lakshminarayanan
al., 2017 Lee et 2018
Liang et al.
2018 Mohseni et al.) 2220 Chen et al.) 20200 Hsu et al. 2020 Liu et al.) 2020) This
can be restrictive in many real-world settings where images often have multiple objects of interest.
For example, self-driving cars must differentiate between the road, traffic signs, and obstacles within
a frame. In the medical domain, multiple abnormalities may be present a medical image (Wang ett al.) 2017). Multi-label classification is desirable since there is no constraint on the number of classes an instance can be assigned to.
Currently, OOD detection in multi-label classification remains relatively underexplored. Out of the multi-label methods evaluated in (Hendrycks et al. 2019), MaxLogit achieved the best performance. However, simply using the maximum-valuee logit is limiting because it does not incorporate infor- mation available from other possible labels. As seen in Figure I MaxLogit can only capture the difference between the dominant outputs for dog (in-dist.) and car (OOD), while positive infor- mation from another dominant label cat (in-dist.) is dismissed. Other baseline methods such as ODIN (Liang et al., 2018) and Mahalanobis (Lee et al., 2018) also derive scores based on the max- imum score (e.g., calibrated softmax score or Mahalanobis distance), and fail to capture the joint information. While energy scores have recently demonstrated superior OOD detection performance in the multi-class setting (Liu et al. 2020), this method does not trivially generalize to multi-label classification setting where labels are not mutually exclusive. Hitherto, a key challenge lies in how leverage information across different labels.
In this paper, we propose an energy-based based method for OOD detection in the multi-label setting, which estimates OOD indicator scores jointly from multiple labels. We propose simple and effec- tive aggregation mechanism, SumEnergy, that seeks to combine the label-wise energy scores derived from individually independent classes. We show that the SumFnergy scores mathematically

---

Under review as a conference paper at ICLR 2021
X I-Dissributio
f(x:0)
Plane
Gar
Bird
Cat
Deer
Dog
Frog
Horee Ship
Truck
Max
Sum
aggregating over many
Out-ot-Distrbutton
In-Distribution
Duttof-DDitributioo
BunBrorgyy
Figure 1: Energy-basse out-of-distribution detection for multi-label classification, During inference time, input x passed through classifier and label-wise scores computed for each label. OOD indicator scores are either the maximum valued score (denoted by green outlines) the sum all scores. Taking the sum results in larger difference scores more separation between in-distribution and OOD inputs (denoted by red lines), resulting in better OOD detection. Plots in the bottom right depict the probability densities of maximum-valtes versus summed scores.
meaningful, and can be interpreted from joint likelihood perspective. Intuitively, input with multiple dominant labels is more likely to be in-distribution, which the key aspect that SumEnergy capitalizes on. As shown in Figure 1 summing label-wise energies over all labels amplifies the difference scores between in-distribution and OOD inputs. Our method is parameter-free and can be conveniently used for any pre-trained multi-label classification model. Below describe our contributions in detail.
Contributions First, we propose theoretically motivated scoring function, SumEnergy, that is based on the aggregation over label-wise energy scores. Extensive experiments show that SumEn- ergy outperforms existing methods on three common multi-label classification benchmarks, estab- lishing state-of-the-art performance. For example, al. 2014), our method reduces the false positive rate (at 95% TPR) by 10.05% when evaluated against ImageNet OOD data, compared to the best performing baselines. Consistent performance improvement is also observed on a different OOD test dataset Texture (Cimpoi et al. 2014a), as well as alternative network architecture.
a DenseNet trained with MS-COCO (Lin
Additionally, we perform a comparative analysis of how an alternative aggregation method affects OOD detection performance. In particular, consider MaxEnergy, which takes the maximum energy score among the individual labels as the OOD indicator score. On a DenseNet trained with the PASCAL-VOC dataset, MaxEnergy yields 4.05% lower FPR95 compared SumEnergy, which underlines the importance of taking into account scores derived from multiple labels.
Lastly, ablation, demonstrate that energy summation aggregation method, compared to previous scoring functions such as logit (Hendrycks) al. 2019), MSP (Hendryeks & Gimpel| 2016),, ODIN (Liang al. 2018), and Mahalanobis dis- lance [Le et al. 2018). To see this, we explore the effectiveness of applying aggregation methods previous popular scoring functions, which helps understand their applicability in the multi labei setting. We find that summing labels' scores using previous methods is inferior to summing la- bels" energies, emphasizing the need for SumEnergy. For example, simply summing over the logits across labels results in up to 51.93% degradation in FPR95 on MS-COCO, since the outputs are mixed with positive and negative numbers. In contrast, SumEnergy does not suffer from this issue because the signs of label-wise energy scores are uniform. More importantly, label-wise energy is provably aligned with the probability density of the corresponding label's 's training data. Our study therefore underlines the importance of properly choosing both the label-wise scoring function and the aggregation method. We show strong compatibility between the label-wise energy function and aggregation function, supported by both mathematical interpretation and empirical results.
more compatible with the proposed
2

---

Under review as a conference paper at ICLR 2021
2 METHODS
2.1 BACKGROUND: ENERGY FUNCTION IN MULTI-CLASS CLASSIFICATION
We consider a multi-class neural classifier f(x) : IR + æ² which maps an imput x E IR to K real-valued numbers as logits. A softmax function is used to derive categorical distribution,
e,v, (x)
plu x)
(1)
Σラ= e (x))
which indicates the probability for an input to be of class ys, with (1,2, .... K} A multi-class classifier can be interpreted from an energy-based perspective (Grathwohl et al. 2019) by viewing the logit fu. (x) of class y: energy function E(x, y1) -Jv (x). Therefore, Equation can be rewritten as:
F(x,5))
p(y x)
(2)
Σ K
y== E(x,)))
E(x,).)
(3)
B(x()
By equalizing the two denominators above, express the free energy function Ex) for any given input x E IR in forms of:
E(x) = log
ex (x)
2.2 ENERGY FUNCTION FOR MULTI-LABEL CLASSIFICATION
In this work, we propose using energy-based method for OOD detection in multi-label classifi- cation networks, where an input can have several labels (see Figure D In what follows, we first introduce label-wise energy function, and then propose aggregation methods that leverage the joint information across labels a theoretically meaningful way.
Label-wise Free Energy In multi-lahel classification, the prediction for each binary label ys is independently made by a binary logistic classifier:
ef,, (x)
p(u x) 1 (x)
where i {1,2, J.,,,K.,K) K). For brevity, we use yi the probabilistic derivations indicate the label being positive, i.e., 1. The logistic classifier output can be viewed as the softmax with two logits and fy. (x), respectively. For each class yr. we can define label-wise free energy as follows:
Fgg. (x) = log(1 + e (x)).
Aggregation Methods We propose and contrast two aggregation mechanisms that seek to combine the label-wise energy scores derived above:
Max
Emax (x) max -Ey, (x)
(6)
Sum
Egum (x)
Ey.. (x)
In particular, MaxEnergy finds the largest label-wise energy score among all labels; whereas SumEn- ergy takes the summation of energy scores across all labels. Note that in the above equations. label- wise energy Ey, (x) by definition is a negative value, and the aggregation methods output a positive value by negation.
3

---

Under review as a conference paper at ICLR 2021
Mathematical Interpretation We provide mathematical interpretations of different aggregation methods. First, by resorting to the energy-based model (LeCun et al.) (006), we show that the label-wise energy score is provably aligned with the conditional likelihood function. The condi- tional likelihood p(x yi) is given by:
e Fi (xx
p(x B4)
Sxlp. e By, (x)
where Zy: Sxlye (x) is the normalized density. Since Zy, is the same with respect to all x with label Y/, the denominator in Equation g does not affect the overall distributional of label-wise energy scores. By taking the logarithm on both sides Equation we have:
y. (x) & logp(x y1).
Given this, MaxEnergy can be interpreted taking the maximum of conditioned log- oggllikeldhood among labels:
Emax(x) or max p(x 9:),
which does not take into account the joint information across labels.
In contrast, we provide mathematical interpretation for SumEnergy, which is the first method to con- sider the joint estimation OOD scores across labels. We show that SumEnergy can be interpreted from the joint likelihood perspective:
Exum (x)
log p(x yi) Zya
(10)
logp(x yi) +
log Zy,
(11)
Z:coostant x
By applying Bayesian rule for each term log p(x (1) in Equation I1 we have
K
p(y
x) p(x)
Esum (x) = log
+ Z
(12)
p(y))
i=1
K
log
I
p(u xx) + K l(g) + (Z log II p(n))
(13)
C: constant for all x
Given all label Y/4 are conditionally independent, we have [ Therefore, Equation 13 equivalent to:
p(y: x) p(v1. U2.,.., VK
x).
Fxum (x) = log p(v1 V2,..., UK x) + K <Koog((x) +
(14)
log p(x 1/. 1/2 uK)
11 p(y))
K. logp(x) +
(15)
x(x)
Logp(x /
yı, U2, --.. yK)
(K (K-1) 1T-1)-logp(s
(16)
joint conditional log likelihood
log density, 1 in-distrbtttion
constant
The equation above suggests that Exum(x is in fact linearly aligned with the joint conditional log likelihood and log data density. The second term is desirable for OOD detection since it is aligned with the underlying data density, which is higher for in-distribution data x and vice versa. The first term takes into account joint estimation across labels, which new to our multi-label setting and was not previously considered in multi-class setting (Liu et al. 2020). The first term allows even further discriminativity between in- vs. OOD data, since OOD data is expected to have lower joint conditional likelihood (i.e., not associated with any of the labels). In contrast, having multiple
4

---

Under review as a conference paper at ICLR 2021
dominant labels is indicative of in-distribution imput, which is characteristic that SumEnergy captures. More importantly, our method does not require estimating the density Z explicitly, as Z is sample independent and does not affect the overall SumEnergy score distribution.
We note that our derivation is based on the assumption that all labels are independent, which is in accordance with standard multi-label classification loss by treating each label as binary prediction problem T'oumakas & Katakis, 2007). We consider this setting for its simplicity and generality. Our work also opens up an interesting future direction of OOD detection by considering the struc- tural dependency among labels (Chen et al. 2017 c)
2.3 AGGREGATED ENERGY FOR MULTI-LABEL OOD DETECTION
We propose using the aggregated energy functions E(x) defined in Section 2.2 for OOD detection:
out if F(() S T,
G(x:T)=
(17)
in
if Ef > T,
where 7 is the energy threshold, and can be chosen so that a high fraction of in-distribution data is correctly classified by G(x; Exx) could take on forms of Max or Sum. A data point with higher aggregated energy Ex) is considered as in-distribution, and vice versa (see Figure []. We explore
and provide the tradeoff of different aggregation methods in Section 3.2
EXPERIMENTS
In this section, we describe our experimental setup (Section 3.1 and demonstrate the effectiveness
of our method on several OOD evaluation tasks (Section 3.2) We also conduct extensive ablation studies and comparative analysis that leads to improved understanding of different methods.
3.1 SETUP
In-distribution Datasets We consider three multi-label datasets: MS-COCO (Lin et al. 2014), PASCAL-VOC (Everingham et al.) (015), and NUS-WIDE (Chua al. 2009). MS-COCO consists of 82,788 training, 40,504 validation, and 40,775 testing images with 80 common object categories. PASCAL- VOC contains 22,531 images across 20 classes. NUS-W WUSE includes 269,648 images across 81 concept labels. Since LUSE has invalid and untagged images, we follow the work (Zhu et al. 2017) and use 119,986 training images and 80,283 test images.
Training Details We train three multi-label classifiers, one for each dataset above. 'Te classifiers have a DenseNet- 121 backbone architecture, with a final layer that is replaced by 2 fully connected layers. Each classifier is pre-trained on ImageNet- KK and then fine-tuned with the logistic sigmoid function to its corresponding multi-label dataset. We use the Adam optimizer (Kingma & Ba 2014) with standard parameters (3) 0.9, B2 0.999). The initial learning rate 10 for the fully connected layers and 10-6 for convolutional layers. We also augmented the data with random crops and random flips obtain color images of size 256 256. After training, the mAP is 87.51 % for PASCAL-VOC, 73.83% for MS-COCO, and 60.22% for NUS-WIDE.
Out-of-distribuution Datasets To evaluate the models trained the in-distribution datasets above, we follow the same set up as in (Hendrycks et al. 2019) and use ImageNet (Deng et al.) 2009) for its generality. Besides, we evaluate against the Textures dataset (Cimpoi et al.) 201 4b) as OOD. For ImageNet, we use the same set of 20 classes chosen from ImageNet-22K as in (Hendrycks et al.) 2019.. These classes are chosen not overlap with ImageNet-Ik since the multi-label classífiers are pre-trained on ImageNet-IK, K.. Specifically. we use the following classes for evaluating the MS- COCO and PASCAL-VOC pre-trained models: dolphin, deer. bat, rhino, raccoon, octopus, giant clam, leech, venus flytrap, cherry free, Japanese cherry blossoms, redwood, sunflower, croissant, stick cinnamon, cotton, rice, sugar cane, bamboo, and turmeric. Since NUS-WIDE contains high- level concepts like animal, plants and flowers, use different set of classes that are distinct from NUS-WIDS asterism, battery, cave, cylinder, delta, fabric, filament, fire bell, hornet nest, kazoo, lichen, naval equipment, newspaper, paperclip, pythium, satellite, thumb, x-ray tube, yeast, zither.
We measure the following metrics that are commonly used for OOD detection: (1) the false positive rate (FPR95) of OOD examples when the true positive rate of in-distribution examples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR).
Evaluation Metrics
5

---

Under review as a conference paper at ICLR 2021
Din
MS-COCO PASCAL-VOC
NUS-WIDE
FPR95 AUROC AUPR
OOD Score
MaxLoogi (Hendrycks et al.
2019)
43.53 / 11889911111 93.74
45.06/
89.22/9.2222/ 83.14
56.46
83.58 94.32
MaxProb
43.53 1931/99111/01 99.74
45.06
89.22 83.14
56.46 83.58 94.32
MSP (enndrycks Gimpel
2016
7 85.37
74.05/79.32
/ 60.81 87.00
ODIN (Liang ct al. 2018)
43.53 89.11 933.74
89.02/
/ 83.16
56.46/ 83.58 99.32
Mahalanobis (Lee et al. 2018)
46.86 / 88 88.59/ 93,85
81.12 41.74/8..65
62.67 / 88,02 / 95,25
LOF (Breunig et al. 2000
80.44 73.95 86.01
86.34/ 69.21/ 58.93
85.21 67.75 89.61
Isolation Forest (Liu al.
2008
94.39 49.04/ 66.87
93.22 50.67 35.78
95.69 53.12 83.32
MaxEnergy
43.53 89.11 93.74 74
45.06 89.22 83.14
56.46 83.58 94.32
SumEnergy
33.48 92.70/ 96.25
41.01 99..1// 86.33
48.98 88.30 96.40
Table 1: OOD detection performance comparison using energy based approaches vs. competitive baselines. We use DenseNet. (Huang et (017) train on the in-distribution datasets. We subset of ImageNet classes as OOD test data, as descrihed in Section 3.T values are percentages. 1 indicates larger values are hetter, and indicates smaller values are hetter. Bold numbers are superior results. Description of baseline methods, additional evaluation results different OOD test data, and different architecture (e.g., ResNet) can be found in the Appendix.
3.2 RESULTS
How do energy-based approaches compare common OOD detection methods? Table I compare energy-based approaches against competitive OOD detection methods in literature, where SumEnergy demonstrates state-of-the-an performance. For fair comparisons, we consider approaches that rely on pre-trained models (without performing retraining fine-tuning) Follow- ing the set up in (Hendrycks et al.] (019). all the numbers reported are evaluated on ImageNet OOD test data, described in Section 3.1 We provide additional evaluation results for the Tex- ture OOD test dataset in Appendix [ Most baselines such as Maxl MoxLo (Hendrycks et al.) 2019), Maximum Softmax Probability (MSP) (Hendrycks Gimpel 2016). ODIN (Liang et al. (0118) and Mahalanobis (Lee et al.) 2018) derive OOD indicator scores based on the maximum-valle valued statis- ties among all labels. Local Outlier Factor (LOF) (Breunig et al. 2000) uses K-nearest neighbors (KNN) to estimate the local density, where OOD examples are detected from having lower den- sity compared to their neighbors. Isolation forest (Liu al.) 2008) tree-based approach, which detects anomaly based on the path length from the root node to the terminating node.
Among different approaches, SumEnergy outperforms the best-performing baseline across all three multi-label clas- sifiers considered, particular, on a network trained
1.00
0.95
Pusltive Rate
0.80
with the MS-COCO dataset, SumEnergy reduces FPR95
0.70
by 10.05%, compared to MaxLogit. We provide the AU-
ROC curves for our method SumEnergy in Figure 2 for all three in-distribution datasets considered. The y-axis is the true positive rate (TPR), whereas the x-axis is the FPR. The curves indicate how the OOD detection per- formance changes we vary the threshold in Equa- tion 17 We additionally evaluate on a different architec- ture. ResNet (He et al. 2016). for which we observe con- sistent improvement and provide details in the Appendix.
0.50
0.40
PASCAL-VOC
0.20
COCO
0.10
NUS-WIDE
0.00
0.2
0.4
0.6
0.8
1.0
False Positive Rate
Figure 2: AUROC curves for OOD detector obtained from three in- distribution multi-label classification datasets.
We also note here that existing approaches using a pre- trained model (such as ODIN (Liang et al.) 2018) and Mahalanobis (Lee et al.) (018)) have hyperparameters that need to be tuned. In contrast, using energy-based method on pre-trained models is parameter-fiee free and easy to use and deploy. In particular, the Mahalanobis approach is based the assumption that feature representation forms conditionti Gaussian distributions, and hence may not be well suited for the multi-label setting (which requires joint distribution to be learned).
How do different aggregation methods affect OOD detection performance? Table I we also perform comparative analysis of the effect of different aggregation functions that combine label-

---

Under review as a conference paper at ICLR 2021
Dm
MS-COCO
PASCAL
NUS-WIDE
FPR95 / AUROC/ AUPR
OOD Score
A Aggregation
Logit
Sum
95.46 61.81 80.39
87.18 72.68 /6 /6.24
96.53
51.75 82.55
Prob
Sum
45.04 / 89.32 94.40
38.57 / 86.53 79.10
50.84
/ 83.82
95.15
ODIN
Sum
56.56/ / 84.62 / 92.24
50.35 / 79.45
70.19
56.26
/ 81.04 94.34
Mahalanobis
Sum
53.43 / 87.52 93.35
44.43 / 87.76
79.86
69.05
/ 80.46
94.09
LOF
Sum
N/A
N/A
N/A
Isolation Forest
Sum
N/A
N/A
N/A
Energy (ours)
Sum
33.48 92.70 / 96.25
41.01 91.10 / 86.33
48.98 / 88.30 96.40
Ablation study the effect of summation for prior approaches. We use DenseNet (Huang et al. 2017) to train the in-distribution datasets. We use ImageNet OOD test data described in Section Note that Sun does not apply to tree-based or KNN-based approaches (e.g., LOF and Isolation Forest).
Table 2:
wise energy scores. Among those, we observe that MaxEnergy does not outperform SumEnergy, which utilizes information jointly from across the labels. The performance of MaxEnergy is on par with MaxLogit since MaxEnergy, given by max, log(1 + efv, (x)), is approximately close to the MaxLogit when (x) is large. The results underline the importance of taking into account information from other labels, not just the maximum-valuee label. This is because, in multi-label classification, the model may assign high probabilities several classes. Theoretically, SumEnergy is also more meaningful, and can be interpreted from joint likelihood perspective as shown in Section 2.2
What is the effect of applying the aggregation method to prior methods? As an extension, we explore the effectiveness of applying the aggregation method to previous scoring functions such as logit (Hendrycks al. 2019), MSP (Hendrycks & Gimpel 2016) and ODIN (Liang et al. 2018). We calculate scores based on the logit f (x), sigmoid of
The results are summarized in Table 2
the logit Ite (x) ODIN score, as well as Mahalanobis distance score Mg. (x) by treating each
label independently. We then perform summation across the label-wise scores the overall OOD score. This ablation essentially replaces the Max aggregation with Sum, which helps understand the extent to which previous approaches amenable in the multi-label setting. Note that the summation aggregation method does not apply to tree-based or KNN-based approaches such as LOF and Isolation Forest.
Interestingly, we found that applying summation over individual IggidM/P/ODDN/MMhlloooho scores from each label does not yield competitive results, and in many worsens the perfor- mance. For example, simply summing over the logits across the labels leads to severe degradation in performance since the outputs are mixed with positive and negative numbers. On MS-COCO, the FPR degrades from 43.53% using MaxLogit to 95.46% (using SumLogit). In contrast, SumEn- ergy does not suffer from this issue since the energy scores for each individual label have uniform sign. More importantly, the label-wise scores derived from energy is theoretically more meaningful than Pggd//MSPPODDNiMMaallaannn since provably aligned with the probability density of the training data corresponding the label. This underlines the importance of properly choosing the label-wise scoring function to be compatible with the aggregation method.
SumEnergy vs. SumProb We highlight the advantage of SumEnergy over SumProb both em- pirically and theoretically. As seen in Table 2 the performance difference between SumEnergy and SumProb is substantial. In particular, 11.56% (FPR95). For threshold independent metric AUROC, SumEnergy consistently outperforms SumProb by 3.38% (MS-COCO), 4.57% (PASCAL), and 4.48% (NUS-WIDE). SumEnergy is mathematically meaningful measurement and can be interpreted from a joint likelihood perspective (see Section (.2), whereas SumProb does not. In fact, one can show that the probability score for each individual label is not aligned with the conditioned data density function. To see this, we can derive the probability for each binary logistic classifier as:
MS-COCO, our method outperforms SumProb by
logp(u: xx) = -log
1 1 efx,()) = Sy. (x) + Ey. (x).

---

Under review as a conference paper at ICLR 2021
-5,00
E,00
16
2,00
14
1.75
12
1.50
10
1.25
08
In- Distrautionn
1,00
Out-af-Distriban
0.75
04
0.00
02
0.25
00
Bied
boot
0.00
accoptanc
heyee
bind
boott
botbl
bus
cat
cat
cool
deg
Figure 3: Lahel-wise energy scores - Eyy, (x) for in-distribution example from PASCAL-VOC (left), and OOD input from ImageNet (right). The OOD input is misclassified using MaxLogit score since the dominant output has high activation, making it indistinguishable from an in-distrihution data's MaxLagit score. In contrast, SumEnergy correctly classifies both images since it results larger differences in scores between in-distribution and OOD inputs.
In fact, the first term fy. (x) is larger for in-distribution data with label yr+ whereas the second term smaller for in in-distribution data with Ey, (x) xx - logp(x y.). This leads to a biased scoring distribution that is no longer proportional the label-conditional log-likelihood log p(x ys):
p(y x) ø class-conditional likelihood of data with label y/
SumProb, result, inherits this weakness theoretically and performs worse than SumEnergy.
Qualitative case study Lastly, to provide further insights on our method, we qualitatively examine examples from the multi-label classification dataset PASCAL-VOR (in-dist.) and OOD imput from the ImageNet that are correctly classified by SumEnergy but not MaxLogit. In Figure (left), we see an in-distribution example is labeled dog, car, chair r and person, with MaxL.ogit score 1.63 and SumEnergy score 3.23. We also show an OOD input (Figure 3 right) with a single dominant activation on the bi rd class, with MaxLogit score 2.14 and SumEnergy score 2.19. In this example, taking the sum appropriately resulted in a higher score for the in-distribution image than the OOD image. Contrarily, MaxLogit's score for the in-distribution image was lower than that of the OOD image, which results in ineffective detection.
4 RELATED WORK
Multi-label classification The task of identifying multiple classes within an input example is of significant interest many applications (Tsoumakas & Katakis 2007). with deep neural networks being commonly used as the classifier. Natural images usually contain several objects and may have many associated tags (Wang et al. 2016). (Gong et al.) 2013) use convolutional neural networks (CNN) to annotate images with 3 tags the
dataset. (Chen et al. 2019a) CNNs to tag images of road scenes from 52 possible labels. In the medical domain, (Wang et al.) 2017) present chest X-ray dataset in which one image may contain multiple abnormalities. Multi- label classification is also prominent in natural language processing (Nam et al. (2014). Recent work also provides a theoretical analysis of multi-label classification under various measures (Wu & Zhu, 2020 Our proposed method is therefore relevant to a wide range of applications in the real world.
Out-of didtr--distiitttion uncertainty for pre-trained models The softmax confidence score has be- come a common baseline for OOD detection (Hendrycks & Gimpel (016)) A theoretical investiga- tion (Hein al. 2019) shows that neural networks with ReI.U activation can produce arbitrarily high softmax confidence for OOD inputs. Several works attempt to improve the OOD uncertainty esti- mation by using deep ensembles (Lakshminarayanan
al. (017)) ODIN score (Liang et al. 2018)) Mahalanobis distance-based confidence score (Lee et al. 2018), and generalized ODIN score (Hsu ct al. 2020). Vrees & Taylor 2018) propose to learn the confidence score by using an auxiliary branch to derive the OOD score. Recent work using energy score demonstrated ofate-of-the-an art per- formance on OOD detection tasks (Liu et al. 2020). However, previous methods primarily focused on multi-class classification networks. In contrast, in work, we propose parameter-fie mea-
8

---

Under review as a conference paper at ICLR 2021
surement that allows effective OOD detection in the underexplored multi-label setting, where the information from across various labels are combined in a theoretically meaningful manner.
Out-of-distribution detection with model fine-tuning While work primarily focused on OOD detection for pre-trained neural networks, a parallel line of research has also explored using auxil- iary outlier OOD data to help the OOD detector generalize better. Auxiliary data allows the model to be explicitly regularized through fine-tuning, producing lower confidence on anomalous exam- ples (Bevandić et al., 2018 Geifman & El-Yaniv 2019; Malinin & Gales) 2018: Mohseni et al. 2020 Subramanya et al. 7). A loss function is used to force the predictive distribution of OOD samples toward a uniform distribution (Lee et al. 2017). Recently, (Mohseni et al. 2020) explore training by adding a background class for an OOD score. (Chen et al., (2020) propose using hard outlier mining which improves the OOD detection performance on both clean and perturbed natural images. However, existing works have primarily focused on the multi-class classification setting. We leave the fine-tuning aspect with auxiliary data for multi-label classification as future exploration.
Detection. Generative models (Dinh et al.) 2016 Kingma Welling 2013; Rezende et al.) 2014, Van den Oord et al.) 2016 Tabak & Turner, 2013) can be alternative approaches for detecting OOD examples, as they directly estimate the in-distribution density and can declare a test sample to be density regions. However, as shown by (Nalisnick al. 2018), deep generative models can assign high likelihood to distribution detection using improved metries (Choi & Jang, 2018). including likelihood ratio (Ren) iet al. 2019| Serrà et al. 2019). Though our work is based on discriminative classification models, we show that label-wise energy scores can be theoretically interpreted from data density perspec- tive. More importantly, generative based models (Hinz et al. 2019) can be prohibitively challenging to train and optimize, especially on large and complex multi-label datasets that we considered (e.g., MS-COCO, SUS-WIDE etc). In contrast, our method relies on discriminative multi-labe classi- fier, which can be much easier to optimize using standard SGD.
Generative Modeling Based
if it lies in the low-
data. Deep generative models can be more effective for out-of-
Energy-based learning Energy-based machine learning models date back to Boltzmann ma- chines (Ackley et al. 1985] (Salakhutdinov & Larochelle, 2010). Energy-based learning (LeCun) et al. 2006; Ranzato et al., 2007a by provides a unified framework for many probabilistic and non- probabilistic approaches to learning. Recent work (Zhao et al.) 2019) also demonstrated using energy functions to train GANs (Goodfellow et al., 2014). where the discriminator uses energy values to differentiate between real and generated images. Xie et al. (2016) first showed that a discriminative classifier can be interpreted from energy-based perspective. Subsequent works (Xie et al. 2017 2019 2018b a) explored video generation, 3D shape pattern generation, and text generation ( Deng at. 2019) through EBMs. Energy- -based methods are also used in structure prediction (Belanger & McCallum 2016 Tu & Gimpel, 2018). Grathwohl et al. (2019) showed that discriminative classi- fier can be interpreted from an energy-basee perspective. The proposed JEM optimization objective estimates the joint distribution p(x, y) from generative perspective, which requires estimating the normalized densities and can be intractable and unstable to compute. Liu et al. (2020) propose to use energy score for OOD detection that is derived from pure discriminatively trained classifier, which demonstrated superior performance for multi-class classification networks. In contrast, our work focuses on a multi-label setting, where we contribute aggregation methods that utilize information jointly from across all labels.
5 CONCLUSION
In this work, we propose energy scores for OOD detection in the multi-label classification setting. We show that aggregating energies over all labels into SumEnergy results in better discrimination between in-distribution and OOD inputs compared to using information from only one label's 's in- formation (i.e. MaxLogit, MSP, ODIN, or Mahalanobis). Additionally, we justify the mathematical interpretation of SumEnergy from a joint likelihood perspective. SumEnergy obtains better OOD de- tection performance compared competitive baseline methods, establishing new state-of-the-an on this task. Applications of multi-labe, classification can benefit from our methods, and we anticipate further research in OOD detection to extend this work. We hope our work will increase the attention toward a broader view of OOD uncertainty estimation from an bbsed perspective.

---

Under review as a conference paper at ICLR 2021
REFERENCES
David H Ackley, Geoffrey E. Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147-169 1985.
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning, pp. 983-992, 2016.
Petra Bevandic, Ivan Krešo, Marin Oršić, and Siniša Segvić. Discriminative out-of-distribution detection for semantic segmentation. arXiv preprint arXiv: 1808.07703, 2018.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference Management of data, pp. 93-104,7 2000.
Tianshi Cao, Chinwei Huang, David Yu-Tung Hui, and Joseph Paul Cohen. A benchmark of medical out of distribution detection. arXiv preprini arXiv:2007.0422. 2020.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detec- tion via informative outlier mining, arXiv preprint htt::20006.12007. 2020.
Long Chen, Wujing Zhan, Wei Tian, Yuhang He, and Qin Zou. Deep integration: A multi-label architecture for road scene recognition. IEEE Transactions Image Processing, 28(10):4883- 4898, 2019a.
Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrett attentional reinforcement learning for multi-label image recognition. arXiv preprint arXiv:1712.07465,00 2017.
Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-speciff graph representation for multi-label image recognition. In Proceedings of the IEEE International Conference Computer Vision, pp. 522-531, 2019b.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5177-5186, 2019c.
Hyunsun Choi and Eric Jang. Generative ensembles for robust anomaly detection. 2018
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng, Nus- wide: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR '0)), Santorini, Greece., 2009.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014a.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613, 2014b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. Ieee, 2009.
Yuntian Deng, Anton Bakhtin, Myle Ott. Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. International Conference on Learning Represen- lutions, 2019.
Terrance DeVries and Graham W Taylor. Learning confidence for duitt---iittituttion detection in neural networks. arXiv preprint 1rXiv:1802.04865 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605 100..08803........ 2016.
10

---

Under review as a conference paper at ICLR 2021
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-136, January 2015.
Yonatan Geifman and Ran EI- Yaniv. Selectivenet: A deep neural network with an integrated reject option. arXiv preprint arXiv:/901.009
2019.
Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey loffe. Deep convolu- tional ranking for multilabel image annotation. arXiv preprint 13 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 645. Springer, 2016.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 13X10116,0.02216,, 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. arXiv preprint 191 2019.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. arXiv preprint arXiv: 190119900.0,6,,. 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-distriuttion data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-4708, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1eXi::141..6980 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.6114, 2013.
arXiv preprint
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--allibraeed classifiers for detecting out-of tit-o---ddtibbttion samples. arXiv preprint 1rXiv:1711.09325, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outtof--iittibutiin samples and adversarial attacks. Advances in Neural Information Process- ing Systems, pp. 7167-7177, 2018.
11

---

Under review as a conference paper at ICLR 2021
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. 2008 Eighth IEEE Interna- tional Conference on Data Mining, pp. 413-422. IEEE, 2008.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in Neural Information Processing Systems, 2020.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distributtin detection. 2020.
Eric Nalisnick. Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't know? arXiv preprint arXiv: 18/0.0911 2018
Jinseok Nam, Jungi Kim, Eneldo Loza Mencía, Iryna Gurevych, and Johannes Fürnkranz. Large- scale multi-label text dassification-reviittinng neural networks. In Joint european conference on machine learning and knowledge discovery in databases, pp. 437-452. Springer, 2014.
Mare' Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann L. Cun. Efficient learning of sparse representations with an energy-based -aaed model. Advances in neural information processing systems, pp. 1137-1144, 2007a.
Mare' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based based framework for unsupervised learning. In Artificial Intelligence and Statistics, pp. 3713399,,9 2007b.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for -fiit--distiitttio detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp- 693-700, 2010.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. In- put complexity and out-of-distributtio detection with likelihood-based generative models. arXiv preprint arXiv: 1480, 2019.
Akshayvarun Subramanya, Suraj Srinivas, and R Venkatesh Babu. Confidence estimation in deep neural networks via density modelling. arXiv preprint arXiv:1707.07013 2017.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
Grigorios Tsoumakas and Ioannis Katakis. Multi-label classification: An overview. International
Journal of Data Warehousing and Mining (LJDWM), 3(3):1-13, 2007.
Lifu Tu and Kevin Gimpel. Learning approximate inference networks for structured prediction. In International Conference Learning Representations, 2018.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pp. 4790-4798. 2016.
12

---

Under review as a conference paper at ICLR 2021
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285- 2016.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Guoqiang Wu and Jun Zhu, Multi-label classification: do hamming loss and subset accuracy really conflict with each other? Advances in Neural Information Processing Systems, 33, 2020.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. (635-2644, 2016.
Jianwen Xie. Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 7093-7101, 2017.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE transactions pattern analysis and machine intelligence, 42(1):27-45, 2018a.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 8629-8638, 2018b.
Jianwen Xie, Cang-Chun Zhu, and Ying Nian Wu. Learning based spatial-temporal gener- ative convnets for dynamic patterns. IEEE transactions on pattern analysis and machine intelli- gence, 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy- based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, 2019.
Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial reg- ularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5513-5522, 2017.
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 EVALUATION ON DIFFERENT ARCHITECTURE
We provide additional evaluation results for ResNet (He et al. 2016). The classifiers have ResNet- 101 backbone architecture, but with a final layer that is replaced by fully connected layers. Each classifier is pre-trained ImageNet- I and then fine-tuned with the logistic sigmoid function to its corresponding multi-label dataset. We use the same training settings as in the main paper. After training, the mAP 87.73% for PASCAL-VOC, 72.77% for MS-COCO, and 61.47% for NUS- WIDE.
In Table 3 show the performance comparison of various OOD) detection approaches, evaluated on ImageNet as the OOD test set. The ablation of applying summation over baseline methods is provided in T
Dm
MS-COCO
PASCAL-VOR VOC
FPR95 / AUROC/ AUPR t
OOD Score
Aggregation
1
Logit
Max
34.54/ /0////9.9///// /94.30
36.32/ /9.04/ 82.68
58.05
83,07 94.21
Prob
Max
34.54 / 90.93 94.30
36.32 / 82.66
58.05 83.07 94.21
MSP
Max
77.92 / 72.43 84.34
67.993
0.75559.199 86.40
ODIN
Max
34.58 90.226 93.69
36.32/91.00
82.68
58.05 0.05//83.00 83.07 94.21
Mahalanobis
Max
94.04 / 49.49 70.71
78.02 / 70.93 59.84
61.33 / 83.75 95.15
LOF
74.30 / 74.87 85.82
76.71 67.54 55.35
85.42/ 69.37 90.36
Isolation Forest
99.06 37.59 63.43
98.64 41.94 33.50
96.59 50.75 82.91
Energy (ours)
Max
34.54 / 90.93 94.30
36.32 91. 004 82.68
58.05 83.07 94.21
Sum
31.51 / 92.68 96.15
31.96 92.32/ 86.87
50.25 88.12 96.34
Table 3: OOD detection performance comparison using energy based approaches vs. competitive baselines. use ResNet (He et al. 20166) train an-distribution datasets. We use subset ImageNet classes as OOD test data, as described [.] All values are percentages. f indicates larger values are better, and 4 indicates smaller values are better. Bold numbers are superior results.
D
MS-COCO
PASCAL FPR95 / AUROC/ AUPR
NUS- WIDE
OOD Score
Aggregation
Logit
Sum
95.63 53.52 73.25
96.36/ 49.44443.07
96.49 / 49.83
81.78
Prob
Sum
43.69 / 87.21 / 93.14
35.97 / 84.68 76.61
55.86 / 82.97
94.92
ODIN
Sum
43.69 87.21
/9.144
53.77 7 67.15
55.24
81.84
94.59
Mahalanobis
Sum
94.47 46.82 / 67.06
78.56 70.84 / 59.34
62.79
83.19
94.96
LOF
Sum
N/A
N/A
N/A
Isolation Forest
Sum
N/A
N/A
N/A
Energy
Sum
31.51 / 92.68 / 96.15
31.96 92.32 86.87
50.25 / 88.12 96.34
Table 4: Ablation study the effect of aggregation methods for prior approaches. We use ResNet (He al.
2016) to train on the in-distribution datasets. We use ImageNet as OOD test data as described Scction Note that Ston is not applicable to tree-based or KNN-based approaches (c.g., LOF and Isolation Forest).
A.2 EVALUATION ON DIFFERENT OOD TEST DATA
In addition to ImageNet, we also evaluate on a different OOD test dataset, Textures (Cimpoi et al., 2014a). The results are reported in 5 and Table
A.3 BASELINE METHODS
In multi-label classification, the prediction for each label y with E 21, K} is independently made by a binary logistic classifier:
exv
p(y: x)
(x)
14

---

Under review as a conference paper at ICLR 2021
Dn
MS-COCO
PASCAL FPR95 AUROC/ AUPR 1 1
NUS-WIDE
OOD Score
A Aggregation
Logit
Max
14.63 / 96.10/ 99.32
12.36 96.22 96.97
38.46 87.42 19.19
Prob
Max
14.63/96.100 99.32
12.36 / 96.22 96.97
00.487.42 97.19
MSP
Max
60.82 / 83..0/997..0 /9..05
41.81 / 89.76 93.00
83.09/ 992.48
ODIN
Max
12.22 / 96.18 99.229
12.36 96.22 96.97
38.46/87.44
197.19
Mahalanobis
Max
44.61 85.71 997.41
19.17 96.23 97.90
36.19/91.36 98.52
LOF
70.16 / 74..3 94,96
89.49 / 60.37 / 76.70
64.27 / 78.23 95.94
Isolation Forest
95.55 53.21 90.45
99.39 / 20.89 50.11
93.07 / 51.01 89.17
Energy
Max
14.63 / 99,16, 10/10/// 99.32
12.36 / 96.22 / 96.97
38.46 / 87.42 .242 97.19
(ours)
Sum
12.82 / 96.84 99.54
10.87 / 96.78 / 97.87
31,68 / 92.43 98.65
Table 5: Texture as data. use ResNet (ee al. 2016 train on in-distribution datasets. values are percentages. 1 indicates larger values are better, and numbers are superior results.
indicates smaller values are better. Bold
Dmn
MS-COCO
PASCAL
NUS-WIDE
FPR95 AUROC/ AUPR
OOD Score
Aggregation
Logit
Sum
95.63 73.25
96.36 49.44/944444 43.07
92.38
52.72 89.21
Prob
Sum
43.69 87.21 93.14
35.97
84.68 76.61
34.88
90.76 98.57
ODIN
Sum
43.69 87.21 / 93.14
53.77 / 74.50
67.15
35.27
/ 89.36 98.31
Mahalanobis
Sum
45.62 / 84.34/97.00 /9700
19.45 / 96.09 99.80
37.55 /91.0 99..47
LOF
Sum
N/A
N/A
N/A
Isolation Forest
Sum
N/A
N/A
N/A
Energy
Sum
12.82 / 96.84 99.54
10.87 / 96.78 97.87
31.68 92.43 / 98.65
Table 6: Ablation study on the effect of aggregation methods for prior approaches. We use ResNet (He et al. 2016) to train the in-distribution datasets. We use Texture (Cimpoi et al. (2014a) as OOD test data as described in Section 3.1 Note that Sum is not applicable tree-based KNN based approaches (c.g., LOF
and Isolation Forest).
We consider the following baselines methods under maximun aggregation:
MaxLogit = max fy. (x)
(18)
e (x)
MaxProb max ply: |x) max
(19)
1 (x)
efv((x)
MSP max
(20)
Σ K elag((x)
edv, (8)/T
ODIN = max
(21)
+ clv. (8)/T
Mahalanobis max (o(x) p
(by.))
(22)
In particular, ODIN was originally designed for multi-class but we adapt for the multi-label case by taking the maximum of calibrated label-wise predictions. The input perturbation is calculated using x <sign(-Veg.) where Eg. the binary cross-entropy loss for the label is with the largest output, i.e., h. = arg max; p(y, embedding ((x) for given sample. Py. is the class conditional mean for label y/, and Đ-1 is the covariant matrix.
x). For Mahalanobis distance, we extract the feature
A.4 VALIDATION DATA FOR BASELINES
We use a combination of the following validation datasets to select hyperparameters for ODIN [Linng et al. 2018) and Mahalanobis (Lee et al. 2018). The validation set consists of:
15

---

Under review as a conference paper at ICLR 2021
Gaussian noise sampled iidd. from an isotropic Gaussian distribution;
uniform noise where each pixel is sampled from U = [],,
In-distribution data corrupted into OOD data by applying (1) pixel-wise arithmetic mean of random pair of in-distribution images; (2) geometrie mean of random pair of in-distribution images; and (3) randomly permuting 16 equally sized patches of an in-distribution image.
A.5
HYPERPARAMETER TUNING FOR BASELINES
ODIN Liang et al. 2018) and Mahalanobis (Lee et al. 2018) require hyper-parameter tuning, such as lemperature and magnitude of noise e. We use the validation data above for selecting the optimal hyperparameters. For ODIN, temperature magnitude is chosen from 21 evenly spaced numbers starting from 0 and ending at 0.004. For Mahalanobis, the perturbation magnitude e chosen from [0, 0.0005, 0.0014, 0.001, 0.002, 0.005). The optimal parameters chosen minimize the FPR at TPR95 the validation
chosen from 1,10,100,1000 and the perturbation
16


Reviews:
[{"review_id": "wqUMaHy89P6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the out-of-distribution detection task in the multi-label classification (MLC) setting. It proposes an energy-based method called SumEnergy, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. Experimental results illustrate its superiority compared with other baselines.\n\n\n##############################################################################################\npros:\n1. Overall, this paper is well-written and organized.\n2. The proposed method achieves promising experimental results.\n\n##############################################################################################\ncons:\n1. The explanation for why SumEnergy works is confusing. Specifically,  in Eq.(13), authors explain $E_{sum}(\\mathcal{x})$ is linearly aligned with the log of joint likelihood, which is confusing, because $p(\\mathcal{x}, \\mathcal{y}) = p(\\mathcal{x}, y_1) p( y_2 \\| \\mathcal{x}, y_1)  ...  p( y_K \\| \\mathcal{x}, y_1, ... , y_{K-1})$ while Eq.(13) has $\\prod_{j=1}^{K} p(\\mathcal{x}, y_j)$. (For clarity, I have changed some notations.) \n2. What is the effect of the hyperparameter $\\tau$ for these two methods (i.e. MaxEnergy and SumEnergy)? Please give more comparisons and clarify how to tune $\\tau$.\n3. The novelty is incremental. Although authors highlight the non-triviality to extend from the multi-class setting (Liu et al., 2020) to multi-label setting lies in leveraging information between different labels,  the proposed method SumEnergy may not reflect this point due to 1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ir96jTCaFL9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "======== original feedback ===================\n\nReview: This paper studies out-of-distribution (OOD) detection for multi-label classification with energy-based models. Specifically, the paper proposes to use SumEnergy, which aggregates energy scores from multiple labels, to estimates the OOD. Empirical studies are performed to validate the proposed framework on several benchmarks.\n\nStrength: \n+ The paper studies out-of-distribution detection for multi-label classification by using an energy-based framework, which is a real and practical problem. \n+ The paper provides comprehensive experiments to show the effectiveness of the proposed energy-based framework for out-of-distribution detection with multi-label classification. The method also establishes the state-of-the-art performance, which is good.\n\nConcerns:\n+ since the paper for energy-based out-of-distribution detection has existed. Generalizing it to multi-label context is incremental. \n+ missing some important references:  the current paper didn’t discuss and cite [1], which is the first paper to show that a discriminative classifier can be interpreted from an energy-based perspective. \n+ the related work about EBM is not comprehensive. Even though the current paper is about discriminative EBM, a discussion about the development of the generative EBMs is desirable. \n\n[1] A Theory of Generative ConvNet (ICML 2016)\n\n======== score changed ==============\n\nMy major concern has been addressed by the reply from the authors.  The revised paper has been improved.   \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FDatSD9S3N0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: In this work, SumEnergy is proposed for out-of-distribution detection for multi-label classification. According to the results of the experiment, SumEnergy performs better than MaxEnergy in several datasets of multi-label classification. \n \n+ves: \n1. This paper is written well. The main part is clear.  With the mathematical equation, it shows that SumEnergy is linearly aligned with the log of joint likelihood. The aggregated include more likelihood information over all labels. \n\n2. The results in Tables 1 and 2 show a clear comparison with other baselines. It also indicates that SumEnergy performs best among all methods. The analysis and Qualitative case study are convincing. \n\nConcerns\n1. Most baselines are designed by the authors. Unfortunately, there are no other baselines that are done by other papers. The contribution of this work could be a little questionable.  \n\n2. The SumEnergy seems to has a minor difference with Sum Prob.  Maybe more reason on why still use energy score could be discussed. \n\nQuestions during the rebuttal period: \n\n\nSome related work on energy-based learning could be mentioned.:\n \n[1] Structured Prediction Energy Networks, ICML 2016\n\n[2] Learning Approximate Inference Networks for Structured Prediction. ICLR 2018\n\n[3] Residual Energy-Based Models for Text Generation. ICLR 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3bLBcp0nXV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "####################\n\nSummary: \n\nThe paper proposes a SumEnergy method to estimate the out-of-distribution indicator scores for multi-label classification.\n\n####################\n\nReason for score:\n\nOverall, the paper is above the borderline. I like the idea of utilizing SumEnergy operation to address the out-of-distribution problem in the task of multi-label classification. My major concern is about some unclear parts in the paper and insufficient experimental comparison (see cons below). Hopefully, it would be grateful that the authors could address my concerns during the rebuttal period. \n\n\n####################\n\nPros:\n\n(1) The motivation of the paper, i.e., out-of-distribution detection in multi-label classification is very important and deserves research further.\n\n(2) Extensive experimental results demonstrate that the SumEnergy based on the aggregation over label-wise energy scores achieves better performance than that of MaxEnergy.\n\n(3) The comparative analysis and ablation study in the paper are convinced and detailed, which can help better understand the applicability in the multi-label setting. \n\n####################\n\nCons:\n\n(1) Even though the experimental results demonstrate that the proposed SumEnergy outperforms MaxEnergy and MaxLogit, it will be better to conduct additional comparisons with other methods that also consider the information of all the labels, such as attention-based methods [1][2] and graph-based [3][4] multi-label classification approaches.\n\n[1] Recurrent attentional reinforcement learning for multi-label image recognition. AAAI 2018\n[2] Decoupling category-wise independence and relevance with self-attention for multi-label image classification. Arxiv 2019\n[3] Learning semantic-specific graph representation for multi-label image recognition. Arxiv 2019\n[4] Multi-label image recognition with graph convolutional networks. CVPR 2019\n\n(2) According to the description in the paper, the OOD problem is similar to the few-shot or zero-shot issue. Could the authors explain more about the difference between OOD and few-shot/zero-shot learning, please? Moreover, more experimental details and results on few-shot/zero-shot problems will be welcome.\n\n(3) How to determine the energy threshold seems very critical to the task, because the scores always various on different dataset and domains with the proposed SumEnergy method. More discussion and the impact for experimental results should be added in the paper.\n\n####################\nQuestions during the rebuttal period: \n\nPlease address and clarify the cons above. Thank you!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to do out-of-distribution (OOD) detection in multi-label classification. However,  the challenges of extending energy-based  OOD methods in multiclass to multi-label setting is not big. This paper just defines the label-wise free energy. The key challenging issues in MLC is the label dependency. The paper did not consider modeling the label dependency and devide it to several binary classification issues. And the paper did not provide the theory gurantee. The paper is far below the bar of top conferences.
Decision: Reject