Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the out-of-distribution detection problem in the multi-label classification setting. The authors propose a scoring function named JointEnergy to measure the prediction uncertainty of a given instance. In particular, the JointEnergy aggregates the label wise free-engery by summation and a hard thresholding on the value of JointEnergy is used to determine whether a given input is OOD or not. It is shown that the propose scoring function can be decomposed into three terms: joint conditional log-likelihood on all classes, un-conditional log data density and a constant term. Empirical results on three multi-label classification datasets show that the proposed method can achieve lower false positive rate (FPR95) while having higher area under curve scores (AUROC, AUPR).
main_review: The scoring function takes a seemingly trivial format which is the simple summation of the per-label free energy for a given input. However, the authors justify the choice by associate the JointEnergy with the joint conditional log-likelihood and the log data density, both are expected to be high for in-distribution samples. The experiments also shows that the choice is valid. In general, I think this paper proposes a simple but effective method to handle an often overlooked problem and vote for accept. Several minor comments:

(1)  The label-wise free energy form is based on that the predictive probability is given by sigmoid(f(x)). For the experiments, models are fine-tuned with the logistic sigmoid functions. What's the connection between how the model is trained and how the JointEnergy estimator would work? Can the JointEnergy estimator work on models trained with other logit smoothing functions?

(2) The two terms, log data density and joint conditional log-likelihood, in the JointEnergy seems to be scaling linearly with the number of labels K. Therefore the contributions of the two terms will remain roughly the same even with very large K. In the case where K is very large, i.e. extreme multi-label classification, only top-k predictions are given. I wonder if the JointEnergy can be generalized to that case where the logit from k+1 to K are truncated.

(3) It would be helpful to do some ablation study on how the log data density and log-likelihood would perform on their own.
limitations_and_societal_impact: This paper studies the out-of-distribution detection problem in the multi-label classification setting. To my understanding, the work poses no negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies a new task that detects out-of-distribution (OOD) data in the multi-label setting. Inspired by recent progress in multi-class classification problems, the authors proposed a joint energy score to achieve their goal. Specifically, the proposed method sums up the label-wise energy score. Then, it is used as a criterion to determine which samples are out-of-distribution by thresholding. Empirical results demonstrate that the proposed method achieved promising results on COCO, NUSWIDE, and VOC benchmark datasets. 
main_review: Pros:
1. While the out-of-distribution detection task has been widely studied in the multi-class classification context, it could be very tricky in multi-label learning, because there can be combinational objects and some combinations may never occur. 
2. The experimental results are thorough to verify the effectiveness of the proposed method, which I believe will be beneficial to the community.

Cons:
1. The definition of energy score in [1] is an unconditional version. However, the authors explain their criterion through the line of the conditional version of energy. They should refer their definition to other works that give the conditional form of energy, e.g. https://arxiv.org/pdf/2002.11537.pdf.
2. The authors claim that in the case of OOD, each of the conditional likelihoods should be small, hence the product. Actually, we are considering a conditional probability $P(X=x|Y_1=1,Y_2=1,...,Y_k=1)$. I suppose the only property of OOD data is: given an OOD data, $P(y_1=0,...,y_K=0|X=x)$ is far larger than other data. It is not an obvious induction to conclude $P(X=x_1|Y_1=1,Y_2=1,...,Y_k=1)<P(X=x_2|Y_1=1,Y_2=1,...,Y_k=1)$, s.t. $x_1$ is OOD while $x_2$ is in-distribution. Actually, we can find the key in Eq. (8) where the production of per-label posterior is low for in-distribution data. Thus, we needn't further explore the conditional density.

[1] LeCun Y, Chopra S, Hadsell R, et al. A tutorial on energy-based learning[J]. Predicting structured data, 2006, 1(0).
limitations_and_societal_impact: I have mixed thoughts on this paper. On one hand, the proposed joint density score is a simple generalization from the previous work. On the other hand, I believe this work can inspire a series of works and appreciate its contribution to well-defining new research topics and establishing a simple yet effective benchmark.

Moreover, I recommend the authors further polished the mathematical interpretation section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a new score - JointEnergy, as a sum of free energy across labels to detect out of distribution inputs in a multi-label classification setting. It presents the definition of this new score, motivates it theoretically by connecting it to joint likelihood of data, and demonstrates the superior performance of this score over existing alternatives on 3 different datasets. The results are interesting and significant.  
main_review: Originality: The problem of OOD detection in multi-label classification setting is under studied. This paper extends existing ideas in a novel way to achieve superior detection performance. 

Quality: While the proposed new OOD  detection score is simple and straightforward extension of the free energy interpretation of logit scores, the new score is better motivated theoretically, and shows superior performance to existing alternatives. It uses more information in the scores of all labels instead of just the dominant label and hence is better able to detect OOD inputs. Despite its simplicity, the new score has better performance. It is also free of hyper parameters, which makes it easier to implement. The method applies to general multi-label classification and hence should be extensible to tasks beyond vision (although the paper does not demonstrate it). 

Clarity: The paper is written well. The new score is motivated well, is theoretically justified, and experiments are simple and conclusive

Significance: Despite its simplicity, the new score seems to be quite powerful in detecting OOD inputs in multi-label classification, and can become a significant evaluation metric for this purpose in the future.
 
limitations_and_societal_impact: It is not clear how the new score performs against baselines in detecting OOD inputs in other tasks (other than vision - like language, speech/audio, structured datasets etc.). It is also not clear how it performs when training a different model for each label (i.e without sharing a pre-trained representation). The authors can better address these 2 aspects through both theoretical analysis and experiments. 

I do not anticipate any significant negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new method for OOD detection in the multi-label setting, called JointEnergy. 
The authors claim that JointEnergy can be mathematically interpreted from a joint likelihood perspective, and that their model achieves sota results on a variety of benchmarks.
main_review: **Significance**: high - the problem of OOD detection is well studied for multi-class classification problems, but there is little work on OOD detection for multi-label classification problems. 

**Clarity**: low - the authors do not provide any background on energy-based models, which is necessary to follow the theoretical analysis.

**Quality**: The experimental analysis shows that the proposed solution is very effective, however I have some concerns with the theoretical analysis.

1. the authors define the label-wise free energy as: $E_{y_i}({x}) = -\log(1 + \exp^{f_{y_i (x)}})$. However, they do not explain neither  why they defined the energy function in that way nor which properties the function has that make it an energy function. 
This does not allow the reviewers to check whether such definition (and hence all the subsequent calculations) are correct. 
I would advise the authors to look into how [1] defined the energy function in the case considered, as it is very clear and easy to follow. 

2. the authors make the strong assumption of label independence. However, if we assume the label independence (which is almost never verified in practice), then we are basically dealing with a series of binary classification problems. The work would be more complete if this assumption was dropped.

**Novelty**: medium - the paper presents a novel and simple idea that is able to beat sota models on the task of OOD detection. 
However, the strong label independence assumption hinders its novelty. 

[1] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.

**After author response**: the authors have addressed my concerns, thus I change my score from 6 to 7. Congrats!

limitations_and_societal_impact: Yes
review_title: null

Meta-review: I recommend to accept this paper

In this paper, the authors proposed a novel method to address an important yet under explored problem: out-of-distribution detection for multi-label classification models. This paper is well-written, addressing important problem, proposing simple-but-effective method. All the reviewers are inclined to accept this paper. I would suggest the authors to add additional experimental results done in the rebuttal phase and take the reviewers' suggestions into the camera-ready version.
Decision: Accept (Poster)