Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ======== original feedback ===================

Review: This paper studies out-of-distribution (OOD) detection for multi-label classification with energy-based models. Specifically, the paper proposes to use SumEnergy, which aggregates energy scores from multiple labels, to estimates the OOD. Empirical studies are performed to validate the proposed framework on several benchmarks.

Strength: 
+ The paper studies out-of-distribution detection for multi-label classification by using an energy-based framework, which is a real and practical problem. 
+ The paper provides comprehensive experiments to show the effectiveness of the proposed energy-based framework for out-of-distribution detection with multi-label classification. The method also establishes the state-of-the-art performance, which is good.

Concerns:
+ since the paper for energy-based out-of-distribution detection has existed. Generalizing it to multi-label context is incremental. 
+ missing some important references:  the current paper didn’t discuss and cite [1], which is the first paper to show that a discriminative classifier can be interpreted from an energy-based perspective. 
+ the related work about EBM is not comprehensive. Even though the current paper is about discriminative EBM, a discussion about the development of the generative EBMs is desirable. 

[1] A Theory of Generative ConvNet (ICML 2016)

======== score changed ==============

My major concern has been addressed by the reply from the authors.  The revised paper has been improved.   


review_title: an interesting paper studying an underexplored problem

final_score: rating: 6: Marginally above acceptance threshold
review_content: ####################

Summary: 

The paper proposes a SumEnergy method to estimate the out-of-distribution indicator scores for multi-label classification.

####################

Reason for score:

Overall, the paper is above the borderline. I like the idea of utilizing SumEnergy operation to address the out-of-distribution problem in the task of multi-label classification. My major concern is about some unclear parts in the paper and insufficient experimental comparison (see cons below). Hopefully, it would be grateful that the authors could address my concerns during the rebuttal period. 


####################

Pros:

(1) The motivation of the paper, i.e., out-of-distribution detection in multi-label classification is very important and deserves research further.

(2) Extensive experimental results demonstrate that the SumEnergy based on the aggregation over label-wise energy scores achieves better performance than that of MaxEnergy.

(3) The comparative analysis and ablation study in the paper are convinced and detailed, which can help better understand the applicability in the multi-label setting. 

####################

Cons:

(1) Even though the experimental results demonstrate that the proposed SumEnergy outperforms MaxEnergy and MaxLogit, it will be better to conduct additional comparisons with other methods that also consider the information of all the labels, such as attention-based methods [1][2] and graph-based [3][4] multi-label classification approaches.

[1] Recurrent attentional reinforcement learning for multi-label image recognition. AAAI 2018
[2] Decoupling category-wise independence and relevance with self-attention for multi-label image classification. Arxiv 2019
[3] Learning semantic-specific graph representation for multi-label image recognition. Arxiv 2019
[4] Multi-label image recognition with graph convolutional networks. CVPR 2019

(2) According to the description in the paper, the OOD problem is similar to the few-shot or zero-shot issue. Could the authors explain more about the difference between OOD and few-shot/zero-shot learning, please? Moreover, more experimental details and results on few-shot/zero-shot problems will be welcome.

(3) How to determine the energy threshold seems very critical to the task, because the scores always various on different dataset and domains with the proposed SumEnergy method. More discussion and the impact for experimental results should be added in the paper.

####################
Questions during the rebuttal period: 

Please address and clarify the cons above. Thank you!
review_title: The paper proposes a SumEnergy method to estimate the out-of-distribution indicator scores for multi-label classification.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper deals with the out-of-distribution detection task in the multi-label classification (MLC) setting. It proposes an energy-based method called SumEnergy, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. Experimental results illustrate its superiority compared with other baselines.


##############################################################################################
pros:
1. Overall, this paper is well-written and organized.
2. The proposed method achieves promising experimental results.

##############################################################################################
cons:
1. The explanation for why SumEnergy works is confusing. Specifically,  in Eq.(13), authors explain $E_{sum}(\mathcal{x})$ is linearly aligned with the log of joint likelihood, which is confusing, because $p(\mathcal{x}, \mathcal{y}) = p(\mathcal{x}, y_1) p( y_2 \| \mathcal{x}, y_1)  ...  p( y_K \| \mathcal{x}, y_1, ... , y_{K-1})$ while Eq.(13) has $\prod_{j=1}^{K} p(\mathcal{x}, y_j)$. (For clarity, I have changed some notations.) 
2. What is the effect of the hyperparameter $\tau$ for these two methods (i.e. MaxEnergy and SumEnergy)? Please give more comparisons and clarify how to tune $\tau$.
3. The novelty is incremental. Although authors highlight the non-triviality to extend from the multi-class setting (Liu et al., 2020) to multi-label setting lies in leveraging information between different labels,  the proposed method SumEnergy may not reflect this point due to 1.
review_title: The explanation is confusing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: In this work, SumEnergy is proposed for out-of-distribution detection for multi-label classification. According to the results of the experiment, SumEnergy performs better than MaxEnergy in several datasets of multi-label classification. 
 
+ves: 
1. This paper is written well. The main part is clear.  With the mathematical equation, it shows that SumEnergy is linearly aligned with the log of joint likelihood. The aggregated include more likelihood information over all labels. 

2. The results in Tables 1 and 2 show a clear comparison with other baselines. It also indicates that SumEnergy performs best among all methods. The analysis and Qualitative case study are convincing. 

Concerns
1. Most baselines are designed by the authors. Unfortunately, there are no other baselines that are done by other papers. The contribution of this work could be a little questionable.  

2. The SumEnergy seems to has a minor difference with Sum Prob.  Maybe more reason on why still use energy score could be discussed. 

Questions during the rebuttal period: 


Some related work on energy-based learning could be mentioned.:
 
[1] Structured Prediction Energy Networks, ICML 2016

[2] Learning Approximate Inference Networks for Structured Prediction. ICLR 2018

[3] Residual Energy-Based Models for Text Generation. ICLR 2020

review_title: clear work, however, the contribution could a little limited. 

Meta-review: The paper aims to do out-of-distribution (OOD) detection in multi-label classification. However,  the challenges of extending energy-based  OOD methods in multiclass to multi-label setting is not big. This paper just defines the label-wise free energy. The key challenging issues in MLC is the label dependency. The paper did not consider modeling the label dependency and devide it to several binary classification issues. And the paper did not provide the theory gurantee. The paper is far below the bar of top conferences.
Decision: Reject