Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents an algorithm two learn both classifier and out-of-distribution sample detector. Instead of learning softmax weights, the proposed approach learns to project the inputs to a latent space, where each class is a Gaussian distribution. Out-of-distribution samples can be detected by the distance between the learnt representation and centers. The proposed approach can be viewed as generalization of Gaussian discriminant analysis and one-class classification. The proposed approach is technically sound, and the experiments do show some improvement over previous algorithm on out-of-distribution detection, especially on tabular datasets. However I think there are some weaknesses of this paper

* The novelty is a little thin. The proposed algorithm is based on just a modification of the learning objective, and there are no theoretical analysis of why the proposed approach can work better.
* Experimental result is somewhat weak. Improvement on the image datasets seems marginal, especially on the SVHN dataset. I also doubt if classifying Cifar10 against TinyImageNet or LSUN challenging enough, because these datasets are fairly different. I am interested in whether the proposed approach can detect novel classes, such as training using only 9 of 10 classes on Cifar10.

Another question: does learning classifier as well as centers need additional optimization techniques, like special initialization?

Update
======= 

After a careful read of the Mahabolis baseline (Lee et al., 2018) I agree with the authors that this paper has some novelty comparing with previous works, i.e., directly learning a generative classifier instead of converting a discrimitively trained classifier into generative. Combined with the good results obtained. I will raise my score to a weak accept (though without a strong belief).
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a metric learning-based generative model for detecting the out-of-distribution examples.  A new objective function is proposed to model class-dependent class-distribution into a Gaussian analysis models. For the proposed objective, the illustration of derived KL divergence under the Gaussian discriminative analysis assumption is well done.  The empirical results conclude the superiority of the proposed loss function in both tabular and image datasets, when comparing the plain network and one with a softmax.

This study aims is to detect out-of-distribution samples for better generalization. However, the related works need to be revised and present the novelty of the work compared to some metric and distance-based learning algorithms. For example, the proposed idea is similar to adding a regularization term to the prototypical network with Euclidean distance (Snell et al. 2016). This aspect is not very well explained.

Another issue is the lack of comparison with state-of-the-art approaches. The Related Work section (Sec. 2) show a baseline (plain) and another one based on softmax. Experimental comparison with state-of-the-art will help to position this work.

** Update ** I read the authors comments and other reviews. Although some clarification were useful, I still maintain my rating of "weak reject", I don't get much excitment and I am not feeling there is something great with this work.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary:

Unlike the softmax classifier, the authors considered the generative classifier based on Gaussian discriminative analysis and showed that such deep generative classifiers can be useful for detecting out-of-distribution samples. For various benchmark tasks, the proposed method outperforms baselines based on the softmax classifier.  

Detailed comments:

The novelty of this paper is not significant due to the following reasons:

1. The main message (i.e. the concept of the deep generative classifier can be useful for detecting out-of-distribution samples) is not really new because it has been explored before [Lee' 18]. Even though this paper considers training a deep generative classifier directly unlike [Lee' 18], the proposed method looks like a simple variant of [Lee' 18].

2. Missing baselines for training the deep generative classifier: training the deep generative classifier directly has been studied by [Guerriero' 18] and [Pang' 18] but the authors did not compare the proposed training method with such baselines. Because of that, it is hard to say that contributions from proposing a training method are significant. 

Questions:

1. Could the authors consider a case without an identity covariance assumption? Most training methods for deep generative classifier assumes the identity covariance matrix because optimizing the log determinant is not easy. So, it would be interesting if the authors can handle this issue. 

2. Even though the authors assume the identity covariance matrix, the covariance matrix of pre-trained features can not be an identity matrix. Could the authors report the performance of Mahalanobis detector using the proposed deep generative classifier? 

[Lee' 18] Lee, K., Lee, K., Lee, H. and Shin, J., 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (pp. 7167-7177).

[Guerriero' 18] Samantha Guerriero, Barbara Caputo, Thomas Mensink, DeepNCM: Deep Nearest Class Mean Classifiers, ICLR workshop 2018.

[Pang' 18] Pang, T., Du, C. and Zhu, J.,  Max-mahalanobis linear discriminant analysis networks. In ICML, 2018.
review_title: Official Blind Review #1

Meta-review: The paper presents a training method for deep neural networks to detect out-of-distribution samples under perspective of Gaussian discriminant analysis.

Reviewers and AC agree that some idea is given in the previous work (although it does not focus on training), and additional ideas in the paper are not super novel. Furthermore, experimental results are weak, e.g., comparison with other deep generative classifiers are desirable, as the paper focuses on training such deep models.

Hence, I recommend rejection.
Decision: Reject