Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper is concerned with how to determine whether a set of data points are from a given distribution. It uses the so-called typical set to transform the problem to determining whether the data points lie in the typical set of the given distribution. It proposes a statistical test using the empirical distribution of model likelihoods to determine whether inputs lie in the typical set if the considered model. 

The motivation of the work is very clear, and the paper is well organized. The basic idea of using the typical set to check whether given data points are from a given distribution seems sensible, as guaranteed by Theorem 2.1.

My concern is about the performance of the proposed method compared to alternatives. First, a standard approach to the considered problems seems to be the two-sample tests (or its approximations or variations), so it would be desirable to compare the typical set-based approach with the two-sample test approaches theoretically. In particular, given that you have to allow some error when using typical sets, what is exactly the advantage of the proposed approach?  Second, according to the empirical results (Section 5), the proposed method does not seem to clearly outperform alternatives such as KS-test. In this case, a better justification of the reliability of the proposed approach would be helpful.

I acknowledge I read the authors' response and other reviews and would like to keep my original rating. (I agree that the t-Test and KS-test were probably first used by the authors, but at the same time it is natural to adopt them; that is why I considered them as baselines.)
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Thanks for the authors for your detailed reviews.

My major concerns about the proposed method are whether "the typicality set" could be faithfully applied in the small data regime. The authors point me to the interesting Figure 4, which shows that it basically achieves converged performance when $m = 50$ or smaller numbers for some problems. I think this experiment is a strong support for the proposed method. 

However, I don't agree that the M=1 Gaussian case acts as a strong support for the method. As I said, for some other wired distribution, it is difficult to interpret what the M=1 Typicality set becomes. 

The authors also clarify the difference between different baselines. 

Overall, I will increase my score to "Weak Accept".

##########################

Recent works have shown that out-of-distribution samples can have higher likelihoods than in-distribution samples for some generative models. To explain this phenomenon and to tackle the problem for OOD detection, this paper adopts "typical sets" for identifying in-distribution samples. Specifically, a "typical set" is a set of examples whose expected log likelihood approximate the model's entropy. For a Gaussian distribution, the paper finds that a single point typical set locates exactly in the \sqrt{d} radius, which is usually favored over the high-likelihood origin. Then the paper uses the "typical set" to perform OOD for a batch of examples. Empirically they demonstrate competitive performance over MNIST and natural image tasks. 

Typical set seems natural for out-of-distribution detection. An important property is that, if one draws a large number of independent samples from the distribution, it is very likely that these samples belong to the typical set (basically Theorem 2.1).  However,  for small n, this property doesn't hold anymore, which leaves here a questionmark whether "Typical set" can be used for OOD detection in small n regime. As the author argues, for Gaussian distribution when n=1 the typical locations are those \sqrt{d} radius points. But this doesn't justify the "Typical set". If the distribution is some non-Gaussian wired distribution, the typical locations doesn't seem to make sense at all.

Following the previous argument above, the Typical set method requires to perform OOD for a batch of examples. In contrast, the Annulus method can be directly applied to one single test example. 

Empirically, the Typically set doesn't demonstrate obvious advantages compared to the baselines. For both MNIST and natural image tasks, it seems that all methods behave similarly. For comparing such big tables, I would recommend adding a column showing the average ranks among all methods. Beyond that, standard OOD tasks usually evaluate methods using AUROC and AUPR (Hendrycks and Gimpel, 2017). Is it possible to also include such metrics ?

Theorem 2.1 is confusing. It is beneficial to define what P is, and verbally state what the theorem conveys. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I've read the authors' rebuttal and other reviews; I'd like to keep my score as is. My main concerns are the novelty of the work, the theoretical soundness of the method for small data settings and its robustness in settings with model misspecification. 

#################################

The paper proposes a new approach based on the notion of typical set in probability and tackles the challenging problem of detecting OOD using deep generative models. The main claim of the paper is that assigning high likelihood to OOD samples in DGMs is due to the mismatch between model’s typical set and its high probability density areas. 

I liked the idea of proposing a hypothesis testing approach for finding OOD samples generated from a model; however, my main concern is that the approach has some major practical limitation that the authors have also rightly mentioned in their discussion. It seems that even with a hypothesis testing tool for OOD detection, the model capacity and other properties of the model are more fundamental and critical for OOD detection in DGMs. In other words, how this tool can be useful in practice if the models are misspecified and how robust is the tool with respect to model properties. This major limitation has not been addressed in the experiments. 

This paper, does a good job in finding the OOD data points if the likelihood histograms do not overlap using the typicality notion. However, this idea had already been proposed and explored in Choi. et al. 2019 (although for a flow-based model). This makes the technical novelty of the work less significant. 

Overall, I think the paper needs some improvement in terms of discussing the robustness of the test with respect to model properties; otherwise, it is just another typicality set explanation of why DGMs may produce high likelihood values for OOD samples which has already been mentioned in previous work. 
review_title: Official Blind Review #2

Meta-review: This paper tackles the problem of detecting out of distribution (OoD) samples. To this end, the authors propose a new approach based on typical sets, i.e. sets of samples whose expected log likelihood approximate the model's entropy. The idea is then to rely on statistical testing using the empirical distribution of model likelihoods in order to determine whether samples lie in the typical set of the considered model. Experiments are provided where the proposed approach show competitive performance on MNIST and natural image tasks.

This work has major drawbacks: novelty, theoretical soundness, and robustness in settings with model misspecification. Using the typicality notion has already been explored in Choi. et al. 2019 (for flow-based model), which dampers the novelty of this work. The conditions under which the typicality notion can be used are also not clear, e.g. in the small data regime. Finally, the current experiments are lacking a characterization of robustness to model misspecification. Given these limitations, I recommend to reject this paper.

Decision: Reject