Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper applies a goodness of fit test (KS test) to the latent space of normalizing flows for purposes of out-of-distribution detection.  To combat high-dimensionality and model misspecification, extensions such as random projections and two-sample tests are also proposed.  The results report AUROC on RNVP (batch size 5 and 10), comparing against a typicality test and a kl-based test.  The paper also tests using autoencoders vs rand projections, alternative divergences, effects of model misspecification, and detection in latent vs original feature space.  The findings are that the KS-test is indeed a practical choice for OOD detection via GOF testing.  Moreover, it is observed that better models don't necessarily mean better OOD detection, as described earlier by Zhang et al. [ICML 2021].
main_review: ## Pros

*Exploration of OOD in practice*:  While GOF-based OOD detection is theoretically well-founded, applying the technique successfully in practice for high-dimensional data is a challenge in its own right.  Thus, this paper provides value to practitioners, guiding them on how to make these crucial implementation choices.

*Ablations*: The paper also does a nice job of testing the efficacy of specific parts of the experimental pipeline---for example, autoencoders vs random projections, GOF vs two-sample tests.

## Cons

*Novel only in implementation*:  The method's novelty is restricted to the implementation details.  Ultimately, the core methodology is to apply the KS test to a reparameterized distribution, which is not an original contribution.  Applying a KS test to flows for OOD purposes is novel, to the best of my knowledge.  Moreover, the paper's other insights such as model quality vs OOD ability mostly back already supported points from earlier work (e.g. [Zhang et al, ICML 2021].
summary_of_the_review: The paper is fairly well executed; however, I can't say that I learned much from the work as its contribution is mostly in the implementation details.  And it is unclear to what extent these details generalize beyond standard image benchmarks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an interesting approach for OOD detection problem for flow models. The approach is based on random projections on the real line where KD statistical test is applied in order to compare two distributions. Two variants of the approach are considered 1 sample test where the comparison is made with respect to flow prior and 2 sample test where the comparison is made with respect to transformed samples from two datasets. The quality of the approach is compared agains standard reference methods. 

main_review: Strengths

The proposed approach is simple and does not require training additional components. It can be applied both in data space and latent of the flow, but according to experimental studies application of latent space gives better results. The results are comparable to the results obtained to the reference approaches - the significant gain is observed on CIFAR10 vs SVHN setting. The paper is easy to follow and the idea of the paper is clear. The proposed approach is easy to applied so the results are easy to reproduce.

Weaknesses 

The contribution of this work is smart but novelty is a bit limited due to the fact that the authors combine two known approaches - KS statistical test and random projections. It seems that this framework is general and can be even applied to the bottleneck models like VAE or even for any low dimensional features extracted from the data. The question is can we apply this framework to any low dimensional representation of data or we need the invertible mapping that does not decrease the dimension and direct access to the likelihood. 

The second question is about Carleman’s condition. Considering the case where we use flow’s latent is it sufficient that it would be satisfied by Gaussian (flow latent distribution) or it should be satisfied by data distribution (the normalised Gaussian by determinant of the Jacobian)?

The third issue is about the results. The model seems to be comparable to KLOD besides the CIFAR10 vs SVHN where it performs better. On the other hand, it has problems with CIFAR vs LeSUN setting. Do you have some intuition what is the reason and how to deal with that issue? Most of the experiments that examine the OOD problem are performed between different datasets. It would be beneficial to see some analysis for inside dataset split, where the split is indicated by two groups of classes. This case is somehow similar to Cifar10 vs. Cifar100 comparison. 
summary_of_the_review: I admire the simplicity of the approach and I tent to recommend to accept this work at this stage.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript addresses groupwise outlier detection with normalizing flows. The main idea is to project latent representations of the input batch onto several random directions and to evaluate the resulting 1D distributions according to the Kolmogorov-Smirnov test either with respect to the prior (GOD1KS) or with respect to the corresponding distribution of the training data (GOD2KS). Finally, OOD ranking is performed according to the average KS value across all random directions. GOD1KS is used when the underlying normalizing flow is well learned, while GOD2KS is an option when there is some uncertainty regarding the learning outcome. Experimental performance decreases  when the method is used in combination with undertrained normalizing flow or a normalizing flow with a larger capacity.
main_review: Strengths:

S1. The proposed approach is interesting, simple, mathematically sound and effective. In comparison with Zhang et al. (2020), the proposed approach does not require estimation of a multivariate distribution.

S2. Experiments show robust overall performance. Notably, there are less failure cases than in related methods.

Weaknesses:

W1: Authors does not propose a compelling motivation for groupwise OOD detection.

W2: The evaluation should also show the OOD AP metric as in Zhang et al. (2020). 

Suggestions
- Can you offer an intuitive explanation for the failures of TyTest and KLOD in Table 1?
- It may be interesting to evaluate the average density across the input batch as a straight-forward baseline. I do not expect this to be competitive, but averaging may improve over the pointwise result.
- Can you provide any insight why the method can not exploit normalizing flows with large capacity?
- Include algorithmic presentation of GOD2KS in the supplement (for completeness).
- Page 2: absolute value of the Jacobian -> absolute value of the determinant of the Jacobian.
- Figure 1, caption: explain S_0 and S_1.
- Page 3, "in order to evade the curse of dimensionality": this becomes especially relevant in contemporary normalizing flows (eg. VFlow, DenseFlow) where latent dimensionality is greater than input dimensionality.
- Page 3, "is distinct from the distributions": make clear that this refers only to Fig1(b).
- Figure 2, caption: does not follow -> follow.

summary_of_the_review: This paper provides interesting insights about normalizing flows and OOD detection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes evaluates using statistical tests on random 1d-projections in the latent space of a flow model for groupwise out-of-distribution (OOD) detection. Concretely, Komolgorov-Smirnov tests are used to compare latent encodings of a given batch of samples to the expected distribution of the in-distribution latent encodings. For the expected distribution, either the predefined latent prior is used, resulting in a 1-sample Komolgorov-Smirnov test, or the empirical distribution of the latent encodings of the training data  is used, resulting in a 2-sample Komolgorov-Smirnov test. The paper reports similar groupwise OOD detection performance as existing methods, with the 2-sample KS-test method having less failure cases.
main_review: I found it easy to understand the ideas and methods of the paper, results are also well-understandable.

Overall, the paper does not bring too many surprises from my view - I would have expected any use of 1d projections for OOD detection to work better in latent space than in input space. Also it seems not too surprising that this very generic method cannot result in very high OOD detection performance (compared to methods exploiting more assumptions about OOD data). But in any case good to see precisely how well OOD detection works in this way. The main surprising part might be that generation quality can be negatively correlated with OOD detection performance.

The manuscript would benefit from discussing other OOD detection approaches in latent space, https://arxiv.org/abs/2006.10848 also present one method using parts of the latent space of the Glow model, https://arxiv.org/abs/2102.08248 uses VAEs, not flows, but also look at parts of the latent space there. In the future, could be interesting to compare to such methods (using parts of latent space/log-likelihood contributions), which exploit more assumptions about where semantic differences are encoded in the latent space, on both semantic and non-semantic OOD detection. Note also that https://arxiv.org/abs/2006.10848 report a related result to the result that generation quality and OOD detection performance can be negatively correlated, finding that fully connected flow models have worse log likelihoods but better OOD detection performance (see Sec 2).

I would like it if caption figures were more self-contained and figures could be understood separately from the text. E.g., figure 3 and figure 5 captions could also talk a bit more about what one could learn from the figures.
summary_of_the_review: The paper in my view does not present too many surprising results, but I also find it valuable to see non-surprising results that are clearly presented and easy to understand. And the result on generation quality and OOD detection performance being negatively correlated is also valuable to know.

**Update Post Rebuttal**
The revisions have further improved the manuscript, I increase my score to 8.
review_title: null

Meta-review: The paper investigates the use of flow models for out-of-distribution detection. The paper proposes to use a combination of random projections in the latent space of flow models and one-sample / two-sample statistical tests for detecting OOD inputs. The authors present results on image benchmarks as well as non-image benchmarks. 

The reviewers found the approach well-motivated and appreciated the ablations. The authors did a good job of addressing reviewer concerns during the rebuttal. During the discussion phase, the consensus decision leaned towards acceptance. I recommend accept and encourage the reviewers to address any remaining concerns in the final version.

It might be worth discussing this paper in the related work: Density of States Estimation for Out-of-Distribution Detection https://arxiv.org/abs/2006.09273
Decision: Accept (Poster)