Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the problem on "posterior collapse" in hierarchical variational autoencoders (HVAE), and provides a theoretical explanation for why this occurs during training based on the ELBO lower bound. It further discusses why posterior collapse can affect the OOD detection performance of the HVAE model. Based on these insights, the paper proposes to enhance the connection (dependence) between the input an its multilayer stochastic latent representations based on an informative HVAE training objective. It also proposes an adaptive likelihood ratio score for detecting OOD inputs, which enhances the separation in in-distribution and OOD inputs, and does not depend on the specific choice of higher-level latent layer representations used. 
strengths_and_weaknesses: Strengths:

The paper addresses an important problem of posterior collapse observed in hierarchical VAEs, which significantly limits the ability of the model to be used for OOD detection. The motivation and background on posterior collapse based on mutual information is interesting, and there is adequate discussion of related work on the problem. Building on the insights, the paper proposes a novel training objective (informative HVAE) which alleviates the issue of  posterior collapse by enhancing the dependence between the input and the higher level latent variables. The experiments are fairly extensive and compare the OOD detection performance of the proposed method with a number of baselines from different categories of OOD detection. 

Weaknesses:

- The technical details are hard to follow in some places and lacks enough discussion. For instance, the likelihood-ratio score in Eq. (6) and its approximation are not discussed clearly. Same comment for the informative HVAE loss in Eqs. (8) and (9). 

- The paper compares with several baselines, but they are not clearly defined in the main paper (please questions on the experiments). 

- In the experiments, only two in-distribution datasets are evaluated, while it is common to evaluate on more ID/OOD dataset pairs. Some additional results, including an ablation study on the adaptive log-likelihood score, could be provided in the appendix to make the evaluation stronger.

- A number of typos and grammatical issues, which could be easily fixed by proof-reading. 

questions: While the paper is technically strong, it is a bit hard to follow and there are some logical jumps that are not obvious. Overall, the presentation of ideas could be improved.

1. It is mentioned that the proposed method is completely unsupervised, i.e., it does not require labels for the in-distribution data, nor does it require auxiliary OOD data for the detection algorithm. The proposed method is different from many conventional OOD detection methods in that it does not depend on a classification model (DNN) for its scoring. In this sense, it is more like an anomaly detection method.
Can the authors comment on whether the proposed method can be improved by utilizing the class labels (if available) by modeling the class-conditional distributions of $x$?  

1. The paper does not discuss a number of prior works on OOD detection in section 2. These include methods not based on deep generative models such as maximum softmax probability [1], Generalized ODIN [2], Deep Mahalanobis [3], Energy-based [4], ReAct [5] etc.

1. On line 89, it is mentioned that likelihood methods based on generative models always assign a higher likelihood to OOD inputs compared to ID inputs. While this happens in some cases, it is not always true.

1. In equation (3), it should be clarified that $z_{L+1} := x$ as a special case. Otherwise the dependence on $x$ is not clear in the second term. 

1. Please provide a discussion on the log-likelihood ratio score in Eqn. (5).

1. On line 161, it should be $q_\phi$ and not $q_\theta$.

1. On line 185, it is mentioned that the entropy of the marginal distribution on $x$ is a positive constant. This does not have to be the case for continuous $x \in \mathbb{R}^d$. If so, is the lower bound on the mutual information correct?

1. The informative loss proposed in Eqn. (8) and its lower bound could be discussed in more detail. 

1. It is not obvious how the authors arrive at the proposed adaptive log-likelihood score. Please provide some intuition about why this is formulated as a weighted difference of log-likelihood ratios. Could this be arrived at in a more principled way? Why does it enhance the separation between ID and OOD inputs?

1.  Minor: On lines 241 - 243, it is mentioned that the metrics are threshold independent, but the `FPR80` metric does depend on the threshold.

1.  The paper compares with different categories of OOD detection methods, but some of them are not defined in the main paper. This makes it a big vague to read the tables and figures. 

1. In Table 1, why is the AUPRC not reported? This metric is sometimes more effective is capturing the separation between the OOD and in-distribution scores, especially at low FPR. 

1. In Table 1, it is not clear why some of the baseline methods are missing for the CIFAR10 / SVHN datasets. The format is a bit confusing to follow. 

1. In Table 2, why is SVHN used as the OOD dataset for FashionMNIST, whereas MNIST is used as the OOD dataset in Table 1? More complete results could be provided in the Appendix. 

1. In Figure 3, could the authors also report the area under the ROC curves, maybe as part of the legend. 


### References

[1] Hendrycks, Dan, and Kevin Gimpel. "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136 (2016).

[2] Hsu, Yen-Chang, et al. "Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[3] Lee, Kimin, et al. "A simple unified framework for detecting out-of-distribution samples and adversarial attacks." Advances in neural information processing systems 31 (2018).

[4] Liu, Weitang, et al. "Energy-based out-of-distribution detection." Advances in Neural Information Processing Systems 33 (2020): 21464-21475.

[5] Sun, Yiyou, Chuan Guo, and Yixuan Li. "React: Out-of-distribution detection with rectified activations." Advances in Neural Information Processing Systems 34 (2021): 144-157.


limitations: The discussion of limitations and broader impact in Appendix H and I is adequate.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a hierarchical VAE (HVAE) for out-of-distribution (OOD) detection. Authors investigate the 'posterior collapse' problem with HVAE models and propose a solution to mitigate the by increasing the mutual information between the input and latent representations. Finally, an adaptive likelihood ratio-based measure is proposed for the HVAE models to detect OOD samples. The proposed approach is evaluated on benchmark datasets and the proposed approach outperforms related variants. 
strengths_and_weaknesses: Strengths:

1. Authors systematically explored the 'posterior collapse' problem with HVAE models. An interesting mitigation strategy is proposed by increasing the mutual information between the input and latent representations. 

2. An adaptive likelihood ratio-based measure is proposed to distinguish the OOD sample using all layers of HVAE.

3. The draft is clearly written and easy to follow.

4. Authors experimentally evaluated various components of the approach justifying their contribution toward the final performance. 


Weaknesses:

1. There are other approaches to prevent posterior collapse such as bidirectional inference [20] or oversmoothing VAE loss function. How does the proposed approach compare with these approaches and what is the fundamental difference? [20]  also considers skip connections. 
Takida et al., "Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE"  

2. In table 1, the approaches are not cited. 

3. Experiments are limited. Authors are encouraged to compare the results against the more recent state of the art such as 

a. Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.

b. Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild, ArXiv, abs/2110.00218, 2021

c. Kaur, Ramneet, Susmit Jha, Anirban Roy, Sangdon Park, Edgar Dobriban, Oleg Sokolsky, and Insup Lee. "iDECODe: In-distribution equivariance for conformal out-of-distribution detection." AAAI (2022).

questions: Please address the comments in the weaknesses section, especially the novelty of the proposed approach with respect to existing approaches to preventing posterior collapse.
limitations: No
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work studies OOD in hierarchical VAEs (HVAE). It connects the earlier observations that HVAEs may yield higher likelihood for OOD samples to the so-called “posterior collapse”, where higher-level latent variables degenerate to the (conditional) prior and hence becomes uninformative w.r.t. the input sample. To alleviate this issue, the work promotes increased mutual information between the input and the higher-level latents. It further develops an OOD score based on a weighted difference of the log-likelihood ratio between subsequent slices in the hierarchy of the latent variables.
Experiments demonstrate improved OOD accuracy on two benchmarks. The proposed OOD criterion is shown to be more stable than the layer-specific scores (which implicitly require the index of the layer as a hyperparamter).

**A post-rebuttal note.**
I thank the authors for their elaboration and I appreciate the effort. I increase my score, since my main concerns have been resolved. Nevertheless, I encourage the authors to improve clarity in the main text, as well as to include the computational considerations entailed by the approach (as provided in the response below) in the final revision.
strengths_and_weaknesses: **Pros:**
- the narrative that OOD accuracy in hierarchical VAEs is connected to the posterior collapse issue is compelling and interesting to read.
- the work appears technically sound and the arguments are appropriately formalised. 
- the proposed OOD score does not have any hyperparameters, but nevertheless appears competitive w.r.t. parametric alternatives.
- the empirical results are strong and support the main claims.


**Cons:**

While the OOD is the focus of this work, the approach leads to increased in-domain likelihood overall, in some cases rather substantially so (c.f. Tab. 4 in Appendix);

Both the proposed training and the adaptive OOD score may be a suboptimal choice. For example, on FashionMNIST->MNIST the likelihood regret approach has a somewhat superior OOD accuracy.

Comparisons could have been a bit more extensive (only 2 training sets). Tab. 3 and 4 in the appendix provide more results, but those are not compared to previous non-VAE and more recent works (e.g. [A])

The idea behind the proposed score seems to be mainly intuition-based and lacks theoretical backing. The explanation in Appendix C does not provide further theoretical insights.

[A] Revisiting Flow Generative Models for Group-wise Out-of-Distribution Detection

**Typos:**
- l. 70 “inference”
- l. 100 “PixelCNN”
- l. 206,213 “ration”
- l. 310 “still” redundant
- l. 311 “preserve”

**Other comments:**
- Tab. 2 left should probably read “FashionMNIST(in)/MNIST(out)”
- The exposition could be improved (e.g. l. 95 “at the cost of bringing heavy burdens”). It is not always clear in notation, if it is a definition or an implied equality (e.g. l. 120,188).
- The acronyms used in Tab. 1 are not all self-explanatory, nor is it very clear which previous works specifically they come from. Please make it clearer in the revision.

questions: - What is the OOD accuracy with the vanilla VAE loss (i.e. without the informative loss), but with the adaptive criterion? 
- Testing the approach on MNIST and CIFAR is great, but what about more natural images (e.g. ImageNet)?
- How does the computational footprint change specifically w.r.t. the baseline?

limitations: The limitations in Sec. H are a bit unspecific ("additional computational burden"). The broader impact in Sec. I is perhaps a bit more technical than it should be.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper aims to detect OOD samples (unsupervisedly) with hierarchical VAEs. The main idea is based on the likelihood ratio in [1], with some interesting modifications. Firstly, the paper demonstrates that alleviating posterior collapse in hierarchical VAEs can help the performance of OOD detection using likelihood ratio. Then the paper proposes a modified training objective that upweights the mutual information between data $x$ and higher level latent variables, which enforces higher level latent variables to contain information about $x$ and hence prevents higher level latent variables to collapse to prior. The paper also proposes a new score that eliminates the need to tune the hyper-parameter $k$ in previous likelihood ratio method. 

[1] Hierarchical VAEs Know What They Don't Know
strengths_and_weaknesses: Strengths: 

1.One thing I particularly like about the paper is that the proposed method is based on solid motivations. The paper clearly demonstrate the impact of posterior collapse to OOD detection in hierarchical VAEs, and then proposes method to alleviate posterior collapse. This makes the  main structure of the paper easy to follow. 

2. The proposed method is simple yet effective. Upweighting the MI between $x$ and $z$ can certainly alleviate posterior collapse, but the magic only happens when it is combined with the analysis on the relationship between posterior collapse and OOD detection.

3. Comprehensive and strong experimental results.

Limitations:

1. Since the method is based on previous likelihood ratio idea, in order to make the paper self-contained, more details of the previous method should be given. In particular, section 3.2 should give a more detailed introduction to [1]. The motivation and intuitive explanation should be provided.

2. Section 3.4 is not very clear to me. Some explanations in the appendix should be moved to the main text.

[1] Hierarchical VAEs Know What They Don't Know
questions: 1. How does the modification in training objective affect the performance of VAE, e.g., reconstruction, test likelihood, sample quality? We want the VAE to be versatile, not just a tool to detect OOD samples.

2. If we alleviate the posterior collapse with simpler approaches, such as simply downweighting the KL term or using some warm-up scheme, can we obtain hierarchical VAEs that do better in OOD detection?

3. In section 3.4, as well as appendix C, if we can use the ration of $R$, where $R$ is the partial generative model's log likelihood, why don't we just automatically pick k with the largest $R$-ratio?

4. In Figure 2, I don't quite understand why the partial reconstructions for $p_{\theta}(x|z_{>3})$ and $p_{\theta}(x|z_{>4})$ have absolutely no variation. Assuming $z_{>3}$ has been collapsed into prior, then $q(z_{>3}|x) \approx p(z_{>3})$. As a result,  sampling $z_{>3}$ from posterior $q(z_{>3}|x)$ and $z_{<3}$ from prior $p(z_{<3}|z_{>3})$ (which is what a partial generative model does) is essentially sampling the whole $z$ from prior. Then the resulting reconstruction should just be some generated samples from the VAE, which should be diverse. But here they are the same. Does the posterior collapse to a single point?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: This paper studies unsupervised out-of-distribution detection based on hierarchical VAE models. In particular, it (1) investigates the posterior collapse issue, (2) proposes a training procedure by increasing the mutual information between the input and latent representations, and (3) proposes an adaptive likelihood ratio score for detecting OOD inputs. Multiple reviewers found the method interesting and technically sound. 

Post rebuttal, all reviewers unanimously supported this paper positively. The contribution and insights presented in this paper will be valuable for the OOD detection community. The AC recommends acceptance. 

Please incorporate the reviewer's requested discussions (e.g. computational footprint) in the final version. Several published papers in the reference sections are in arXiv format, which necessitates proper citations in camera ready. 






confidence: Certain
Decision: Accept