Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper makes an attempt to address the problem of large-scale inference for hierarchical probabilistic graphical models that contain both global and local latent variables by proposing to use Amortized VI instead of VI in [1] for more efficient and scalable inference. The authors first define a general class of hierarchical models which they call Hierarchical Branch Distributions (HBDs). They then propose a variational family matching this class of models for performing amortized inference. The authors then perform a series of experiments on synthetic datasets and the MoveLens dataset where they evaluate different inference models in terms of the final ELBO and convergence.  
main_review: [General Comment]

In my opinion, the main weakness of the current manuscript is the lack of novelty or at least the lack of transparency of the contributions. While I very appreciate the importance of accurate/scalable inference for hierarchical graphical models and therefore am a big fan of this line of research, I found it very difficult to pin-point the exact contributions of this paper. Amortized variational inference has been applied not only to models with solely local variables, but also models with global and local variables as well, so this is not a new idea by any means. Here are just a few examples: [2-6] that all use AVI and most contain local and global variables. In fact, [2] also uses pooling to extract group content from local features. So if the main contribution can be described as Figure 3 (‘d’ in particular), it’s hard for me to see the novelty here as I would classify that as basically choosing an appropriate conditional variational family for your model. Ofcourse if this was an application paper, where an extensive set of experiments had been done  with the specified model to a particular task to achieve good results, it would have been a different story but this is not the case here. 



[Clarity]

I did not find the paper easy to read, which I believe also partially contributed to the contributions not being very clear. I would encourage the authors to proofread the paper once more. I also do not find all the mathematical statements clear or necessary. For example, the “symmetric“ definition in page 2 is redundant as far as I can see (a=b ⇒ p(a|c) = p(b|c)). Theorem 2 (& paragraph 4 in page 1) is also obvious and redundant I would argue. Choosing a variational family that factorizes just your target would of course yield a more accurate inference. 

-There is a font switch for theta on page 3 (compare equation 4 & 5)
-Typo in equation (16)



[Novelty & Prior work]

I don't think the current manuscript did a good job of discussing related work. See my general comment for more details. 
Another missing reference is [7]


[Minor comments]

In order to properly test the inference, I also would advise the authors to test the model containing no ‘x’. While the authors state that “results extend easily to case where there are no local covariates”, this is not necessarily the case in practice depending on  how much information the covariates provide about ‘y’. In such cases, the model does not have to rely on the latent variables so accurate inference might not matter as much to achieve a good likelihood. 




[References]

[1] Hoffman, Matthew D., et al. "Stochastic variational inference." Journal of Machine Learning Research 14.5 (2013).

[2] Edwards, Harrison, and Amos Storkey. "Towards a neural statistician." arXiv preprint arXiv:1606.02185 (2016).

[3] Lee, Dong Bok, et al. "Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning." International Conference on Learning Representations. 2020.

[4] Esmaeili, Babak, et al. "Structured neural topic models for reviews." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

[5] Ilse, Maximilian, et al. "Diva: Domain invariant variational autoencoders." Medical Imaging with Deep Learning. PMLR, 2020.

[6] Bouchacourt, Diane, Ryota Tomioka, and Sebastian Nowozin. "Multi-level variational autoencoder: Learning disentangled representations from grouped observations." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.

[7] Johnson, Matthew J., et al. "Composing graphical models with neural networks for structured representations and fast inference." Advances in neural information processing systems 29 (2016): 2946-2954.
limitations_and_societal_impact: To the best of my knowledge, this work does not raise any major social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a Gaussian naive bayes approximate posterior for a 2-level hierarchical naive bayes model. Various parameterizations for the variational posterior are explored, including a fully coupled joint posterior, a factored branch posterior, and an amortized version of the latter. Results and theory indicate that having an approximate posterior that makes the same independence assumptions as the true posterior does not hurt, and that parameter sharing improves performance at scale.
main_review: *Originality*
* It is unclear whether the contribution of this paper, the exploration of the parameterization of the Gaussian naive bayes approximate posterior, is novel. To remedy this, the related work must cover similar models and inference methods. Currently the related work is too high-level, and offers little context. Similar generative models must have been used elsewhere, such as hierarchical naive bayes [1]. Please find and cite the relevant work in order to convince us that the model is useful and that amortized variational inference has not been applied in this exact setting.
* The preference for the branch posterior rather than the fully-connected joint can be motivated by Webb et. al. [2].

*Quality*
* The exploration of the approximate posterior is thorough. However, not enough context is given for experimental results. How do the models perform relative to other baseline models? How is evaluation normally conducted on MovieLens?

*Clarity*
* The paper is very clearly written, and seems technically sound.

*Significance*
* Providing more context will help show the significance of the work. The methods and experiments section are complete, but giving more context for the background and results will complete the paper.

*Comments*
* The title is a bit too high-level and vague as well. This paper proposes amortized variational inference for a very particular model, I suggest including that in the title.

[1] Langseth, H. and Thomas D. Nielsen. “Classification using Hierarchical Naïve Bayes models.” Machine Learning 63 (2006): 135-159.

[2] Stefan Webb, Adam Golinski, Robert Zinkov, N Siddharth, Tom Rainforth, Yee Whye Teh, and Frank Wood. Faithful inversion of generative models for effective amortized inference. In Advances in Neural Information Processing Systems, 2018.

Edited score: After calibrating across other reviews, the issues I raised are either not high priority (experimental) or can easily be fixed (more context), and have lowered confidence accordingly. I would be willing to raise the score higher if those are addressed.
limitations_and_societal_impact: * The appendix includes some discussion of the limitations of the different posterior parameterizations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: An approach vor variational inference in large-scale hierarchical models is presented that is more efficient than the base lines. This is achieved using shared parameters (amortization) and a feature pooling network. Experiments confirm the efficiency of the approach on synthetic data as well as the MovieLens dataset.
main_review: The paper is original and clearly written (all terms are defined, although some only later after the first mention).

Regarding the significance my thoughts are as follows:
This paper has a focus on the theory and thus does not spend too much space on the experiments. The theory regarding the branch approximations is interesting and seems to be relevant. What is not entirely clear to me from the paper is how widely applicable the proposed methods are. The cited application papers in the introduction are all older works before the boom of neural network based approaches. In my view the main limitation of the paper is thus an appropriate comparison to a state-of-the-art model (such as VAE maybe, or a NN approach for learning hierarchical models) to show, either theoretically or practically, how the proposed method compares to more recent methods. This would make the results, which are more or less just comparing variants of the same method, much more convincing. In general, though, I think this is a good paper and the above reservations should maybe not be weighted too heavily.

Corrolary -> Corollary
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper deals with scalable approximate inference for hierarchical models and proposes a framework for hierarchical approximations that exhibit the same factorization as the exact posterior of the hierarchical model. This efficiently enables subsampling the data when evaluating the evidence lower bound and its gradients. The authors further propose a pooling-based strategy for amortized inference to avoid the usual linear scaling in the number of approximation parameters.  The paper is concluded with a set of numerical experiments with both synthetic and real data demonstrating the merits of the proposed framework.



main_review: The paper proposes a novel (to the best of my knowledge) and elegant solution to an important problem and I think the proposed methods will be of general interest to the approximate inference community. 

All technical details in the paper appear to be correct and the claims in the paper are theoretically motivated and reasonably justified using empirical experiments. However, the empirical experiments could be further improved by

- The results for the synthetic experiment are only compared using the lower bounds. Since the target distribution is Gaussian, why not 1) compute and plot the true log marginal likelihood in Figure 7 in the appendix for comparison 2) compute and evaluate the error on the estimated moments/parameters for completeness since the true quantities are available.

- Use replications and add error bars to all experiments

Overall, the paper is well-written and well-structured and I really enjoyed reading it. However, I do think the notation can be slightly improved. For example,

- I am a little bit puzzled by the exact meaning of the definition in Eq. (2). Perhaps, this is caused by the usual casual notation in probability, where there is no distinction in the notation for a random variable and the values it takes. Can this be rephrased or made more precise?

- Along the same lines, in eq. (6), (8)-(10) it would be a nice service to the reader if the authors would distinguish between the ranom variables and samples of the random variables. For example, use \theta^s or \hat{theta} to represent a sample of the random variable \theta and the same for the other variables.

- I think there may be missing a minimization operator on the left hand side of eq. (14) since currently, the left hand side is a function of v (and the existence of a uniform bound seems unlikely). I noticed that \min_v is also present in the proof in the appendix, so it is most likely just a typo.


Other comments:

- the xlabels in fig 6 are missing.

 - extra parenthesis in eq. (16)

- It's nice that the authors included a section describing some of the practical challenges and considerations.



limitations_and_societal_impact: Not applicable
review_title: null

Meta-review: The authors consider variational inference for a hierarchical model where each observation is generated using a local/observation-specific latent variable, which in turn depends on the global latent variable. They prove that using a variational posterior with the same dependence structure as the model results in inference accuracy comparable to a general joint posterior approximation at a much lower computational cost and allows observation subsampling for an additional speedup. Experiments confirm the theoretical results and show that amortizing local inference enables training on much larger datasets without affecting accuracy.

The paper describes a simple but effective idea clearly and shows that it works in practice. The discussion of related work is insufficient however, and should be extended to include the Neural Statistician and its descendants and their relationship to this work. Also, as pointed out by a reviewer, the title of the paper is too general and should really be made more specific, as the method applies only to hierarchical models with a very specific, if common structure.
Decision: Accept (Poster)