Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles approximate inference for hierarchical Bayesian models with fully nested structure. The specific approach taken is variational inference with a q-distribution that iteratively applies conditional normalizing flows to derive a hierarchical representation, and factorizes in a manner parallel to the generative model by reusing flow parameters, thus having a number of parameters that does not grow with cardinality of each plate. The benefits of the model are illustrated in a few synthetic experiments as well as in application to a human neuroimaging dataset. 
main_review: # Strengths
The motivation for the work is clear and it's likely to be of broad interest. Factorizing the approximation network to match the structure of the model is interesting and novel to my knowledge, and the experiments build up from synthetic toy examples to a real-world problem. 

# Opportunities for improvement

## Overall framing and relation to broader approximate inference
The overall framing of the paper starts from the perspective of improving the tractability of normalizing flows for posterior approximation rather than improving approximate inference, and takes its aim primarily at flow-based methods (notably, cascading flows) rather than approximate inference more broadly. I think broader engagement with the VI literature may improve the paper, including particularly some mention of the approximation vs amortization gap. This then situates the MF VI baseline as not just a "common practice" method but (as far as I can tell) the upper bound on performance of the remaining methods, which share its approximation gap but add an amortization performance gap. I'm not sure I understand the non-conjugate results in this context, though -- is it because the link function is applied to the flow approximation but not in the MF context? Relatedly, I think the discussion from supplement B2 is worth including in the main paper -- the HVM and SSVI approaches among others seem like a natural complement to the present work. Making these connections seems more important than engaging with SBI and likelihood-free inference -- I don't think likelihood-free approaches are ever competitive with likelihood-based approaches if likelihoods are available (nor is this their intent), so I'm not sure why they are an important baseline or particularly relevant to the current work. Finally, as a minor point, I wonder if the more typical reader here would come from the VI literature, for whom leading with the ELBO (rather than Reverse KL) framing would be more intuitive. 

## Pushing the synthetic examples to regimes where ADAVI wins
None of the examples actually show that ADAVI is a superior approach: in the NC case it is roughly equivalent in performance to CF. In GM ADAVI may achieve the best ELBO out of the amortized methods, but takes an order of magnitude more time than just running MF-VI, so the benefit of amortization is not shown. In the gaussian random effects example (fig3) it's clear that ADAVI has favorable scaling, but even in the largest example MF-VI has fewer parameters and achieves higher accuracy in less time. Furthermore, I'm not sure if the selected metrics truly show the benefit of amortization: rather than showing optimization time for all the models, shouldn't the paper show the predictive inference time (and performance) for held out data for all the models? Then the amortized models should be dramatically faster by not requiring any optimization. Finally, I'm not sure why timings are not done on consistent hardware -- it seems like everything in the paper should run on both CPU and GPU, so picking one (or showing timing for both) would be better than trying to compare timings between two different sets of hardware. 

## Including quantitative evaluation on the real problem
While the real-world results match qualitative patterns expected from prior work, the paper needs to do more to quantitatively show how ADAVI is indeed superior for these kinds of models. The best way to do this would be reporting cross-validated predictive loss, e.g. by holding out subjects, times, or connectivity measures, and comparing both the loss and runtimes against other models. 

## Minor comments and typos

Section 2.2. could be clarified further. For example: 
- Is the set transformer really elementwise? Or is it "elementwise" w.r.t. the leaf random variables in the graph? 
- If the "contraction" operator isn't to tensor contraction, could a different word be used, considering that tensor notation is already used and "contraction" has a common meaning in that setting? 
- $\mathcal{B}_h$ is defined twice above and below expr. 1. 
- Both upper and lowercase $x^{i,j}$ is used -- is it the same or different? 
- The meaning of the superscripts on $x$ is not explained (I think one is the plate index and the other is the datapoint index, but not sure). 

In addition: 
- The paper should clarify the full algorithm: is everything trained end-to-end? Or are the STs trained first, and then the model parameters? 
- Section 3.2 could potentially pick a more practically-relevant non-conjugate example from the literature (there are plenty of examples in neuroscience, though I'm not sure if any have non-discrete latents). 
- Fig2 and 3 are missing error bars/ribbons. 
- $x \sim \mathrm{Mix}(\ldots)$ is not defined and not standard to my knowledge -- I'm assuming this is multinomial over mixture components given $\pi$ and each component is Gaussian? 
- Expressions 9a-9b are not arranged in an intuitive way (definitions and priors are intermixed, everything is in one giant block). An annotated figure (even a plate diagram) would be better, and the specifics can be left to the supplement, especially since the paper doesn't give enough info to understand the model beyond the Kong et al. citation. 


summary_of_the_review: Hierarchical graphical models of the sorts considered here are in wide usage by scientific practitioners, and automatic approximate inference methods that are efficient and accurate are of longstanding interest -- thus the motivation of the work is clear. The nature of the contribution is likewise clear. My primary concern is that the evaluation doesn't fully demonstrate the benefits of the approach (e.g. using predictive out-of-sample log-likelihoods rather than training ELBOs, including a quantitative evaluation of the real-world problem, providing consistent timings, pushing the scale until ADAVI dominates). Secondarily, I think the overall framing, clarity, and writing could be improved. I think the work is below the bar now and am rating accordingly, but I think my concerns should be sufficiently addressable in rebuttal for the work to be appropriate for the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces ADAVI, an approximate inference algorithm for hierarchical Bayesian models (HBMs). The approach is similar to NPE from simulation-based inference but exploits the hierarchical structure of the forward model to generate an efficient variational family automatically. Experiments demonstrate the applicability of the method on HBMs of increasing complexity, including a challenging neuroimaging model. Results indicate good performance against other amortized methods. 
main_review: Strengths:
- The method deviates from black-box simulators and instead exploits the hierarchical nature of many forward processes to scale to high-dimensional parameter spaces.
- Experiments are detailed, and results are discussed carefully. ADAVI shows good performance, in particular against (S)NPE, but non-amortized methods are shown to achieve higher ELBO. 
- The supplementary materials are thorough and include helpful experimental details as well as further discussions.

Weaknesses:
- Population studies are taken as an example of large HBMs with millions of parameters. I believe it would be fair to mention that oftentimes only a few of those many parameters are actual of interest for scientific inquiry. In this case, most of the parameters are latent variables for which no explicit estimation is necessary. Would you argue that ADAVI is still relevant in this case? [I would say it may still be relevant, but I would be curious in having your opinion.] Could you also better motivate in which scientific cases inference over millions of parameters is strictly necessary?
- In Section 2.3, the variational distribution is defined as a mean-field approximation. Can you comment on the constraints that result from this assumption?
- Results are discussed in terms of the ELBO only. The quality of the approximate posteriors is never diagnosed explicitly (at least as far as I can see). 
- I would appreciate some comments on the impact of the size of the encoding space produced by the SetTransformers. How shall one set this size? Should it be the same across all levels?

Small remarks:
- Avoid footnotes if you can.
- Replace "fig. X" with "Fig. X". 
summary_of_the_review: This paper addresses a common problem of simulation-based inference and proposes a sound and efficient solution to enable inference in high-dimensional parameter spaces. Experiments show convincing results.

My main issue is the lack of quality checks of the approximate posteriors produced by ADAVI. This should matter the most, in my opinion, well above efficiency, wall-clock times, and the number of parameters. If the approximate posteriors are wrong, none of those matter. For now, I do not recommend this paper for acceptance, but I will be pleased to change and increase my evaluation if the authors can present diagnostics of the resulting approximate posteriors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes an amortized variational inference to produce a dual variational family for hierarchical Bayesian models (HBM) that can be represented as pyramidal Bayesian models. The presented method exploits the exchangeability of parameters in HBM to reduce its parametrization for faster inference on high-dimensional data such as neuroimaging. The authors compare empirically the proposed method with several amortized and non-amortized alternatives and on several experimental data in terms of the size of parametrization, inference time, and quality of inferences. 
main_review: Strength:

- A good motivation: Extending the applications of simulation-based inference and structured variational inference to high-dimensional data settings (such as neuroimaging data) is interesting.

Weaknesses:

- The proposed ADAVI method only applies to a pyramidal class of Bayesian networks in which dependency structure follows a pyramidal graph. The authors need to motivate in the text that this class of problems covers most cases in the target applications in neuroimaging studies. 
- The proposed method in its ultimate performance becomes similar to a simple mean-field approximation. The authors claim the proposed method instead provides more expressivity while failing to show this point in the experiments. 
- The performance improvements (in terms of quality of inference and inference time) compared to other alternatives remain marginal (results in Table 1,2 and Figure 3).
- Despite the promising abstract and introduction setting a high expectation on the high-dimensionality of target problems (thousands of brain locations), the experiment on neuroimaging data is merely conducted on a relatively small dataset with 30 subjects and 1483 measures. This barely fits with the requirements in the field. These days we need methods that can deal with considerably larger datasets consisting of thousands of subjects with thousands of measures (for example UKBiobank). Furthermore, it would be nice to also compare quantitatively the time complexity and inference quality of ADAVI with other alternatives when applied to neuroimaging data.

Minor suggestions, comments, and questions:
- It may be nice to motivate the importance of HBR in the neuroimaging context. Why it is important to be able to handle the hierarchically organized data in the neuroscience context? 
- Section 2.1: Direct Acyclic Graphs => Directed Acyclic Graphs
- Throughout the text: fig. => Fig.
- Section 2.1: A RV's hierarchy => An RV's hierarchy
- What do you mean by "symmetry" in "... exploiting the symmetry induced by the plates ..."?
- Section 3: "a a hierarchy" => "a hierarchy"
- Is the curves in Fig. 2a derived empirically? Then why the wall time for ADAVI remains fixed until $10^5$ examples? Please explain.
- While the main arguments are around the lower computational complexity of the proposed method, it is very difficult to judge the time improvements when diverse hardware are used in a heterogeneous cluster of computers (for example two types of CPU). Would be nice to fix the hardware setup to ensure fairness and reliability of comparisons.
- I suggest moving sections B.2 and B.3 to the main text (possibly to the final discussion) as they contain important information about the presented method.
- Would be great also to see some results in section E.5 in the main text (on how the individual network can deviate from the population network). That would be of high interest to the audience from the neuroscience community. 

summary_of_the_review: Despite a nice motivation, the empirical results show a minor improvement over the already existing methods. The results are also weakly presented for the target problem in the neuroimaging context (small data, no quantitative comparison, lack of discussion). The method is not tested on very high dimensional settings as they are in the neuroimaging context.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to derive an automatic methodology that takes as input a generative HBM and generates a dual variational family able to perform amortized parameter inference. The proposed method can be used to the context of  pyramidally structured data with good inference quality and a favorable training time. 
main_review: Overall, I am not an expert in HBMs and Neuroscience, but based on the introduction in this work, the proposed method is meaningful to the 
field of Bayesian modelling for the neuroimaging, as it can estimate posterior distributions for a generative model of pyramidal structure, which is difficult to achieve. Authors provide sufficient experiments to support their claim in terms of both the inference quality and the speed. So I hold a positive attitude toward this paper. The paper is not very reader-friendly, so it would be better that authors can provide the introduction of some basic concepts and more related works in their final version to improve the readability of their paper.  
summary_of_the_review: Please refer to the Main Review section.
review_title: null

Meta-review: The paper provides a unique contribution to the scalability of Bayesian inference to Pyramidal Bayesian Models with application to neuroimaging. The major point of concern by the reviewers is around how close is the inference approach to the more classical Mean-Field VI. However, in my opinion, the authors have addressed these concerns in the rebuttal. Therefore, I recommend Accept.
Decision: Accept (Poster)