Title: Model-Informed Flows for Bayesian Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian Inference

Autoregressive Flows

Non-Centering Parameterization
Abstract: Variational inference often struggles with the posterior geometry exhibited by complex hierarchical Bayesian models. Recent advances in flow‐based variational families and Variationally Inferred Parameters (VIP) each address aspects of this challenge, but their formal relationship is unexplored. Here, we prove that the combination of VIP and a full-rank Gaussian can be represented exactly as a forward autoregressive flow augmented with a translation term and input from the model’s prior. Guided by this theoretical insight, we introduce the Model‐Informed Flow (MIF) architecture, which adds the necessary translation mechanism, prior information, and hierarchical ordering. Empirically, MIF delivers tighter posterior approximations and matches or exceeds state‐of‐the‐art performance across a suite of hierarchical and non‐hierarchical benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a flexible family of variational posteriors, termed Model-Informed Flow (MIF), which combines full-rank variational Gaussian approximations with partially non-centered parameterizations (NCP) on conditionals via Variationally Inferred Parameters (VIP). The key idea of VIP is to better handle \"funnel\"-shaped posteriors through reparameterization, introducing auxiliary CP-NCP interpolation parameters (λ) that are learned by optimizing the variational objective, but the prior work that is limited to mean-field Gaussians. By incorporating a lower-triangular Cholesky factor, the proposed method can be naturally interpreted as an autoregressive flow, but this paper has a clear motivation of NCP adjustment along with several other carefully chosen design elements, making the approach competitive even with one-layer MIF with affine functions. The method is applicable to Bayesian hierarchical models with Gaussian priors, where the mean and variance parameterized by a deep net. Ablation study and experimental results on benchmark datasets demonstrate the  effectiveness and broad applicability of the proposed method compared to other variational inference baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### **Pros:**\n\n**Quality**: This paper presents a flow-based approach with clear motivation from NCP via VIP, and it captures complex dependencies using a full-rank Gaussian. The research question is intriguing, as the auxiliary parameters could, in principle, be subsumed by the flow-based mapping and powerful neural network parameterizations already used in autoregressive flow approaches. While the mathematical expressions are equivalent, including NCP, there are differences in sampling and optimization. The effectiveness and added complexity are justified by ablation experiments. The authors also discussed the limitations such as the sequential nature. \n\n**Clarity**: The paper is well-written and neatly presented. The technical derivations are easy to follow yet insightful.\n\n**Significance**: Although sufficiently expressive architectures such as IAF can achieve strong results, this paper offers a compact and flexible alternative, with careful design specifically aimed at handling “funnel”-shaped posterior geometries. The work highlights that thoughtful methodological design—beyond simply relying on powerful neural networks—can be valuable. Even mathematically equivalent representations may differ in their ease of sampling or optimization. This perspective invites future innovations borrowing from classical probabilistic modeling and statistical science in designing better flow mappings, and it encourages a shift in focus within the ML community from prediction to inference. \n\n**Originality**: The connection to autoregressive flows is nice, though not particularly surprising. The use of VIP with full-rank Gaussian, and the extension of standard autoregressive flows with translation and prior inputs, are novel contributions. \n\n### **Cons:**\n\n**Quality**: The experimental evaluation is somewhat limited (explained in Questions).\n\n\n**Clarity:** The notion of “prior information” needs further explanation. Does it mean that the parameters are informed by previous variables in the specified hierarchical model, or that the hierarchical model itself already includes neural network (NN) components? In the context of probabilistic programs, the model can be user-specified. Is the proposed approach applicable to generic hierarchical models?\n\nIt seems that MIF serves both as a way of specifying hierarchical priors and as a method for flexible variational posterior, but the topological order of variables may differ. Have you tried reversing the topological order (e.g., bottom-up vs top-down)?\n\n**Significance**: The real-world impact is unclear. (1) Due to the use of full-rank Gaussians, scaling to high-dimensional problems may be challenging. (2) The experiments are mostly limited to a few simple models and datasets, and the evaluation relies solely on ELBO. \n\nCould the authors evaluate the method’s ability to capture complex dependencies in the context of variational inference, and using a density estimation benchmark on, for example, the two-moons dataset? It would also be helpful to comment on the applicability of MIF to generic variational inference in probability programming, likelihood-based density estimation and sample generation.\n\nQUESTIONS:\n(1) The experimental evaluation is somewhat limited. Is it possible to demonstrate that MIF offers general benefits, such as improving variational autoencoders, by evaluating on binarized MNIST (e.g., reporting marginal log-likelihood log p(x) using importance sampling)? Table 1 and Table 2 essentially use the same set of datasets, although the ablation is different. \n\n(2) With a limited-capacity conditioning network (e.g., a one-layer affine transformation), does MIF capture posterior dependencies better than alternative flow-based approaches such as coupling flows, MAF, or IAF?\n\n(3) Equation 2.2 and 2.3, 2.8, the likelihood term does not depend on z_i and the joint density factorizes between z and x? The way of writing it is confusing. The parent variables pi(x_j) are the z s?\n\n(4) Line 135: What is the quadratic form in Equation 3.4?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works introduces Model-Informed Flows (MIF), a new VI architecture aimed at addressing the challenges posed by hierarchical Bayesian models, which often exhibit complex posterior geometries. The authors demonstrate theoretically that a combination of a full-rank Gaussian and Variationally Inferred Parameters (VIP) can be exactly represented by a *generalised* forward-autoregressive flow. Their proposal generalises FAF by including model prior information. The authors then conduct various investigations to validate the performance of MIF including ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n- Sound theoretical foundation has been provided by the authors. I am convinced that this work is well motivated.\n- The architecture introduced in this work is novel to the best of my knowledge and the authors have shown clear intent to test it's performance across different benchmarks.\n- The work is clearly presented and well written.\n\n**Weaknesses**\n- The inherent sequential nature of MIF limits it's scalability and practicality. This has been recognised by the authors but it'd would been nice to see them hint towards resolutions for this major limitation.\n- The evaluations, though seemingly thorough, lack error bars or standard deviations. The numbers presented in table 1, for example, are not very conclusive as there are minute differences b/w MF-VIP and FR-VIP. Similarly I cannot ascertain the value of different components of MIF since all the numbers are very close to each other. This, in my opinion, is another major limitation.\n- For Figure 2, why have the authors not included any other variational approximations here? There are several complex and highly-expressive variational approximations that could've been considered here as baselines. I can read that they intended to compare this with IAF (Line 204) but this result is not present.\n- There are no runtime comparisons. Since IAF and FAF have difference in their ability to capture VIP but the one's faster than the other, the authors should have considered an accuracy to runtime comparison for these two methods.\n\n\n**Minor**\nWhy did the authors choose to label the y-axis in Figure 2 as 'Best Elbo'? The notion of 'best' is very ambiguous and inconsistent from the rest of the paper where the authors quote Negative ELBO.\n\nQUESTIONS:\nKindly see *Weaknesses* section above. I have detailed my concerns there and would consider increasing my score if the authors address some of those points.\n\nLIMITATIONS:\nThe authors should have commented on the runtime performance. It is important that they elucidate to the reader the differences between the runtimes/latency of MIF vs IAF", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents Model Informed Flows (MIF) for Bayesian inference. In Bayesian hierarchical models, the dependence structures between latent variables can lead to geometries in the posterior distribution that are difficult to explore by standard variational families. This paper leverages the idea of modelling non-centered latent variables in Variationally Inferred Parameters (VIP) and embeds it in a Gaussian full-rank autoregressive flow, which leads to MIF. MIF is shown to achieve state-of-the-art performance or better across a wide range of hierarchical and non-hierarchical models.

STRENGTHS AND WEAKNESSES:
This paper is very well and clearly written. How the funnel type geometry arises in hierarchical models is very intuitively described. The introduction of VIPs and its connections to autoregressive flows are easy to follow, even for someone likely myself who is learning about the VIPs for the first time.

QUESTIONS:
* The experiment section mostly contains comparisons to mean-field Gaussian VI, full-rank Gaussian VI, and VIPs. However, in the related work section, several other related algorithms are brought up (i.e. ASVI, Cascading Flows, EMF). However, these are not included in the comparison. Can the authors comment on, if any, the difference in performance that they expect relative to MIF?
* In Algorithm 1, we see that MIF involves training three neural networks. This could be quite computationally heavy. Can the authors comment on the relative computational cost between MIF and other competing methods?
* The MIF structure seems rather flexible, and I wonder if it could be helpful for problems with other interesting structure/geometry (i.e. multi-modal posteriors). Can the authors comment on this a bit on what they think?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces model-informed flows (MIF), a novel autoregressive normalizing flow formulation that extends masked autoregressive flows/forward autoregressive flows (FAF) by (i) introducing an additional translation term that depends on the previous noise variables; and (ii) introducing the model’s prior distribution as additional inputs to the conditioner. It is shown that this is sufficient to represent useful variational posteriors, in particular distributions given by multivariate Gaussians transformed through a (partial) non-centering transform, whereas FAF/IAF may not be able to represent/learn these posteriors effectively.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper makes some interesting insights into how known limitations of simple variational families/inference strategies (such as the funnel problem) are solved by using sufficiently expressive normalizing flow families. In particular, showing that multivariate Gaussians under a variationally inferred parameterization is sufficient for many existing models is a nice result, along with its representation as a closed-form MIF. Empirically, the evaluation is thorough and shows that MIF performs competitively with or outperforms existing inference approaches. The authors also conduct interesting ablation studies to show the effects of each individual component of the MIF, including showing that MIFs do indeed outperform IAF-like formulations with limited NN capacity due to the superior inductive bias.\n\nAs far as weaknesses, it’s unclear if the full MIF is currently a practical model family due to its sampling time scaling in variable dimensionality compared to the parallel IAF-like families (while also losing efficient likelihood evaluation compared to FAFs). The paper could also do a better job in comparing and contrasting with related work in its presentation, in particular providing a more detailed technical comparison to the structured or NF-based variational approaches to inference in probabilistic programs, to clarify the novel contributions of the paper in context.\n\nQUESTIONS:\nQuestions:\n- It seems from the formulation in Definition 3 that one could alternatively absorb the (scaled) translation term into the shift function $m$, by making $m$ dependent on $\\epsilon$ in addition to $z$. Is there any reason why an explicit translation function is preferred? \n- In Figure 2, the Funnel plot seems to show that the performance of MIF and MIF ($\\epsilon$-cond) are similar for larger hidden sizes, but Table 2 shows a significant difference. Is this because Table 2 uses smaller hidden sizes?\n- It’s a little unclear to me if the conditioning on prior distributions is a novel contribution of of this paper or if it has previously been used in other works on NFs for inference in probabilistic programs/hierarchical Bayesian models. Could the authors clarify this?\n- Could the authors provide a citation for the term “forward autoregressive flow”? It appears to be essentially the masked autoregressive flow of (Papamakarios et al. 2017).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}