Model-Informed Flows for Bayesian Inference
Authors: Joohwan Ko
Justin Domke
Venue: NeurIPS 2025 poster ()
Keywords: Bayesian Inference
Autoregressive Flows
Non-Centering Parameterization
Primary area: probabilistic_methods

Abstract: Variational inference often struggles with the posterior geometry exhibited by complex hierarchical Bayesian models. Recent advances in flow‐based variational families and Variationally Inferred Parameters (VIP) each address aspects of this challenge, but their formal relationship is unexplored. Here, we prove that the combination of VIP and a full-rank Gaussian can be represented exactly as a forward autoregressive flow augmented with a translation term and input from the model’s prior. Guided by this theoretical insight, we introduce the Model‐Informed Flow (MIF) architecture, which adds the necessary translation mechanism, prior information, and hierarchical ordering. Empirically, MIF delivers tighter posterior approximations and matches or exceeds state‐of‐the‐art performance across a suite of hierarchical and non‐hierarchical benchmarks.

Model-Informed Flows for Bayesian Inference
Joohwan Ko, Justin Domke
Manning College Information and Computer Sciences University of Massachusetts Amherst {joohwanko, domke) domkee
fcs. umass edu
Abstract
Variational inference often struggles with the posterior geometry exhibited by complex hierarchical Bayesian models. Recent advances flow-based variational families and Variationally Inferred Parameters (VIP) each address aspects of this challenge, but their formal relationship is unexplored. Here, we prove that the combination of VIP and full-rank Gaussian can be represented exactly as a forward autoregressive flow augmented with translation term and input from the model's prior. Guided by this theoretical insight, we introduce the Model-Informa Flow (MIF) architecture, which adds the necessary translation mechanism, prior information, and hierarchical ordering. Empirically, MIF delivers tighter posterior approximations and matches or exceeds state-of-the-art performance across a suite of hierarchical and non-hisn non-hierarchical benchmarks.
Introduction
Inference remains challenging for many complex Bayesian models. Variational Inference (VT) casts posterior approximation minimization of the Kullback-Leibler divergence between tractable approximating distribution and the true posterior 9 25 23]. Recent advances have focused on
increasing the expressiveness of variational families. Flow-based models, which apply sequence of invertible transformations to simple base distribution, have shown particular promise [34 36 28
Variationally Inferred Parameters (VIP) [19 address key challenge in VI: The posteriors of hi- crarchical Bayesian models often exhibit pronounced curvature or "funncl-like" geometry 7 24 that standard families like Gaussians cannot capture. VIP builds insights from non-centered parameterization (NCP) in Markov chain Monte Carlo (MCMC) 35), adaptively learning the optimal degree of non-centering for each latent variable during the VI optimization process.
Our preliminary investigations reveal a striking observation: When VIP is combined with full-rank Gaussian variational families, it achieves performance comparable to state-of-the-art methods from recent evaluation [T]] (See Tables [ and 3) Since VIP and flow-based models both excel at navigating posterior geometry, might they be manifestations of a common principle? In particular, can flow-based distribution encapsulate the benefits of VIP?
Our core theoretical result, formalized in Theorem 4 shows that every full-rank VIP transformation can be represented exactly as a forward autoregressive flow (FAF), provided that the flow (i) respects the topological order of the latent variables, (ii) augments the usual affine mapping with new "translation" " term, and (iii) receives the model's prior mean and scale functions additional inputs.
Guided by these principles, we introduce the Model-Informed Flow (MIF), a forward autoregressive architecture that includes the necessary translation of the model's prior. Across a range of benchmark models, MIF compares well to state-of-the-ar VI methods. We also perform targeted ablations to verify the predicted implications and practical consequences of our theory.
and incorporates the distributional form
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

FR FAF MIF
-2
-1-
-5
6
2:
Figure 1: "Funnel"-type distributions commonly arise in hierarchical model. This figure shows the funnel distribution (gray contours) approximated with 5000 samples from three families (blue points): A Full-Rank Gaussian (FR) is very poor (KL-divergence of 1.86 nats). A standard Forward Autoregressive Flow (FAF) is much better (0.38 nats) but still imperfect. Our proposed model- informed flow (MIF) achieves Kl-ddivergence of effectively 0.
2 Preliminaries
2.1 Variational Inference
In Bayesian inference, given model (22,.)). the goal is approximate the posterior distribution p(z[æ) given observed data Direct computation of this posterior is typically intractable, so variational inference (VT) introduces tractable family (z) and minimises the Kullback-Leibler (KL) divergence from qu (z) to the true posterior. The marginal likelihood then decomposes as
logp(x) (x)) log
p(a,z)
KL (L(()((z) p(+
(2.1)
divergence
ELBO (=) || p(..a))
Since logp(x) constant with respect to the variational parameters w, maximising the ELBO 2.11
is equivalent to minimising the KL divergence between goe and the posterior.
2.2 Hierarchical Bayesian Models
Hierarchical Bayesian models common in domains such ecology [12 43 38 and epidemiology
16 37 22)] Following Gorinova et al. [19), we consider a hierarchical model which each latent
variable z; is conditionally Gaussian with a mean and variance that arbitrary function of parent variables, while the observed data æj is generated from an arbitrary likelihood p(I; | (x,)). )). In particular, the likelihood for v, may depend on one or more latent variables through its parent т (Ty)). Formally, write
2, N N (I(2))
~y~ ((([,ll((z)))
(2.2)
where (-) denotes the set of parent variables. Throughout, T--) may include both observed and latent variables; that is, variable can be conditioned on any subset of the remaining variables permitted by the model graph. In practice, the functions fi and (which return the mean and standard deviation of 2,, respectively) often arise directly from a probabilistic programming system, encoding how the mean and variance of each latent variable depend on its parent variables.
The joint density corresponding Eq 2.2 is
p(z, x) = II A (I (=)))((()))) (
I ((())((a))) j=1
(2.3)
Example.
As concrete instance of Equation
2.2 consider the model
21 \O 1),
22 M(a21, exp(321)),
Ø1 E1 22 ~ Poisson (exp(21 + )
22 22, 2i ~ Binomial(n = I1, = o (22)),

---

where (u) [ue To cast this as an instance of Equation 2.2 define the parent sets
T (21)
0. T(22) { ((11) {21, 22). (xx2)) {22, 21),
and the functions
fi (0) 0, 91 (0) -1,
f2(21) = a21, g2(21) cxp(
In Equation
2.2 we make no assumptions about the distributional form of the observed variables
Fy, p(a1 | (21) Poisson (exp(= + 22)) and p(22) p( (x:)) Binomial(n 31, p (22))
can be arbitrary.
2.3 Non-Centered Parameterization
Hierarchical models often exhibit difficult curvature and "funnel-like" geometries. These are a
challenge for both sampling and variational inference approaches 7 24
22 A common strategy
in MCMC to alleviate these issues is the non-centered parameterization (NCP) 33). To illustrate, consider two latent variables 21 and 22 having prior of
21 ~ (H,
22 ~ ~V cxx(21)/))))
(2.4)
In the standard (centered) parameterization, the goal is to approximate p(=[ær. However, the nonlinear dependence of #2 on 2 can yield a sharply funnel-shaped joint posterior. As illustrated Figure (left panel), full-rank Gaussian struggles to capture this funnel structure.
The non-centered approach introduces auxiliary variables e (21. N(O,I) and defines transformation 2 = TNCP (e) that recovers (Z1, 22) via
21
21 = μr + 0c1,
22 = exp
c2.
(2.5)
2
Instead of p(2,,), inference is done on 2)),2) where p(e) No 1) and p(zle) = p(a|= (x)p(e)). The intuition is that x relatively uninformative, then p(c (z)) will be close a standard Gaussian, even when p(=[z) might have funnel-type geometry.
2.4 Variationally Inferred Parameters
While NCP helps alleviate pathologies in hierarchical models, it is not ideal for every latent variable. Non-centering can prevent funnel-shaped geometries cases with limited data, but when data is highly informative, centered formulation may be better. Variationally Inferred Parameters (VIP) addresses this trade-off by learning a partial non non-centered parameterization for each variable [19].
VIP reparameterizes E RR P by introducing an auxiliary variable = (E,,..., Ep). and by defining a parameter A E [0, 1] per dimension. Rather than drawing 2; directly from its prior, VIP samples
4 N (Aifi (π (z:)) gi (T (z
(2.6)
then recovers 2; via VIP transformation, TvIP, as in Definition Note that A interpolates between the centered (A; = 1) and fully non-centered (A; = 0) extremes.
... Definition 1 (VIP Transformation) For fixed ordering of latent variables = (.,...,, ED), and auxiliary variables ă = (ž,,..., Ep)....,pp..... let A E [0. 1] be the partial non-centering parameter for each coordinate i, and fi())) and gi (*x())) be continuous functions that return the mean and standard deviation of z; (respectively) based on the parent nodes (2:). Then, the VIP transformation TvIP (ž) is defined coordinate- wise by
= f(((((())) + (T((()))) 1- (E A f((())))
(2.7)
for 1, 1,...,.. D.
Substituting TvIP into the joint distribution (Equation 2.3 yields partially non-centered represen- tation
(V
N -I (a/A
x ( [(()))()))
(2.8)
==1
==1
VIP then learns a variational distribution gnr by minimizing KT, (gm (F)]lvvr((() x)), optimizing both the variational parameters and the partiality parameters X.
3

---

2.5 Equivalence of VIP in Model vs Variational Spaces
VIP applies TvIP, in the model space. However, there is straightforward analog in the variational space. Suppose (2) is any distribution over the non-centered latent variables 2. We define new distribution VIIVII(2 by letting
(2, with I (2).
(2.9)
In other words, VIP VIP is the distribution induced on 2 by applying the VIP transformation TVIP to samples from qu Concretely, its density follows from the change-of- varriables formula
qu, (I) 9w (Tvip (z, 2)) ddet VETVIP (2,z)1
(2.10)
Since the KL. divergence is invariant under such invertible transformations
we have that
KL ((w,)r(())ll((l)))) = KL (3)]))))))))))))))
(2.11)
Thus, optimizing min KL (4w, vir (=))x))p(((zz)) with the posterior fixed is equivalent to reparame- terizing the posterior in 2.4, In this paper, choose Gaussian distribution for (ho following the original framework [19]. In practice, we can choose any base family gw (2) and still reap the
benefits of partial non-centering. For more details please refer to Algorithm 2 in the Appendix
2.6 Forward Autoregressive Flows
Flows offer powerful way to extend the expressiveness of variational families, thereby reducing the KL divergence between the variational approximation and the true posterior [36] [8] 2 I]. In flows, simple base distribution is gradually transformed into a more complex one through a sequence of invertible mappings. One such transformation the forward autoregressive flow.
Definition 2. A forward autoregressive flow (FAF) is an invertible transformation TEAP(E), which maps a base random variable e = (c) ~ Pe to the target variable 2 = (Z1,..., 2D) via autoregressive transformation
2 m,(=1aa1) + 8;(215-1) G. i i= 1, 1.....,, D,
(2.
where is base distribution, m, is a shift function and s; is scaling function. Furthermore, an affine forward autoregressive flow is one which m, and log s; are affine functions.
In practice, the scaling function is often parameterized in terms of log 8; and subsequently recovered using the exponential function, which ensures positivity; however, alternative nonlinear functions can also be used. Also, multiple layers of the transformation can be composed to build more flexible variational familics.
Iodee--nformedd Flows: Bridging VIP and Autoregressive Flows
We return our central question: Can a flow-based distribution encapsulate the benefits of VIP? The VIP process, when applied in the variational space, involves two key stages: first, sampling an auxiliary variable ō from base variational distribution (ž), and second, transforming é into the latent variable using the VIP transformation, TvIP-
The original VIP framework [19 primarily considers qu (2) to be mean-field Gaussian (i.e., with a diagonal covariance matrix). Our investigation extends this by considering (3) as a full-rank Gaussian.
We first establish that full-rank Gaussian a special case of FAF. full-rank Gaussian variational distribution ~ N(,, T) ) is generated by an affine transformation TA. applied to standard normal variable e (0,
= T=) Lc,
(3.1)
where μ R is the mean vector and L E DxD is lower-triangular Cholesky factor of the covariance matrix. This affine transformation, TA, can be replicated by an affine FAF by defining the shift and scale functions of the FAF (recall Definition 2
m2(2+=11) p4 + Li1--11 L 111-111-111 (ā #1a-1), log (ā log Lis
(3.2)

---

(see Lemma inn Appendix C Thus, full-rank Gaussian distribution can be viewed as a specific instance of an affine FAF.
While a standard affine FAF can represent the initial transformation Ta (generating å from 6), cannot, by itself, fully represent the composite transformation TVIP Ta that yields To see why, observe that the composite transformation reduces to
2 f((m(2))) + (x(s))))) (N= +
Σ Lug Laq (xf(()))))
(3.3)
we attempt to cast this composite transformation into the standard affine FAF form from Equa- tion 2.12 the terms would need to be
m2(21a-1)= 1 (T(())) + (.(((()))) (1s + Σ - - M.f(((2))))
(3.4)
log (z log (FT2))
log Lu
(3.5)
In principle, sufficiently powerful networks in the FAF could represent these forms. However, they could not be captured in an affine FAF due to dependence on e and prior functions. In addition, Equation (3.4) contains the product ge( (z()),, which induces quadratic dependence that an affine mapping cannot represent. These nonlinearities may be difficult to capture even with an FAF that uses, e.g., multi-layer perceptrons. To remedy this, introduce generalized flow.
Definition A generalized forward autoregressive flow (GFAF) is an invertible transformation TGFAF(C), which maps base random variable (C))...,CD) Pe the target variable 2 (21,..., zp) via an autoregressive transformation
2, (2 + s<(21a-1)(c; t,(61.1-1, (14-1)), i= i= D,
(3.6)
where Pe is base distribution, m, is shift funetion, s; is scaling function, and is translation function. Furthermore, an affine generalized forward autoregressive flow is one in which m,, log s,, and t, are affine functions.
Our main theoretical result, stated in Theorem demonstrates how the composite transformation T = TvIp& Ta (a full-rank Gaussian transformation followed by the VIP transformation) can be exactly represented by such generalized forward autoregressive flow.
Theorem 4. Let T = Tvip TA where TvIP is the VIP transformation (Definition 7 and TA is the affine transformation from full-rann Gaussian. If fa and logg; in the hierarchical Bayesian model (Equation (.2 are arbitrary continuous functions, and the parent nodes (1(2) are subset the preceding variables 21:1-1 (respecting the causal dependency structure of the model), then T can be represented affine generalized forward autoregressive flow with
m,(21a-1) fi(m(2)));
(3.7)
(2 = log Lis + (1 - Ar) (T(())
(3.8)
(((e1=-15 Toss
(As((((()))) - pa Li11-1e12-1)
(3.9)
Theorem 4 therefore demonstrates that GFAF, by leveraging its novel translation term & (which depends past noise 61x--1 and by incorporating the model's prior functions, can replicate the full-rank VIP mechanism.
3.1 Model-Informed Flows
Building on the previous ideas, we introduce simple generalization of FAFs designed to capture the VIP mechanism even the conditioning networks are affine.
Definition 5. A Model-Informee Flow (MIF) is an invertible transformation TMuF (e), which
maps a base random variable e = (EI...., ~ Pe to the target variable = = -.1,..., ..i...,,pp) via an autoregressive transformation
z; m,,u,)) sz(u)(Es (()m--1;4)), i= L...., D,
(3.10)
= = f((T(())), (((((z)))
(3.11)
5

---

Algorithm Model-Informed Flow (MIF)
Input: (EI:-.. ".......000 .En). prior functions fi," fn and 91.-.. : 99n (or their logs for scale) & Process pre-defined topological order of latent variables Construct input incorporating prior information
1: for to do
2: "4 +- 1, ((((=(==))))))))))) (o:)), (((()(
3:
m, < NN.(u.) (u:)
P Neural network for the shift function Neural network for the log-scale function Neural network for the translation term with f inputs
4: logs; (
5:
4 f (N,, E1u--1)
6: 2, f m, + exp(logs:) (G-ti)
Generalized autoregressive transformation
7: end for
8: return 2 (Z)
where Pe is base distribution, m, is shift function, 8; a scaling function, and i, a translation term. Furthermore, an Affine Model-Informed Flow is one in which mi. log 8;, and i, are affine functions.
Algorithm D presents the pseudocode for implementing MIF, highlighting in blue the components that extend standard forward autoregressive flow; omitting these reduces MIF to such a standard FAF. The key architectural differences are, first, MIF employs conditioning networks not only for the shift m, and log-scale log 8; but also for the translation term tú with the latter explicitly taking previous noise variables 61:1-1 inputs, motivated by Theorem to incorporate the model's prior functions fa and log 9i as additional inputs to these conditioning networks, providing them with valuable, explicit structural guidance from the target model. Third, the generation process in MIF (the loop over i) must adhere to pre-defined topological order of the latent variables consistent with the hierarchical Bayesian model.
Second, MIF is designed
Before moving on to experiments, we discuss two practical issues.
3.2 Incorporating Prior Information through and log 9i
Our preliminary experiments revealed that omitting explicit prior function inputs (fe, log g:) from MIF often minimally impacted performance. This occurs when these priors are affine, as the target generalized forward autoregressive flow (GFAF) representing full-rank VIP (per Theorem 4 then simplifies to an affine GFAF An MIF, even without these explicit prior inputs, can learn to reproduce this affine GFAF if its own conditioning functions for m,, log 8;, and f; possess at least affine capacity. This is shown in the following corollary.
Corollary Let T TvIP o TA where TvIP is the VIP transformation (see Definition 7 and T' is the affine transformation from full-rank Gaussian. If fa and log gi in the hierarchical Bayesian model (see Equation 2.2 are affine functions, then T be represented an affine generalized forward antoregressive flow.
This corollary shows that when the prior functions are affine, affine MIFs have enough capacity to learn them even if they are not provided, meaning there is no impact on the optimal KL-divergence. Still, in practice providing fi and logg; g helpful in practice to accelerate convergence and reduce training complexity. We suspect the same pattern holds more generally-iff a MIF is trained with high-capacity networks, it may be able to learn to represent the prior functions if not provided, at some extra training cost.
3.3
The Necessity of Conditioning on Previously Generated Latent Variables
Inverse Autoregressive Flows (IAFs) 28 represent an important class of normalizing flows, lauded for the computational efficiency of their sampling process. Unlike FAFs, conditioning networks of IAFs depend solely on the base noise variables 614-1, allowing parallel computation of all z,
2, (1) + s;(Ci=1))
1...,,
(3.12)
This structure makes it challenging to accurately represent the VIP mechanism. To capture VIP, an IAF's conditioning networks would need to replicate target forms that intrinsically depend on previously generated latent variables 21:-1 via the model's prior functions (T(()) and g: (T(2)))

---

Table Negative ELBOs (-ELBO) for hierarchical benchmark models using mean-field Gaussian (MF), mean-field Gaussian with VIP (MF-VIP) full-rank Gaussian (FR), and full-rank Gaussian with VTP (FR-VIP). Lower values indicate tighter posterior approximations.
Model
MF
FR
FR-VIP
8Schools 34.80
31.89
33.85
31.86
Credit
548.65
533,90
535.88
525.02
Funnel
1.86
0.00
1.86
0.00
Radon
1267.99
1215.17
1220.73
1213.52
Movielens
870.93
850.49
856.11
844.51
IRT
823.15
822.58
817.38
816.64
Specifically, they would need to satisfy
m f((((2))) + (T(()) (p+ Σ Lyg (agg->((T(()))) Ar((((2))))
(3.13)
log 81((::11)1 (1)); log + log Lui.
(3.14)
where 2, (and thus :T2:)) the right- hand side themselves complex functions of 61:-11 Asking m. m=(c1=-1) and (c((is-1) to represent this transformation is challenging as it essentially requires the network to reproduce the entire process that produced 21 in previous steps.
Particularly, an IAF with affine conditioning networks for my, log &; (and term like 4, if added) cannot capture the VIP transformation, since #1:i-1 and 61:1-1 have complex non-linear relationship. This limitation underscores the advantage
forward autoregressive structures. We show in our experiments (e.g., results related to Figure 2 that IAFs require substantially greater model capacity to approach the effectiveness of MIF in representing VIP.
4 Related Work
Early flow-based generative models models-inclluing MADE [18), Real NVP [15). Glow (27), and Masked Autoregressive Flow estabbiiidd that autoregressive masking, coupling transforms, and invert- ible 1 Ix convolutions enable tractable likelihoods with strong high-dimensional density estimation. Normalizing flows were subsequently adapted for variational inference, first as post-hoc enrichments of diagonal Gaussians [36] and later as architectures that scale in latent dimensionality. most notably Inverse Autoregressive Flow (IAF) [28]. These developments demonstrated that flexible, invertible parameterizations can substantially tighten the ELBO without sacrificing computational efficiency.
For hierarchical Bayesian models, Variationally Inferred Parameters (VIP) introduced gradient-baat based partial con-centering to alleviate the funnel geometry typical of deep hierarchies [19]. Building on VIP's idea of injecting model structure into the variational family, hybrids such as Automatic Structured Variational Inference (ASVI) [3, Cascading Flows [4], and Embedded-Moode Flows (EMF) 41 couple model-aware parameterizations with flow flexibility. Our Model-Inforne Flow continues this line by proving that full-rank VIP can be represented in forward autoregressive flow augmented with translation and prior inputs, yielding compact yet expressive variational family for hierarchical models.
Experiments
5.1 Experimental Setup
To validate our theoretical results, we evaluate six different hierarchical Bayesian models: 8Schools, German Credit, Funnel, Radon, Movielens, and IRT. These models exhibit vary- ing degrees of funnel-like posterior geometries, making them ideal testbeds for examining the benefits of VIP-inspired flow designs. our final comprehensive benchmark experiments, we also include more models such as Seeds, Sonar, and Ionosphere.
7

---

Table 2: Negative ELBOs (-ELBO) for ablation study comparing affine
Flow (MIF) against variations and baselines across hierarchical models. MIF includes z-conditioning,
translation term, prior info, and correct order. Please refer to Appendix D for more details of variants of MIF. Lower values indicate tighter posterior approximations.
MIF (e-cond)
MIF
MIF (w/o Prior)
MIF (w/o Order)
Model
FR-V VIP
MIF
IAF
(w/o 4)
8Schools 31.86
31.74
32.04
31.86
31.83
33.83
32.22
Credit
525.02
520.72
523.92
525.87
522.75
534.63
523.85
Funnel
0.00
0.01
0.38
0.38
0.38
1.86
0.37
Radon
1213.52
1213.11
1215.11
1214.31
1213.45
1259.13
1215.23
Movielens
844.51
842.91
844.01
846.66
843.59
854.10
844.23
IRT
816.64
815.49
815.70
815.74
815.53
815.58
815.66
For all experiments, we utilize the Adam optimizer [26] and initialize all learnable parameters from standard Gaussian distribution with standard deviation of 0.1. For our Model-Informed Flow (MIF) evaluations, distinct configurations are used: the experiments in Section 5.3 primarily consider single-layer affine MIF to clearly assess its structural components. In subsequent experiments, including the network expressiveness analysis and comprehensive benchmarks, MIF's conditioning networks are implemented multi-layer perceptrons with ReLU activation functions to explore the impact of increased representational capacity via hidden units.
We perform 100,000 optimization iterations, using 256 Monte Carlo samples to approximate the ELBO during training. After training, the final ELBO is evaluated using 100,000 fresh samples for reliable estimation. For each model configuration, six learning rates (logarithmically spaced from 10-1 to 10-6). Additional details regarding the benchmark models, specific variational family configurations, training time comparison, and MLP architectures are provided in Appendix D Our implementation is available at hhttps://githen com/ htt/jooohanko/Moeell. onforrdd-FFl Flow
report the best ELBO achieved after exploring
5.2 Preliminary Observations: Full-Rank VIP Outperforms Baselines
Prior work on VIP focused exclusively mean-field Gaussian approximations 19 To assess whether the advantages of VIP extend to richer variational families, we apply VIP a full-rank Gaus- sian setting and compare four configurations-- -MF, MF-VIP, FR, and FR-VIP-acnde
six canonical hierarchical models. As shown in Table] T FR-VIP achieves the lowest negative ELBO in every case, outperforming both its nIn counterpart and mean-field VIP, and achieving performance comparable to more complex state-of-the-ar art methods (cf. Table These results highlight the power of combining full-rank with VIP and motivate our design of VIP-inspired low architectures.
5.3 Impact of Design Components
In this section, we empirically validate the contribution of MIF's distinct design elements through targeted ablation study, with results presented in Table 2 We compare the affine MIF against variants that individually alter or remove its key theoretically-motiiatee components: the c-dependent translation term (t,). by evaluating MIF (w/o t,): the of explicit prior information, by assessing MIF (w/o Prior) which omits ft, log as inputs; and adherence to the hierarchical variable processing an IAF-like variant, MIF (e-cond), which
order, examined via MIF (w/o Order). Additionally, conditions primarily on base noise 61:-1 instead of previously generated latents 21:1-1, alongside relevant baselines like PRPVIP and a standard IAF. The consistent outcome is that deviating from the full MIF design by removing or altering these components generally degrades performance, underscoring their collective importance its effectiveness.
5.4 Impact of Network Expressiveness
Recall from Section 3.3 that while IAFs face fundamental challenges in representing the VIP mecha- nism due to their conditioning solely on noise (e), it was posited that sufficient network expressiveness might partially overcome these limitations. We empirically investigate this by comparing our full

---

MIF
MIF (e-cond)
Eightt Schools
German Credit
Funne]
-315-
0.0
-500
Best EL ELBO
-31.7
-318-
-0.2
-319
-0.3
-320
-534
Radon
MovielLens
Item Response Theory
1211
-442.0
-513
Best ELBB
1213
-43.
1214
943.5
101
100
10F
10
30F
Number of Hiddon Units
Number of Hidden Units
Nomber of Hidden Units
Figure 2: Best ELBO achieved by full MIF (blue) and the variant without latent-variable conditioning (orange) as function of MLP hidden-unis count. Higher capacity allows the no-latent variant to close the gap, showing that sufficiently expressive networks can implicitly learn dependencies otherwise provided by explicit latent inputs.
MIF, which conditions on previously generated latents 21:-11, against MIF (e-cond), an IAF-like variant where the conditioning networks primarily use 61:1-1 instead of 21:1-1.
Figure 2 illustrates the performance of MIF versus MIF (e-cond) as the expressiveness of their conditioning networks (e.g., by increasing the number of hidden units) is varied. Across most benchmarked models, enhancing network capacity allows MIF (c-cond) to substantially improve its ability capture complex dependencies. Consequently, the performance gap between this IAF-like variant and the standard MIF significantly narrows with increased expressiveness, demonstrating that sufficiently powerful IAF-like architectures can indeed achieve strong results, in many cases closely approaching or nearly matching the performance of MIF, which itself may maintain a slight lead or offer benefits parameter efficiency.
5.5 Comprehensive Evaluation on Hierarchical and Non-hierarchical Benchmarks
To assess its generality and competitive strength, we benchmark MIF with 1024 hidden units per conditioning network, denoted MIF(A 21 against several leading variational inference methods. These baselines, primarily drawn from the recent benchmark by [0] and detailed with their acronyms in the caption of Table 3 provide a robust comparison. The evaluation spans the six previously discussed hierarchical models and three additional datasets (Seeds Sonarí, Ionosphere 1) to test MIF's 's broad applicability. Results in the Table 3 show that MIF(h = 21o) achieves the best or equal-best performance on most of the benchmarks. These strong results, obtained using just single layer of our expressive MIF transformation, demonstrate that its theory-driven design effectively delivers state-of-the-an performance with notable architectural simplicity.
Discussion and Limitation
This paper introduces the Model-Informed Flow (MIF), new architecture for variational inference designed for complex hierarchical Bayesian models. MIF originates from connection between VIP and FAFs. A significant finding is that even single-layer affine MIF can achieve surprisingly competitive performance against other methods. A limitation of MIF its reliance on the FAF structure. Although FAFs may not inherently demand more floating-pott pointt operations (FLOPs) per sample than Inverse Autoregressive Flows (IAFs), with variational inference, IAFs are better suited for modern parallel computing environments. The VIP mechanism appears to be essentially sequential in nature. Thus, our research suggests consideration for flows and VI: to capture difficult posterior geometry may require either sequential processing in MIF or high-capacity conditioning networks.

---

Table Negative ELBOs (-ELBO) for various methods on hierarchical and non man-hiearrchica benchmarks. Compared methods include: Gaussian Mean-Fitn Field (MF) [8), Gaussian Mixture Model VI (GMMVI) [6] Sequential Monte Carlo (SMC) [4), Annealed Flow Transport (AFT) (5), Flow
Annealed IS Bootstrap (FAB)
32 Continual Repeated AFT (CRAFT) 31] and Uncorrected
Hamiltonian Annealing (UHA)
42
Lower values indicate tighter posterior approximations:
denotes results taken from 10]
Model
MF
GMMVI
SMC
AFT
CRAFT
UHA
MIF 240)
8Schools 34,80
31.70
31.80
31.81
31.79
31.90
33.09
31.78
German Credit
548.65
519.35
531.42
527.60
519.39
524.59
536.30
519.29
Funnel
1.86
0.01
0.29
0.30
0.01
0.02
0.38
0.00
Radon
1267.99
1213.67
1221.63
1236,42
1215.92
1229.77
1260.38
1211.83
Movielens
870.93
848.00
849.68
853.84
845.32
850.20
861.22
842.09
IRT
823.15
815.05
820.46
819.56
816.44
818.66
821.84
813.17
Seeds
76.73
73.41
74.69
74.26
73.41
73.79
N/A
72.00
Sonart
137.67
108.72
111.35
110.67
108.59
115.61
N/A
108.85
Ionosphere
123.41
111.83
114.75
113.27
111.67
112.38
N/A
111.85
Acknowledgement
This material is based upon work supported in part by the National Science Foundation under Grant No. 2045900.
10

---

References
[1] A. Agrawal and Domke. Disentangling impact of capacity, objective, batchsize, estimators, and step-size flow vi. arXiv preprint arXiv:2412.08882 2024.
[2] A. Agrawal, D. R. Sheldon, and J. Domke. Advances in black-box vi: Normalizing flows, importance weighting, and optimization. Advances in Neural Information Processing Systems, 13:1735--1736 2020.
[3] L. Ambrogioni, K. Lin, E. Fertig, Vikram, M. Hinne, D. Moore, and M. van Gerven. Automatic structured variational inference. In International Conference on Artificial Intelligence and Statistics, pages 676-684. PMLR, 2021.
[4] L. Ambrogioni, G. Silvestri, and van Gerven. Automatic variational inference with cascading flows. In International Conference on Machine Leurning, pages 254-263. PMLR, 2021.
[5] M. Arbel, A. Matthews, and A. Doucet. Annealed flow transport monte carlo. In International Conference on Machine Learning, pages 318-330, PMLR, 2021.
[6] O. Arenz, P. Dahlinger, Z. Ye, M. Volpp, and G. Neumann. unified perspective on natural gradient variational inference with gaussian mixture models, arXiv preprint arXiv:2209. 11533, 2022.
[7] M. Betancourt and M. Girolami. Hamiltonian monte carlo for hierarchical models. Current trends Bayesian methodology with applications, 79(30):2 (1330)224,40 4, 2015.
[8] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume Springer, 2006.
[9] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, (12(518)8559-877 2017.
[10] D. Blessing, X. Jia, J. Esslinger, F. Vargas, and G. Neumann. Beyond elbos: A large-scale evaluation of variational methods for sampling. arXiv preprint arXiv:2406.074235 2024.
[11] M. Cover and J. Thomas. Elements of information theory. Wiley-Interscience, 1991.
[12] N. Cressie, C. Calder, J. Clark, M. Ver Hoef, and C. K. Wikle. Accounting for uncertainty in ecological analysis: the strengths and limitations of hierarchical statistical modeling. Ecological Applications, 19(3):553-570 2009.
[13] M. Crowder. binomial anova for proportions. Applied statistics, pages 34-37, 1978.
[14] Del Moral, A. Doucet, and A. Jasra. Sequential monte carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, (6(3):11-4366 2006.
[15] L. Dinh, J. Sohk-Dicks, and Bengio. Density estimation using real NVP. arXiv preprint 2017.
[16] F. Dominici, J. M. Samet, and S. L. Zeger. Combining evidence on air pollution and daily mortality from the largest us cities: hierarchical modelling strategy. Journal of the Royal Statistical Society Series A: Statistics in Society, 163(3):263-302 2000.
[17] A. Gelman and J. Hill. Data analysis using regression and multilevel/hierarchhial models. Cambridge university press, 2007.
[18] M. Germain, K. Gregor, Murray, and H. Larochelle. Made:
Masked autoencoder for distri- bution estimation. In International conference on machine learning, pages 881-889, PMLR, 2015.
[19] M. Gorinova, D. Moore, and M. Hoffman. Automatic reparameterisation of probabilistic programs. In International Conference on Machine Learning, pages 3648-366 PMLR, 2020.
[20] R. Gorman and T. Sejnowski. Analysis of hidden units in a layered network trained to classify sonar targets. Neural networks, 1(1):75-89, 1988.

---

[21] F. M. Harper and J. A. Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis). 5(4):1-19, 2015.
[22] M. Hoffman, P. Sountsov, V. Dillon, I. Langmore, D. Tran, and S. Vasudevan. Neutra- lizing bad geometry in hamiltonian monte carlo using neural transport. arXiv preprint arXiv:1903.03704 2019.
[23] M. D. Hoffmann, D. M. Blei, C. Wang, and Paisley, Stochastic variational inference. Journal of Machine Learning Research, 2013.
[24] J. Ingraham and D. Marks. Variational inference for sparse and undirected models. In Interna- lional Conference on Machine Learning, pages 1607-1616. PMLR, 2017.
[25] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37:183-233 1999.
[26] D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:4412.690,, 2014.
[27] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
[28] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved vari- ational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
[29] A. B. Lawson. Bayesian disease mapping: hierarchical modeling in spatial epidemiology. Chapman and Hall/CRC, 2018.
[30] F. M. Lord and M. R. Novick. Statistical theories of mental test scores. IAP, 2008.
[31] A. Matthews, M. Arbel, D. J. Rezende, and A. Doucet. Continual repeated annealed flow transport monte carlo. In International Conference on Machine Learning, pages 15199. PMLR, 2022.
[32] L. I. Midgley, V. Stimper, G. N. Simm, B. Schölkopf, and M. Hernández-Lobesd Flow annealed importance sampling bootstrap. arXiv preprint arXiv:2208.00 0/800, 2022.
[33] R. M. Neal. Slice sampling. The annals of statistics, 31(3):705-767 2003..
[34] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. Advances neural information processing systems, 30, 2017.
[35] O. Papaspiliopoulos, G. O. Roberts, and M. Sköld. A general framework for the parametrization of hierarchical models. Statistical Science, pages 59-73, 2007.
[36] D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1530-1538, 2015.
[37] S. Richardson and N. Best. Bayesian hierarchical models in ecological studies of health- environment effects. Environmetrics: The official journal of the International Environmetries Society, 14(2):129-14) 2003.
[38] J. A. Royle and R. M. Dorazio. Hierarchical modeling and inference in ecology: the analysis of data from populations, metapopulations and communities. Elsevier, 2008
[39] D. B. Rubin. Estimation in parallel randomized experiments. Journal of Educational Statistics, 6(4):377-40) 1981.
[40] V. G. Sigillito, S. P. Wing, L. V. Hutton, and K. B. Baker. Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest, 10(3):262-266, 1989.
[41] G. Silvestri, E. Fertig, D. Moore, and L. Ambrogioni. Embedded- model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling. arXiv preprint 2rXiv:2110.06021 2021.
12

---

[42] A. Thin, N. Kotelevskii, A. Doucet, A. Durmus, E. Moulines, and M. Panov. Monte carlo variational auto-encoders. In International Conference on Machine Learning, pages 10247- 10257. PMLR, 2021.
[43] C. K. Wikle. Hierarchical bayesian models for predicting the spread of ecological processes. Ecology, 84(6):1382-1333 1394, 2003.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 2 defines the scope of the paper and Section gives evidence the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We state possible limitations of MIF compared to inverse autoregressive flows in terms of speed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification:
See Section 3 and the corresponding proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental details in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide the code for the experiments in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Every details experiments is in Experiments section and Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not provide error bars, but the experiment results show average values across different runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All of the experiment details are in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////eerrppp.cpppppbbtttiiia
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We did not state broader impacts since there are no significant impacts we anticipate that need to be described in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
20

---

Supplementary Materials
A
Acronyms and Notations
This section provides summary of acronyms and mathematical notations frequently used throughout the paper to aid the reader.
Table List of Acronyms
Acronym
Definition
CP
Centered Parameterization
ELBO
Evidence Lower Bound
FAF
Forward Autoregressive Flow
FR
Full-Rank (Gaussian)
GFAF
Generalized Forward Autoregressive Flow
IAF
Inverse Autoregressive Flow
KL
Kullback-Leibler (divergence)
MF
Mean-Field (Gaussian)
MIF
Model-Informed Flow
MLP
Multi-Layer Perceptron
NCP
Non-Centered Parameterization
VI
Variational Inference
VIP
Variationally Inferred Parameters
Table 5: List of Notations
Symbol
Definition
Observed data
Latent variables, 2 (21, (21,..., 2D)
D
Dimensionality of the latent space
(2))
Set of parent variables for 2; a hierarchical model
fi())
Prior mean function for i.e., ~ (T()(=())))) Prior standard deviation function for 24, i.c., 2 ~ ((1
g.(-)
Base noise variable, typically ~V0 Non-Centered Parameterization transformation Auxiliary base variable in VIP/CC, T(ž) Partial non-centering parameter for 2; in VIP, X E 00 VIP transformation
TNCP()
Z
A
TvIP()
TAO)
Affine transformation, e.g., + Le for a Gaussian Shift function for z; an autoregressive flow Scale function for z; in an autoregressive flow Translation function for e; in GFAF/MIF
m)))
se(-)
4())
Input to conditioning networks in MIF: [R14-1, f
B Algorithm
Algorithm details the procedure for generating a sample z from the variational distribution VI, VIP and then evaluating its corresponding log-density.
21

---

Algorithm 2 Sampling from and Evaluating (I,)
Input: Parameters w of the base distribution 9u (3); VIP parameters AI,...,Api model prior
functions fz(-). g;()), and parent dependencies ()); dimensionality D; a topological ordering for E1:...;2D)
1: Sample ă ~ qw (2)
2: log f
3: for = 1,..... D do
4: 2, c (TT + gi (T)i)) (
{.f((((z))))
5: log t log J + (1- 2,) log 9 (T(2+)
6: end for
7:
Qw. Tw. (ir.viI(2) t log (3) - log J
8: return 2, log Qu,, (2)
C Proof
Lemma 7. The affine transformation TA, from full-rank VI can be represented as an affine forward autoregressive flow with
m,(21:-11 = 14 + L1,11-1 L 11-1111-11 (1 PL:-1),
log 8; (215-1) = log Lis.
Proof. We begin by expressing the affine transformation TA(c) = pl + Le coordinate wise form:
i-1
24 u4 + Lue + ELyS i 1=....,. D.
(C.1)
#=1
For any forward autoregressive flow to be equivalent to this transformation, need to derive appropriate functions m.(-) and s;(-) such that 2; = m2 m,(21s-1) + $1(21=1)< for each i.
First, observe that for any 2 1, the variables 21:-11 depend only 61:1-1 due to the lower-triangule structure of L. Since -1 is lower-triangular submatrix with positive diagonal entries (as is a Cholesky factor), it is invertible. Thus, we can solve for é1:--1 in terms of 21:-1:
614-1 LT 1 (21--1 -1--1)
(C.2)
Now, substituting this expression into original coordinate-wise equation:
2; = 11s + Listts
Σ
Lajes
(C.3)
f=1
= μs + Lues + Li1N-161--11
(C.4)
14 + Ligts + Ia I 111--11-11 (211-1 1-11-1-21---1 m1--1)
(C.5)
This naturally suggests defining:
m,(21-11)= 14 Li.10-1 T 1 (211-1 p1a-1),
(C.6)
s: (1(21=11) Li--
(C.7)
With these definitions, the forward autoregressive flow transformation becomes:
2; m,(#1:i-1) + s;(21s11)G;
(C.8)
which is identical to our coordinate-vve expansion of the affine transformation TA- Since this holds for all = I,.... D, we have shown that the full-rank VI affine transformation can be exactly forward autoregressive flow with the specified shift and scale functions.
represented
Note that m. (=1--11) is affine function of ø15-11, and s, (=1:--1) is constant function, confirming that this is indeed an affine forward autoregressive flow.
22

---

Theorem 4. Let TvIp o AA where TvIP is the VIP transformation (Definition T7 and T is the affine transformation from full full-rank Gaussian. If f: and logg, 9; in the hierarchical Bayesian model (Equation (..2) are arbitrary continuous functions, and the parent nodes (=2)) are subset of the preceding variables 21:--1 (respecting the causal dependency structure of the model), then T can be represented as affine generalized forward autoregressive flow with
ma(21:i 1) = fi(m(Zz)))
(3.7)
log (2(:i-1) log Lis + (1 - Xa) (ogg(((()))),
(3.8)
(1)11) = Lss (Arf((((zi))
- 4411-1618-11
(3.9)
Proof. The affine transformation TA defined as Ta(c)= = μ + Le, where μ the mean vector and L is a lower- triangular matrix with positive diagonal elements. In coordinate form, this gives us
2
#4
Lug
+
Σ Ligo, 3=1
(C.9)
where & denotes the intermediate result after applying Ta but before applying Tx-
Now, the VIP transformation 72 is defined in Definition D as
2; (T)) + (T(2)) -A= (ž+ {xf((((2))).
(C.10)
where 2 is the output of TA Substituting, get
2 [+(((=)) +x (T((=) (>a >eJ((T(z()))
(C.11)
-i((((2))) + + i4 Lise +
Lijes -hass--Afile (xl((()))
(C.12)
j=1
From our assumption that (2x) C 21:1-1, we know that f((T(2))) and gi(T(2;)) are functions of only the preceding variables 21:1-1-
To express this in the form of a generalized forward autoregressive flow, we need functions m, 8;. and I; such that:
2, = m; (=1x-1) + 81(21:1-1 () (1 1,
(C.13)
Let's rearrange our expression for 2 match this form:
2; + (ps + Lug +
Σ
Log (xf(-((((T(())))
(C.14)
j=1
(((()))))((())))))) Lis Lis(E
1
(-M-
Σ
Lisey + (x((((21)))
(C.15)
Lis
j=l
By identifying terms, define:
m; = fi(T(Zz));
(C.16)
log 8i (2151111) log Lis + (1 >x)) (
(C.17)
t4 t((E11-112
Li (M)f((((2))) pi -1,10-1118-1)
(C.18)
Thus, we have shown that the composite transformation T = Ta o Ta can be represented as generalized forward autoregressive flow with the specified functions 8i, and 4, which completes proof.
23

---

Corollary 6. Tvip o T where Tvip is the VIP transformation (see Definition 7 and TA is the affine transfornation, from full-rank Gaussian. If f, and log gi the hierarchical Bayesian model (see Equation (.2 are affine functions, then T can be represented an affine generalized forward autoregressive flow.
Proof. This result follows directly from Theorem 4 which established that the composite transforma- tion T TvIP o Ta can be represented as a generalized forward autoregressive flow with specific functions m,, and t,. We now show that when f; and log 9i are affine functions, all components of this flow become affine, reducing a standard affine forward autoregressive flow.
Since Si and log 9: are affine functions, we can express them as
[1 = 0; b/ #1:1-1, log (2 = q: + de ø1s-1,
(C.19)
(C.20)
where di, are scalar constants and 6;, d, are vectors of appropriate dimensions.
From Theorem we have
(1(a-11)) = fi(21=-1) = a; + by 214-1,
(C.21)
log (213511 log Lii + (1 (1-b) 2,) log 9 (214-1)
(C.22)
log Lut + (1-b))(c + df 21:-1);
(C.23)
(((15-1, 215-1) =
14 14,14-1 61-11 + Xafi(215-1)
(C.24)
Lii
1
-M; L 618-1 + >1(a; + 67 211-1))
(C.25)
Lii
(-#4 + ).a,)) -
Lu L. 61u-1 1 +
Li 6, 21--1
(C.26)
Lu
Clearly, m; (2 is an affine function of 21:1-1, and log (21x-1) is also an affine function of 21s-11 Furthermore, t4(C1:111, 21:-1) is affine function of both 61:1-1 and 21:--1-
Since my, log 8t, and f; are all affine functions of their respective inputs, this formulation constitutes an affine generalized forward autoregressive flow.
D Experimental Details
All experiments were run on single server with Intel Xeon Platinum 8352Y CPU (128 hardware threads at 2.20 GHz), 512 GiB of RAM. For each experiment (i.e., each training or evaluation run), used one NVIDIA A100 (40 GiB) under CUDA 12.8.
D.1 Models
Eight Schools The Eight Schools model 39 estimates treatment effects 0 for Nachools 8 schools, given observed effects y; and known standard errors de. log ~ NO, 5) Vi ~ N(,,, i= 1, 1...., Nachools-
5), 0. (14, exp(logT))
(D.1)
(D.2)
Latent variables: = = log 01,.... ONassss
German Credit A logistic regression model 11 with Nectures K predictors Wikk for Nots individuals, predicting binary outcomes yn. uses hierarchical priors on coefficient scales Tk.
log ~ N0 10)
(D.3)
log T (log To. 1). Bx ~ (0, exp(logT)))
k 1 K
(D.4)
k = 1 1,...,A K
(D.5)
Tho Sranki
1,..., N
(D.6)
Vn ~ Bernoulli(1/(1 + exp(-1m)))
12 1 N obs.
(D.7)
24

---

Latent variables: = (log To, (log ( R==1 {8t]k=]
Funnel Neal's Funnel distribution 33 in D dimensions for x (I1,...,ID))
21 3))
(D.8)
3k (0,00 exp(sn/2),
h: == 2.......
(D.9)
All variables 21,...;ED latent. (The variance for x1 is 3 9; the variance for 24, k 2 is cxp(x1).)
Radon A hierarchical linear regression 17 modeling log- log-radon levels log in Nnomes homes. Data includes z, (floor indicator for home j). ug (uranium reading for county k), and Cy (county for home j). Priors are inferred from the provided code structure.
Po ~ NN 1)
(D.10)
~ ((0,)) 1)
(D.11)
10,
(D.12)
log Jm. ~ NN(00, 10).
1: 1,
N counties
(D.13)
log oy ~ NN(0, 10)
(D.14)
my ~ NNoo + a uk. exp(log mz )),
k: = 1,
N Nountics
(D.15)
logr, [me, + b 2y exp(log oy)),
1..... homess
(D.16)
Latent variables: z = (Po, a, b, (log mm Naunia log y: {m2
Nomni k=
Movielens
The Movielens model [21 is a hierarchical logistic regression for predicting movic ratings. Each rating n is given by a user Un E {I...., M) for a movie described by a binary attribute vector x, E {0, 1 (with D 18 attributes). The rating is yn E {0, 1}.
.....,,M,
Each user m has an attribute preference vector Zm (Zm.1.... D) E R.D. The model is:
Mg N(O, 1), A, N),,
I,...,D D
(D.17)
1,.... D
(D.18)
Zm. ((H,,
m = 1, M; = 1..,,,
D
(D.19)
Un Bernoulli (sigmoid (x,, Zo,)),
N.
(D.20)
Here, (4 the mean preference for attribute j across users, and Xy is the log standard deviation of preferences for attribute j. Latent variables: 2 ((15) (2,} (Zm. M.D
m==1j==)
IRT
The Item Response Theory (IRT) two-parameter logistic (2PL) model 30 is used to model
student responses to test items. Let N's be the number of students, No be the number of questions (items), and NR be the total number of responses. For each response r, 8p the student ID and gr is the question ID.
a, 1).
1. .....NN
(D.21)
My 1)
(D.22)
log ~ 1)
(D.23)
log 0 1)(0,1)
(D.24)
3, "N(uss exp(log
=
NQ
(D.25)
log 74 NO00, exp(log
NQ
(D.26)
logit, exp(log 74. )a. + Bg.,,
Nn
(D.27)
3r (ernoull((ignooic((ooi,,
NR-
(D.28)
Here, as the ability of student 8, 3, the easiness question q, and exp 7a) is its discrimina- Nq. tion. Latent variables: ({a,).N Ma, log σg, log o., (8, | (log ))
25

---

Seeds The Seeds model 13 is random effects logistic regression for analyzing seed germination data. models the number of germinated seeds r; out of N; seeds in G 21 experimental groups. using group-specific predictors 21, and 22,4-
T 1aamma(0.01.) 0.01)
(D.29)
ao, a1 02, 012 (0, 10)
(D.30)
6: N' (0.1/VT),
i 1, ..... G
(D.31)
7/ + 01 + 02 + 01231,142, + bi, Ti ~ Binomial (Ni, sigmoid(1)))
i
G
(D.32)
G.
(D.33)
Here, 11, and 22,4 indicator variables for experimental conditions (c.g.. seed type and root extract). is the precision for the random effects b,. Latent variables: (T, do, 01; d2, 112 {6: (b:)(=))
Sonar The Sonar model 20 a Bayesian logistic regression. It uses = features/coefficients x to classify N = 208 sonar returns My into two categories E 1 (e.g., mines vs. rocks).
N' (0,
(D.34)
y/ Bernoulli (sigmoid (x Txuu u.)) i= i
(D.35)
Here, u, E IRP the feature vector for the i-th observation, Ip the x D identity matrix, and o is predefined prior variance for the coefficients x. The latent variables are 2 (x).
The Ionosphere model [40] is a Bayesian logistic regression. It uses D = 35 fea- tures/coefficients to classify N 351 radar returns u, from the ionosphere into two categories y: E {).
lonosphere
x N (0,0,,1))
(D.36)
y: ~ Bernoulli (sigmoid (x [u)),,,, i i= I.
(D.37)
Here, u, E IR is the feature vector for the i-th observation, ID is the D D identity matrix, and of a predefined prior variance for the coefficients x. The latent variables are 2 = (x).
D.2 Implementation Details
The conditioning networks for the shift (m,), log-scale (log s;), and translation (t,, for MIF and its variants) terms in both forward autoregressive flows (FAF) and inverse autoregressive flows (IAF) are constructed follows, depending the experimental context.
For affine flows, such as those used in the ablation studies Section 5.3 each conditioning network (for m,, log 8;, or t.) consists of single linear layer. This layer directly maps the conditioning inputs the required output parameters.
For flows requiring greater representational capacity, specifically in experiments evaluating network expressiveness (Figure and for the comprehensive benchmarks (T the conditioning networks are implemented using a two-component MLP-based architecture. Each parameter (m, log 8;, or 4;) computed by summing the outputs of two parallel components, both of which operate on the same input conditioning variables:
1. A linear component:
This is direct linear transformation of the inputs, identical in architecture to the conditioners used in the affine flows.
2. An MLP component: layer with ReLU activation functions, followed by linear output layer. The size of the hidden layer (number of hidden units) varied in experiments that assess the impact of network capacity (e.g., from 20 to 20 as illustrated in Figure 2).
This is multi-layer perceptron (MLP) featuring a single hidden
The final value for m. log st, or t; is the sum of the outputs derived from these linear and MLP components.
For experiments involving baseline models from Blessing et 10 we utilized the implementations provided in their publicly available code repository to ensure consistency and fair comparison.
26

---

Table 6: Training time comparisons (seconds) between the default Model-Informed Flow (MIF) and
variant that conditions on the base noise ("MIF with c").
Model
D
Hidden Units
MIF (s)
MIF with (s)
17.06
13.88
64
28.40
23.22
Eight Schools
10
256
28.04
22.18
512
28.21
22.03
1024
30.30
22.61
18.23
14.70
64
27.38
23.16
Funnel
10
256
27.04
22.42
512
28.57
22.30
1024
30.77
22.69
63.81
16.06
64
148.73
27.81
German Credit
125
256
208.44
54.92
512
275.74
93.26
1024
417.05
173.51
0
84.25
15.78
64
238.04
37.32
Radon
174
256
331.24
94.82
512
453.70
168.28
1024
699.73
338.63
72.26
17.59
64
196.64
33.15
IRT
143
256
253.17
67.95
512
457.56
119.95
1024
516.98
222.65
458.28
28.97
64
1981.14
384.45
MovieLens
882
256
4847.2 23
1387.64
512
8634.91
3156.82
1024
873.444
6749.33
D.3 Training Time Comparisons
report wall-clock training times (in seconds) across six benchmarks and varying hidden sizes. comperss our default Model-Informed Flow (MIF) against variant that conditions on the base noise M"MI with e"), while Table 7 uses (m, t) and "2 networks (no t)" uses only (m, s). Overall, "MIF with e" is consistently faster than the default MIF at comparable capacities, and removing the translation network (t) yields additional speedups with similar trends across datasets. We use D to denote latent dimensionality.
number of conditioning subnetworks: "3 networks"
27

---

Table 7: Ablation on the number of conditioning subnetworks (seconds). "3 networks" uses shift, scale, and translation (m, 8, (). "2 networks (no ()" removes the translation network and uses only (m, 8) under the same training setup.
Model
D
Hidden Units
Networks (s)
2 Networks (no 2 (8)
17.06
16.98
64
28.40
24.13
Eight Schools
10
256
28.04
24.81
512
28.21
25.80
1024
30.30
25.76
0
18.23
18.63
64
27.38
25.89
Funnel
10
256
27.04
25.85
512
28.57
26.27
1024
30.77
26.01
63.81
52.68
64
148.73
115.39
German Credit
125
256
208.44
139.90
512
275.74
175.94
1024
417.05
258.98
0
84.25
77. 77.52
64
238.04
220.17
Radon
174
256
331.24
306.44
512
453.70
419.42
1024
699.73
647.75
0
72.26
66.73
64
196.64
181.92
IRT
143
256
253.17
234.43
512
457.56
423.00
1024
516.98
478.45
458.28
423.65
64
1981.14
1832.85
MovieLens
882
256
4847.23
4486.69
512
8634.91
7992.34
1024
15 873.45
687.99
28


Reviews:
[{"review_id": "VuVEC0UjTN", "reviewer": "Reviewer_Z9uq", "summary": "This paper introduces model-informed flows (MIF), a novel autoregressive normalizing flow formulation that extends masked autoregressive flows/forward autoregressive flows (FAF) by (i) introducing an additional translation term that depends on the previous noise variables; and (ii) introducing the model’s prior distribution as additional inputs to the conditioner. It is shown that this is sufficient to represent useful variational posteriors, in particular distributions given by multivariate Gaussians transformed through a (partial) non-centering transform, whereas FAF/IAF may not be able to represent/learn these posteriors effectively.", "questions": "Questions:\n- It seems from the formulation in Definition 3 that one could alternatively absorb the (scaled) translation term into the shift function $m$, by making $m$ dependent on $\\epsilon$ in addition to $z$. Is there any reason why an explicit translation function is preferred? \n- In Figure 2, the Funnel plot seems to show that the performance of MIF and MIF ($\\epsilon$-cond) are similar for larger hidden sizes, but Table 2 shows a significant difference. Is this because Table 2 uses smaller hidden sizes?\n- It’s a little unclear to me if the conditioning on prior distributions is a novel contribution of of this paper or if it has previously been used in other works on NFs for inference in probabilistic programs/hierarchical Bayesian models. Could the authors clarify this?\n- Could the authors provide a citation for the term “forward autoregressive flow”? It appears to be essentially the masked autoregressive flow of (Papamakarios et al. 2017).", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper makes some interesting insights into how known limitations of simple variational families/inference strategies (such as the funnel problem) are solved by using sufficiently expressive normalizing flow families. In particular, showing that multivariate Gaussians under a variationally inferred parameterization is sufficient for many existing models is a nice result, along with its representation as a closed-form MIF. Empirically, the evaluation is thorough and shows that MIF performs competitively with or outperforms existing inference approaches. The authors also conduct interesting ablation studies to show the effects of each individual component of the MIF, including showing that MIFs do indeed outperform IAF-like formulations with limited NN capacity due to the superior inductive bias.\n\nAs far as weaknesses, it’s unclear if the full MIF is currently a practical model family due to its sampling time scaling in variable dimensionality compared to the parallel IAF-like families (while also losing efficient likelihood evaluation compared to FAFs). The paper could also do a better job in comparing and contrasting with related work in its presentation, in particular providing a more detailed technical comparison to the structured or NF-based variational approaches to inference in probabilistic programs, to clarify the novel contributions of the paper in context.", "was_revised": true, "final_justification": "I recommend acceptance of the paper based on the novel insights of the paper on the connection between affine autoregressive flows and Gaussian VIPs. The rebuttal has adequately clarified my questions about the technical formulation (affine) and positioning of the work.", "rebuttal": "Thank you for your thorough review and valuable feedback on our manuscript. Below, we address each of your comments in turn. If anything remains unclear, we would be grateful for any further questions or clarifications.\n\n> (Weakness 1) it’s unclear if the full MIF is currently a practical model family due to its sampling time scaling in variable dimensionality compared to the parallel IAF-like families\n\nWe have measured the computational cost compared to IAF, please see the following table below.\n\n| **Model**       | **D** | **Hidden Units** | **MIF (s)** | **IAF (or MIF with ε-cond) (s)** |\n|:---------------:|:-----:|:---------------:|:-----------:|:--------------------------------:|\n| **Eight Schools**    |  10   |       0         |    17.06    |             13.88               |\n|                 |       |      64         |    28.40    |             23.22               |\n|                 |       |      256        |    28.04    |             22.18               |\n|                 |       |      512        |    28.21    |             22.03               |\n|                 |       |     1024        |    30.30    |             22.61               |\n| **Funnel** |  10   |       0         |    18.23    |             14.70               |\n|                 |       |      64         |    27.38    |             23.16               |\n|                 |       |      256        |    27.04    |             22.42               |\n|                 |       |      512        |    28.57    |             22.30               |\n|                 |       |     1024        |    30.77    |             22.69               |\n| **German Credit** | 125 |       0         |    63.81    |             16.06               |\n|                 |       |      64         |   148.73    |             27.81               |\n|                 |       |      256        |   208.44    |             54.92               |\n|                 |       |      512        |   275.74    |             93.26               |\n|                 |       |     1024        |   417.05    |            173.51               |\n| **Radon**      |  174  |       0         |    84.25    |             15.78               |\n|                 |       |      64         |   238.04    |             37.32               |\n|                 |       |      256        |   331.24    |             94.82               |\n|                 |       |      512        |   453.70    |            168.28               |\n|                 |       |     1024        |   699.73    |            338.63               |\n| **IRT** |  143  |       0         |    72.26    |             17.59               |\n|                 |       |      64         |   196.64    |             33.15               |\n|                 |       |      256        |   253.17    |             67.95               |\n|                 |       |      512        |   457.56    |            119.95               |\n|                 |       |     1024        |   516.98    |            222.65               |\n| **MovieLens** | 882 |    0         |   458.28    |             28.97               |\n|                 |       |      64         |  1981.14    |            384.45               |\n|                 |       |      256        |  4847.23    |           1387.64               |\n|                 |       |      512        |  8634.91    |           3156.82               |\n|                 |       |     1024        | 15873.45    |           6749.33               |\n\n\nWe note that MIF imposes little extra computational cost in terms of flops, but the sequential nature of sampling does impose some limitations on parallelization. However, as the above table shows, this is usually quite modest. Given the large improvements in accuracy, we believe there are many situations where it would be worth paying this cost.\n\nMoreover, the primary goal of our work is to expose the previously unexplored relationship between flow‑based models and variational inferred parameters (VIP), and to introduce MIF as a family that captures all the benefits revealed by this connection. As discussed in Sections 4.4 and 5, there is a clear trade‑off between MIF and IAF‑like families:\n\n- When network capacity is limited (e.g., few hidden units), MIF often outperforms IAF‑style flows, as shown in Figure 2.\n- When ample capacity is available and fast sampling is a priority, the $\\epsilon$‑conditioned variant of MIF (our IAF‑like family) is preferable.\n\nMoreover, even if one opts for an IAF‑style architecture, it remains beneficial to incorporate the other key components from our VIP analysis into the variational family.\n\n> (Question 1) It seems from the formulation in Definition 3 that one could alternatively absorb the (scaled) translation term into the shift function $m$, by making $m$ dependent on $\\epsilon$ in addition to $z$. Is there any reason why an explicit translation function is preferred?\n\n\nAn affine FAF (i.e. one where both m and log s are affine functions, see Definition 2) cannot capture the composite mapping $T = T_{\\mathrm{VIP}} \\circ T_A$ (Equation 3.3) without an explicit translation term (please see Section 3, just before Definition 3). Concretely, even if we allow the shift function $m$ to depend on both $\\epsilon$ and $z$, it still cannot reproduce the quadratic dependence on functions of $z$ required by Equation 3.4. By introducing a separate translation function $t$, we can match that quadratic form exactly and thus express $T = T_{\\mathrm{VIP}} \\circ T_A$ as an affine generalized FAF, as proven in Theorem 4. Although it is possible to omit the translation term when there are no constraints on the shift and log‑scale functions, our experiments show that including it consistently improves performance.\n\n\n> (Question 2) In Figure 2, the Funnel plot seems to show that the performance of MIF and MIF ( $\\epsilon$-cond) are similar for larger hidden sizes, but Table 2 shows a significant difference. Is this because Table 2 uses smaller hidden sizes?\n\nApologies for the confusion—Table 2 was indeed run with smaller hidden sizes than Figure 2, which explains the larger performance gap.\n\n> (Question 3) It's a little unclear to me if the conditioning on prior distributions is a novel contribution of this paper or if it has previously been used in other works on NFs for inference in probabilistic programs/hierarchical Bayesian models. Could the authors clarify this? \n\n\nThe idea of conditioning on prior distributions has indeed been used in the VIP paper, albeit only in concert with a mean-field Gaussian variational distribution (Section 2.4). As we stated above, the main goal of our work is to analyze the previously unexplored relationship between flow‑based models and VIP, which we found helps address sampling from posteriors with challenging curvature. Previous works on normalizing flows for inference have typically focused on creating more expressive variational families and improving efficiency. Our method is novel in that we explicitly incorporate not only prior information but also the other key components from our analysis to build a variational family that is proven to perform well on posteriors with funnel‑like geometries. We will expand the Related Work section in the final version of the manuscript to compare in detail with prior NF‑based inference methods and highlight our distinct contributions. \n\n\n> (Question 4) Could the authors provide a citation for the term \"forward autoregressive flow\"? It appears to be essentially the masked autoregressive flow of (Papamakarios et al. 2017).\n\nApologies for the omission. Indeed, by \"forward autoregressive flow\", we mean to refer to the work of Papamakarios et al. (2017). While they refer to this simply as an \"autoregressive flow\" we add the term \"forward\" to contrast more clearly with \"inverse autoregressive flows\". To make this explicit, we will add this clarification after Definition 2 in the revised version. Also, please note that our Definition 2 coincides with the autoregressive flow formulation given in Section 3.2 of Papamakarios et al. (2017).\n\nG. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. *Advances in neural information processing systems*, 30, 2017."}, {"review_id": "xNecNswc3W", "reviewer": "Reviewer_DgS9", "summary": "This paper presents Model Informed Flows (MIF) for Bayesian inference. In Bayesian hierarchical models, the dependence structures between latent variables can lead to geometries in the posterior distribution that are difficult to explore by standard variational families. This paper leverages the idea of modelling non-centered latent variables in Variationally Inferred Parameters (VIP) and embeds it in a Gaussian full-rank autoregressive flow, which leads to MIF. MIF is shown to achieve state-of-the-art performance or better across a wide range of hierarchical and non-hierarchical models.", "questions": "* The experiment section mostly contains comparisons to mean-field Gaussian VI, full-rank Gaussian VI, and VIPs. However, in the related work section, several other related algorithms are brought up (i.e. ASVI, Cascading Flows, EMF). However, these are not included in the comparison. Can the authors comment on, if any, the difference in performance that they expect relative to MIF?\n* In Algorithm 1, we see that MIF involves training three neural networks. This could be quite computationally heavy. Can the authors comment on the relative computational cost between MIF and other competing methods?\n* The MIF structure seems rather flexible, and I wonder if it could be helpful for problems with other interesting structure/geometry (i.e. multi-modal posteriors). Can the authors comment on this a bit on what they think?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper is very well and clearly written. How the funnel type geometry arises in hierarchical models is very intuitively described. The introduction of VIPs and its connections to autoregressive flows are easy to follow, even for someone likely myself who is learning about the VIPs for the first time.", "was_revised": true, "final_justification": "The authors addressed my questions and concerns. And I am keeping my positive score.", "rebuttal": "Thank you for your thorough review and valuable feedback on our manuscript. Below, we address each of your comments in turn. If anything remains unclear, we would be grateful for any further questions or clarifications.\n\n\n>The experiment section mostly contains comparisons to mean-field Gaussian VI, full-rank Gaussian VI, and VIPs. However, in the related work section, several other related algorithms are brought up (i.e. ASVI, Cascading Flows, EMF). However, these are not included in the comparison. Can the authors comment on, if any, the difference in performance that they expect relative to MIF?\n\nPlease refer to Table 3, this presents a broad set of comparisons: Gaussian Mean‑Field VI (MF, [1]), Gaussian Mixture Model VI (GMMVI, [2]), Sequential Monte Carlo (SMC, [3]), Annealed Flow Transport (AFT, [4]), Flow Annealed Importance‑Sampling Bootstrap (FAB, [5]), Continual Repeated AFT (CRAFT, [6]), and Uncorrected Hamiltonian Annealing (UHA, [7]). We chose these baselines because they are the most closely related to our approach and were identified as top performers in the benchmark study of [8]. While we believe this suite of experiments already demonstrates the significance of our algorithmic contributions, if the reviewer would like to see comparisons to any other particular algorithms, we would be happy to include these (even during the discussion period, if possible).\n\n>In Algorithm 1, we see that MIF involves training three neural networks. This could be quite computationally heavy. Can the authors comment on the relative computational cost between MIF and other competing methods?\n\nIn practice, adding the third conditioning network has minimal impact on runtime. It introduces only an $O(D)$-sized parameter block-negligible relative to the overall model-and the dominant cost remains evaluating the log-density, which all autoregressive flows share. As a result, per-iteration runtimes for MIF are effectively unchanged compared to two-network variants or other flow-based VI methods. Below, we present the wall-clock comparison between MIF with two versus three conditioning networks, demonstrating that the extra network adds negligible overhead while enhancing expressiveness.\n\n| **Model**           | **D** | **Hidden Units** | **3 Networks (s)** | **2 Networks (s)** |\n|:-------------------:|:-----:|:---------------:|:-----------------:|:-----------------:|\n| **Eight Schools**        |  10   |        0        |       17.06       |       16.98       |\n|                     |       |       64        |       28.40       |       24.13       |\n|                     |       |      256        |       28.04       |       24.81       |\n|                     |       |      512        |       28.21       |       25.80       |\n|                     |       |     1024        |       30.30       |       25.76       |\n| **Funnel**     |  10   |        0        |       18.23       |       18.63       |\n|                     |       |       64        |       27.38       |       25.89       |\n|                     |       |      256        |       27.04       |       25.85       |\n|                     |       |      512        |       28.57       |       26.27       |\n|                     |       |     1024        |       30.77       |       26.01       |\n| **German Credit**   |  125  |        0        |       63.81       |       52.68       |\n|                     |       |       64        |      148.73       |      115.39       |\n|                     |       |      256        |      208.44       |      139.90       |\n|                     |       |      512        |      275.74       |      175.94       |\n|                     |       |     1024        |      417.05       |      258.98       |\n| **Radon**          |  174  |        0        |       84.25       |       77.52       |\n|                     |       |       64        |      238.04       |      220.17       |\n|                     |       |      256        |      331.24       |      306.44       |\n|                     |       |      512        |      453.70       |      419.42       |\n|                     |       |     1024        |      699.73       |      647.75       |\n| **IRT**     |  143  |        0        |       72.26       |       66.73       |\n|                     |       |       64        |      196.64       |      181.92       |\n|                     |       |      256        |      253.17       |      234.43       |\n|                     |       |      512        |      457.56       |      423.00       |\n|                     |       |     1024        |      516.98       |      478.45       |\n| **MovieLens** |  882  |        0        |      458.28       |      423.65       |\n|                     |       |       64        |     1981.14       |     1832.85       |\n|                     |       |      256        |     4847.23       |     4486.69       |\n|                     |       |      512        |     8634.91       |     7992.34       |\n|                     |       |     1024        |    15873.45       |    14687.99       |\n\n\nWe will add these measurements to the appendix of the paper.\n\n>The MIF structure seems rather flexible, and I wonder if it could be helpful for problems with other interesting structure/geometry (i.e. multi-modal posteriors). Can the authors comment on this a bit on what they think?\n\n\nThank you for the question. MIF is inherently flexible: it extends autoregressive flows with an architecture already proven to sample effectively in funnel-like geometries, and because autoregressive flows capture complex, multimodal posterior shapes we expect MIF to perform well on other challenging structures too. Looking ahead, we plan to study tree-structured hierarchical models where independent branches can be sampled in parallel, thereby reducing—or even removing—the sequential sampling cost and further speeding up inference.\n\n\n[1] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.\n\n[2] O. Arenz, P. Dahlinger, Z. Ye, M. Volpp, and G. Neumann. A unified perspective on natural gradient variational inference with gaussian mixture models. arXiv preprint arXiv:2209.11533, 2022.\n\n[3] P. Del Moral, A. Doucet, and A. Jasra. Sequential monte carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3):411-436, 2006.\n\n[4] M. Arbel, A. Matthews, and A. Doucet. Annealed flow transport monte carlo. In International Conference on Machine Learning, pages 318-330. PMLR, 2021.\n\n[5] L. I. Midgley, V. Stimper, G. N. Simm, B. Schölkopf, and J. M. Hernández-Lobato. Flow annealed importance sampling bootstrap. arXiv preprint arXiv:2208.01893, 2022.\n\n[6] A. Matthews, M. Arbel, D. J. Rezende, and A. Doucet. Continual repeated annealed flow transport monte carlo. In International Conference on Machine Learning, pages 15196-15219. PMLR, 2022.\n\n[7] A. Thin, N. Kotelevskii, A. Doucet, A. Durmus, E. Moulines, and M. Panov. Monte carlo variational auto-encoders. In International Conference on Machine Learning, pages 1024710257. PMLR, 2021.\n\n[8] D. Blessing, X. Jia, J. Esslinger, F. Vargas, and G. Neumann. Beyond elbos: A large-scale evaluation of variational methods for sampling. arXiv preprint arXiv:2406.07423, 2024."}, {"review_id": "iyD8VCI9Ws", "reviewer": "Reviewer_FEHh", "summary": "This works introduces Model-Informed Flows (MIF), a new VI architecture aimed at addressing the challenges posed by hierarchical Bayesian models, which often exhibit complex posterior geometries. The authors demonstrate theoretically that a combination of a full-rank Gaussian and Variationally Inferred Parameters (VIP) can be exactly represented by a *generalised* forward-autoregressive flow. Their proposal generalises FAF by including model prior information. The authors then conduct various investigations to validate the performance of MIF including ablation studies.", "questions": "Kindly see *Weaknesses* section above. I have detailed my concerns there and would consider increasing my score if the authors address some of those points.", "limitations": "The authors should have commented on the runtime performance. It is important that they elucidate to the reader the differences between the runtimes/latency of MIF vs IAF", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strenghts**\n- Sound theoretical foundation has been provided by the authors. I am convinced that this work is well motivated.\n- The architecture introduced in this work is novel to the best of my knowledge and the authors have shown clear intent to test it's performance across different benchmarks.\n- The work is clearly presented and well written.\n\n**Weaknesses**\n- The inherent sequential nature of MIF limits it's scalability and practicality. This has been recognised by the authors but it'd would been nice to see them hint towards resolutions for this major limitation.\n- The evaluations, though seemingly thorough, lack error bars or standard deviations. The numbers presented in table 1, for example, are not very conclusive as there are minute differences b/w MF-VIP and FR-VIP. Similarly I cannot ascertain the value of different components of MIF since all the numbers are very close to each other. This, in my opinion, is another major limitation.\n- For Figure 2, why have the authors not included any other variational approximations here? There are several complex and highly-expressive variational approximations that could've been considered here as baselines. I can read that they intended to compare this with IAF (Line 204) but this result is not present.\n- There are no runtime comparisons. Since IAF and FAF have difference in their ability to capture VIP but the one's faster than the other, the authors should have considered an accuracy to runtime comparison for these two methods.\n\n\n**Minor**\nWhy did the authors choose to label the y-axis in Figure 2 as 'Best Elbo'? The notion of 'best' is very ambiguous and inconsistent from the rest of the paper where the authors quote Negative ELBO.", "was_revised": true, "final_justification": "I am satisfied with the extra evaluations the authors have conducted during the rebuttal period. I maintain that this is a novel work and is of interest to the community.", "rebuttal": "Thank you for your thorough review and valuable feedback on our manuscript. Below, we address each of your comments in turn. If anything remains unclear, we would be grateful for any further questions or clarifications.\n\n> (Weakness 1) The inherent sequential nature of MIF limits it's scalability and practicality. This has been recognised by the authors but it'd would been nice to see them hint towards resolutions for this major limitation.\n\nFunnel geometries are notoriously difficult for any inference method, and while MIF's sequential updates do add some non-parallelisable cost, the overhead is modest in practice (see the runtime table below). Looking forward, we plan to exploit conditional independencies among latent variables-thereby reducing the $D$-dependence of forward autoregressiveness while preserving the MIF-derived benefits. For example, in a tree-structured hierarchy $p(\\theta, z, x)$ with global latent $\\theta$, all local factors $q(z|\\theta)$ can be sampled in parallel. We leave a full exploration of these parallel schemes to future work.\n\n> (Weakness 2) The evaluations, though seemingly thorough, lack error bars or standard deviations. The numbers presented in table 1, for example, are not very conclusive as there are minute differences b/w MF-VIP and FR-VIP. Similarly I cannot ascertain the value of different components of MIF since all the numbers are very close to each other. This, in my opinion, is another major limitation.\n\nThank you for raising this point. We agree that quantifying variability is crucial, so we've added standard deviations to Table 1 in this rebuttal (based on five independent runs) and will include the full error bars in the final manuscript. As in the manuscript, for each run, we perform 100,000 optimization iterations using 256 Monte Carlo samples to approximate the ELBO during training, then evaluate the final ELBO with 100,000 fresh samples. We also sweep six learning rates (log-spaced from $10^{-1}$ to $10^{-6}$ ) and report the best result. \n\n| Model      | MF (mean ± sd)    | MF-VIP (mean ± sd) | FR (mean ± sd)   | FR-VIP (mean ± sd) |\n|:----------:|:-----------------:|:------------------:|:----------------:|:------------------:|\n| 8Schools   |   34.75 ± 0.85    |    31.90 ± 0.53    |   33.82 ± 0.94   |    31.85 ± 0.40    |\n| Credit     |  549.20 ± 1.20    |   534.10 ± 0.89    |  535.20 ± 1.15   |   525.05 ± 0.52    |\n| Funnel     |    1.86 ± 0.10    |     0.00 ± 0.00    |    1.86 ± 0.10   |     0.00 ± 0.00    |\n| Radon      | 1266.80 ± 1.35    |   1215.02 ± 0.77   | 1221.11 ± 1.95   |   1213.05 ± 0.51   |\n| Movielens  |  871.61 ± 1.20    |    851.00 ± 0.95   |  855.93 ± 1.19   |    844.35 ± 0.85   |\n| IRT        |  822.95 ± 0.95    |    822.05 ± 0.53   |  817.35 ± 0.59   |    816.35 ± 0.50   |\n\nWe will add these to the revised version of the paper.\n\n> (Weakness 3) For Figure 2, why have the authors not included any other variational approximations here? There are several complex and highly-expressive variational approximations that could've been considered here as baselines. I can read that they intended to compare this with IAF (Line 204) but this result is not present.\n\nWe apologize for the confusion but MIF with $\\epsilon$-conditioning is in fact just an inverse autoregressive flow-by removing forward dependence on previous latents. We will make sure to clarify this in the revised version of the paper. The sole aim of Figure 2 is to isolate and compare $z$ -conditioning versus $\\epsilon$-conditioning under varying network capacity. As expected, $z$ -conditioning ensures accurate sampling in funnel-like geometries even with simple (affine) nets, while $\\epsilon$-conditioning only matches that performance once the nets become expressive enough to implicitly learn the same dependencies. We therefore restrict Figure 2 to these two closest algorithmic variants; broader comparisons against other highly expressive VI families are provided later (e.g., Table 3 and the review-paper benchmarks).\n\n> (Weakness 4 & Limitation 1) There are no runtime comparisons. Since IAF and FAF have difference in their ability to capture VIP but the one's faster than the other, the authors should have considered an accuracy to runtime comparison for these two methods. / The authors should have commented on the runtime performance. It is important that they elucidate to the reader the differences between the runtimes/latency of MIF vs IAF\n\n\nApologies for not including runtime comparisons earlier. As Algorithm 1 shows, vanilla MIF and its $\\epsilon$-conditioning variant (i.e., IAF) require essentially the same total FLOPs, but vanilla MIF executes them sequentially-incurring an $O(D)$ wall-clock-whereas IAF can launch those operations in parallel and achieve $O(1)$ wall-clock per sample. This reflects an explicit trade-off: full MIF guarantees accurate sampling in funnel-like geometries even with shallow conditioning networks, while $\\epsilon$-conditioning achieves constant-time complexity at the cost of needing higher-capacity nets to implicitly recover those dependencies. In response to your suggestion, we have now measured wall-clock runtimes for both variants on our benchmark models and will include a comprehensive runtime comparison in the final manuscript. Below is the runtime table where $D$ is the number of latent variable:\n\n\n| **Model**       | **D** | **Hidden Units** | **MIF (s)** | **IAF (or MIF with ε-cond) (s)** |\n|:---------------:|:-----:|:---------------:|:-----------:|:--------------------------------:|\n| **Eight Schools**    |  10   |       0         |    17.06    |             13.88               |\n|                 |       |      64         |    28.40    |             23.22               |\n|                 |       |      256        |    28.04    |             22.18               |\n|                 |       |      512        |    28.21    |             22.03               |\n|                 |       |     1024        |    30.30    |             22.61               |\n| **Funnel** |  10   |       0         |    18.23    |             14.70               |\n|                 |       |      64         |    27.38    |             23.16               |\n|                 |       |      256        |    27.04    |             22.42               |\n|                 |       |      512        |    28.57    |             22.30               |\n|                 |       |     1024        |    30.77    |             22.69               |\n| **German Credit** | 125 |       0         |    63.81    |             16.06               |\n|                 |       |      64         |   148.73    |             27.81               |\n|                 |       |      256        |   208.44    |             54.92               |\n|                 |       |      512        |   275.74    |             93.26               |\n|                 |       |     1024        |   417.05    |            173.51               |\n| **Radon**      |  174  |       0         |    84.25    |             15.78               |\n|                 |       |      64         |   238.04    |             37.32               |\n|                 |       |      256        |   331.24    |             94.82               |\n|                 |       |      512        |   453.70    |            168.28               |\n|                 |       |     1024        |   699.73    |            338.63               |\n| **IRT** |  143  |       0         |    72.26    |             17.59               |\n|                 |       |      64         |   196.64    |             33.15               |\n|                 |       |      256        |   253.17    |             67.95               |\n|                 |       |      512        |   457.56    |            119.95               |\n|                 |       |     1024        |   516.98    |            222.65               |\n| **MovieLens** | 882 |    0         |   458.28    |             28.97               |\n|                 |       |      64         |  1981.14    |            384.45               |\n|                 |       |      256        |  4847.23    |           1387.64               |\n|                 |       |      512        |  8634.91    |           3156.82               |\n|                 |       |     1024        | 15873.45    |           6749.33               |\n\nUnfortunately, we cannot provide figures as part of this discussion. However, for a full accuracy to runtime comparison, we will provide another version of Figure 2 with the same set of results, but with the x-axis measuring the runtime for each method. This will still show some benefits of using MIF.\n\n> (Minor 1) Why did the authors choose to label the y-axis in Figure 2 as 'Best Elbo'? The notion of 'best' is very ambiguous and inconsistent from the rest of the paper where the authors quote Negative ELBO.\n\nThank you for pointing out the inconsistency in our labeling. We used \"BestELBO\" to emphasize that, at each iteration, the curve shows the maximum ELBO achieved over our learning-rate sweep, rather than the negative ELBO that we minimize elsewhere. We will clarify this in the revised figure."}, {"review_id": "kMgrQhzXLS", "reviewer": "Reviewer_gniu", "summary": "This paper presents a flexible family of variational posteriors, termed Model-Informed Flow (MIF), which combines full-rank variational Gaussian approximations with partially non-centered parameterizations (NCP) on conditionals via Variationally Inferred Parameters (VIP). The key idea of VIP is to better handle \"funnel\"-shaped posteriors through reparameterization, introducing auxiliary CP-NCP interpolation parameters (λ) that are learned by optimizing the variational objective, but the prior work that is limited to mean-field Gaussians. By incorporating a lower-triangular Cholesky factor, the proposed method can be naturally interpreted as an autoregressive flow, but this paper has a clear motivation of NCP adjustment along with several other carefully chosen design elements, making the approach competitive even with one-layer MIF with affine functions. The method is applicable to Bayesian hierarchical models with Gaussian priors, where the mean and variance parameterized by a deep net. Ablation study and experimental results on benchmark datasets demonstrate the  effectiveness and broad applicability of the proposed method compared to other variational inference baselines.", "questions": "(1) The experimental evaluation is somewhat limited. Is it possible to demonstrate that MIF offers general benefits, such as improving variational autoencoders, by evaluating on binarized MNIST (e.g., reporting marginal log-likelihood log p(x) using importance sampling)? Table 1 and Table 2 essentially use the same set of datasets, although the ablation is different. \n\n(2) With a limited-capacity conditioning network (e.g., a one-layer affine transformation), does MIF capture posterior dependencies better than alternative flow-based approaches such as coupling flows, MAF, or IAF?\n\n(3) Equation 2.2 and 2.3, 2.8, the likelihood term does not depend on z_i and the joint density factorizes between z and x? The way of writing it is confusing. The parent variables pi(x_j) are the z s?\n\n(4) Line 135: What is the quadratic form in Equation 3.4?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Pros:**\n\n**Quality**: This paper presents a flow-based approach with clear motivation from NCP via VIP, and it captures complex dependencies using a full-rank Gaussian. The research question is intriguing, as the auxiliary parameters could, in principle, be subsumed by the flow-based mapping and powerful neural network parameterizations already used in autoregressive flow approaches. While the mathematical expressions are equivalent, including NCP, there are differences in sampling and optimization. The effectiveness and added complexity are justified by ablation experiments. The authors also discussed the limitations such as the sequential nature. \n\n**Clarity**: The paper is well-written and neatly presented. The technical derivations are easy to follow yet insightful.\n\n**Significance**: Although sufficiently expressive architectures such as IAF can achieve strong results, this paper offers a compact and flexible alternative, with careful design specifically aimed at handling “funnel”-shaped posterior geometries. The work highlights that thoughtful methodological design—beyond simply relying on powerful neural networks—can be valuable. Even mathematically equivalent representations may differ in their ease of sampling or optimization. This perspective invites future innovations borrowing from classical probabilistic modeling and statistical science in designing better flow mappings, and it encourages a shift in focus within the ML community from prediction to inference. \n\n**Originality**: The connection to autoregressive flows is nice, though not particularly surprising. The use of VIP with full-rank Gaussian, and the extension of standard autoregressive flows with translation and prior inputs, are novel contributions. \n\n### **Cons:**\n\n**Quality**: The experimental evaluation is somewhat limited (explained in Questions).\n\n\n**Clarity:** The notion of “prior information” needs further explanation. Does it mean that the parameters are informed by previous variables in the specified hierarchical model, or that the hierarchical model itself already includes neural network (NN) components? In the context of probabilistic programs, the model can be user-specified. Is the proposed approach applicable to generic hierarchical models?\n\nIt seems that MIF serves both as a way of specifying hierarchical priors and as a method for flexible variational posterior, but the topological order of variables may differ. Have you tried reversing the topological order (e.g., bottom-up vs top-down)?\n\n**Significance**: The real-world impact is unclear. (1) Due to the use of full-rank Gaussians, scaling to high-dimensional problems may be challenging. (2) The experiments are mostly limited to a few simple models and datasets, and the evaluation relies solely on ELBO. \n\nCould the authors evaluate the method’s ability to capture complex dependencies in the context of variational inference, and using a density estimation benchmark on, for example, the two-moons dataset? It would also be helpful to comment on the applicability of MIF to generic variational inference in probability programming, likelihood-based density estimation and sample generation.", "was_revised": true, "final_justification": "The authors clarified the positioning of this work as a model-informed variational inference method for probabilistic programs, rather than a general-purpose density estimator, along with a few other points. The rebuttal has resolved my concerns, and I will maintain my positive evaluation.", "rebuttal": "Thank you for your thorough review and valuable feedback on our manuscript. Below, we address each of your comments in turn. If anything remains unclear, we would be grateful for any further questions or clarifications.\n\n\nBefore addressing the individual reviews, we wish to clarify that MIF is not restricted to—nor does it rely on—full-rank Gaussian variational inference. Theorem 4 shows that MIF can *recover* the full-rank-Gaussian + VIP transformation, but Algorithm 1 makes clear that MIF itself is an autoregressive flow model, independent of any full-rank Gaussian assumption.\n\n> (Clarity 1) The notion of “prior information” needs further explanation. Does it mean that the parameters are informed by previous variables in the specified hierarchical model, or that the hierarchical model itself already includes neural network (NN) components? In the context of probabilistic programs, the model can be user-specified. Is the proposed approach applicable to generic hierarchical models?\n\nBy prior information we mean only the mean $\\mu_i$ and scale $\\sigma_i$ that define each conditional prior $p\\left(z_i \\mid z_{<i}\\right)$. Our algorithm needs nothing else, so it works for any hierarchical model that can expose these two functions-whether the user writes them directly or we auto-extract them from frameworks such as PyMC or NumPyro.\n\n\nWe are willing to add text like below to the paper for clarity:\n\n**Prior information.**\nFor each latent variable $z_i$ we require two callables $\\mu_i(z_{<i})$ and $\\sigma_i(z_{<i})>0$ that output the mean and scale of its conditional prior $p(z_i \\mid z_{<i}) = \\mathcal N(\\mu_i,\\sigma_i^2)$. These are the only model‑specific inputs our flow uses.\n```\nLatentSpec = {\n    \"name\": \"z_i\",                  # variable name\n    \"parents\": [\"z_j\", ...],        # z_<i>\n    \"mean_fn\": lambda parents: ..., # returns μ_i\n    \"scale_fn\": lambda parents: ... # returns σ_i\n}\nmodel_spec = [LatentSpec, ...]      # one per latent\n```\nThe callables can be hand‑written or auto‑parsed from frameworks such as PyMC or NumPyro; any hierarchical model that yields these $(\\mu_i,\\sigma_i)$ pairs is compatible with our Model‑Informed Flow.\n\n\n> (Clarity 2) It seems that MIF serves both as a way of specifying hierarchical priors and as a method for flexible variational posterior, but the topological order of variables may differ. Have you tried reversing the topological order (e.g., bottom-up vs top-down)?\n\nYes. We provide this experiment in Sec. 4.3: the row “MIF (w/o order)” in Table 2 trains the flow with the reverse (bottom‑up) ordering of latent variables. Performance drops on every dataset, confirming that matching the model’s natural (top‑down) order is important for capturing conditional structure efficiently.\n\n\n> (Significance 1  & Question 1)) The experiments are mostly limited to a few simple models and datasets, and the evaluation relies solely on ELBO. Could the authors evaluate the method’s ability to capture complex dependencies in the context of variational inference, and using a density estimation benchmark on, for example, the two-moons dataset? It would also be helpful to comment on the applicability of MIF to generic variational inference in probability programming, likelihood-based density estimation and sample generation. The experimental evaluation is somewhat limited. Is it possible to demonstrate that MIF offers general benefits, such as improving variational autoencoders, by evaluating on binarized MNIST (e.g., reporting marginal log-likelihood log p(x) using importance sampling)? Table 1 and Table 2 essentially use the same set of datasets, although the ablation is different.\n\nThank you for the suggestion. We’d like to clarify that MIF is not designed as a general-purpose density estimator but as a model-informed variational inference method for probabilistic programs. It cannot be applied to density estimation problems, as there is no prior or posterior.\n\nIn particular:\n\n1. **Different training objective.** MIF is trained by maximizing$\\mathbb{E}_{q(z)}[\\log p(x,z) - \\log q(z)],$ which requires an explicit latent-variable model $p(x,z)$. Pure density-estimation tasks (e.g., two-moons, binarized MNIST) optimize $\\log p(x)$ directly and lack this programmatic structure.\n\n2. **Model-informed architecture.** Our shift/scale/translation networks consume “prior inputs” $f_i, g_i$ derived from the known generative model. Without a specified probabilistic program, those modules have no signal to condition on, and MIF’s inductive bias disappears.\n\n3. **Workload scale.** Probabilistic inference is inherently more expensive than density estimation, and state-of-the-art PPL benchmarks typically involve only tens to low-hundreds of latent dimensions. That is why we focused on six canonical hierarchical models (8 Schools, Credit, Funnel, Radon, Movielens, IRT) in Table 1 and 2 and 3 more models (Seeds, Sonar, Ionosphere) in Table 3, which capture challenging posterior geometries yet reflect real-world PPL use cases.\n\nWhile it would be interesting in future work to adapt MIF for pure density estimation (e.g., by re-casting it as a normalizing flow or integrating it into a VAE encoder), those extensions lie outside our current scope. We will clarify these points in the revised manuscript.\n\n\n> (Question 2) With a limited-capacity conditioning network (e.g., a one-layer affine transformation), does MIF capture posterior dependencies better than alternative flow-based approaches such as coupling flows, MAF, or IAF?\n\nYes. Even with a minimal conditioning network (a single affine layer), MIF still captures posterior dependencies more effectively than other methods. Tables 1 and 2 demonstrate that this low-capacity MIF matches—or exceeds—the accuracy of those alternatives, which require deeper, higher-capacity networks to reach similar performance.\n\n> (Qeustion 3) Equation 2.2 and 2.3, 2.8, the likelihood term does not depend on z_i and the joint density factorizes between z and x? The way of writing it is confusing. The parent variables pi(x_j) are the z s?\n\nIn equation 2.3, the likelihood term would be $\\prod_{j=1} p\\left(x_j \\mid \\pi\\left(x_j\\right)\\right)$. As the reviewer suggests, here the set of parent variable $\\pi\\left(x_j\\right)$ would include the latent variables $z_j$. Our reason for writing the model this way is that we wish to consider models where arbitrary variables could be observed. So, latent variables might be conditioned on both observed and latent variables, and similarly for observed variables. For this reason, we settled on our (admittedly subtle) notation where the parent sets $\\pi(z_i)$ and $\\pi(x_i)$ include both latent and observed variables. We will clarify this in the revised version of the paper.\n\n> (Question 4) Line 135: What is the quadratic form in Equation 3.4?\n\nWe believe the reviewer is referring to our comment about \"the quadratic form in Equation (3.4)\" on line 135. We apologize for this somewhat ambiguous phrasing. By “quadratic form” we mean the product of $g_i(\\pi(z_i))$ and $f_i(\\pi(z_i))$, which introduces a quadratic dependence on $z$ that an affine function of $z$ cannot capture; we will clarify this wording in the revision."}]

Author rebuttal:

Meta-review: The paper's main result is a connection between the Variationally Inferred Parameters (VIP), a prominent method that adapts to sharp posterior geometries. The main insight is that VIP in combination with a full rank Gaussian can be exactly represented as an autoregressive normalizing flow. The paper further introduces model-informed flows (MIFs) that can incorporate the model prior as an input to the variational model. Experiments show that this approach is competitive with state of the art variational inference techniques.

The reviewers were all positive about the paper and found in particular the connection between VIP and flows interesting. The concerns (mostly about the practicality of the approach) of the reviewers were largely resolved in the rebuttal.
Decision: Accept (poster)