Variational Transdimensional Inference
Authors: Laurence Davies
Dan MacKinlay
Rafael Oliveira
Scott A Sisson
Venue: NeurIPS 2025 spotlight ()
Keywords: Bayesian inference
variational inference
transdimensional inference
normalizing flows
Bayesian optimization
Primary area: probabilistic_methods

Abstract: The expressiveness of flow-based models combined with stochastic variational inference (SVI) has expanded the application of optimization-based Bayesian inference to highly complex problems. However, despite the importance of multi-model Bayesian inference for problems defined on a transdimensional joint model and parameter space, such as Bayesian structure learning, flow-based SVI has been limited to problems defined on a fixed-dimensional parameter space. We introduce CoSMIC normalizing flows (COntextually-Specified Masking for Identity-mapped Components), an extension to neural autoregressive conditional normalizing flow architectures that enables use of a single flow-based variational density for inference over a transdimensional (multi-model) conditional target distribution. We propose a combined stochastic variational transdimensional inference (VTI) approach to training CoSMIC flows using ideas from Bayesian optimization and Monte Carlo gradient estimation. Numerical experiments show the performance of VTI on challenging problems that scale to high-cardinality model spaces.

Variational Transdimensional Inference
Laurence Davies
Dan Mackinlay² Rafael Oliveira Scott A. Sisson 18
Universiit of New South Wales
'CSIRO Data6l
"laurence@latentlogiic com. au, sccott. sisson@unsw... edu. au
dan mantmaackilaaay. rafael dww.santoodeelliiiiraaaaaaaeeee csiro.ar
Abstract
The expressiveness of flow-based models combined with stochastic variational inference (SVI) has expanded the application of optimization-basee Bayesian inference to highly complex problems. However, despite the importance of multi- model Bayesian inference for problems defined on a transdimensional joint model and parameter space, such as Bayesian structure learning, flow-based SVI has been limited to problems defined on fixed -ideensiiii parameter space. We introduce CoSMIC normalizing flows (COntextually--ppp Speecffidd Masking for erentity-mapper Components), an extension to neural autoregressive conditional normalizing flow architectures that enables use a single flow-based variational density for inference over a transdimensional (multi-modee a combined stochastic variational transdimensional inference (VTT) approach training CoSMIC flows using ideas from Bayesian optimization and Monte Carlo gradient estimation. Numerical experiments show the performance VTI on challenging problems that scale to high-cardinality model spaces.
conditional target distribution. We propose
Introduction
Variational inference via stochastic optimization [23, 37] has surged in interest since the introduction of normalizing flows [43]. Flow-based densities can be used for variety of downstream tasks, such
as importance sampling [43], simulation- based inference [39, 56), adaptive Markov chain Monte Carlo (MCMC) [17], and generative modeling [27]. While many existing approaches only consider continuous supports, there is a growing interest in applications where the support is either discrete discretely indexed [11]. One such application concerns a targel transdimensional probability distribution with support X = Ummg (fm) x Om). where M is finite discrete index set, Om C dR., and the dimension dm of bm may vary with m. Hence X' is a transdimenssional space [16, 21, 48]. Such spaces arise in Bayesian model inference, where θm correspond to model parameters, and E MM is model index. Discrete indices parameterize many practical inference problems, including variable selection [16], mixtures of-rreessi--rrersooooooo learning directed acyelic graphs (DAGs) from data [52], phylogenetic tree topology search [15], mixture-commee component inference [8], geoscientific inversion [45], and change-pon point models [21]. This article is concerned with estimating the target distribution æ with associated density function T(m, 0m), 0m @m. whose dimension depends on For simplicity we refer to <m, 0m) and related functions as density functions, even though they are not continuous. Typically, this density only available in conditional unnormalized form, (0 m) (m. m)) where 2,m Se... ( m)d0m..
The factorization (7m, 0m) = 17( (0mm (m) implies there is a discrete target probability mass function over models, (m)) 2m. 2-1 where 2 ΣmEM Zm. Estimation of n/m,, 0m) then becomes estimation of both e,,.. m) and π (m.).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In the presence of likelihood function p(D m. Om) for data D, and priors p(Om m) and p(m).
the target distribution defined by the D-conditional transdimensional posterior π (m. p.. D) x pl (D] m. fm (m..mm)p(())..... In the context of variational Bayesian inference (VI; see [2, 25 approx- imation of the transdimensional posterior x (m, 6m (D) has not been addressed generality. Such a scheme would approximate some unnormalized target density n(m, Pm (D) = 2m(m, 6m (D) by choosing parameters & E , 1b E R of tractable variational density family qo. 0m) =
(0 (m) to minimize
Bi
arg min ((v) Ø),
[(V, 6) DKI.(u..oll).
vi
where DKL is the Kullback-Leibler (KL) divergence. There are two impediments to constructing such a variational approximation: (i) defining and optimizing qs as On may vary in dimension conditional on m, and (ii) the inference of go: for discrete latent variables m during the optimization of qo a non-stationary objective as Φ - o" and d + vp* are interdependent.
Background: Flow- models for stochastic variational inference: Rezende and Mohamed [43] showed that normalizing flow for qu (with fixed m) is able to approximate many challenging ixce-dimensional distributions that are not well approximated by common parametric families. A normalizing flow is defined by diffeomorphism To : R² - [Rd between two random vectors ~ q and x ~ Ma. such that their distributions q and Va are absolutely continuous with respect to d-dimensional Lebesgue measure, have well-defined densities q()) and vz(z), respectively, and can be related by = (0) so that q(0; 6) ( det VT (0)). O E IRd, As is typical of normalizing flow-based models, we refer to Un as the reference distribution and assume it factorizes into a product of d identical marginal distributions Va= V 8 ... @ v Dav. Construction of To is typically achieved by defining d bijective, univariate functioon Tp, R + R, 24 Tp. (0.) for i E {I..... d}. The parameters P. NN (0,s) for the 2th transformation are determined by a neural network NN o such that p; is not dependent on O2, so that the inverse T (.) can be calculated without requiring inversion of NN g. This independency remains if the neural network NN, is autoregressive with respect to the inputs o...... Aa [38]. Benefits of autoregressive flows are their higher-overall expressiveness and efficiency in the variational inference setting versus e.g. coupling flows [7]. For these reasons, this paper employs autoregressive NN A conditional normalizing flow extends is a natural extension of a normalizing flow with a conditioning variate, &. passed contextual input to the NN,, such that P: = NN (f, y: E). Applications include classification, where & is an index, or likelihood estimation [55] where & encodes the parameters of the likelihood function.
The MADE encoder [19] enables autoregressive neural flow architectures, which can be coupled with any such as affine [38] and spline [13] transformations. The cost of autoregressive flow depends on the direction. In the forward (sampling) direction, it evaluates each dimension sequentially, for time complexity of (d). In the the inverse (likelihood) direction, computation can be parallel. The inverse autoregressive flow (IAF) [28) reverses this dependence, setting = Ta(2), yielding the variational density 4s (0m) (T (0)) det VT (0) (=)))) det (Tx(2)
tantexttally-Speeiffee Masking for Identity-mapped Com- ponents) flows, widely applicable and simple modification conditional neural flow architectures (Section 2). CoSMIC flows fundamentally expand the use cases for normalizing flows to encompass variational inference applications, so that a single flow-based variational density can be used for variational inference over a transdimensional (multi-model) target distribution. In effect, this extends the reparameterization trick exploited by IAF-based VI to the transdimensional setting. In Section 3, we demonstrate the efficacy of COSMIC transformations within novel variational transdimensional inference (VTI) framework with two implementations. The first builds upon principles of Bayesian optimization [50], and the second uses Monte Carlo gradient estimation [34]. We also provide a theoretical analysis of VTI approximation error bounds under a Gaussian process surrogate, and convergence guarantees for the marginal model distribution under convergent optimization steps. Finally, we demonstrate the applicability of VTI to problems with model spaces that cannot be easily enumerated within the memory limitations of current computing architectures. In particular, Section 5 explores problems in Bayesian robust variable selection [35] and Bayesian causal discovery [22].
Contributions: We introduce CoSMIC
ToyTh CUDA code for all experiments is available https://ittubb.com/avviee

---

2 Formulating a transdimennional variational density
Rather than constructing a variational density separately for each model m E M, it is preferable to construct single density on the transdimensional support X'. To account for the varying dimension of Om. we adopt the dimension saturation approach of Brooks et al. [5], where the dimension of the parameter space conditional on each model is unified across all models. This is achieved by augmenting the space of model-conditional parameters with auxiliary variables u v, as discussed below. We use /m to identify auxiliary variables of dimension dmax dm. where dmax max,, {dm }. We define the saturated support (0m; u (mm) E Om x Um C Rdmax with unnormalized, dimension dimension-saturated. saturated, conditional target density
i O m. Wim m)) m mm)mmm (m)m)
Defined on the same augmented support is the family of saturated variational densities
(m, On (mm) Ts (O Wmm (m)g
where, noting the availability of a transport (0m: Wmm) To(= ()) Uddax, we define the IAF
Tsh Om Vmm m) = Viduua (T 1 (0m, u m)) det VTZ (0ms Wim m)
-Vdmax (2) det VT
(=| m)l
(4)
Our goal is to construct the IAF so that equation 4 factorizes into active and i..dd. auxiliary parts, i.e. m)
2o(Om. Umm
go 0m. m) Vdd...( (m),)
(5)
and to exploit this factorization in the construction a transdimensional loss function. To achieve this factorization, define the following notation. Let A; . [(,0 1] flag whether latent coordinate i appears in model m, and let B; {0, - f0. 1))r)]]] f B,(b)
(b. ................................ b), broadcast this bit to
the corresponding parameter block. Their composition C == B, o A; M {0, 1)le.l therefore activates exactly the autoregressive parameters p; needed by Tp. (z(1) under model Concatenating
the blocks gives the global context-to-mant map (see Figure 1(b) for a visualization):
dmax
C(m) == (m). Coiaae (m)) E 10,1)
|el
Σ lpl.
(6)
Similarly, A and denote the respective coordinate--ooccaenaicc maps similar in form to equation 6. After fixed left-align permutation aligning latents with On. Proposition 2.2 proves this factorization exact for any autoregressive network NN, that parametrizes the transport To.
Recalling the univariate bijective maps of the inverse autoregressive flow as Tp. R ~ O for i .,.... dmaax we assume the existence of static point ااd such that Tpla (=) = for all = R, i... the transform becomes the identity map at p'd, For example, simple affine transformation (scale and location shift) is 0 Tp. (z) p (0)) + p1)) z, where p; (p()) (00pp()) .))) this case, the static point is ptd = (0, 1) as then 0 = z. We can then construct simple mechanism for "choosing" between p and p for each individual transform T, 1l..... dmaxx via the convex combination
pf = =1 ((())) + (m))) m E M.
Each coordinate- wise transform then becomes of = T (2(0), E (1..... dmax). That is, the transformation parameters become context- mentext-dependent composition of the elements of p; and the static point pd (Figure I(C)). A composition transforms parametrized according to equation is a Contextually-Speciffe Masking for mapped Components (CoSMIC) normalizing flow.
Lemma 2.1. For CoSMIC transform (0m, wim) = Tø(zm, p. Umm = 2m Vm E M.
Proposition 2.2. Fix m E M. Let P., be the permutation matrix that places the coordinates indexed by I (m) (from the proof of Theorem 2.1) before those in ['mm while preserving the original order inside each group. Define the leff-align--ermutee permuted flow T == P p-1 To Pm and the corre- sponding density df
(0m: um) = Vd (z) det VT m)l 2 T - (0m. u, u.)) Redefine
C. : C B o Pm A. Then (a) g (0... Wmm) factorizes as per equation 5 with the substitution
Ts = T and (b) the marginal 90 (0... m) is consistent.
3

---

(b)
COSMIC FAF sap
CHSMIC JAE
CUSMIC JAE
(OCI P
Pm
Pin
Mimm
Lef-aliga permitation
lnvese algn penmzzatio
Figure I: (a) CoSMIC flow composition, (b) Context-to-mask map, (c) single CoSMIC IAF step.
From here on, we use the notational convenience To == T and Oo := q denote the composition of transforms and associated variational density that include the left-align permutation P.,. required by Proposition 2.2. We also write the partitioning 2 = (Zm= mm as explicitly obtained by [zm Amm]
Z. By construction, Vdenax Vd @ Vm ie. Vdmax (z) = Vdn.. (zm Vid (2)m)
Corollary 2.3. Given Lemma 2.1 and Proposition 2.2, then
Vd (det VTo(2 m)]l"
Vd (Zm) det VT [m)] == h x m),
(8)
Tj(To(z m) m)
n10m m)
and, substituting {(m; +) = Ezwesmax hogh m)). the loss in equation becomes
L( 6) Em [e(m; Φ) log p(m) + log qu (m)].
(9)
Proposition 2.2 states that, conditional on model m, the CoSMIC IAF (Figure achieves the factorization of the saturated-space variational approximation in equation 5. From Corollary 2.3, this means that when computing the loss function in equation 1, the ratio of the pnneension-saturaaee variational density q and conditional target 7 (equation 8, LHS) which are both dmax diinensoiai and which involve the auxiliary variables, collapses down direct comparison only on the dm-- dimensional model specific densities To (fm m) and 1)(O,.. m) (equation 8, RHS), and without the involvement of any auxiliary variables. That is, the CoSMIC flow enables the IAF calculate on diee-dimensional space, while permitting the model-specific comparison within the loss function to operate on the natural dm dimeensiona space.
The implementation of a CoSMIC inverse autoregressive flow step T; as part of a composition of transforms Ty, Ti is visualized in Figure (aa) Individual architectures for affine and rational quadratic spline transforms [13] and compositions are described in Appendix A.2.
3 Formulating a model weights distribution
Formulating and estimating de is not as straightforward as that of de because the discrete random variables m ~ qu are not automatically linked to the density parameters v by automatic differentiation. This problem naturally lends itself to methods developed in black-box variational inference [41, 53, 54] and multi-armed bandits [6], as described below. The representation of m is any discrete random variable on a finite space M. Writing the true distribution of as Tms a finite M implies the existence of a categorical distribution тद which is bijectively equivalent to 7m- The random variables Ç ~ Ny. exist on the finite support Ç E C C N, thus [C] [.]. This property is used by the surrogate-based approach described in Section 3.1.We formalize this concept via Theorem D.I.
consider two approaches to model Ze- Firstly, we derive non-parametric surrogate-based approach which comes equipped with theoretical convergence guarantees and is applicable to model spaces M of low cardinality. We then present an approach based on parametric models that can scale to arbitrarily large spaces M that are trained using doubly stochastic gradient estimators.

---

3.1 Estimation via surrogate
The objective in Equation 9 can be rewritten as single-variable objective with respect to di:
Φ² E arg min min [(v, Φ) arg max max Bm-rn [-((m; 6) log + H[ge],
(10)
quEPa
where Pv denotes the space of probability measures over M parameterized by d E v C R"v, and H denotes entropy. If we replace Pv by P(M), i..., the whole space of probability measures over M, the solution to the inner optimization problem admits a closed-form expression:
Li (m) = p(m))xx(((((m)) x))
(11)
Computing the expression above within an optimization loop over o in practice would, however, require the evaluation of flow-based densities over the entire model space. We may, instead, follow a eneeape-ttoevaluuee density qu,o which approximates Fe for a given Φ, by means of learning a surrogate model over within the same optimization loop". In particular, we derive Gaussian process (GP) upper confidence bound [49], which provides the following approximation to the optimal model probabilities:
p(m) exp (m (m)) Emrem p(m') exp ue (mm)²
(12)
where (m(m) == (m, Φt) + [m, with He and of representing the posterior mean and variance of a GP model conditioned on all mini-batches of data B, fōr-1, Me,,, log h (2t,s/me,s)) ! available at iteration & of stochastic gradient descent, and oi denotes the current flow parameters. In this form, u, provides an upper confidence bound (UCB) over-E(m, Φ,) determined by the choice of confidence parameter B 2 0. The GP posterior mean and variance can derived in closed form if the observation noise is Gaussian with, e-g., variance o² We, however, show that sub-Gaussian noise assumption is sufficient to use conventional GP model. In addition, if Φe follows a convergent sequence (e.g., by ensuring diminishing step sizes during gradient-basee optimization), we have the following guarantee.
Corollary 3.1. Let E ~ GP(O, r),, where : (M x 4)2 + R is bounded, continuous positive- semidefinite kernel over M x Assume log ho(z|m) - {(m;; h) o pe--uu--Cr sub-Gaussian with respect to r~w V. Then, if &t follows convergent sequence, the following also holds:
DrL.gu.ellae... E0((1 D
(13)
where O₂ characterizes convergence in probability.
The result above tells us that the UCB-based models distribution approaches the optimal distribution rate of Op((t 1/2) and ultimately converges to it as - co. Therefore, a stochastic gradient optimizer using samples from the surrogate density 9u.t should asymptotically converge to the optimization path determined by the optimal Lz That is, under appropriate settings for, e.g.. its learning rate schedule, the optimization will converge to Φ². Lastly. note that the result in Theorem 3.1 is independent of the choice of B, which can be set to 3 = O. Our analysis is mainly based on obtaining enough samples almost everywhere across the model space, which can be ensured by sampling according to the predictive mean μ4, of the surrogate, as exp M > 0 under mild assumptions. However, in practice, a non-zero value of 3 helps to accelerate convergence finite time by encouraging exploration. Corollary 3.1 is direct application of Theorem C.3, proved in Appendix C.5, where we also contrast it with existing results [36].
Due to the reliance on GP-based approximations, a naive implementation of this approach would incur a cost of O (3) per stochastic gradient step, where B is the mini-batch size, due to matrix inversions [42]. However, for model spaces of moderate cardinality [M| = M, we can keep compute
We are here assuming that the prior p(m) is cheap to evaluate. If not. we can model -6(m; Φ) + log p(m). instead, with surrogate, which leads to similar theoretical guarantees after minimal adjustments.
.G E Or(ge) if limc-+o limsup P(ç.g. > CJ- 0.
5

---

costs linear with the number of optimization steps by applying recursive equations to evaluate the GP posterior mean and covariance over the model space (see Eq. 21 and 22). leading to a cost of O + M + B) O(M²B) per step, as B < M, totaling (TT over 7 steps. Sparse approximations to GPs can further reduce this cost. [20, 42] make it practical for larger model spaces. For our purposes, we implemented a diagonal Gaussian approximation, which makes the cost linear in the batch size and constant t via a mean-field approximation.
3.2 Categorical and neural probability mass functions
By Theorem D.I, we may represent probability distributions over the model space M by arbitrarily parametrized categorical distributions. A drawback of the surrogate is the need to maintain and update estimates over the entire model space, which can be impractical for spaces of very large cardinality. such as DAG discovery. Hence, we introduce two parametric alternatives.
Categorical: Assume IM = M E N. Then, for db E TR the distribution over . is defined by M
M qo (o(m) ا-ن exp wy ) i= 1[m; = m] exp th: The logit weights vector is unconstrained in A and can be jointly optimized with ø by gradient methods. Density evaluations and the entropy can be computed with memory cost ((.M)).
If the model space is too large we may use a structured sample generation process which allows for the number of parameters
Autoregressive
be smaller than cardinality of the model space i.e., dim( < M. For instance, Germain et al. [19] proposed an autoregressive parametrization for distri- butions over binary strings 8 E {0, 1]d. via the decomposition ps. (s)) )) = II = Pe (s:[s)..... 5;-1). For each 8, assign a unique E M and define q (m) := p& (m))). The conditional densities and sampling can be implemented via MADE, allowing us to map the entire model space with fewer parameters when 2d 2 [M]. The same reasoning can be applied to DAG via decomposition of its adjacency matrix. Details of MADE are in Appendix G.8, and for DAGs in Appendix G.7.
3.3 Estimation via Monte Carlo gradients
When [.M is too large to use a surrogate-basel approach, or to even parameterize entire vector of categorical weights in physical memory, we can employ neural-based methods that use gradient descent and estimation of the gradients of d via Monte Carlo estimation of gradients (MCG) [34]. Using v.go(m) = qu (m)V, to sh is
V. v log (m)). the gradient of the expectation in equation 9 with respect
CeL( h) = Em-gy [e(m; の) V s log zo (m)] + Emagg log (s(m) V log (o)g)(m))
(14)
p(m)
In practice, the variance of this estimator can be very high. However, techniques exist reduce this variance [34, 41? J for general applications. We use a control variate in the form
V LW Φ) Em (9(Ø, 1. s) v. log qu (m)].
(15)
where ( s) Ezwesuan [log ha (2(22)) (m) log qo (m) log p(m) s). We compute ç using the method described in Appendix E. (full description in Appendix E).
The benefit of using MCG for variational parameter estimation is the flexibility of choice for qa- We compare two: (1) MCG of the logits of standard categorical distribution, and (2) MCG multi-layer perceptron weights that parameterize a configuration of the MADE neural autoregressive density estimator of Germain et al. [19] (see Appendix G.7). When |M| is large, such implementations of ty permit efficient approximate representation of the model distribution.
3.4 Information-Limiting the optimization
The convergence of v/ - a is dependent on the convergence of á - 0², and optimal sample efficiency for the inference of is achieved when & xe v." Intuitively, go should focus primarily

---

on the higher-probabiee probability models that contribute most to estimator variance. but discovering these models requires stable approximation of each ( m) inform v... This circular dependence motivates practical regulation of the optimization of v when estimating V via Monte Carlo gradient estimates, addressing an instability similar to that discussed the reinforcement learning literature [46], but without modifying the objective. Our approach is to reduce the variance of the estimates of qu by bounding the information gain in the transition qu. - - which determines the step size, thereby stabilizing the optimization (detailed in Appendix E.2).
4 Related work
Conditional normalizing flows [13, 55] have emerged powerful tools for incorporating conditioning information. Existing methods use the context variable as a conditioning input, but fewer adapt the flow architecture itself. An exception is the transport-basee based reversible jump MCMC method [9], which learns proposals for transdimensional moves, but does not readily allow its use as an inverse autoregressive flow [28]. In contrast, introduce an partttttyaammmteezeeedd CoSMIC transformation without identity-map training. We bypass path-wise approximations to discrete distributions [24, 33], instead comparing Monte Carlo gradient estimation [34] with Bayesian optimization [47]. We adopt an information-basee approach to scale gradient steps using "small steps," inspired by reinforcement learning [46]. Bayesian methods for model selection and optimization have advanced with black-box variational inference [41, 53, 54] and flexible flows [13, 38, 43]. Recent work in amortized Bayesian mixture models [29] shows amortization over multiple mixture components using conditional normalizing flows, but not for variable dimensions. Conversely, Li et al. [31] introduces an architecture for learning imputation over transdimensional inputs, but lacks immediate application as variational density. Our approach unifies transdimensional inference with flow-based variational methods, bypassing the need for tailored dimension jumps and broadening applications.
5 Experiments
We present experiments involving synthetic and real data on two representative applications: ro- bust variable selection and directed acyclic graphs. To evaluate the quality of the approximation (m, i..)) to the target distribution π (m, 0m) for a relatively small [M| < 2 model space, we use the average negative log- log-likel (NLL) computed over set of samples drawn from π via baseline sampling method, in this case reversible jump MCMC [48]. Let { ) denote N independent samples from (m, 0m). The average NLL corresponds to the cress-entropy H(T, 9o.o) between и and Te.o- which quantifies the expected number of bits needed to encode samples from
using dv.g. and is defined as NLL- 1 E i== log qu.o (m', ( O Comparison of VTI DAG inference quality with baseline frequentist and Bayesian approaches use standard metrics [30].
5.1 Bayesian misspecified robust variable selection
We study robust Bayesian variable selection problem where the response y IR is related to predictors æ E IR (including an intercept) through a linear model. 'he innovation is a mixture- of-Gaussians noise specification, accommodating outliers via heavy-tailed component. A subset indicator 7 E {0, 1)28 selects which predictors enter the model. If ,3 E TRP are the coefficients, only the components where 7, 1 contribute to the linear predictor. In particular, for data {(x,, y:) ):)) the prediction function is (xx) x: T[B @ y). the likelihood is
P(u: Ti, 3,, = =(1 a1 (un (UEM(2)))of) -1-aa\MMsHt)) +
(16)
and priors p(7) = P and p(,3)= = No0,, T). Here, a controls the fraction of outliers, and (af. az) encode the variances of in-distribution and outliers, respectively. To complicate the inference problem, two misspecified aata-generating processes were used (medium- and high-misspecification) which encourages multi- multi-modality in the approximating posterior (00m
7

---

Mispeciizziin (5)
12
1991
Figure 2: Quality of VTI approximation for Bayesian misspecified robust variable selection. Outer columns denote medium (left) or high (right) likelihood misspecificaton, inner columns indicate different normalizing flow constructions, increasing flow expressivity from left to right. Flow types are described in Appendix A.2. Top row: Estimated model probabilities qu (m) vs true model probabilities π (m) on the log scale. Bottom row: Cross entropy between individual model estimates go (0m (m) and true density T (0mm (m7) versus true model probability. Colors indicate 10 replicated analyses, each with |M] = 2 models.
VT
Muspochkauaan level Medium
Maapestiutae.coom L0gg eerupy
Spllen byen
Splrs)
[M]
Figure 3: Left: A simulation study of the robust variable selection example showing the cross entropy (NLL) between RJMCMC samples and an flow-based variational transdimensional density using rational quadratic spline CoSMIC flows under fixed number of iterations (30,000). Each cardinality was run with 10 independently sampled synthetic data sets. Right: Comparison of bivariate plots of variables 0 0% obtained by RJMCMC and VTI for a single [M] = 27 problem.
Table in Appendix F summarizes the full experiment configuration. Figure 2 offers a holistic assessment of inference quality relative sampling baseline using RJMCMC, where cross-entropy reduces as flow expressivity increases. It shows two problem settings, mid and high misspecification, and for each setting shows how increasing complexity of the variational density (left-to-right panels) improves the quality of the approximations of both π (00m (m) (bottom row) and estimated model probabilities (top row), and that the approximation quality of क ((0m (m) is higher for higher probability models.
Cardinality sweep: Using the focused prior setup on both the medium and high misspecification level targets, we sweep the cardinality of the model space |M from 29 to 221 and compute the cross entropy H [ qv.s), where samples (m, 6m)
are obtained via RJMCMC (see Appendix F.3). Figure 3 (left) compares the cross entropy between the three (m) types discussed in Sections 3. and 3.3 in simulated problems of increasing [M). As expected, H ( (e.o) generally increases with [M| when the flow architecture is held fixed. The surrogate method (blue bars) performs comparably with the other methods for the smaller model spaces (IM 2"), whereas the neural density (orange bars) performs consistently as IM] increases. Figure 3 (right) shows two bivariate plots of selected variables (0%)) 0,2 from the posterior inferred using RJMCMC and VTI. This qualitative visual comparison shows how well the CoSMIC flow is able capture non-trivial model distributions versus the sampling approach (for the full multivariate comparison see Figure 11). Appendix describes the experiments in further detail and demonstrates VTI robustness to diffuse priors.
8

---

VI
14-
AUROOC
243
12
Figure 4: Simulation study comparing VTI DAGMA [1], DiBS/DiBS+ [32], and JSP- GFlowNets [10] for discovery of a 10-node non-linear DAG visualized using standard metrics (Appendix G.2, left to right, where better is: higher, lower, lower, higher). Bars display mean and standard error over nine i...d. repetitions for each data set size.
5.2 Bayesian structure learning via non-linear directed acyelic graph discovery
We consider a dataset of real-valued observations, denoted by X E IR where n is the number of data samples and Na is the number of nodes. Our goal perform Bayesian inference over a space of non-linear structural equation models (SEMs) which is isomorphic to a space of directed acyclic graphs (DAGs) and non- non-linear functions over the active edges. DAG is represented by directed adjacency matrix A E {0, [0,1] Nax where Asj = 1 indicates a directed edge from node to node and A, = 0 otherwise. The acyclicity constraint requires that the directed edges in A do not form any directed cycle. In non-linear SEM, each node X, depends non- linearl on its parents in the form X = f(X) + C, NO, d²), where f R Na + NR is a nonlinear function possessing an acyelic Jacobian matrix. We follow [1, 52] whereby f is multi -ee--llyyeee-ppppe
(MLP) structured as fN.((NN... (X)) We implement ff using single hidden layer, with rectified linear unit (ReL.) activation functions used to model non-linearity where the bias term can be optionally included (see Appendix G). By introducing a topological ordering of the N₂ nodes, we simultaneously enforce acyclicity and a consistent mapping of parameters to each graph. Let P be a permutation matrix that reorders nodes into valid topological order and define U to be strictly upper-triangular By construction, any acyclic adjacency matrix can be represented as = P U.. Each edge is guaranteed to point from lower-indexed nodes to higher-indexed nodes in the topological order [3]. Note that this parametrization does not conform to Theorem D.I, as the correspondence between (P, U) and A is use a MADE- space (see Appendices G.8 and G.7 for details). The simulation study in Figure 4 contrasts VTI with state-of-the-art Bayesian and non- -an-aaae baselines (DiBS/DiBS+ [32], JSP-GFlowNets [10], and DAGMA [1]) with the aim of demonstrating that the performance of the generic VTI approach can be competitive with application-speccin
f(X)
However, this does not violate the consistent parameter mapping. We discrete distribution [19] for gs for inference over a very high cardinality model
approaches, where one would expect the latter to have better performance. Evaluation of each method is depicted using the commonly accepted F1 score, structural Hamming distance (SHD), Brier score, and area under receiver operating curve (AUROC) (see Appendix G.2). complete description of this study is in Appendix G.4.
Real data example in flow cytometry: Sachs et al. [44] use Bayesian networks to analyze multi- parameter single-cell data for deriving causal influences in cellular signaling networks of human immune cells. Causal interactions are validated by comparing to a Comain-agreed adjacency matrix representing causality within the data, establishing baseline for causal prediction accuracy. We use VTI to discover the distribution of non-linear DAGs for these data, comprising n = 7466 entries over
Table 1: Comparison of DAG discovery on flow cytometry data [44]: VTI versus baselines
Method
F1
SHD
Brier
AUROC
VTI non-linear DAG
0.44
23.0
23.0
0.68
DAGMA non-linear
0.32
25.0
25.0
0.60
DiBS+ non-linear
0.22
28.0
17.0
0.54
JSP-GEN non- -on-linear
0.23
54.5
44.0
0.51

---

Na 11 nodes, and benchmark this against the agreed adjacency. Table shows strong performance of VTI compared to state of the art methods. A complete description is in Appendix G.5.
6 Discussion
We have introduced CoSMIC normalizing flows as means to implement variational transdimensional inference (VTT), the approximation of target density over a transdimensional space with a single variational density. VTI is broadly applicable to a wide class of transdimensional inference problems. Although the specification of CoSMIC flow requires augmenting all model dimensions to dmax. VTI is not sensitive to these added dimensions during training and inference due to the construction of the auxiliary variable transforms. We presented two approaches for simultaneously optimizing the variational parameters v/, Φ. The Gaussian surrogate- approach benefits from our derivation of the approximation error bounds and established convergence guarantees for the marginal models distribution under convergent optimization steps. The two approaches that use Monte Carlo gradient estimation for SGD optimization benefit from recent advances in neural architectures and neural approximation of very large model spaces. The choice of model sampler is dependent on both the cardinality of the model space and the structure of the problem. When [M] is small, the Gaussian process surrogate- basee sampler or the categorical sampler using Monte Carlo gradients are both appropriate, although in practice it usually safe to default to the latter approach. For high cardinality problems, we recommend a neural model sampler for approximate inference on the distribution of model weights.
The quality of the VTI approximation possesses two notable characteristics. Firstly, those models m E .M estimated to have large posterior model probabilities will contribute most significantly to the loss. Hence the CoSMIC flow will produce relatively more accurate (in the KL sense) approximation of such models, compared to models with low probabilities. This effect is seen in Figure 2 (bottom row). While one might prefer greater accuracy on more dominant models, structured changes to [(vb,)) ()) could give greater control over where the quality of the variational approximation should focus. The second characteristic is that when the normalizing flow is unable to approximate the conditional target T (0mm m) well, a smaller loss can be achieved by shrinking the estimated model probability (mm(m) to zero. This effect is seen in Figure 2 (top row), which lessens flow expressivity increases. Here the question is how to design the normalizing flow, i.e. the flow context E and the mapping C(m), to best allocate resources to produce good approximations of models likely to be of relatively high posterior model probability. In many transdimensional problems, two models could be considered adjacent by structural similarity (e.g. in variable selection where they differ by one included covariate) and so may have similar posterior model probabilities. This could be achieved by e.g. extending the architecture of the context encoder (Appendix A.2) capture similarities between models that generalize over the model space, and learning structural similarity within the surrogate-based model sampler itself via a reward-based criterion.
The left-align permutation used in Proposition 2.2 raises the question of whether the alignment of variables across models in the normalizing flow is important for computational efficiency. More broadly: (a) is there shared information between models? And, if so, (b) would careful manual construction of the flow improve exploitation of this versus allowing the optimization to determine it agnostically? (a) is answered by the robust variable selection example in Figures 3 (right) and 11 where high probability mass regions for each model do not overlap and thus there there is no such shared information. To answer (b) would be an avenue for future research. Future work could also derive convergence rates, which will depend on the choice of optimization algorithm for the flow parameters, and extension of the CoSMIC architecture to coupling flows, in applications outside variational inference. addition, our analysis for the surrogate-base based approach general enough to be extended to a variety of methods for approximating distribution over models. Finally. in extending to the transdimensional setting we inherit the same strengths and weaknesses of single- model VI, including the challenges of mode-collapse. Users would need to take the same steps to manage as in the standard setting. Mode collapse in the model distribution is mitigated by the exploration versus exploitation strategies discussed in Section 3.
10

---

Acknowledgments
This research was made possible by funding support from the CSIRO Machine Learning and Artificial Intelligence Future Science Platform. The authors would also like to thank Ryan Thompson for critical advice the construction of the non-linear DAG example, and Daniel M. Steinberg and Edwin Bonilla for advice Monte Carlo gradient estimation methods. SAS is supported by the Australian Research Council.
References
[1] Kevin Bello, Bryon Aragam, and Pradeep Ravikumar. DAGMA: Learning DAGs via m-matrices and log-determinant acyclicity characterization. In Advances in Neural Information Processing Syxtems, 2022.
[2] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859 877, 2017. ISSN 0162- 1459, 1537-274X. doi: 10. 10.1009/16221459900 221.222111.2111 URL https:///wwt http://wwwwaoooooon./ooooooo/om htt.a//oonieee.coooooo//oo//eeee conn/oooffull// 1080/ 01621459 2017 1285773.
[3] Edwin V. Bonilla, Pantelis Elinas, He Zhao, Maurizio Filippone, Vassili Kitsios, and Terry O'Kane. Variational DAG estimation via state augmentation with stochastic permutations, 2024. URL http: perrxiv.orr/aee abs//////222..///// 2264.02644.. 02644.
[4] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford University Press, 2013.
[5] S. P. Brooks, P. Giudici, and G. O. Roberts. Efficient construction of reversible jump Markov chain Monte Carlo proposal distributions. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, 65 (1):3 39, 2003. ISSN 1369-7412. doi: 10.1111/1467-9861 07777788880077111 03711.
URL https: otg////00...o..//.//// 1111/
[6] Sébastien Bubeck and Nicolò Cesa-Bianchi Regret analysis of stochastic and nonstochastic multi-armed bundit problems, volume Now, 2012. ISBN 978-1-60198-626-4 doi: 10.1561/2200000024 URL OCLC: 931223688.
http: /ttp///axvvv///////////ee//a.ava wtg.aiiwwi/aaaaaaaaooooooeoooeee abs ab2/1204. 1204. 5721.
[7] Andrea Coccaro, Marco Letizia, Humberto Reyes-Gonzále and Riccardo Torre. Comparison of affine and rational quadratic spline coupling and autoregressive flows through robust statistical tests. Symmetry, 16(8):942, 2024. ISSN 2073-8994, doi: 10.3390/sym1608000 URL https: /wwp://wwww.co// http:/wwwwppppptpm////// c0733 1994/16/8/942. Publisher: MDPI AG.
[8] Moumita Das and Sourabh Bhattacharya. Transdimensional transformation--aase Markov chain Monte Carlo. arXiv preprint 1rXiv:1403.52000 2014.
[9] Laurence Davies, Robert Salomone, Matthew Sutton, and Chris Drovandi. Transport reversible jump proposals. In Proceedings of The 26th International Conference on Artificial Intelligence and Staristies, html..
pages 6839-6852.. PMLR, 2023. URL https::/prroceddnng mlr. ISSN: 2640-3498.
html.
[10] Tristan Deleu, Mizu Nishikawa- Vishikawa-Toomey. Jithendaraa Subramanian, Nikolay Malkin, Laurent Char- lin, and Yoshua Bengio. Joint Bayesian inference graphical structure and parameters with a single generative flow network. Advances in Neural Information Processing Systems, 36: 31204-31233 2023. https: htttpp::/procceedngg..eeo
www.neeerii..oo cttpeeeeddiiieeeeeesee..eceeeeee Abstract- Conference hhmml. html.
[11] Gian C. Diluvi, Benjamin Bloem-Reddy and Trevor Campbell. Mixed variational flows for discrete variables. In Proceedings of The 27th International Conference Artificial Intelligence and Statistics, pages 2431-2439, PMLR, 2024. URL ISSN: 3498.
mlr. wtt...ppres//v////ieeee htl////////lii2/../ html.
[12] Lester E. Dubins and David A. Freedman. A sharper form the Borel-Cantelli lemma and the strong law. The Annals of Mathematical Statistics, 36()),,0000000000 36(3)80008877 1965. ISSN 00034851. URL http:://t/// org/ stable/ 2238191. Publisher: Institute of Mathematical Statistics.
[13] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.

---

[14] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Nflows: Normalizing flows in PyTorch. Zenodo, 2020.
[15] Richard G. Everitt, Richard Culliford, Felipe Medina-Aguayo, and Daniel J. Wilson, Sequential Monte Carlo with transformations. Statistics and Computing. 30:663-676, 2020.
[16] Yanan Fan, Scott A. Sisson, and Laurence Davies. Reversible jump Markov chain Monte Carlo and multi-model samplers. In Handbook of Markov Chain Monte Carlo. Chapman and Hall/CRC, 2026. URL https://arxiv. wwrg/bbs/1000 2055.
[17] Marylou Gabrié, Grant M. Rotskoff, and Eric Vanden-Eijnden. Adaptive Monte Carlo augmented with normalizing flows. Proceedings of the National Academy of Sciences, (001100)))22002001100 2022. doi: 10. 10.000/pas...00422000 19. URL https: https:://uwt p 2ttp:///0......a......aaaaa0000 2109420119. Publisher:
Proceedings of the National Academy of Sciences.
[18] Andrew Gelman and Yuling Yao. Holes in Bayesian statistics. Journal of Physics G: Nuclear and
Purticle Physics, 4000) 2020. ISSN 0954-3899, doi: 10.1088/136/1 1088/1166-667711111 1/abcaa3. URL https: //dz. doi org. 10tg//10.0o 0tt///136666671//aa
Publisher: IOP Publishing.
[19] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. for distribution estimation. Proceedings of the 32nd International Conference on Machine Learning. pages 881-889, PMLR, 2015. URL https: /ppp//eeeoo.gs..///////////...ee mml. press//337ggermainn55 html. ISSN: 1938-7228.
MADE: Masked autoencoder
[20] Arjan Gijsberts and Giorgio Metta. Real-time model learning using incremental sparse spectrum Gaussian process regression. Neural Networks, 41:59-69, 2013. ISSN 08936080. doi: 10016/..euuntt22112....0001 URL https: pettpa://linkinggee
elsevier. http://r.....rreeeeeee//////////
[21] Peter J. Green. Reversible jump jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82(4):111-732, 1995. ISSN 0006-3444. doi: 10.1095/biommee/224.111 URL https:
//doi. org/10. 1093//i0mt//////// 4...1.............11...
[22] David Heckerman, Christopher Meek, and Gregory Cooper. A Bayesian approach causal discovery. In Dawn Holmes and Lakhmi C. Jain, editors, Innovations in Muchine Learning: Theory and Applications, pages 1-28. Springer, 2006. ISBN 978-3-540-33486-6. URL https://doi. ottp:/...////..../// 33486-6_1.
[23] Matthew D. Hoffman, David M. Blei, Chong Wang. and John Paisley. Stochastic Variational Inference. Journal of Machine Learning Research, 14(40):1303-133 -3377 2013.
[24] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In International Conference Learning Representations, 2017. URL attra:///pennevviev.neeeeeeee id=rkE3y85ee.
[25] Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999. ISSN 1573-0565. doi: 10.1023/A:1007665990 178. URL attpps//doo org/10. 1023/ A: 1007665907178.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL http: //arxiv. wwt.//////////ea.ee.oeee 6980.
[27] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 13126113 2013. URL httpps://ppenreee// hwtp://www...eoooor////////o....
[28] Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems,
29, 2016.
[29] Simon Kucharsky and Paul Christian Burkner. Amortized Bayesian Mixture Models, 2025.
[30] Erich Kummerfeld and Alexander Rix. Simulations evaluating resampling methods for causal discovery: ensemble performance and calibration. In 2019 JEEE international conference on bioinformatics and biomedicine (BIBM), pages 2586-2593 2593. IEEE, 2019.
[31] Yang Li. Shoaib Akbar, and Junier Oliva. ACFlow: Flow models for arbitrary conditional likelihoods, In Proceedings of the 37th International Conference on Machine Learning, pages 5831-5841, PMLR, 2020. URL https::/ //ppceed///ccceinng. mlr, wewess//119/111// html. ISSN: 2640-3498.
12

---

[32] Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. DiBS: Differentiable bayesian struc- ture learning. In Advances Neurai Information Processing Systems, volume 34, pages 24111-24123, Cur- ran Associates, Inc., 2021. URL https: https://proccedddnng mewpeddiiiinnnnnnssneeeeeeeee http:///pppp...lleeepppeereeeeee Abstract .ttml.
hash/ 1tta://6666335544466633333366aa3
[33] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017. URL
https://opente
met/forum?id-S1 jE5L.55..
[34] Shakir Mohamed, Mihacla Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1 2020. ISSN 1533-7928. URL
http://prrwpppppp./eeeeeaeeeeeee http://jmlr.org www.oogg.pe.eeee... pttpppppppreee/-/eeeeee
html.
[35] R. O' D'Ha and M. Sillanpää. review of bayesian variahle selection methods: what, how and which.
Bayesian Analysis, 4( ((1):85-117, 2009. ISSN 1936-0975, 1931-6690. doi: 10.1214/09-A4003 URL
https://preejcctuucldd.oom htwp:///d.....ooooo////////aeeee analysiiiaaaaaaaaaaaalaaaaaseeea w 4/isssee--eeeeeeeeee 1/A-review- of -a--ayyeiaaaaaaiaaaaeeeeeeeeddde wwwpeeeeeeeeoooeeeeeeeee-eeeeeae International Society Bayesian Analysis.
1214/19-111100 0001144099-940.... full. Publisher:
[36] Rafael Oliveira, Lionel Ott, and Fabio Ramos. No-regret approximate inference via bayesian optimisation. In Proceedings of the Seirth-Seentth Conference on Uncertainty in Artificial Intelligence, pages 2082 2092. PMLR, 2021. URL https://proceed// .ml. mmr. 3498.
html. ISSN: 2640-
[37] John Paisley, David M. Blei, and Michael I. Jordan. Variational bayesian inference with stochastic search. In Proceedings of the 29th International Coference International Conference on Machine Learning, ICML' 12, pages 1363-1370 Omnipress, 2012. ISBN 978-1-4503-1285-1
[38] George Papamakarios, Theo Pavlakou, and Tain Murray. Masked autoregressive flow for den- sity estimation. In Advances in Neural Information Processng Systems, volume 30. Cur- ran Associates, Inc., 2017. URI. pttpa://proceediiie penooeeeingn.neurpps...... www.ness.ccppppre....eeeeeeeeee. /011//21///// 2017/hash/ Cctta888822277822633697990008 htt.re/ae...0eeteeeaaatttttt...e Abstract. html.
[39] George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. Proceedings of the Twenty-Secone International Conference on Artificial Intelligence and Statistics, pages 837-848. 848, PMLR, 2019. URL https: //procee//os mlr.
press/v89/ pwwpmaaariosseeee. html. ISSN: 2640-3498.
[40] Gilles Pisier. Subgaussian sequences in probability and fourier analysis. Gruduate Journal of Mathematics. 1:59-78, 2016. URL http:htttaiii..coto org/ abs tabs//60 00///70070000..... Publisher: Mediterranean Institute for the Mathematical Sciences tex.arxivid: 1607,01053.
[41] Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS), 2014.
[42] Carl E. Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006. ISBN 2262-18253-2
[43] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference Machine Learning, pages 1530-1538. PMLR, 2015. URL https: httpaa//pprooe //tppe///proceeingg.a mlr, wttpppeeee//reeeeeeeeepp _html. ISSN: 1938-7228.
[44] Karen Sachs, Omar Perez, Dana Pe'er, Douglas Lauffenburger, and Garry P. Nolan. Causal Protein- Signaling Networks Derived from Multiparameter Single-Cell Data. Science, 308(5721):523-529 2005. doi: 10.1 126sccinceeee...... 105809..
[45] Malcolm Sambridge, Thomas Bodin, Kevan Gallagher, and Hrvoje Tkalcic, Transdimensional inference in the geosciences. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, (171(1984)201105442 10547, 2013.
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithus, 2017. URL attt:://arxixt org/ abs//7// 06347.
[47] Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P. Adams, Nando Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148 175, 2016. ISSN 2256. doi: 10.1 (09/JPROC.201) 201.000.....24444 2ocument/7352306. Conference Name: Proceedings of the IEEE.
201..44211..44411... 2494218. URL https: /ttp:://eeeeeerrreeeeee ieee. org/ //grrtttttt//////o/o
13

---

[48] Scott A Sisson. Transdimensional markov chains: A decade of progress and future perspectives. Journal of the American Statistical Association, 100(471):1077-1099, 2005. ISSN 0162-1459, 1537- 274X. doi: 10.1 19800166245000000006664 http:///w/wae httptt:://www... www.:/wwwwwt.ooooeeoooooooa com http///oo/a////aaaaa 1198/ 016214505000000664
[49] Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger. Gaussian process optimization the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML" '00, pages 1015-1022, Omnipress, 2010. ISBN 978- 1 1-60558 -605558-90777. URL https: /ttp///axvvvv///////a///a./..waa org/ wttp//12...... 3995.
[50]
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. IEEE Transactions Information Theory. 58(5):3250-32655 2012. ISSN 0018-9448, 1557-9654, doi: 10.1109/TTT.201122253 1.2182033. URI. http:
//arxiv.orr .arvvooope wttp:/r////////////////// 3995..
[51] Daniel Steinberg, Rafael Oliveira, Cheng Soon Ong, and Edwin Bonilla. Variational search distributions. In NeurlPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024. URL
https://openrevieet watt//oormmidddV/xSSiiii.
[52] Ryan Thompson, Edwin V. Bonilla, and Robert Kohn. ProDAG: Projection-Inducee Variational Inference for Directed Acyclic Graphs, 2025.
[53] Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. In Proceedings of the 31 International Conference on Machine Learning, pages 1971-1979 PMIR, 2014. URL https::/pproeeetee
mlr www.mmllpppeeeeee presss/32/titssaasse html. ISSN: 1938-7228.
[54] David Wingate and Theophane Weber. Automated variational inference in probabilistic programming,
2013. URL http://arxiv. http://aaaaiw...g/aba/////////// 1191
[55] Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows. CoRR, 2019. URI /tttps://ppenrrreete net//forumPdd-rrg3zztttt
[56] Andrew Zammit-Mangion, Matthew Sainsbury-Dale, and Raphaël Huser. Neural methods for amor- tized inference. Annual Review of Statistics and Its Application, 2024. ISSN 2326-8298, 2326-831X. doi: 10. 144/anurrev-stttiiicce--- 112723-034123. URL http:://ww.nmmmarevieee.. annualreviews. org/ content/ cort/ctteet/////////
journals/10 10. 1146/ "aaaannnuvvveeeeaaaeaaaa www.sssttttststiieeeeeeee stwtisticss-112222-00522 034122.
14

---

A
Implementation of CoSMIC normalizing flows
A.1 Inverse autoregressive flow sampling procedure
Draw reference samples
(z ((mmax)) ~ Vimaax
given m, define the permutation matrix Pm that groups active coordinates first:
(Zm; pm)) = Pm.
Zm E Rm.m iim
R idm Vdm = dmax dm-
Concatenate the coordinate-wer -iie transforms Tp. into the map TV and bookend with permutations Pm. give the strict CoSMIC bijection
T (m (z) = Pm (T cono (2 ))) .. C(mm) (2 (mmae))) P..
A.2 Experimental CoSMIC transform compositions
The experiments the below compositions of transforms as inverse autoregressive flows To m) where are the imputs from the reference distribution is context input. compositions except for the diagonal Gaussian are assumed have the strict left-align permutations discussed Appendix A. The term "block" is defined in Appendix A.3.
Experiments will sometimes use context encoder that projects the context input to a higher dimensional space. Typically this will take the form a multi-layered perceptron with hidden layers of increasing size (fixed to powers of 2) and terminating in an activation layer at the largest size, say 2 nodes.
Context encoder:
Model-specific reverse-pen perursttermmttaionn: Flow compositions commonly include reverse permutations to ensure expressibility of an MutoregresiveeNNN-base flow is (approximately) the same for all coordinates. Denoting the generic reverse permutation for all coordinates peev assume the strict left-right permutation Pm (as per Appendix A) has been applied, and hence define the left-most dm coorddiaat reverse permutation Pddm
Affine(5,5): The learned component the affine masked autoregressive transform (38), denoted here as Afffie for transforms 1,....., 5, each having blocks. We set Ta = TAffine prey o...o prey Affine
Spline(4,6): The learned component is the rational quadratic spline masked autoregressive flow architecture (13), denoted here as RQ--ppe Each RP-SSpe 6 blocks. Additionally, we define fixed global affine 0% transform T Miging that is not dependent on inputs context and hence has only two learnable parameters: scale My and shift ag.. We set Tp == Tyyin o RQ--pplnne prev o ...
prayy 5.mm o I RQQ-pline
ds
<dm
A.3 Autoregressive flow definitions
We use the residual variant of the Masked Autoencoder for Distribution Estimation (MADE) [19], implemented in Torch by [14]. Each block maintains the autoregressive property by assigning degrees {1....,d d} inputs and propagating them forward.
Given input Rd and optional context z, each residual block computes:
h = MaskedLLneara (a (BN₂ (MaskedLinearr (a (BN) + 8(2))))
Here, MaaseddLnnear are masked linear layers respecting the autoregressive structure, BN; are optional batch norm layers, and 6(z) is an optional context projection. All layers preserve feature dimensionality and respect degree ordering ensure autoregressive validity.
15

---

B Analysis of a CoSMIC normalizing flow
Proof of Theorem 2.1. Let [(m) {i E (I...., dmnxx A.(m) = 1). and 1 be the complement. The result holds from equation 7 as, for all coordinates i E (m)), umm Tp. (2 = Tpte (x mm
(6)
Proof of Theorem 2.2. (a)-(b) Density factorization and marginal consistency.
We aim to prove
(a) To uim m) = Qu (8.. m) Vid (m
/ To uim m) dum.m qu (0.. m).
and (b)
Write 1/.11 (1 Id), where Id denotes the identity transform, and let the permuted reference vector be Pm. (zm; 2,m) E R p Because the masking function every transform Tp, with C:(m) the identity, inverse flow splits as
T (6m. Wmm) (T m (0m), ukm))
where T O. - Rd the active block and the dummy block is exactly the identity. Consequently, the Jacobian matrix of 25- is block upper- triangular with det V det VIJ 1.
Apply ohange-of variables with Vdmax = Vdm 2 dim to obtain
go (0m: umm [m) Vidma (1 (Bm, u det V
Vit (T (0.n) )) va (m) det VT (0m))
(0m (m) ve (u))m))
which proves the factorization (a).
Integrating the right-hand side over um recovers (b) (0m m.) completing the proof.
Proof of Theorem 2.3. is sufficient show (Qc. DKL (av.o = [(v,)) Note by equation
fj(m, pm, umm) = p(m) ( Umm m) = p(m)n((m.. m))d, (2))
log [ (o()m)s (Om: umm m))
Drr (|ñ)
Emogg
p(ma)i(O u:m m)
(0m, 2im m²)
Emsag
E E(om,armm
log
7/(0m, 2im m))
+ log (m)) log (p(m))
Emaqy
[log (qw (m))
log (p(m))) +
Vdm (Zam M [m)m] det (To(2)]
I1
Em-ras
Rawdsaax
log
by Proposition 2.2
m7 m)Me (MM
DKL (9o.o (m) ())
Corollary B.1 (Computational complexity).
Sampling (forward IAF); all coordinates can be updated in parallel
O wall-time depth.
Evaluation (inverse direction); must populate (<4) sequentially identical to standard IAF.
O (dmax) arithmetic operations,
Proof. The forward IAF updates 6., via closed-form 7, that read previous outputs- all available after one pass through the network- which are therehy fully parallelizable. Conversely, evaluating ((m))-1 at an arbitrary point O... M must reconstruct sequentially, exactly for any IAF, giving (ddmax) time.
16

---

C Theoretical analysis of the model weights distribution
We consider the following bi-level stochastic optimization problem over function f M x 1 - as:
Ø' E arg max
max 4EW qtEP(M)
(m,, @) +lo p(m)]++++ + = arg max Em [f 6) + +ogg (17)
6E#
where P(M) denotes the space of probability measures over M, H is the entropy, and the optimal or for given @ can be shown be:
p(m) exp f(m,,
(m))m) Em'Em p(m') exp f(m', 6)
E M.
(18)
This formulation corresponds stochastic optimization problem optimum for q/ has closed-form expression 4j.o for every given e E To solve this problem, we will follow a sequential optimization process over & (c.g., stochastic gradient deseent). However, sampling from the optimal model distribution 2/.0 (above) requires evaluating the summation in the normalization constant, which is expensive. Therefore, we will instead approximate each qi.0. with distribution Qu,t composed of and
variables o and qy. where the
a cheaper-to-cvaluate surrogate based noisy observations -t-1.+ f(z1; My, Φt-1), where my ~ qu.t-1,2 E (1..... B}, such that Ez [f<= m, Φ3] = fmm, Φ). If we ensure that qu.t approaches g7.0. over time, optimization steps hased on Ju,t will eventually follow of.d. and allow for the optimum o² to be reached.
C.1 Regularity assumptions
We make the following assumptions about the function f and the observation noise. Assumption C.1. The objective f a sample from zero-mean Gaussian process prior with a bounded, positive-semidefintt covariance function x (M x 2)² + IR, which is continuous over 4.
The GP assumption allows us to derive closed-form expressions for predictions over I and their associated uncertainty. The continuity assumption is easily satisfied by most practical covariance functions and ensures that, if de converges some s² GP-based estimates (m, 6²) will also converge for every m M. To model predictions over f with closed-form GP updates, we also need Gaussian assumptions about the observation noise, which is given by:
C f(z, m, b) f(m,
m E M, á +
(19)
However, we will show in our analysis, sub-Gaussian tails are enough for GP modeling, which we formalize next. Assumption C.2, The observation noise is of oä-sub-Gaussian, i.e., given any m E M and ø E 4, we have:
Ys E R, Elexp(scm. S exp 13"00")
(20)
This mild assumption is satisfied, for example, when v is zero-mean Gaussian distribution and f is Lipschitz continuous on its first argument, which case a. only depends on of through its Lipschitz constant [4, 40).
C.2 Gaussian process model
Under the GP assumption f (O, rt), posterior over f is again Gaussian process. Suppose at each iteration 2 1 of stochastic gradient descent we sample mini-batch (m. from variational posterior approximating di.s at = ót-1. Given a batch of observations B, := ( Me,s Ve. h the GP posterior f|BI....t ~ GP (ue, Ke) has its mean and covariance described by the following recursive equations:
Me +) = (m-1 Φ) km-1(m,,)) x) (K,-1 +101 21 - Mc-1)
(21)
Ne(m, b m' 0²) (m- Φ. m², nn) 1m-1(m, Ø) (K, + I²1) R (m',
(22)
where Rm-1(m, Φ) == [se-1 me... 2z- 1) E R Ki-1 ( br- me.s, 1))
BBxB and pi- := (mu, 1) E R with μn = 0 and Ko = K. Any pointwise prediction is then modeled as f'm, Φ) Bi..... ~ (m 6). of (m, 6)), where om Φ) == ka(m, o, m, σ). for
(m, Φ) E M x Φ.
17

---

Algorithm Stochastic optimization with UCB sampling
for I {1, ..... N do
[m.,i]== [=1 ~ qu,t-1
(2t,i) ~u
Dr, [(Zi,i, me,i, DI-1), for
Φ - UPDATEPARAMETERS ΦC--1, (f(zz, mi,i. 01 9u,-1) [4:, Kt + UPDATESURROCGATE((MI: yt. di- 11,,,,,, ( end for
C.3 Upper confidence bound (UCB) algorithm
Given the GP posterior, we formulate an upper confidence bound algorithm [49] with:
u+ (m)m = Jee [m,, Φt) (eo Φc), .,,
(23)
where Bt >> is parameter controlling the size of the confidence bound, which we will discuss in analysis. We then derive sampling distribution based using the UCB surrogate for as:
Gu,t E arg max Emm--/uem))) + p(m) log q(m)]. 4EP(M)
(24)
The solution to this optimization is available in closed form as the UCB softmax:
p(m) exp exp (m)
gu (m)
m M.
(25)
Em'EM p(mr') exp ur(r"")
Equipped with this UCB-hased sampling distribution, follow the generic procedure outlined in Algorithm 1. The algorithm starts by sampling from the current UCB distribution. A sample-hased estimate of the optimization objective Em-e, [f,m 1)) = Σe f(z,,, Ma,4, oi-1) is then passed the algorithm responsible for updating parameters Φt. e.g., stochastic gradient descent update. Once the parameters are updated, we reevaluate the objective and update our GP. The procedure then repeats up to a given number of iterations N E N.
C.4 Approximation errors under sub-Gaussian noise
In the following, we derive generic concentration bounds for GP predictions under sub-Gaussian observation noise. We start by showing that the approximation error between the GP mean and the true function is sub- Gaussian.
Lemma C.1. Let f (O K) be zero-mean Gaussian process with given positive-defiite covariance function S x S R. Assume are given sequence obxervations Yn = f(x,) en. where In S and en of "-sub-Gaussial noise, for n E Let pa and o²; denote the predictive mean and variance. respectively, of the GP posterior under the assumption that the noise is zero-mean Gaussian with variance given by á. Then, for all 2 0 and all E S. we have that [xx) (x) is o (x)-sub- (z)-suu-Gaausinn.
Proof. For n. 0. the proof is trivial as, without observations, we only have the prior with pa(x) = and xā(z) x(x, 27. Now let X, : {x, H C S denote set of n 2 1 observed locations. For any given E S, expanding the GP posterior mean from its definition, the approximation error be decomposed as:
An(x) f(z) (x) f(x) x(x, X.,) (K. 11) (f.. En)
(26)
f(x) x(=, x K, -. fr k(z, Xn, dT En:
where N(x, Xx) = [<(2 21)... <<xx J., K. [x(,;,,;)];;
== [(<:)]] and == [e:] M-
The last term on the right-hand side above is sub-Gaussian, since E(e..) O and, letting ax, (K,. I²I) -1 -x .r), we have sum independent sub-Gaussian random variables, see egg. [40], Lemma 1.1:
Elexp(a.Te.))
exp Σ an,iEnne
Elexp(an,scn.))) S exp
50. Σ ond
(27)
i=1
18

---

which follows from the definition of sub-Gaussian noise (ef. Assumption C.2). The remaining term the right-hand side of equation 26 is zero-mean Gaussian random variable with variance given by:
(arlf()) - (,,, + [n]
=(x,z) 2x(x,, X.. (K. + o. x( (Kn; x) + n( (K, + of (K.(K, a = n(z,)) x) 2x(x,, (K., + I. x x) + an K.....
"x((2o,
(28)
As equation 26 describes the of two independent sub- Sab-Gaussian random variables, can follow similar reasoning to the one applied in equation to show that An (xx s (a) (u))-su-Gauusian for some s (z) > 0. The resulting uab-Gaussian parameter s (x) then bounded by the sum of the individual sub- Gabssanssinn parameters in equations 27 and 28 as:
(x) S x(2, x) 2x(2, Nn (Kn + "((an, + a Knan + a.a, an x)) + a (K, T.T) x(z, x) 2n(x, N,) (K, + -
<(2, 2x(x, X..) (.,, - x(e, (K,, +,,1) x((nn,)
(29)
=<(,,)) n(2,, (K,, + -
=x)
which concludes the proof.
C.5
Convergence guarantees
Now apply the error bounds above to the general optimization problem in equation 17. Assumption C.3. The sequence parameters {dr Cauchy sequence, i.e.:
> 0, ANĄ :: || - dell S X, 2 Nx.
(30)
assumption above can be guaranteed by, e-g., diminishing step sizes during (stochastic) gradient descent. essentially means that de will converge some å E d C 12 though not requiring it be the optimum. Assumption C.4. The prior p(m) has full support over M.
Such assumption ensures that the prior would not wrongly assign zero probability to plausible models.
Lemma C.2. Let assumptions C.4 hold, and set 3a B O, for all E 20,1,2,... Then following almost surely holds:
(m, O(t Vm M.
(31)
Proof. Consider the following upper bound on the predictive variance of a GP model [51, Lem. D.3];
0f o.a Φ) E N, o2 o) S o? + o (llmm)))N..) (m,,)) V(m, Φ) E M 2
(32)
where (m, 3) denotes the number of observations collected at (m, Φ) E M x + up to time 2 1. In addition, letting Hh denote the alagbrr generated by the history of all random variables measurable at time t, and setting = lim 6r. the second Borel-Cantelli lemma [12] tells us that.*
Vm E M,
(mm,, ②) = lim
E = m 5.-1]-
(33)
C+00
1-300
Therefore, for (m, δ) need the series above diverge. To ensure the latter, show that the conditional probabilities Equation (35) have nonzero lower bound or, if they converge zero, that they do so slowly enough.
We now derive a lower bound on the sampling probabilities. First, observe that:
Vi E N, E [[lm+( Ile)] ] [[] (.) Sc]llo]] s E|E[I](( d)
S.]]. VΦ E 4,
(34)
'More precisely, the second Borell-Cantell Cantelli lemma shows that the two sides of Equation 33 are proportional each other, while equality holds if the right-hand side diverges.
19

---

where [I((, suPmEM (mm, 6) denotes the supremum norm of f(, 6), and we applied Jensen's s inequality in the last step. Since the kernel x is continuous and bounded, the sub- uub-Gaussian parameter of (., Φx) has a maximum in M. which is finite, As the expected value of the maximum of finite collection of sub- Gaussian random variables is bounded [see, e.g., 4, Thr. 2.51, it follows that the GP mean Jt is almost surely bounded all times (by, e-g., Markov's inequality). Considering the model sampling probabilities and that pmin = minmeM p(m) > 0 by Assumption C.4, we then have that the following almost surely holds:
P [m = m Sii] = qu.e(m) 2 Punin exp( (Il oe) + [m, (m))
v 2 0,
Lm, EM exp(Ila((s
+ (m, (x))
(35)
Puin exp(-2)lu((. + (m,,
[M maX (xp. x4))
As, for every m E M, sequence {o] (m, ())) non-negative and non-increasing. has limit by the monotone convergence theorem. Let c. = lim,-+o0 maXmeM o Φe), and let m, E M be one of the maximizers of 4o0 (r 0r). If 0- 0, by Equation 35, we have for m. that:
Pmin exp( 2ll de)l + (m,, (i))
lim P [mc+1 m. 55] lim $1000
11000
[M maxmEM exp(So, 04))
Pmin exp(
(: de) (0.)
lim
IM exp(33r.)
Pmin exp(
( #b)
(36)
lim
[M
Puin exp(-2EIIIft ~)ll 50xl)
[M
5m 0,
which implies (. @) 00 by Equation 33.
However, in that case. must have of lime++oo of (m., 6e) 0 by Equation 32, which is contradiction. Therefore, a. and consequently lime-+oo a (m, Φe) S o. = 0, all m E M.
Finally, we show that (,, )b) E O(t As we have seen that lim 0i(- 6x) = 0 above, applying the limit
to equation 35, we see that IP Im = m 5t-1] + bm > 0. for each E M. Hence, Ne(m, oe) oc) E O(t
implying that o3 (-, (b)) is (t asymptotically by Equation 33, which concludes the proof.
Definition C.1. Let {Ee hew be a raalevalued stochastic process. We say that & E O>(g(t)), positive function N + (0, 20). if:
15+ [g(t)
Ve > 0, 3C2 E co), No EN:
IF
>
C.
Se.
VI 2 No.
(37)
equivalently that:
&. [g(t)
lim limsup sup P C-oo t-+00
=.
(38)
Theorem C.3. Under the assumptions in Theorem C.2, we have that the following holds in probability:
DKL ( Or(t-1/3)
(39)
Proof. Expanding from the definition of the KL divergence and the variational distributions, we have that: t 1 0, DKL. (9,, 147.0r)
Emnogu [log qu, (m) log 97.00 (m)]
Emn-qu (m)) [m,, Φc)
+ log Σ p(m') exp f(m', ((m')))) log m
Σ
p(m') exp ue(m')
Am'E.
(40)
Under assumptions C.1 and C.2, given any $ 0. applying standard sub-Gaussian concentration results [4] and a union bound, we have that, for all 2 0:
P(3m E M [f Φt) pm .e)) > (0, (=))
Σ
PIIf(m, (t) pe(m, Φe)1 > Ba. (m, Φt)]
mEM
< 2| exp (_
82
ös
(41)
20

---

With probability at least 1 63. it then follows that:
ue(m) fm,, Φx) peemm, Øb) + (o, 61) f(m, x6) S 2m0
(42)
log
Σ p(m') exp fim ((m) log Σ
p(m') exp')expulll ux u(mmm) 0
(43)
MM
for all m E M. Hence, with the same prohability, it holds that:
Dka(qu,:ll97.d4 S 23 lor ot) & 231lor(s, Ai)llao v2 0,
(44)
By Theorem C.2, we know that (m Φ+)) E O(t-1/2), so that there exists C 0 such that (m Φ+) S
Ct C/21/2 for m E M. We then have that:
DKL (qu.1117700)) > 2/
lim lim 8-+00 t-000
S lim lim supp [DLL > ())))
Ct-1/2
80000
+
(45)
(- 88 = -0
$ lim 2| exp 1
which concludes the proof.
Remark C.4. The result in Theorem C.3 is similar to Corollary in Oliveira et al. [36], which also derives the concentration bound for the KL divergence between a turrogate-basee approximation of posterior and the truc posterior. However, Oliveira et al.'s 's result only provides an asymptotic convergence rate requires an upper bound on the information gain of the surrogate model order (vt) and an appropriately scaled UCB parameter Sr, whereas our result shows that we do not need either these assumptions whenever sampling lower bound can be guaranteed, i.e., infseN,me.. Pfme+1 = m m/))) > In addition, Oliveira et al. [36] only deals with the static setting where the target posterior does not change over time, while in our case we have changing de that leads to different targets per optimization step. This non-stationarity requires additional with the convergence analysis.
21

---

D Bijective equivalence between discrete distributions
Proposition D.1. Every finite discrete distribution over finite support M [mL, m2..... my has unique representation categorical distribution. Specifically, there exists bijective mapping between the all finite discrete distributions on M and the set of categorical distributions parameterized by probability vectors vp over M.
Proef. Let P denote the set of all finite discrete distributions over M, and let C denote the set of categorical distributions parameterized by
Injectivity: Suppose 9 and are distinct probability vectors in C. Then, there exists least one index such that + Φs. Consequently, the corresponding distributions assign different probabilities to my, implying & $
Surjectivity: For any finite discrete distribution E P, define 0 = p. Since p satisfies 0, 2 0 and valid parameterization Thus, every p corresponds to some
Σ 0.
Since mapping both injective surjective, is bijective. Therefore, every finite discrete distribution a unique categorical distribution representation.
L
22

---

E
Monte Carlo gradients via score function estimation
An alternative the reparameterization trick is the score function estimator (SFE), which circumvents the issue of non-differentiable samples from discrete distrihutions by using the log trick to compute the gradients of function with respect to variational parameters. the case the distribution models, we have identity
(m) qu (m)) V log (m).
(46)
By the Leibniz integral rule, the gradient of the expectation in equation 9 with respect to the parameters of the discrete distribution is
V L(.. 15) = V..m-gg [[(m)] + V Banigg log (m)
p(m)
log to (m)) p(m)
Emmrg V log go (m)] Emrig
V log qu (m)
In practice, the variance of this estimator can be very high when the batch size is not large. However, there techniques reduce this variance for general applications. The simplest of which is the form
control variate in
VuL(o v)
Em-yy [Ex~uamar [1v, o, =) <) V log gm
(47)
By simply choosing (2....... [L(Ø, v.J, where the expectation is estimated online the iterations of the optimizer, we can reduce variance of V L(0. $). See Appendix for implementation details.
E.1 Control variate for function estimator
We adopt the approach used Kingma and Ba [26] for obtaining an unbiased running first moment of loss function. At iteration we draw B samples {mrm LB and compute
tnn
Cr n l, B.
With fixed decay / E (0, 1), update the (biased) first moment exactly as in the approach of the Adam opti- mizer [26]:
je + Bie-1 (1 - E)
k..-
B
nm1
To remove the initialization bias,
ftx
Pe
g
Using St 1e as baseline, the Monte Carlo gradient estimator becomes
1 (la. log (m)m..))
V
B
Because a independent of each my,, the estimator remains unbiased while the baseline substantially reduces its variance.
E.2 Controlling learning rate via the information gain
When using stochastic gradient descent for optimization over parameters of both qu and it is necessary to use careful scaling of the estimated gradients to ensure the optimizer does not "drop off clift" into a local minimum. Such phenomena has been observed in related fields such proximal policy gradients [46] where the authors demonstrate empirically such necessity reinforcement learning problems. essence, want to control the learning rate of z with respect controlling this rate and leave mathematical properties for the optimal scaling to future research.
convergence of á + Φ², We show empirical results for
One approach is to control the rate of information gain (IG) of qu during the simultaneous optimization over both and é. By assuming bounded rate of information gain for (achieved via gradient clipping) only
23

---

T. Defining the IG in terms of need to consider computing the IG over successive d% for steps 1 I,..., entropy, have
TG
(411) 2 H( (1) H(g
(48)
When qu is categorical distribution, this quantity is available analytically. However, in general this is not available, but it estimated via Monte Carlo integration and importance sampling using available quantities (see Appendix F.3). We choose set a threshold for the between steps, denoted Bio(e) and then at each successive step we scale v using an iterative method such as bisection'.
E.3
Monte Carlo estimation of information
The helow procedure assumes ge represents distribution over strings of Bemouulii variables. Let ab E R parameterize a masked autoencoder that determines logits for a product Bernoulli distribution
To (m) II (NN (m)) m1() [1- o(NN()) (m))]]=m²
m E {0. 13
with MADE logits N (-) and (NN = (1 + t = (N
After an SGD proposal = 1b - a Ve, we
estimate the entropy reduction
AZ(a) = H(a)) - H(ar),
H(p) = - log((m)
(49)
To reduce computation at the expense introducing some estimation bias, employ importance weights re-use the current sample of model indicators in an iterative search to scale the gradient step. Draw mini-bateh [m (m)))) ) ~ qu once; no re-sampling is needed afterwards. Because the expectation in equation 49 switches from qe gur, rewrite
HI H(g..) Em [war. (m)) log gor (m)), (p)r Por (m (m))
(50)
For Bernoulli product the weight factorizes:
(NN (m00)))))1 [1 (NNN 11
Wor.e v (m7) =
(NN p (NN (NN (NN() = NNN(((m) NNO = -NN(#) (m))
(51)
(NN(()) implemented stably via (NN
-NN(1) expp[oo(() +e NN log(11 +e" (r)))
The mini-batch estimator is therefore
N
Ax(*)) = Σ
uyy (m) log (m)
Ax(b) Σ log (m)
(52)
Given tolerance 0, reduce 0.5 until
(x() - xx S..
(53)
no a > 10 satisfies equation 53, discard the update by setting the gradient to 0. Otherwise, apply the accepted scaled gradient.
'In preliminary investigations, other approaches for implementation of this threshold such as constrained optimization and computation of Lagrange multipliers were trialed without success, possibly due to the geometry of the optimization landscape.
24

---

F
Robust variable selection example details and additional results
The likelihood is
p(u: wi ((
+ a Mui
(54)
with priors p(n) = 2 and p(.3) NO I..). Each of the parameters in the likelihood are described in Table under the Misspecificatio::Nnne column. The data generating setup in Table 2 describes three levels misspecification to induce poor identifiability and thus a posterior that is challenging to fit using simple variational density families, such as mean field inference. This exemplifies the use of normalizing flows for this experiment. While many parameters are shared. some differ strongly between the likelihood and DGP. In particular, notice the difference in d1, 02. Also, for the highly misspecified DGP, correlation between included covariates and excluded covariates j induced by factor of Pt.j 0.1 for proportion of j, making the recovery the DGP using any inference method challenging and improbable task. For every data set, B will be either B1 or 32 with probability 0.5.
Table 2: Data generating setup
Parameter
Misspecification to likelihood
None
Mid
High
Number of data points x
50
Dimension of |
Dimension y
[ Probability of inclusion 1'(7%-- = 1)
27
128
0.4
Non-outlier 01
4
Outlier
10
4
Probability of correlation P(Pij
0/% 1. = 0)
0
0.4
Total correlation factor Ej Pi.j
0
0.1
B B2 Outlier probability a
0.5
0.5
1.5
0.1
Lastly, during the inference process, we consider two separate experiments for each DGP: a "focused-prior" experiment where о₁ = 1.5, and a "wide-prior" experiment where од = 10. These two scenarios cause a significant difference between the inferred reversible jump MCMC model probabilities and the inferred VTI model probabilities,
be seen in the subsequent figures.
VTI inference was conducted cluster of GPU nodes with mixed Nvidia RTX3090 and H100 cards. On the former we used float32 precision for MLP architectures, the latter used float64.
F.1 Focused versus wide priors
Each Figures 1-10 replicate of Figure 2 in the main text, showing sweep of 10 randomly generated data sets (indicated by different colours) according to corresponding setup Table using three different variational families: diagonal Gaussian MIP (a CoSMIC mean-field variational family), a composition of 5 affine masked autoregressive flows each with 5 hidden blocks, and composition of 4 rational quadratic spline masked autoregressive flows each with 6 hidden blocks. The expressiveness of each variational family increases from left right each figure.
the 0y = 1.5 focused prior setting (Figures 7, 9) performance is generally good, as per Figure text: (i) the model probability estimates (top row) tend to move closer to the y = line as the expressiveness of the variational family increases (left to right plots): (ii) the slight S-shape of the model probability estimates around the y line easily interpretable the the variational objective L(v, Φ) (equation 9) will naturally favour models with higher posterior model probability over those with lower probabilities; (iii) true data generating process models (triangles) are generally given high posterior model probabilities; and (iv) individual model posteriors are better estimated for higher probability models (negative slope on the bottom rows).
main
For the 10 wide prior setting (Figures 6, 8, 10) performance at first glance appears much worse, particularly terms of estimating model probabilities. However, on closer inspection this not the case. It is well known
25

---

(e.g. [18]) that the marginal likelihood (a.k.a. model evidence; component of the posterior model probability) highly sensitive diffuse priors. In such cases with σ 10) the posterior will tend unreasonably favour those models with fewer parameters, and particularly (in the case regression models) null model with no predictors, even the presence a very clear relationship between predictors and response. This effect can be clearly seen in Figures 6, 8, 10 (top row), where the null model (indicated by circle) is given far higher posterior model probability the true data generating process (triangles) is generally given high posterior model probability (comparable VTI approximation. From these results we conclude
the r(m) axis than the actual data generating process (triangles). In contrast,
with the focused prior setting in Figures 7, 9) under that: (i) the posterior model probabilities that depend on the marginal likelihood (i.e., the estimates of π ((m) on the rr-xii) have been affected by the wide prior to unreasonably favour models with less parameters; (ii) the VTI-based posterior model probability estimates suggest that they less sensitive the undesirable effects of prior; and (iii) combination the resulting plots Figures 6, 10 (top row) only appear to indicate worse performance of VTI compared to the gold standard than is actually the case.
F.2 Within model comparison
Figure 11 illustrates typical comparison between the reversible jump MCMC estimated posterior distribution and the VTI approximation. The figure shows the posterior of the data generating process model from the first high misspecification dataset Figure (main text). While there are some small difterences, the main features of the posterior appear to be well captured.
Facrova prier, mispecificatiu loach Nove Diagonal Guussian MLP
Focnoud price mixevzifizriim loacl: Nenr Affine MAP (5.5)
Focrsed price misepcificciion livel: Nuvo Spline MAF (4.0)
10"1
16"1
1075
10-12
10-1
5.10
(0 (n). (P v0))
102
135
300
107
122
100
[(m)
m(m)
Figure 5: As Figure 2 (main text), but under: no misspecification (01 1,02 = 10), focused prior (03 = 1.5). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
26

---

Widk print, mispellicatio lodi: None Diigonal Conssaan MLP
Wiik mioqucilicaiin leval: Noaz Aftine MIP MAP(5.5) (5,5
Wale peicx, misspeeiiioiio level: Naxt Spline (AA
107
127
10"11
10"11
10m1
10
10"
10mm
1FCe ve (P 1977
167
10P
107
102
10
10
10
S
161
[(m)
Figure 6: As Figure 2 (main text), but under: misspecification (01 1,02 10), wide prior (0g = 10). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
Frossse prict, misqecificttion lovcl: Modium Diagonal Guussian MLP
Faceosed mispecificatii loudl Medium Affine MAP (5,5)
Fiemndd prior, miserriffcatio loacl Medliam Spline MAF (4.0)
10"1
10"1
10m5
00
10--7
10-17
5.10
(2 mn). (P m0))
10-
135
300
107
100
[[m)
r(m)
Figure 7: As Figure 2 (main text), but under: mid misspecification (0) = 2, 5), focused prior (03 = 1.5). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
27

---

Wide peice, misppsillaaiion lovel: Maliun
Widk insoosiiciiinn loxel Malium Aftine MNP (5,5)
Wide miopecilicaiin levdl Medium
Dizgonal Conssaan MLP
Spline (4A
107
127
10"11
10"1
10TT
10
10
10m3
1FCe ve (Po 1977
107
10
100
10
10
5
5 16
k(m)
Figure 8: As Figure 2 (main text), but under: mid misspecification (01 2, d2 5), wide prior (0g = 10). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
Foursed price. missecifiction leneh: High
Ficesed print mispecdiiation lovel: High Affine MAP (5.5)
Fostrood prior, misqurificttio lovel High Spline MAF (4,0)
Diagonal Guussian MLP
10"1
16"1
1075
10:17
10-1
5.10
(2 (n). (8 m0))
102
135
200
107
100
[[m)
m(m)
Figure 9: As Figure 2 (main text), but under: high misspecification (0, = 4, 4). focused prior (03 = 1.5). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
28

---

Wale price, nisppcllicaiion levcl: High Diagonal Gonssaan MLP
Wide mispucilictiion lovel: High Aftine MAP (5,5)
Widk prior, mispostication lodd High Spline (AA((
107
10"1
10-11
1011
10
10m3
2FCe ve (Pom 197
107
107
100
10
10
5
k(m)
Figure 10: As Figure 2 (main text), but under: high misspecification (01 4,02 = 4), wide prior (0g = 10). Circles indicate the null model (constant only, no predictors); triangles indicate the data generating process.
Figure 11: Multivariate plot comparison between reversible jump MCMC (left) and VTI (right) using spline flow composition four layers and six blocks on the first synthetic narrow-prior high- misspecification data set from the Figure (main text) example.
29

---

F.3 Baseline reversible jump MCMC for robust variable selection
Consider the lincar model y X3 with (00, We introduce binary mask {0, 1 to indicate active coefficients in 8 E Rº, The reversible jump MCMC algorithm explores the model space by proposing bit-flips m, corresponding to adding (birth) or removing (death) predictors.
Jacobian Determinant: For bit- Sit-flipping moves in saturated space where the dimensionality remains constant (dim(m') = dim(m)), the transformation is bijective with a Jacobian determinant of 1:
a(m' .
a(m, 8)
Thus, the Jacohian does not affect the acceptance prohability.
Birth Move (m, + (m',,) :): A birth move flips bit in - mn' from 0 to 1. Given the birth/death move ratio
ply 3,m²) (mm)p(C m²) m(m()
Tba(m, m', B) pyy B. m) p(3 m) ((m)
the acceptance probability is:
Omam(m, m', 3) = min (1, roamm, m',
Death Move (m, 3) (ml' 9): death move flips a bit in - m' from to O. Using same birth/death move ratio, the acceptance probability is:
ddeath (m, m', 9) = min {1, m'm
Within-Model Gaussian Proposal 8y:
Within fixed model m. propose 8 using symmetric
random-walk:
p(y 8" m) P( m)
Owithin (m, 8' min
ply B8 p, m)
Since the proposal is symmetric, the proposal densitics cancel out in the acceptance probability.
30

---

G Example description: Bayesian inference of multi-layer-pereeptoon directed acyclic graph discovery
Notation:
Na
number of nodes in graph
numher of data samples (rows are i.i.d. samples)
X TERE R
P E PNa
permutation matrix (node order)
U E 10 Nax
strictly upper-triangular edge mask
A P UP
adjacency in canonical order (code default)
pa (
11
parents of node in the sorted order.
Node-wise conditional mean: Fix hidden width H and model indicator (P,U) U). For cach non-root node = 2, .,... Na define parameters
0 (W ( w bl.)) R
with w ER b1th ERR W ER
ER. Let u, = U1: the 1e-tttt of active parents. Writing X1g-1 to denote the 1, l,....j , columns of X.
1(11-11: 06) U) = w( ReLU(w (X (X u.) + b b
fi()= 0.
(S5)
Gaussian likelihood:
Let be the permutation associated with P (so X x is column after sorting).
With homoscedastic noise o²,
Na
nNd log log(2mo²)
[xx(1) (2) (1:-11); ((1)))))) u))²
log 1l X P.U.O))
20
(
Parameter prior (masked i.idd. Gaussian): the mask. Then
Let C(m) S {l..... dim 0} be the index set that survives
p(O U. II
N(Ox;0,0ō)
LEC(m)
(parameters outside ((m) are handled by reference density).
Structural prior:
p(P.U) U)
exp( (UI))
A X20, 0,
with P permutation matrix and U strictly upper triangular.
The un-normalised neg-posterior is the sum of the three boxed terms above.
G.1 Data generating process
The data generating procedure generally follows the simulation design in Thompson et al. [52].
Global hyper-parameterss
Na number of nodes, H hidden width, x² noise variance,
PEdge E (0, 1): edge probability, do parameter prior scale.
31

---

Sample graph structure:
P Uniform (Px,))
Us Bernoulli (pEdge)
siijje & Na.
A = UP.
Sample node parameters: Let the bias flag 8 E (8 = keeps both bias vectors, , = 0 scts them 0). For cach non-root node j .,..., Na draw independently
0 (wi!) (bb w (b
[
ii [-0.7, -.33 0.7].
while the root node has (*)) Ø. Note active parameters are drawn uniformly from non-zero range rather than from the prior.
Context-to-mase map: For m (P, U), = C(U) S (I,..., dim 0) keeps exactly the coordi-
nates satisfying the conditions:
1. Column of w! is active iff Ut, 1;
If E Uy = then parameters g are masked.
(The permutation has no effect on the mask.)
Data generation (topological order): to the permutation induced by For each sample
1y... Ti generate sequentially
x(*)
=(1) E161
x x()
(a)
[x(x (1 U)) t j 2....,
where Ej. iid N/ 1) and
fz2;000 U) w ReLU( (elU(W +3,14 u, = U10-n5
Collecting the draws gives
x
X
E
R
stored in topological order (X Xa(Na)).
x
G.2
Comparison metrics
Given knowledge of "true" adjacency matrix A, each experiment uses four for comparison with the estimated postcrior: F1, structured Hamming distance (SHD), Brier score, and area under the receiver operating characteristic curve (AUROC). This follows the experiment setup in Thompson et al. [52].
G.3
Common inference setup
For each data set in both the simulation study and real data example, VTI is run total of 10 replicates using different random seeds, and the posterior is selected where the terminal loss is minimized. For DAGMA, the sparsity hyperparameter swept from X min 10-3 to m 1 over 10 logarithnically spaced values. For the autoregressive flow, we use Affine(5.5) (see Appendix A.2) with context encoder designed as follows:
8(P. U) o [x2]
[x2] (P UP).
where of [x2] (x) = Wæ + broadcasts from [x] to the first power of greater than equal to 2/x]. The final dimension of OP, U) is 4096.
32

---

G.4 Simulation design
simulation study, configuration MLP follows. We set hidden layer width H 10. We set the number nodes to Na 10. We omit the bias parameters (/t) b5a) for all edges, i.e. set The edge inclusion probability is set PEdge 0.5. For VTI, the model prior p(m) is uniform (i.c. the sparsity parameter is set
0).
We generate 10 i.i.d. complete data sets of length Tmma 20 from the above process. The experiment compares data size against metrics from Appendix G.2. The range of data sizes are n 16, 32, 4, 128, 256, 512, 1024, where n < Tmmax simply takes the first samples.
VTI inference was conducted on cluster of GPU nodes with mixed Nvidia RTX3090 and H100 cards. On the former used float32 precision for MLP architectures, the latter used float64.
In the DAGMA setup, sweep of the regularization tuning parameter À was conducted for each dataset. The resulting adjacency matrix with the closest number active edges This resulted in a higher than-usual usual score for DAGMA results in the simulation study when compared to other methods. For DiBS/DiBS+, the inference ran for 5, 000 steps over 10 "particles" (each an individual Stein variational gradient descent optimization, see [32]). JSP-GEN was configured to use batch size of 1024 over 50, 000 iterations.
data- getterennraatng graph was selected.
G.5 Real data example
For VTI, chose to penalized structural model prior p(m) that induces "extra" sparsity via further down-weighting the probability of graphs with edges in order reach acceptable level of closeness the "consensus" graph in Sachs et al. [44]. should noted that no other experiment do we use sparsity-inducing priors. We set\ = 200 and set the number of hidden nodes per edge to = 5 and include the bias terms, i.e. g = 1.
For DAGMA non-linear, DiBS/DiBS+, and JSP-GFN, we use 10 hidden nodes per edge and no bias term.
G.6 DAG Model indicator construction: Lehmer Code Decoding
permutation of the ordered set {1, 2,....... Na)is represented by a Lehmer code e = (c1, C .N,), where E (0. 1....,Na +). At step i S Na) we choose the (c: + 1)-th unused index in the remaining ascending list.
Example. For Ng 5 and (2,1,0,0,0)
= 2: (1,2,344,5)-
1: (1,2,45))
c 0: (1,4,5)
e = 0: (1,5)
= 0:
[5]
Permutation-matrit representation. P.. iff row chosen at column i.
The permutation is stored as a one-hot P E {0, Nax with
Algorithm decodes each column in parallel. For column the code & [0, Na i]specifies "pick the (k+1)--th leftover row." The Boolean mask marks currently unused rows; broadcasting the flattened one-hot vector onto the corresponding (batch, row) pairs writes the unit entries. Column Na is filled by the single row that remains unassigned. This implementation gives compact (B, .N.) tensor, expanded by the decoder to (B, Na, Na) for efficient batched linear algebra in DAG-inferena pipeline.
33

---

Algorithm 2 Vectorized Lehmer decode via leftover mask
Require: Podee NBxNa (batch of Lehmer codes) )
Ensure: P {0, (0,1)BxNaXN))
I: bs + B
2: P -- 0bsx Na
3: for to Na do
4: k + Poode i
5: OneHot & one_ one_ho(kk Na i+ (shape = bs x (Na - i+ 1))
6: Used Σ cl c=1 7: Mask + (Used = 0)
8: Idx < nonzero nonzero(Mask)
9: P P| Idx/.11. i]
Eeshape(OneHon -1)
10: end for
Na-1
11: Used 4- Σ Pland
cml
12: Last < nonzero (Used = 0)
13: P P[Last [ast Na) <-
14: return
G.7 Model identifier for directed acyelic graphs
We encode a permutation matrix P (0. [J.1NNaxN, NaxNa using a compressed Lehmer code consisting of Na 1
categorical variables (pfat ..... PNa-13 }.. Here cas has N₂ -i outcomes.
Concretely, gi E {0, I,..., Na 1}, ps E {0, 1, 1...,,,.,,,,,,. Na 2}..... PN columns are fixed, the last column is forced.
cut
10, Once the first Na -
Each p = is mapped to one-hot vector of length Na. The value k selects the (k+1)--tt available row for the i-th column; previously taken rows remain zero, preserving the permutation property.
Given we form an upper-triangular mask E {0, NaNN with zero diagonal. Each entry above diagonal (i < f) is Bernoulli variable, U flattens Ma(N,-1) hits. The adjacency matrix P P UP, giving a DAG.
We concatenate the Na- categorical codes with the 12 Bernoulli bits, yielding vector of length (Na 1) -1) MADE consumes together with multiplier_ft fn specifying the parameter count for each entry.
Let 2, denote the j-th component of z:
Na J. j 1...., No 1, 1.
multiplier_fn(f
j Na..... No 1 Na(Na-1)
The architecture yields the autoregressive factorization
Na-14 Ma(Ma-1)
p(z)
II
p(z) 221).
The identifier (pi ..... PN Uinaay is modelled autoregressively by single MADE network, yielding A PP UP upon sampling.
34

---

We employ a structural prior over the space of models with the edge-penalty term
1
p U 1) = No! 2 MaiN
exd(-ynEdges(U)
(56)
nEdges (U) Σ
(57)
sKj
Na(N, 1) log 2 - >nEdges( (U).
logp = log((N)))
(58)
Note that when 0, the prior is uniform.
G.8 Neural probability mass function for model indicators over large spaces: MADE*
To represent a distribution over binary strings, we use the Masked Autoencoder for Density Estimation (MADE) [19] implementation found in the Durkan et al. [14] repository. To represent a more complex discrete distribution such as that required hy the P, U representation of directed acyclic graph, apply simple extension this architecture allow us to vary the output dimension multiplier. For presentational clarity call this extension MADE+. The key change in MADE the introduction of per-dimension output multiplier function r(i) that determines how many parameters are emitted for the i-th input dimension in the autoregressive factorization.
In the original MADE, all features share a common multiplier k; yielding an output dimensionality of & x d when there are input features. Mathematically. if x E R. the network outputs (h1,h2,.... h2...., hhd) E T kd
In MADE*, function r : {0, l,...... - 1
N is provided, and the final output dimension dd-1 r(t). For cach input dimension z, the network outputs r(i) parameters. Concretely, where is the number of input features, the final output dimension becomes total_out, totooouutuuurreee = Σd-1 r(i). other words, each input x, can be associated with custom number of distributional parameters (c.g., to handle discrete variables of different cardinalities). The masking logic is preserved by replicating each degree, deg(æ;), exactly r(t) times in the final layer.
Below is simplified, side-by-side pseudocode comparing MADE (left) and MADE (right). Changes in MADE are highlighted in green.
Algorithm 3 Original MADE (Final Layer Construction)
Algorithm 4 MADE
(Final Layer Construction)
out features features * output multiplier final Jayer_ = ManaedLinear(
total out features = 2i=0 final mnyl_ MaskedLinear(
features- r(i)
in in_degrees- out fer_feetures- out features,
in in_degrees= = prev prev_out__et _ot__deerees, out features total out features,
autoregressive features features, is_ ss_output= True
autoregressive_feane features = features, is "_outppt True,
output_multipiir__ff r())
By allowing each input dimension X; to have its own output multiplier r()), the MADE architecture provides more flexible autoregressive decomposition:
p(x)
II p( 215-..,
where now conditional distribution for ir; can parameterized by r(1) parameters (e.g.. logits for categorical variable of size r(i), or a mcan/variance pair, etc.).
Hence, one can naturally combine discrete variables of varying dimensions such as Bernoulli and categorical variables. For example, if xı is categorical with 10 categories and 12 is a Bernoulli variable, one can specify r(0) 10 and r(1) 1, that the overall conditional densities (or probability mass functions) multiply form richer joint model adapting precisely each variable's nature.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introductioo/ccontrrbttioo are met in Sections 2, 3, and demonstrated in numerical experiments in Section 5.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While there is explicitly -tilee Limitations section, (i) the effects of using normalising flows with poorer/stronger degrees expressivity on the quality
VTI approximation are explicitly discussed in Section 5 (Experiments) and Appendix F (Additional Results); the nature of the VTI approximation over transdimensional model space tending to focus more on high posterior model probability modcls is discussed in Section (Discussion); and the limitations of particular methods of approximating qu (m.) depending on the cardinality of the model space IM] are discussed in Sections 3.2, 3.3 and (Discussion).
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The methods presented in Section 2 and Section 3 are formally analyzed in Section A and Section C respectively. The authors unaware of any unaddressed assumptions.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Every effort was made to disclose all of the information needed to reproduce the experimental results. Scction F and Section G contain details of the setup of experiments that are not included in Section 5.
Justification:
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes]
Justification: Code will be provided the supplementary materials.
6. Experimental setting/details
Question: Does the paper specify training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Almost all details are provided in the main paper or supplementary material appendices. Any smaller details not provided in these sources can he determined from the supplementary code itself. These include the hyperparamcters of the optimizer, learning rate schedulers, and
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiment design runs multiple data sets and optimizer seeds. Figures and 3 show стror bars. No statistical tests are performed.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
36

---

Answer: [Yes]
Justification: Compute time and resources were not considered be the main focus the paper's contribution (which is the development of the first approach to extend variational inference to the transdimensional setting), however Section F and Section G.7 mention the GPU architectures used for each experiment.
9. Code of ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code Ethics https://neurip.. htcpp///e..ccieaaaa.......eeeee csGuidelinss?
Answer: [Yes]
Justification: To the best knowledge of the authors, the paper conforms, in every respect, with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: To the best knowledge the authors, paper describes a novel methodology that no direct societal impact.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The experiments use only freely- available data sources.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data sources are referenced and the licence and terms are properly respected. Only open source librarics were used in the code.
13. New assets
Question: Are alongside assets?
introduced in the paper well documented and is the documentation provided
Answer: [NA]
Justification: No new assets are released apart from the eventual release of the code used for experi- ments with the appropriate CC license.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: crowdsourcing experiments were conducted.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: No crowdsourcing experiments conducted.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it important, original, or non standard component of the core methods in this research? Note that if the LLM used only for writing. editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
37

---

Answer: [NA] Justification; LLM technology was not used for any purpose other than grammar and minor textual edits.
38


Reviews:
[{"review_id": "XLQ7QCBfXx", "reviewer": "Reviewer_cEAN", "summary": "This paper introduces an extension of conditional normalizing flows for transdimensional target distributions, where the parameter dimensionality $d_m$ depends on the discrete model index $m$. The authors propose a novel variational transdimensional inference (VTI) framework with two implementations: one leveraging Bayesian optimization via Gaussian process surrogates, and the other using Monte Carlo gradient estimation, which is more suitable for large-scale settings. Theoretical results are provided under certain assumptions, including convergence guarantees for marginal target distributions. The method is validated through numerical experiments on Bayesian robust variable selection and structure discovery in directed acyclic graphs (DAGs), demonstrating its effectiveness in challenging transdimensional inference problems.", "questions": "- Are there other existing methods that could be adapted to the tasks used in your experiments? Including a broader set of baselines would help clarify whether the proposed approach achieves state-of-the-art performance.\n\n- The problem setting of transdimensional target distributions is compelling. Could you provide additional real-world application domains where such distributions naturally arise, beyond the two studied examples?\n\n- How is the variational distribution amortized over the model space $\\mathcal{M}$? More specifically, how is the conditional dependence on the model $m$ incorporated into the formulation of $q_\\phi(\\theta_m, u_{\\backslash m}|m)$? A more detailed description of the conditioning mechanism would be helpful for understanding and reproducing the method.", "limitations": "- The method relies on careful design of conditional flows (CoSMIC flows) and masking strategies, which may require nontrivial tuning and domain knowledge to generalize to new tasks.\n- The empirical evaluation is limited in baseline comparisons and application domains, making it difficult to assess how broadly the method generalizes or whether it consistently outperforms existing approaches.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- As far as I am concerned, designing variational inference approaches for transdimensional posterior estimation where models have varying parameter dimensionality, is new. This represents a novel and important contribution to the field.\n- The paper also provides theoretical results regarding the convergence guarantees for the marginal distribution over the model space $\\mathcal{M}$.\n\n**Weaknesses**:\n\n- The experimental evaluation is limited in scope. The proposed method is not compared with alternative approaches in the first application (Bayesian misspecified robust variable selection), and only a single baseline is considered in the second (Bayesian nonlinear DAG discovery). Including comparisons with additional relevant baselines (e.g., [1], [2]) would better contextualize the method's performance.\n- The paper offers limited discussion of the computational cost and scalability of the method, particularly for the Bayesian optimization variant involving Gaussian process surrogates. A more detailed analysis of runtime, memory requirements, and scaling behavior—especially in high-dimensional model spaces—would enhance the practical relevance of the work.\n- Variational inference methods are known to be prone to mode collapse, potentially underrepresenting multi-modal posterior distributions. It remains unclear whether the proposed approach is susceptible to this issue, and if so, whether any design choices (e.g., CoSMIC flows or masking strategies) help to mitigate it. A discussion or empirical check on this point would be valuable.\n\n[1] Cheng, C., & Shang, Z. (2022). Robust Bayesian variable selection with model misspecification. Journal of Multivariate Analysis, 190, 104960. https://doi.org/10.1016/j.jmva.2022.104960\n\n[2] Miller, J. W., & Dunson, D. B. (2019). Robust Bayesian inference via coarsening. Journal of the American Statistical Association, 114(527), 1113–1125. https://doi.org/10.1080/01621459.2018.1429275", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns, and I will keep my positive score.", "rebuttal": "We thank the reviewer for taking the time to engage with our manuscript and for their feedback. We appreciate the effort to provide a thorough assessment of the work, and we value the opportunity to improve our submission. All comments and questions will be addressed individually in an inline fashion.\n\n> **Baseline comparisons:** Including comparisons with additional relevant baselines ... would better contextualize the method’s performance.\n\nWe agree that baselines must be appropriate to the developed methods. The goal of this paper is to introduce the very first method to achieve a flexible, *trans-dimensional* variational algorithm (VTI), which we obtain by developing the CoSMIC architecture for an inverse autoregressive flow. In this sense, the relevant baseline is the ground-truth, to show that the method both works and performs as expected. This is the case with the first experiment (Bayesian misspecified robust variable selection) in which we compare to the ground truth obtained by RJMCMC output. To our knowledge, there are no other baseline methods capable of producing a variational approximation to the transdimensional posterior.\n\nFor the second experiment (DAGs), we agree that additional baselines are more important. Here, because of the immense size of the model space a ground truth is difficult to obtain, and so we evaluated VTI performance via typical DAG diagnostics in Figure 3 and Table 1. We agree that in this setting comparison to a single method (DAGMA) is not sufficient. So far, we have run the additional benchmarks DiBS/DiBS+ (Lorch et al. 2021) (please see the revised results in the tables in response to zSgg). We\nexpect to run further benchmarks before the end of the reviewing period. We expect the additional benchmark performance to be within the range of DiBS/DIBS+ and DAGMA. What we are aiming to demonstrate with this analysis is that the performance\nof a generic methodology like VTI can be competitive with an\napplication-specific methodology. One would expect the application-specific methodology to perform best. We will improve the discussion on this point in the Experiments section. The additional results will be added to Figure 3.\n\nWe will use the additional page for the camera-ready version\nto expand upon experimental details and content. Some of this will\nentail bringing some of the content currently in the Appendix into the\nmain paper (e.g. Figure 4 plus discussion, and if possible, Figure 11\nplus discussion). It will also involve improving the quality of the\ndiscussion (emphasising important points), and bringing missing detail\nregarding the experimental setup (currently in Table 2) to enable more\ntangible understanding and discussion of the most relevant points\n\n> **Computational costs:** The paper offers limited discussion of the\ncomputational cost and scalability of the method, particularly for the\nBayesian optimization variant involving Gaussian process surrogates. A\nmore detailed analysis of runtime, memory requirements, and scaling\nbehavior—especially in high-dimensional model spaces—would enhance the\npractical relevance of the work.\n\nAlthough we do not provide a comprehensive study of the\ncomputational costs and scaling in all settings (model space\ncardinality, maximum dimension of the model parameters), we do provide a\nlimited study of performance when scaling the model space and parameter\ndimension simultaneously in Figure 4, Appendix E.1 (as discussed in the\nresponse to reviewer ctpC). Since our work builds on IAFs it is\nmore natural to refer to e.g. (Papamakarios et al. 2021) which provides\na general overview of the computational costs. As a rule-of-thumb, we found in practice\nthat the size of and choice of transform for the flow composition in\n$q_\\phi$ is subject to the problem type, parameter dimension, and\nmodel space cardinality. Problems requiring a highly expressive\nparameter posterior should use rational quadratic\nspline transforms in the flow composition. Problems with high parameter\ndimension may require additional transforms in the composition. Problems\nwith high model space cardinality may require a higher number of blocks\nin MADE (hidden layers) and/or additional transforms in the flow\ncomposition, as well as a context encoder. A brief description of the\nchoice of flow architecture is in Appendix A.2.\n\nFor the surrogate-based estimator, we apply incremental, fixed-cost\nupdates over the model space at each time step of the stochastic\ngradient descent loop. Although a naïve implementation of GPs would\nincur a cost $\\mathcal{O}(t^3 B^3)$ per SGD step $t$, equations 25\nand 26 in the appendix describe the updates for the posterior mean and\nvariance of a GP surrogate which we used in practice. In them, the\nkernel matrix is only evaluated over a $B$-sized batch, hence leading\nto a $\\mathcal{O}(B^3)$ matrix inversion cost per SGD step. Evaluating\n$M^2$ entries of the posterior predictive variance over a model space\nwith cardinality $M$ costs\n$\\mathcal{O}(B^3 + M B^2 + M^2 B) = \\mathcal{O}(M^2 B)$ per SGD step,\nas $B \\ll M$. Over a total of $T$ steps of SGD, we have a cost of\n$\\mathcal{O}(T M^2 B)$ due to surrogate updates. Hence, for moderately\nsized model spaces, these fixed-cost updates incur a lower cost if we\ncan update the posterior covariance matrix over the entire model space.\n\nWe agree these ideas need greater prominence, and so will provide both\nenhanced and additional discussion as part of moving Figure 4 content\n(plus discussion) from the Appendix to the main paper.\n\n> **Mode collapse:** Variational inference methods are known to be prone\nto mode collapse... . It remains unclear whether the proposed approach is\nsusceptible to this issue, and if so, whether any design choices (e.g.,\nCoSMIC flows or masking strategies) help to mitigate it. A discussion or\nempirical check on this point would be valuable.\n\nThis is an important question that we did not address in detail. \nIn extending VI to the transdimensional setting we inherit the same strengths and weaknesses of classical VI, and we make no claim to have eliminated the challenges of mode-collapse. Users of this method would need to take the same steps to manage it in the flows as in the non-transdimensional setting. In the case of mode collapse for the model distribution , we address this (within the limits of the specification of $q_\\psi$) with our exploration versus exploitation strategies discussed in Section 3.\n\nThank you for raising this point. We will include a detailed discussion\naround this issue in the revised manuscript.\n\n> **Other applications:** Could you provide additional\nreal-world application domains where such distributions naturally arise,\nbeyond the two studied examples?\n\nTrans-dimensional posteriors commonly arise in\nmany other areas, including phylogenetic tree topology search (Everitt\net al. 2020), geoscientific inversion (Sambridge et al. 2013), gravitational wave detection (Littenberg and Cornish 2009). The revision will include these citations.\n\n> **Amortization:** How is the variational distribution amortized over the model space $\\mathcal{M}$?\n\nAs discussed in Section 3, we train one shared recognition\nnetwork $f_{\\phi}$ that (i) is conditioned on model index $m$ to\noutput $q_{\\phi}(\\theta_{m},u_{\\setminus m}\\mid m)$ in a single\nforward pass, and (ii) undergoes heavy optimization during\ntraining so that subsequent inference—across any model or\ndataset—requires only a relatively cheap feed-forward evaluation (no\nper-model VI runs). We will improve the text in Section 3 to make this\nclearer.\n\n> **Conditioning mechanism detail:** how is the conditional dependence on\nthe model $m$ incorporated into the formulation of\n$q_\\phi(\\theta_m,u_{\\setminus m}|m)$?\n\nAn equation describing the conditioning\nmechanism is provided in Appendix A.3, lines 564-571, given inputs $x$ and context\n$z$, where the context is what we use as the conditioner. This approach\nis inherited from the *nflows* package and based on the work in Papamakarios et al. (2017) and Durkan et al. (2019). Additionally, we use a\ncontext encoder to enhance expressivity over the model space, \ndescribed briefly in Appendix A.2 lines 550-552. We appreciate that this\ncould be better explained, and so we will expand the description in the\nAppendix to more clearly describe the architecture of the context\nencoder for each experiment.\n\n> **Tuning complexity:** The method relies on careful design of\nconditional flows (CoSMIC flows) and masking strategies, which may\nrequire nontrivial tuning and domain knowledge to generalize to new\ntasks.\n\nThank you for raising this point. We respectfully disagree\nthat the design demands non-trivial masking strategies; in our opinion\nit is substantially less specialized than competing methods. We provide\na cookbook approach to designing the masking (Figure 1 and Appendix A.2) enhancing automatability. \nExtending to other problem types is more \nstraightforward than, for example, designing new RJMCMC \nproposals. The need for domain\nknowledge extends only to that required for the set up and specification\nof a typical Bayesian transdimensional inference problem in the choice\nof priors and likelihood. \n\nIn order to make this clearer for the reader, we will improve the\nDiscussion on this point in the revised version.\n\n# References\nEveritt, R.G., Culliford, R., Medina-Aguayo, F., Wilson, D.J. 2020. “Sequential Monte Carlo with Transformations.” *Statistics and Computing* 30: 663–76.\n\nLittenberg, T.B. and Cornish, N.J., 2009. Bayesian approach to the detection problem in gravitational wave astronomy. Physical Review D—Particles, Fields, Gravitation, and Cosmology, 80(6), p.063007.\n\nSambridge, M.,  Bodin, T., Gallagher, K., Tkalcic, H. 2013. “Transdimensional Inference in the Geosciences.” *Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences* 371 (1984): 20110547."}, {"review_id": "L4btMWlWzR", "reviewer": "Reviewer_zSgg", "summary": "The paper introduces a flow-based framework for variational inference on a space composed of discretely indexed parameter sets of differing dimensions, i.e., for _transdimensional inference_. To achieve this, independent _auxiliary variables_ are introduced to increase the dimension of each set up to a prescribed value (defined as the maximum dimension within the transdimensional space). The authors refer to this approach as CoSMIC. \n\nBy demonstrating that the learned distribution is factorizable with respect to these auxiliary variables (Corollary 2.3), the authors construct a Gaussian Process (GP)-based surrogate for approximating the posterior distribution (Section 3.1). They also introduce a low-variance gradient estimator for the learning objective for when the discrete component of the transdimensional space exceeds the computational memory limit (Section 3.3). An empirical analysis of the proposed methods in simulated experiments confirms their effectiveness (Section 5). \n\nAll in all, the work is well-articulated and clearly motivated. In spite of the limited experimental evidence, I believe that CoSMIC paves the road for further research on amortized VI for problems like variable selection and structure learning.", "questions": "See Weaknesses. \n\nAlso, when discussing reversible jump MCMC (Section E.4), the defined ratio $\\alpha$ for the death move is always greater than or equal to one. I believe $\\alpha_{\\text{birth}}$ should refer to the ratio within the minimum operator. Is that right?", "limitations": "Limitations of CoSMIC have been properly addressed throughout the manuscript.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n\n1. The paper is well-written, clearly motivated, and mathematically precise. \n\n2. The authors propose a simple and effective solution to an important problem in the field of variational inference. \n\n3. The authors clearly distinguish the use cases for the surrogate-based and the Monte Carlo approximations of the target distribution - depending on the cardinality of the discrete component of its support.\n\n4. Applications on variable selection and structure learning highlight the method’s usefulness on well-known problems.\n\n5. Experimental results attest the effectiveness of CoSMIC. However, the empirical analysis could be more comprehensive; see below.\n\n6. Reproducibility: the code has been publicly released in the supplementary material. \n\n## Weaknesses \n\n1. We are often interested in sparse solutions to the presented applications of variable selection (Section 5.1) and structure learning (Section 5.2). However, CoSMIC requires padding each parameter set to a prescribed dimension, which may unnecessarily increase the method’s complexity. Could the authors elaborate on the introduction of sparsity priors to the proposed solutions?\n\n2. I did not completely understand Figure 2’s bottom row. Is there a discernible pattern when looking at the plots from left to right? I would expect the cross-entropy to decrease as we increase the complexity of the variational density. \n\n3. Bayesian structure learning and causal discovery with non-linear mechanisms has been a topic of major interest in recent years. VGB [1], JSP-GFN [2], DiBS [3], and BCD Nets [4] are just a few examples. Authors should discuss the relatively restricted baseline choice (DAGMA) in further detail. \n\n4. [Minor] Proposition 3.1 is straightforward; any distribution over a discretely indexed set can be uniquely represented as a distribution over the set of indices. The paper would be strengthened if the corresponding space had been used for a discussion on, e.g., further baseline methods.\n\n[1]  Bayesian learning of Causal Structure and Mechanisms with GFlowNets and Variational Bayes. arXiv.\n\n[2] Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Network. NeurIPS 2023. \n\n[3] DiBS: Differentiable Bayesian Structure Learning. NeurIPS 2021. \n\n[4] BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery. NeurIPS 2021.", "was_revised": true, "final_justification": "This paper introduces a novel and principled approach for approximated Bayesian inference in spaces of varying dimension. The authors have clearly addressed my concerns, and I will keep my positive score.", "rebuttal": "We sincerely thank the reviewer for their detailed and insightful\ncomments, as well as for their strong interest in our work and\nmethodology. Their expertise is evident in the depth of the review, and\nwe are grateful for the opportunity to respond. We have carefully\nconsidered each of the points raised and have done our best to address\nall concerns. All comments and questions will be addressed individually\nin an inline fashion.\n\n> **Experimental detail:** Empirical analysis could be more comprehensive.\n\nWe agree, and we will use the additional page for\nthe camera-ready version to expand upon experimental details and\ncontent. Some of this will entail bringing some of the content currently\nin the Appendix into the main paper (e.g. Figure 4 plus discussion, and\nif possible, Figure 11 plus discussion). It will also involve improving\nthe quality of the discussion (emphasising important points), and\nbringing missing detail regarding the experimental setup (currently in\nTable 2) to enable more tangible understanding and discussion of the\nmost relevant points.\n\n> **Sparsity:** If we’re interested in sparse modelling solutions, do we\nreally need to increase the modelling complexity to $d_{max}$? Can we\nelaborate on introducing any \"sparsity priors\" to the proposed\nsolutions?\n\nThank you for this insightful question. In terms of what\nis implemented in the experiments: We use a prior that induces “extra”\nsparsity in the real-data example for non-linear DAGs (Sachs et al\n2005). The use of this prior is to *further* down-weight the probability\nof graphs with more edges in order to reach an acceptable level of\ncloseness to the “consensus” graph in Sachs et al 2005. In no other\nexperiment do we use sparsity-inducing priors.\n\nAs to the wider question: Introducing sparsity-inducing priors will tend\nto produce a posterior with little weight on non-sparse solutions\n(unless justified by the data). Because we know (see experiments and\nDiscussion section) that most of the effort in approximating the\nposterior focuses on high posterior model probability models, this means\nthat even though the flows are defined in $d_{max}$-dimensional space,\nthe extra dimensions used in the higher-dimensional but\nlower-probability models will have little impact on minimising the loss.\nFrom this perspective the ability of VTI to approximate the posterior is\nnot sensitive to this extra dimensionality, even though in practice the\nCoSMIC setup requires its specification. In general though, this\nquestion does open the door to future research on how to reduce these\npractical requirements (beyond hard-specification of a reduced\n$d_{max}$), perhaps via a more dynamic (run time) setup, though this\nmay present implementational difficulties. Thank you for raising this\nquestion – we will include a discussion on this point in the Discussion\nof the revised version.\n\n> **Figure 2:** I did not completely understand Figure 2’s bottom row. Is\nthere a discernible pattern when looking at the plots from left to\nright? I would expect the cross-entropy to decrease as we increase the\ncomplexity of the variational density.\n\nWe apologise for the lack of clarity in Figure 2. You are\ncorrect to expect the cross-entropy to decrease when looking from left\nto right. In Figure 2 there are actually two problem types: medium\nmisspecification (panes 1, 2, 3), and high misspecification (panes 4, 5,\n6). Left to right (increasing flow expressivity) for each problem type\nwe observe decreasing $H$. We will improve the plot with a clear\ndividing line between the two problem types to improve readability, and\nalso improve the caption and surrounding text to make this more clear.\n\n> **DAG experiment baselines:** Authors should discuss the relatively\nrestricted baseline choice (DAGMA) in further detail.\n\nThank you, we agree and will include the cited papers in\nthe discussion. So far, we have run additional benchmarks using\nDiBS/DiBS+ (Lorch et al. 2021) in addition to DAGMA for the 10-node MLP DAG synthetic study (please see the revised figures in the\ntables below, these have been added to Figure 3 in the\nrevised paper), and we expect to run further benchmarks before the end\nof the reviewing period. We expect the additional benchmark performance\nto be close to VTI performance. (As an aside: we have found a small\nerror in our plotting code, and DAGMA performance is now much closer to\nVTI in line with expectations.) What we are aiming to demonstrate with\nthis analysis is that the performance of a general methodology like VTI\ncan be competitive with an application-specific methodology (like DiBS,\nDAGMA). In general, of course, one would expect the application-specific\nmethodology to perform best. We will improve the discussion on this\npoint in the Experiments section, along with the new benchmark results.\n\n> **Proposition 3.1:** \\[This\\] is relatively straightforward ... The\npaper would be strengthened if the space had been used for a discussion\n...\n\nWe agree, and will recover this space to be used for the improvements discussed above/below.\n\n> **MCMC acceptance probability:** Also, when discussing reversible jump\nMCMC (Section E.4), the defined ratio for the death move is always\ngreater than or equal to one. I believe should refer to the ratio within\nthe minimum operator. Is that right?\n\nThank you for picking up on this error. We mistakenly\nwrote that the acceptance probability is the reciprocal of\n$\\alpha_{\\mathrm{birth}}$, whereas it should read\n\n$r_{\\mathrm{birth}} = \\dfrac{p(y | \\beta, m')p(\\beta | m') p(m')}{p(y| \\beta, m) p(\\beta | m) p(m)}$,\n\n$\\alpha_{\\mathrm{birth}} = \\mathrm{min} \\(1,r_\\mathrm{birth}\\)$,\n\n$\\alpha_{\\mathrm{death}} = \\mathrm{min}\\left\\(1,\\dfrac{1}{r_{\\mathrm{birth}}}\\right\\)$.\n\nWe will correct this in the manuscript. The acceptance probability is implemented correctly in the code.\n\n# Extra baseline results for figure 3\n\n**F1**\n| Sample size | VTI | DAGMA | DiBS | DiBS+ |\n|-------------|------------|------------|------------|------------|\n| 16 | 0.165±0.021 | **0.275±0.034** | 0.013±0.006 | 0.035±0.014 |\n| 32 | 0.248±0.029 | **0.376±0.030** | 0.056±0.019 | 0.100±0.030 |\n| 64 | 0.370±0.042 | **0.467±0.032** | 0.098±0.028 | 0.162±0.049 |\n| 128 | 0.374±0.035 | **0.454±0.028** | 0.164±0.035 | 0.236±0.049 |\n| 256 | 0.420±0.035 | **0.472±0.031** | 0.227±0.035 | 0.338±0.033 |\n| 512 | 0.466±0.033 | **0.499±0.035** | 0.266±0.033 | 0.405±0.045 |\n| 1024 | 0.516±0.038 | **0.522±0.045** | 0.309±0.028 | 0.437±0.025 |\n\n**SHD**\n| Sample size | VTI | DAGMA | DiBS | DiBS+ |\n|-------------|------------|------------|------------|------------|\n| 16 | 24.143±0.610 | 32.889±1.149 | 22.089±0.869 | **21.857±0.941** |\n| 32 | 24.274±0.856 | 29.667±2.629 | 21.689±0.912 | **21.136±0.941** |\n| 64 | 22.218±1.045 | 23.111±1.746 | 21.278±0.954 | **20.226±1.195** |\n| 128 | 24.226±0.931 | 24.444±1.268 | 20.622±0.991 | **19.486±1.262** |\n| 256 | 24.908±1.040 | 22.778±1.412 | 20.067±0.955 | **18.090±0.957** |\n| 512 | 26.296±1.190 | 22.333±1.918 | 20.100±0.932 | **16.889±1.201** |\n| 1024 | 25.610±1.576 | 20.889±1.908 | 19.800±0.844 | **17.000±1.018** |\n\n**Brier**\n| Sample size | VTI | DAGMA | DiBS | DiBS+ |\n|-------------|------------|------------|------------|------------|\n| 16 | **20.844±0.792** | 32.889±1.149 | 21.993±0.901 | 21.811±0.961 |\n| 32 | **20.102±0.820** | 29.667±2.629 | 21.264±1.010 | 21.101±0.941 |\n| 64 | **18.428±0.962** | 23.111±1.746 | 20.526±1.088 | 20.168±1.193 |\n| 128 | 19.966±0.653 | 24.444±1.268 | **19.298±1.209** | 19.394±1.248 |\n| 256 | 21.283±0.975 | 22.778±1.412 | 18.049±1.171 | **17.923±0.968** |\n| 512 | 23.956±1.257 | 22.333±1.918 | 17.268±1.131 | **16.888±1.201** |\n| 1024 | 24.173±1.587 | 20.889±1.908 | **16.224±1.022** | 17.000±1.018 |\n\n**AUROC**\n| Sample size | VTI | DAGMA | DiBS | DiBS+ |\n|-------------|------------|------------|------------|------------|\n| 16 | 0.527±0.008 | **0.531±0.020** | 0.503±0.002 | 0.509±0.004 |\n| 32 | 0.553±0.013 | **0.593±0.022** | 0.515±0.005 | 0.527±0.008 |\n| 64 | 0.612±0.021 | **0.657±0.018** | 0.526±0.008 | 0.547±0.015 |\n| 128 | 0.611±0.018 | **0.651±0.016** | 0.545±0.011 | 0.570±0.017 |\n| 256 | 0.633±0.021 | **0.662±0.019** | 0.564±0.012 | 0.603±0.013 |\n| 512 | 0.663±0.021 | **0.676±0.023** | 0.576±0.013 | 0.632±0.020 |\n| 1024 | **0.706±0.026** | 0.693±0.027 | 0.590±0.012 | 0.641±0.011 |"}, {"review_id": "RCImxoWH54", "reviewer": "Reviewer_ctpC", "summary": "The paper proposes a flow-based variational approximation method for inference over models with varying parameter dimensionality, supporting targets where some parameters can be present or non-present and where the dimensionality of the each parameter group can vary as well. Varying dimensionality within each group is handled by 'dimension saturation' that expands the dimensionality to some assumed maximum, whereas a few alternative methods are proposed for addressing the discrete choice over the groups. The method is demonstrated in two experiments but its properties are not analysed in detail.", "questions": "- How well the Monte Carlo estimator for high cardinality works in practice? If used for problem of low cardinality, is it roughly as good as the GP solution or substantially worse? If worse, should we expect it to be accurate for high cardinality?\n\n- Corollary 3.2 and the paragraph below that talks about convergence of the UCB-based solution but makes no reference to $\\beta$, implying that already the mean (using $\\beta \\approx 0$) would work as well. Can you clarify what is happening here? I probably missed something about the claim.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper addresses an important problem formulation and extends the flexibility of flow-based approximations. The solution is novel and highly non-trivial, despite largely leveraging existing tools as part of the solution. The paper provides good formal treatment of the topic, defining all necessary quantities clearly and provides useful theoretical guarantees for some non-trivial components. For modelling the weights over the parameter groups the authors provide multiple alternative solutions even though already a single one would have been sufficient for publication.\n\nThe breadth of the technical contribution is hence commendable, but this comes with a cost: The presentation if very dense and the paper is difficult to read for anyone who is not already familiar with all of the technical components and theories needed for the development. Moreover, the point of presenting three alternative solutions in Section 3 is a bit lost when the evaluation largely skips this question and just presents the overall performance without even clearly telling which variant was used. This is a bit problematic also in terms of the conclusions: Section 6 states the choice depends on the cardinality of the model space, but this is not really backed up by empirical evidence. Since the MC estimator does not seem to be empirically validated in detail, I wonder whether it is useful to describe it in the main paper. Perhaps that method variant itself could be in the appendix, leaving space for bringing in some of the extra experimental evidence from the appendix to the main paper, or for opening up the method description for a broader audience?\n\nThe experimentation is done nicely on two rather different kind of problems, but the presentation in the main paper is a bit superficial and we do not learn very much about *how* the method really works since the experiments mostly just confirm it work. For example, for Section 5.1 there are missing details that are only in the Appendix (the true $\\sigma$) and Figure 2 only compares alternative versions of the proposed method (that all look fairly similar) without providing any context in form of alternative methods like sampling. The appendix includes such comparison (Figure 11 showing the approximation is extremely similar to MCMC) that would have been highly useful already in the main paper, or at least the main paper should clearly tell the result is there.", "was_revised": true, "final_justification": "The paper is in general of high quality with no major open issues. The authors addressed my questions during the review well, but they were minor aspects and there is no need for me to change the overall evaluation.", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments,\nand we appreciate their interest in our paper. The feedback provided has\nhelped us refine our approach, and we hope that our responses clarify\nand strengthen the manuscript. All comments and questions will be\naddressed individually in an inline fashion.\n\n> **Presentation and clarity:** Presentation is dense and paper is\ndifficult to read. The paper should open up to a broader audience.\n\nOn reflection we agree, and will aim to use the\nadditional page for the camera-ready version to improve overall paper\naccessibility and readability for a wider audience. This will be in\naddition to the specific changes discussed below.\n\n> **Model weights variants:** The point of 3 alternative solutions is lost\nwhen the evaluation largely skips this question. The performance of the\nMC gradient estimator as depending on model space is also not validated\nin detail.\n\nThank you for this insight. We will a) improve the text in Section 3 to\nmake clearer the potential value and need for using different model\nweights estimators, and b) explicitly state in all experiments which\nparticular estimator was used. Here, every experiment uses the Monte\nCarlo gradient estimator, whether as the main approach (Figures 2 & 3,\nTable 1) or in comparison to the variants (Figure 4). We discuss the MCG\nestimator performance in a separate comment below.\n\n> **Detail of experimentation:** The presentation in the main paper is a\nbit superficial and we do not learn very much about how the method\nreally works.\n\nThank you – we agree. In line with the above responses we\nwill use the additional manuscript space to expand upon experimental\ndetails and content. Some of this will entail bringing some of the\ncontent currently in the Appendix into the main paper (e.g. Figure 4\nplus discussion, and if possible, Figure 11 and discussion). It will\nalso involve improving the quality of the discussion, and bringing\nmissing detail regarding the experimental setup (such as the true\n$\\sigma$) into the main paper.\n\n> **High cardinality MC estimator:** How well the Monte Carlo estimator\nfor high cardinality works in practice? If used for problem of low\ncardinality, is it roughly as good as the GP solution or substantially\nworse? If worse, should we expect it to be accurate for high\ncardinality?\n\nIn Appendix E.1, Figure 4, we conducted a comparison\nbetween the diagonal surrogate, categorical Monte Carlo gradient, and\nneural (MADE, Germain et al. 2015) Monte Carlo gradient (MCG)\napproaches on problems of increasing model space cardinality on two\ndifferent levels of misspecification for the robust variable selection\nproblem type. We fixed the number of iterations and the architecture of\nthe normalizing flow (RQS). From this, we observed that the MCG methods\nare competitive with the Gaussian surrogate approach, and in some cases\nperformed slightly better (though this is subject to tuning of both\napproaches). The benefits and drawbacks of each approach cannot be made\nin a broad statement, but rather are problem specific. For instance, we\nprovided theoretical guarantees for the surrogate approach in Section 3.\nAlso, there could be scope for future research into exploiting model\nspace structure for the surrogate approach. The benefit of the MCG\nmethods is, as you summarised, applicability to very high cardinality\nmodel spaces where we use a neural (MADE) architecture for the model\ndistribution. Figure 4 shows comparable empirical performance on the\nrobust variable selection problem in model space cardinalities up to\n$2^{24}$. The drawbacks are not concrete: we do not show the same\ntheoretical guarantees, but this could be a topic of future research.\n\nIn order to communicate this comparison more clearly, we will move this\ncontent (Figure 4 plus discussion) to the main paper, and improve the\nclarity and quality of the discussion.\n\n> **UCB convergence:** Corollary 3.2 and the paragraph below that talks\nabout convergence of the UCB-based solution but makes no reference to\n$\\beta$, implying that already the mean (using $\\beta\\approx 0$)\nwould work as well. Can you clarify what is happening here? I probably\nmissed something about the claim.\n\nIndeed, the value of $\\beta$ should not affect the\nasymptotic convergence rate, according to our theoretical analysis,\nthough, in practice, it helps the algorithm in the short term. This\nindependence on $\\beta$ is due to the result in Corollary 3.2 mainly\nonly depending on maintaining a positive sampling probability over all\nmodels in the model space, which then allows us to apply the second\nBorel-Cantelli lemma to bound the posterior variance of the GP surrogate\n(Lemma B.2) and control the KL divergence of the approximation."}, {"review_id": "Z1aKhw4qpf", "reviewer": "Reviewer_W8Fg", "summary": "The paper proposes a new normalizing flow architecture to handle “transdimensional posteriors”, that is, posteriors across models (with potentially different support) and the model index. They provide some theoretical insights on their models and evaluate it on some relatively simple benchmarks.", "questions": "- In which sense is the approach “amortized”. Do you mean amortized over models? Or some other space? I am asking because the word “amortized” is used is many different ways in the literature.\n- Do I understand it correctly that we are using reverse KL for learning and hence need an analytic likelihood?\n- I consider the use of autoregressive flows as a weakness (see above). Why is that necessary? Could we use other flow architectures or do we need the specific autoregressive structure for the transdimensional part?\n- You say that rather than training a density for each individual model seperately, it is preferable to construct a single density. In which circumstances specifically is this likely to be true?", "limitations": "Limitations were not really mentioned in the discussion I believe. E.g., what are potential scalability issues to high dimensional model and/or parameter spaces? I assume there are some limitations in that regard but I don't see it discussed. Similarily, the limitations of the chosen architectures doesn't seem to be discussed really.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "tengths:\n\n- The paper addresses an important problem of doing efficient inference across a large model space\n- The theory seems sound based on the limited time I took to study it.\n\nWeaknesses:\n\n- autoregressive flows are in some sense outdated compared to coupling flows and nobody really uses autoregressive flows in practice anymore. I don’t quite understand why this kind of flows where chosen. Is this necessary or just some arbitrary decision.\n- The solution to the model index surrogate seems a bit unncessarily technical. why would we consider a GP vs. also just using a neural network as your second proposal seems to suggest?\n- The experiments are not super convincing to me. Are the achieved results good enough to justify the use and interpretation in practice? E.g., how good is the entropy achieved in Experiment 1?", "was_revised": true, "final_justification": "Autheros responded thoughtfully to my concerns and partially addressing them.", "rebuttal": "We thank the reviewer for their time in evaluating our submission. While\nwe note the critical tone of the review, we appreciate the opportunity\nto respond and clarify aspects of our work. We have made a sincere\neffort to address all points raised. All comments and questions will be\naddressed individually in an inline fashion.\n\n> **Analytic likelihood:** Do I understand it correctly that we are using\nreverse KL for learning and hence need an analytic likelihood?\n\nThe paper does indeed use a reverse KL loss for learning\nas mentioned in equation 2. The implementation requires a target\ndensity that can be evaluated up to a normalizing constant and is\ndifferentiable (usually using standard autograd tooling built into\nwidely accepted libraries like PyTorch, TensorFlow or JAX). In the\nBayesian setting, where the target is proportional to the product of the\nlikelihood and prior, this requirement translates to only needing the\nlikelihood to be differentiable and evaluable up to normalization. The\nlikelihood does not need to have a closed‐form density; in fact, this is\na motivating problem behind transdimensional inference, where the\nnormalizing constant of each model is intractable. We will aim to\nclarify the text on this point.\n\n> **Flow choice:** I consider the use of autoregressive flows as a\nweakness... Why is that necessary? Could we use other flow architectures\nor do we need the specific autoregressive structure for the\ntransdimensional part?\n\nIn the variational inference setting, inverse\nautoregressive flows are preferred over coupling flows due to:\n\n1.  Faster sampling during training (Kingma et al. 2017; Durkan et al.\n    2019). Autoregressive transformations use MADE which has an\n    asymmetrical computational cost depending on the direction of the\n    transformation. Inverse Autoregressive Flows are implemented to have\n    computational cost $\\mathcal{O}(1)$ (and are parallelizable) for\n    sampling and evaluation of those samples as a by-product, which is\n    the only operation used in the variational inference optimization\n    loop. This means conversely the evaluation of out-of-distribution data points\n    has the sequential cost $\\mathcal{O}(d_{\\mathrm{max}})$ due to sequential inverse\n    evaluation operation that is often cited as the drawback of these\n    flows for generative models. Since the use case is primarily to\n    search over a large model space in a variational inference setting,\n    and to obtain expectations with respect to the amortized variational\n    density, this is not a limitation in our setting.\n\n2.  Higher expressiveness overall (Coccaro et al. 2024).\n\nThe desired factorization that is proved in Proposition 2.2 does rely on\nusing the autoregressive property of the flow, however we do not see a\nbarrier to deriving a CoSMIC architecture specifically for coupling\nflows in applications outside variational inference, such as\nsimulation-based inference. This would be an interesting topic for\nfuture research. We will add some comments along these lines in the\nDiscussion.\n\n> **Jointly learning densities for models:** You say that rather than\ntraining a density for each individual model separately, it is\npreferable to construct a single density. In which circumstances\nspecifically is this likely to be true?\n\nThis approach is preferable in large model‐space settings.\nFor instance, the model space of directed acyclic graphs (DAGs) scales\nsuper‐-exponentially: in the non‐linear DAG example, the model space of\n10‐node DAGs has cardinality $|\\mathcal{M}|=4 175098976430598143$. In\nsuch scenarios, it is infeasible to store individual normalizing flows\nfor each model. We will revise our statement to include this explicit\nexample.\n\n> **Entropy gap + architectural limitations:** How good is the entropy\nachieved in Experiment 1?\n\nFigure 2 offers a holistic assessment of inference quality\nrelative to a sampling baseline using RJMCMC. The entropy is reduced as\nflow expressivity increases; it also reduces with the size of the model\nprobability (Figures 2, 5-10). The entropy gap scales with the\nmodel‐space size (Figure 4) when the flow architecture is held fixed.\nThis matches expectations about neural density expressivity and\nindicates that flow design should be tailored to the problem domain. We\nwill improve our description of this in the Experiments by bringing Figure 11 plus discussion to the main paper as a visual indicator of how good it is, and clarify the\nlimits of the flow architecture in the Discussion.\n\n> **Simplicity of benchmarks:** They ... evaluate it on some relatively\nsimple benchmarks.\n\nWe agree that the models used are initially relatively\nsimple. That said, the robust variable selection problem is designed to\nbe easily scaled to increased complexity in dimension and model space\ncardinality (see Figure 4, Appendix E.1). We also tackle non-linear\nDAGs, which are NP‐hard and non‐trivial in both computational complexity\nand setup. Our goal is not to dominate state‐of‐the‐art benchmarks but\nto introduce a novel variational method generally applicable to challenging domains with few\ncomparable approaches. We will include a statement along these lines in\nthe revision.\n\n> **Scalability:** What are potential scalability issues to high\ndimensional model/parameter spaces?\n\nThank you for raising this question. Our response to\nreviewer cEAN on scalability provides some background on potential\nscalability issues to high dimensional parameter spaces and high\ncardinality model spaces. In addition, in Section 3 we derive scaling\nproperties for the GP-surrogate model distribution. In particular, high\ncardinality model spaces do require careful construction of a context\nencoder $\\delta(m)$. This encoder is learned along with the flow, and\nis intended to non-linearly project $m$ to a higher dimension to\nimprove expressiveness of the residual block described in Appendix A.3.\nWe will add a paragraph to the discussion and experiment setup\ndescribing this consideration in the revised version, and refer the\nreader to the appendices for details.\n\n> **Model index surrogate:** \\[A GP\\] seems ... unnecessarily technical vs\njust using a neural net\n\nThank you for this astute observation. We agree that in\npractice it is safe to default to using Monte Carlo gradient estimation\nusing a categorical distribution in low cardinality problems, or a\nneural net like MADE in high cardinality problems. However, the GP\napproach offers a different framework with its own flexibility, offers\ncompetitive performance, and has the benefit of theoretical convergence\nproperties that we provide in Section 3. We will add a summary of this\nclarification to the discussion in the revised paper.\n\n> **Amortisation:** In which sense is the approach “amortized”\n(clarification)?\n\nHere we mean that, as discussed in Section 3, we train one\nshared recognition network $f_{\\phi}$ that (i) is conditioned on model\nindex $m$ to output $q_{\\phi}(\\theta_{m},u_{\\setminus m}\\mid m)$ in\na single forward pass, and (ii) undergoes heavy optimization during\ntraining so that any subsequent inference over the posterior/target\nsupport requires only relatively cheap evaluation of the model\ndistribution and evaluation of the IAF (no per-model VI runs). We will\nimprove the text in Section 3 to make this clearer.\n\n# Additional References\n\nCoccaro, Andrea, Marco Letizia, Humberto Reyes-González, and Riccardo\nTorre. 2024. “Comparison of Affine and Rational Quadratic Spline\nCoupling and Autoregressive Flows Through Robust Statistical Tests.”\n*Symmetry* 16 (8). https://doi.org/10.3390/sym16080942."}]

Author rebuttal:

Meta-review: The paper tackles the challenging question of how to perform amortized variational inference in settings where there is a prior over parameter spaces of different dimensions. The approach leverages existing tools like Gaussian process surrogates, the overall methodology is highly novel and enables amortized inference methods to directly tackle challenging and important model selection problems like feature selection and structure learning. The paper is well-written although fairly dense, so the authors have promised to use the additional camera-ready page to improve the accessibility of the writing. 

The paper stands out for an spotlight presentation because
1. It makes multiple technical contributions that could be of wide interest, including a new type of trans-dimensional normalizing flow and the new amortization method 
2. It tackles a problem not currently addressed in the literature (using amortized inference for model selection)
3. The empirical results are compelling and demonstrate the breadth of application
Decision: Accept (spotlight)