Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference
Authors: Shumao Zhang
Thomas Hou
Pengchuan Zhang
Venue:  (2021)

Abstract: High-dimensional Bayesian inference problems cast a long-standing challenge in generating samples, especially when the posterior has multiple modes. For a wide class of Bayesian inference problems equipped with the multiscale structure that low-dimensional (coarse-scale) surrogate can approximate the original high-dimensional (fine-scale) problem well, we propose to train a Multiscale Invertible Generative Network (MsIGN) for sample generation. A novel prior conditioning layer is designed to bridge networks at different resolutions, enabling coarse-to-fine multi-stage training. Jeffreys divergence is adopted as the training objective to avoid mode dropping. On two high-dimensional Bayesian inverse problems, MsIGN approximates the posterior accurately and clearly captures multiple modes, showing superior performance compared with previous deep generative network approaches. On the natural image synthesis task, MsIGN achieves the superior performance in bits-per-dimension compared with our baseline models and yields great interpret-ability of its neurons in intermediate layers.

Under review as a conference paper at ICLR 2021
MULTISCALE INVERTIBLE GENERATIVE NETWORKS FOR HIGH-DIMENSIONAL BAYESIAN INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
High- digh--dimensional Bayesian inference problems cast long-standing challenge in generating samples, especially when the posterior has multiple modes. For a wide class Bayesian inference problems equipped with the multiscale structure that www-imensional (coarse-scale) surrogate can approximate the original high- dimensional (fine-scale) problem well, we propose to train a Multiscale Invertible Generative Network (MsIGN) for sample generation. A novel prior conditioning layer is designed to bridge networks at different resolutions, enabling coarse-to- fine multi-stage training. Jeffreys divergence adopted as the training objec- tive to avoid mode dropping. On two high-dimensional Bayesian inverse prob- lems, MsIGN approximates the posterior accurately and clearly captures multiple modes, showing superior performance compared with previous deep generative network approaches. On the natural image synthesis task, MsIGN achieves the superior performance in bits-per-dimensoon compared with our baseline models and yields great interpret-ability of its neurons in intermediate layers.
INTRODUCTION
Bayesian inference provides a powerful framework to blend prior knowledge, data generation pro- cess and (possibly small) data for statistical inference. With some prior knowledge p (distribution) for the quantity of interest z E IRd, and some (noisy) measurement y E IRd,, y, it casts on æ posterior
(1) where L(y[x) is the likelihood that compares the data y with system prediction F(x) from the candidate x. here F denotes the forward process. We can different distributions to model the mismatch E = y - F(x), and for illustration simplicity, we assume Gaussian in Equation ! For example, Bayesian deep learning generates model predicted logits F(x) from model parameters x, and compares it with discrete labels y through binomial or multinomial distribution.
((xly) ((x)x( where L(ylx) N2y F(x);
Sampling or inferring from q is long-standing challenge, especially for high-dimensional (high-d) cases. An arbitrary high-d posterior can have its importance regions (also called "modes") anywhere in the high-d space, and finding these modes requires computational cost that grows exponentially with the dimension d. This intrinsic difficulty is the consequence of "the curse of dimensionality" which all existing Bayesian inference methods suffer from, e.g., MCMC-based methods (Neal al.
2011 Welling & Tch 2011 Cui et al. 2016), SVGD-type methods (Liu & Wang 2016 Chen et al.
2018
2019a), and generative modeling (Morzfeld al. 2012 Parno et al.) 2016 Hou et al. 2019)
In this paper, we focus on Bayesian inference problems with multiscale structure and exploit this structure to sample from high-d posterior. While the original problem has high spatial resolution (fine-scale), its low resolution (coarse-scale) analogy is computationally attractive because it lies in a low-dimension (low-d) space. A problem has the multiscale structure such coarse-scale low-d surrogate exists and gives good approximation to the fine-scale high-d problem, see Section 2. Such multiscale property is very common in high-d Bayesian inference problems. For example, inferring 3-D permeability field of subsurface at the scale of meters is reasonable approximation of itself at the scale of centimeters, while the problem dimension is 10 -tt-times fewer.
We propose Multiscale Invertible Generative Network (MsIGN) to sample from high-d Bayesian inference problems with multiscale structure. MsIGN is a flow-based generative network that can both generate samples and give density evaluation. It consists of multiple scales that recursively

---

Under review as a conference paper at ICLR 2021
lifts up samples to finer-scale (higher-resolution)) except that the coarsest scale directly samples from a low-d (low resolution) distribution. At each scale, a fixed prior conditioning layer combines coarse-scale samples with some random noise according to the prior to enhance the resolution, and then an invertible flow modifies the samples for better accuracy, see Figure | The architecture of MsIGN makes it fully invertible between the final sample and random noise at all scales.
Z2
Z
P
P2
F3
L
Gaussian Noise
Invertible
Prior Conditioning
X1
X
X2
X3
XL
Figure 1: MsIGN generates samples from coarse to fine scale. Each scale, as separated by vertical dash lines, takes in feature x,-1 from the coarser scale and Gaussian noise 2,, and outputs a sample X₁ of finer scale. The prior conditioning layer PC, lifts up the coarser-scale sample X;-1 to finer scale X1, which is the best guess of xı given coarse-scale value X the prior. An invertible flow Fi further modifies to better approximate X,. See Section 2. for detailed explanation.
MsIGN undergoes multi- state training that learns hierarchy of distributions with dimensions growing from the lowest to the highest (the target posterior). Each stage gives good initialization the next stage thanks to the multiscale property. To capture multiple modes, we choose Jeffreys divergence D2 (pllg) as the training objective at each stage, which is defined as
Da(pll4) = DKL (pllg) + (xl(y)))) Ex-p [log (p(x)/g(x)) + Ex-q [lg (q(x)/p(x))) (2)
Jeffreys divergence removes bad local minima single-sided Kullback-Leibler (KL) divergence to avoid mode missing. We build an unbiased estimation of it by leveraging prior conditioning layer in importance sampling. Proper loss function and good initialization from multi-stage training solve the non-convex optimization stably and capture multi modes of the high-d distribution.
In summary, we claim four contributions in this work. First, we propose Multiscale Invertible deep Generative Network (MsIGN) with novel prior conditioning layer, which can be trained in coarse-to-fing scale manner. Second, Jeffreys divergence is used as the objective function to avoid mode collapse, and is estimated by importance sampling based on the prior conditioning layer. Third, when applied to two Bayesian inverse problems, MsIGN clearly captures multiple superior
modes in the high-d posterior and approximates the posterior accurately, demonstrating performance compared with previous methods via the generative modeling approach. Fourth, we also apply MsIGN to image synthesis tasks, where achieves superior performance in bits-per- dimension among our baseline models, like Glow (Kingma & Dhariwal] (208)). FFJORD (Grathwohl et al. 2018), Flow++ (Ho et al. 2019), i-ResNet (Behrmann et al. 2019), and Residual Flow (Chen
al. 2019b). MsIGN also yields great interpret-ability of its neurons in intermediate layers.
METHOLOGY
We will abbreviate q(xly) in Equation 1 q(x) for simplicity in the following context, because y only plays the role defining the target distribution qxx) MsIGN. In Section 2.1 we discuss the multiscale structure in detail of the posterior gxx) and derive a scale decoupling that can be utilized to divide and conquer the high-d challenge of Bayesian inference.
As flow-based generative model like in Dinh al. (2016), MsIGN models a bijective that maps Gaussian noise z to sample x whose distribution is denoted as pe (x)), where 0 is the network pa- rameters. MsIGN allows fast generation of samples x and density evaluation pe px(x). we train our working distribution (x(x) to approximate the target distribution (xx). We present the architecture of MsIGN in Section 2.2 and the training algorithm in Section
2

---

Under review as a conference paper at ICLR 2021
2.1 MULTISCALE STRUCTURE AND SCALE DECOUPLING
We say Bayesian inference problem has multiscale structure if the associated coarse-scale likeli- hood Le approximates the original likelihood L well:
[yyx) L=(x)(x), where (e(v)x)) == Nuy Fc(x)): 0,T
Here x Rd is a coarse-scale version of the fine-scale quantity E Rd (de d), given by deterministic pooling operator A: X. = A(x). The map F: Rd + Rd is forward process that gives system prediction based on the coarse-scale information x A popular case of the multiscale structure is when A is the average pooling operator, and F(x) ~ Fx(xx), meaning that the system prediction mainly depends on the lower-resolution information x,. Equation B motivates us to define surrogate distribution ((x) x (z))) that approximates the target posterior ((x) well
q(x) = p(x) (y) ((x)) = - Lx)))) p(x) -(x).
We also notice that the prior p allows an exact scale decoupling. To generate a sample x from p. one can first sample its coarse-scale version x. A(x), and then replenish missing fine-scale details without changing the coarse-scale structure by sampling from the conditional distribution
p(x xx) p(x]A(x) xe). Using Pe to denote the distribution of x, A(x), the conditional probability calculation summarizes this scale decoupling process as p(æ:) = p(æ) ((tela)))p.(2
Combining the scale effect in the likelihood and the scale decoupling in the prior, we decouple the surrogate (xx) = (ey)) into the prior conditional distribution p(x x.) and a coarse-scale posterior, defined as (x.) == (g)) xx). The decoupling goes as
q(x) = px = |x0_:/x..___x(((((__0_00__p_>x)1 (x.) = p(x(xx))x
(5)
The prior conditional distribution p(x (x)) bridges the coarse-scale posterior gel (xx) and the surrogate ((x), which in turn approximates the original fine-scale posterior (x). Parno et al. (2016) proposed similar scale decoupling relation, and we leave the discussion and comparison to Appendix A
Figure D shows the integrated sampling strategy. To sample an x from q, we start with an x, from de. The prior conditioning layer then performs random upsampling from the prior conditional distri- bution p(- and the output will be a sample x of the surrogate q. Due to the approximation @ = from Equation 4 we stack multiple invertible blocks for the invertible flow F to modify the sample ~ 7 to a sample x ~ q: x = F(X). F is initialized as an identity map in training. Finally, to obtain the x, from qe, we apply the above procedure recursively until the dimension of the coarsest scale is small enough so that de can be casily sampled by standard method.
2.2 MULTISCALE INVERTIBLE GENERATIVE NETWORK: ARCHITECTURE
Our proposed MsIGN has multiple levels to recursively apply the above strategy. We denote the number of levels, x E dR the sample at level I, and A Rdd + did the pooling operator from level 7 to 1 1: X = Axxx). Following the idea in Section 2.1 we can define the I-th level target qi (xx) and surrogate q ). and the last-level target qL is original target q in Equation The 1-th level of MsIGN uses a prior conditioning layer PC₁ and an inverse transform Fi to capture qt-
Prior conditioning layer. The prior conditioning layer PC₁ at level lifts coarse-scale sam- ple X E Rd-1 up to a random fine-scale one X E Rd following the conditional distribution p( The difference in dimension is compensated by Gaussian noise z₁ E d which is the source of randomness: x (C Zi). PC; depends only on the prior conditional dis- tribution p(x; 1)) and thus can be pre-computed independently for different levels regardless of the likelihood L. When the prior is Gaussian and the pooling operators are linear (e.g., average pooling), the prior conditional distribution still Gaussian with moments specified follows.
Lemma 2.1 Suppose that pl (x1; O, Ex), and Ar(x)) Aixi for some At E Rdi- then with U-1 := A A Σ AT and Egj-1 = Σ E, (AIE,)))) A,E, we have
p(x = Aii)) = N U1-1X(11, E11-1)
We omit normalizing constants. Equivalence and approximation are up to normalization in the following.
3

---

Under review as a conference paper at ICLR 2021
With the Cholesky decomposition (or eigen-decomposition) Emje-1 B.B," we design the prior conditioning layer PCr below, which is invertible between x; and (xI-1, 2:):
X = PC((X((1,, == 1 + Biz 21 ~ (0, -d:-
(6)
We refer readers to Appendix B for proof of Lemma 2.1 and the invertibility in Equation
When the prior is non-Gaussian or the pooling operators are nonlinear, there exists a nonlinear invertible prior conditioning operator x PC zi) such that x follows the prior conditional distribution ((x|Xx-1) given X,-1 and Z ~ (O, -di-d:1) ). We can pre-train an invertible network to approximate this sampling process, and fix as the prior conditioning layer.
Invertible flow. The invertible flow Fj at level modifies the surrogate qi towards the target qt- The more accurate the multiscale structure Equation is, the better ÿ approximates qt, and the closer Fi is to the identity map. Therefore, we parameterize Fi by some flow-based generative model and initialize it as an identity map. In practice, we utilize the invertible block of Glow (Kingma & Dhariwal 2018). which consists of actnorm, invertible 1 x convolution, and affine coupling layer, and stack several blocks as the inverse flow Fi in MsIGN.
Overall model. MsIGN is bijective map between random noise inputs at different scales {z/ /= and the finest-scale sample xz. The forward direction of MsIGN maps {zi) to X below:
x = Fi (z)). x (C) X (i)) 2 2515L.
As flow-based generative model, sample generation as in Equation and density evaluation (x) by the change-of-variable rule is accessible and fast for MsIGN. In scenarios when certain bound needs enforcing to the output, For example, image synthesis can use the sigmoid function so that pixel values lie in [0, 1]. Such activations should be bijective keep the invertible relation between random noise the sample.
append element-wiss output activations at the end of MsIGN.
2.3 MULTISCALE INVERTIBLE GENERATIVE NETWORK: TRAINING
Since the prior conditioning layer PC is pre-computed and the output activation G' is fixed, only the inverse flow F contains trainable parameters in MsIGN. We train MsIGN with the following strategy so that the distribution pe of its output samples, where 0 is the network parameter, can approximate the target distribution defined in Equation [ well.
Multi-stage training and interpret-ability. The multiscale strategy in construction of MsIGN en- ables a coarse-to-fine multi-stage training. At stage I, we target at capturing Qi and only train invertible flows before or this level: Fr l. Equation 4 implies that qi can be well approxi- mated by the surogate Tt, which is the conditional upsampling from Qi-1 as in Equation 5 So we use q to initialize our model by setting Fe, l' l the trained model stage - and setting F as the identity map. Our experiments demonstrate such multi-stage strategy significantly stabilizes training and improves final performance.
Figure M and Equation 7 imply that intermediate activations, i.c., X; and X who are samples of predefined posterior distributions at the coarse scales (see Equation 5), are semantically meaningful and interpret-able. This is different from Glow (Kingma & Dhariwal 2018), whose intermediate activations are not interpret-able due to the loss spatial relation.
Jeffreys divergence and importance sampling with the surrogate. The KI. divergence is easy to compute, and thus is widely used as the training objective. However, its landscape could admit local minima that don't favor the optimization. Nielsen & Nock (2099 suggests that DKL (pellg) is zero-forcing, meaning that enforces ps be small whenever q is small. As a consequence, mode missing still be local minimum, see Appendix C Therefore, we turn to the Jeffreys divergence defined in Equation 2 which penalizes mode missing much and can remove such local minima.
Estimating the Jeffreys divergence requires computing expectation with respect to the target q, which normally prohibited. Since MsIGN constructs a good approximation ğ to q, and q can be constructed from coarser levels in multi-stage training, we do importance sampling with the
4

---

Under review as a conference paper at ICLR 2021
surrogate q for the Jeffreys diveregence and its derivative (see Appendix D for detailed derivation):
log Po po(x)
g(x)
g(x)
Da(pellu)= Exaup
+ Exxu
log
(8)
x(x)
((x)
px(x)
(
11 log Po(x)
θ log po (x)
x() d OlogPo
(x
00 PP(elll)) Exapo
Ex-q
(xx)
00
((x)
00
With the derivative estimate given above, we optimize the Jeffreys divergence by stochastic gradient
descent. We remark that log po (x)/00 00 is computed by the backward propagation of MsIGN.
RELATED WORK
Invertible generative models (Deco & Brauer 1995 are powerful exact likelihood models with efficient sampling and inference. They have achieved great success in natural image synthesis, see, e.g.. Dinh et al. y) Kingma & Dhariwal 2018); Grathwohl et al. (201 8); Ho et al. 2019); Chen al. (2019b), and variational inference in providing a tight evidence lower bound (ELBO), see, e.g. Rezende & Mohamed (201)). In this paper, we propose a new multiscale invertible generative network (MsIGN) structure, which utilizes the invertible block in Glow (Kingma Dhariwal 2018)
as building piece for the invertible flow at each scale. The Glow block can be replaced by any other invertible blocks, without any algorithmic changes. Different from Glow, different scales of MsIGN can be trained separately, and thus features in its intermediate layers can be interpreted as low- resolution approximation of the final high-resolution output. This novel multiscale structure enables better explain-ability of its hidden neurons and makes training much more stable.
Different from the image synthesis task where large amount of samples from target distribution are available, in Bayesian inference problems only unnormalized density is available and i..d. sam- ples from the posterior are the target. This paper's main goal is to train MsIGN to approximate certain high-d Bayesian posteriors. Various kinds of parametric distributions have been proposed to approximate posteriors before, such as polynomials (Hl Moselhy & Marzouk] 2012). non-invertible generative networks (Feng et al. 2017 Hou et al. 2019), invertible networks (Rezende & Mo- hamed, 2015) Ardizzone et al.) 2018 Kruse et al. 2019) and certain implicit maps (Chorin & Tu 2009 Morzfeld et al. 2012). Generative modeling approach has the advantage that i.i.d. samples can be efficienlly obtained by evaluating the model in the inference stage. However, due to the tricky non-convex optimization problem, this approach for both invertible (Chorin Tu 2009 Kruse et al.) 2019) and non-invertible (Hou et al. 2019) generative models becomes increasingly challenging as the dimension grows. To overcome this difficulty, we propose (1) to use the Jeffreys divergence as loss function, which has fewer shallow local minima and better landscape compared with the commonly-used KL divergence (see Appendix C for a concrete example), and (2) to train MsIGN in coarse-to-fine manner with coarse-scale solution serving as initialization to fine-scale optimiza- tion problem. In Kruse et al. 2019)) authors list some recent models for low-d inverse problems. We remark that their formulation of posterior assumes observation model error in Equation D and is different from ours. See Appendix D for detailed discussion and experimental comparison.
Other than the generative modeling, various Markov Chain Monte Carlo (MCMC) methods have been the most popular in Bayesian inference, see, e.g., Beskos et al. (2008); Neal et al. (201)); Welling & Tch (0LT); Chen et al. (2044) 2013); Cui al. (2016). Particle-ppimmizztion-baae pling is a recently developed effective sampling technique with Stein variational gradient descent (SVGD) (Liu & Wang 2016 and many related works, e.g., Liu (017) Liu & Zhu (018); Chen et al. [2018 2019a). The intrinsic difficulty of Bayesian inference displays itself as highly corre- lated samples, leading to undesired low sample efficiency, especially in high-d cases. The multiscale structure and multi-stage strategy proposed in this paper can also benefit these particle-based meth- ods, as we can observe that they henefit the Amortized-SVGG (Feng et al. 2017| 7 Hou et al. 2019
sam-
in Section
We leave more thorough study of this topic future work.
Works in Parno et al. 2016); Matthies et al. 2016) utilize the multiscale structure in Bayesian inference and build generative models with polynomials. They suffer from exponential growth of parameter number for high-d polynomial basis. The Markov property (Spantini et al. 2018) used to alleviate this exponential growth. Different from these works, leverage the great capacity of invertible generative networks to parametrize the high-d distribution, and we design novel network architecture to make use of the multiscale structure. The multiscale structure is
general
5

---

Under review as a conference paper at ICLR 2021
structure than commonly-usee intrinsic structure (Spantini 2017 2019a), which assumes that the density of high-d posterior concentrates in a low-d subspace.
Cui et al. 2016 Chen al.
In the image synthesis task, this multiscale idea incorporates with various generative models. For example. Denton et al. (201)); Odena et al. (017)): : Karras et al. (017); p Xu et al. (011)) uses it in generative adversarial networks (GANs) grow high-resolution image from low-resolution ones. But the lack of invertibility in these models makes difficult for them to apply to Bayesian infer- ence problems. Invertible generative models like Dinh al. (2016); Kingma & Dhariwal (2018); Ardizzone et al. (2019) adopted this multiscale idea, but their multiscale strategy is not in the spatial sense: the intermediate neurons are not semantically interpret-able, as we show in Figure
EXPERIMENT
We study two high-d d Bayesian inverse problems (BIPs) known to have at least two equally important modes in Section 4. as test beds for distribution approximation and multi-mode capture: one with true samples available in Section L. one without true samples but close to real-world applications in subsurface flow Section In addition, we apply MsIGN to the image synthesis task to benchmark with flow-based generative
We also report the ablation study of MsIGN in Section 4.1.3
models and demonstrate its interpret-ability in Section 4.2 We adopt the invertible block in Glow Kingma & Dhariwal 2018) the building piece, and stack several of them build our invertible flow F. We utilize average pooling with kernel size and stride 2 as our pooling operator A.
4.1 BAYESIAN INVERSE PROBLEMS
Sample x of our target posterior distribution q is a vector on a 2-D uniform 64 x 64 lattice, which means the problem dimension d is 4096. Every is equivalent to a piece-wise constant function on the unit disk: x() for E Ω [0, 1)² and don't distinguish between them thereafter. We place centered Gaussian with Laplacian-type covariance the prior: N (0, ( -)---)), which is very popular in geophysics and electric tomography. See Appendix for problem settings in detail.
The key to guarantee the multi-modalls of our posteriors is the symmetry. Combining properties of the prior defined above and the likelihood defined afterwards, the posterior is mirror-symmetric: (x((11, 82)) = (x((1,)) 1 s2)). We carefully select the prior and the likelihood that our posterior q has at least two modes. They are mirror-syy mirror-symmetrie to each other and possess equal importance.
As in Figure D we plan to learn our 4096-D posteriors at the end of 6 levels, and set problem dimension at each level as d; = 2' 2 = 4', The training follows our multi-stage strategy, and the first stage is initialized by minimizing the Jeffreys divergence without importance sampling, because samples to Q1 is available since d1 4 is relatively small. See Appendix E for details.
We compare MsIGN with representatives of major approaches: amortized-SVe SVGD (short as A- SVGD) (Feng et al.) (017) and Hamilton Monte Carlo (short as HMC) (Neal et al. (01T). for high-d BIPs, see our discussion in Section 3 We measure the computational cost by the number of forward simulations (nFSs), because running the forward simulation F occupies most training time, We budget same nFS for all methods for fair comparison.
especially in Section 4.1.2
4.1.1 SYNTHETIC BAYESIAN INVERSE PROBLEMS
This problem allows access to ground-truth samples so the comparison clear and solid. The for- ward process is given by F(x) (4, x)2 (Sp (s)))) where (p(s) sin(T)) sin(2r)2)) Together with the prior, our posterior can be factorized into one-dimensional sub-distributions.
namely xxx) I (( x)) for some orthonormal basis {mx}]= This property gives us access to true samples via inversion cumulative function sampling along each direction ur. Fur- thermore, these 1-D sub-distributions are all single modal except that there's 's one, denoted as qee, with two symmetric modes. In other words, the marginal distribution along wK. is double-model and the rest are uni-model. This confirms our construction of two equally important modes. See Appendix for more details in problem settings. The computation budget is fixed at 8 x 10" nFSs.
Multi-mode capture. To visualize mode capture, plot the marginal distribution of generated samples along the critical direction w.,, which by construction is the source double-modalle modality of the posterior. The (visually) worst one in three independent experiments is shown in Figure 2 a).

---

Under review as a conference paper at ICLR 2021
10
Mean
Standard Deviation
10
Correlation
Ground-truth MaIGN
Took Pouar Eriee Aening Drnatoaru
1.75
1.50
A-SVGD
1.25
10-1
0.30
M-ttN
HBCC
0.25
A.SVGD
Figure 2: Results of the synthetic BIP. (a): Distribution of 2500 samples along the critical direction wk. MsIGN more robust capturing both modes, its samples are more balanced. (b): Error mean and its 95% confidence interval. MsIGN is more accurate in distribution approximation, es- pecially at finer scale when the problem dimension is high. The margin is statistical significant as shown by the confidence interval. For more experimental results, please refer to Appendix F
Distribution approximation. To measure distribution approximation, we report the error of mean, variance and correlation or between all sub-distributions, as well as the Jeffreys divergence. Thanks to the factorization property, we compare the mean, vari- ance and correlation estimate with theoretical ground- truths, and report the root mean square of error at dimensions in Figure (b). For MsIGN and A-SVGD that gives access to not only samples but also density, we also report the Monte Carlo estimates of the Jeffreys divergence with the target posterior in Table I
Method
MsIGN
A-SVGD
Feng 2017
Error
56.77=0.15
3372+-21
Table 1: Distribution approximation error by Jeffreys divergence with the target pos- terior in three independent runs
We can see that MsIGN has superior accuracy in approximating the target distribution.
4.1.2 ELLIPTIC BAYESIAN INVERSE PROBLEMS
This problem originates from geophysics and fluid dynamics. The forward model is given by linear measurement of the solution to an elliptic partial differential equation associated with x. We define
F(x)= [Ja pa (s)u(s)ds Sn (=((ss(s))
Sa km (sm))s(ss)ss
where you are fixed measurement functions, and u(s) is the solution of
(e
[()) 8 E with boundary condition u(s) = 0, 8 E 0n
(10)
This model appears frequently in real applications. For example, can be seen as permeability field and pressure in geophysics. However, there no known access to true samples of q. Again the trick of symmetry introduced in Section 4.1 and explained in Appendix E guarantees least two equally important modes the posterior. We put a 5 x 10 10S-nFS budget on our computation cost.
PCA and Clustoring
Cluster 1 Mean
Cluster Mean
MSSGN
MMIGNN
HMC
Komal Dusatty Eiimoi
32
HMC
-50
A-SVCO
-24
Figure 3: Results of the elliptic BIP. (a): Distribution of 2500 samples along a critical direction.. MsIGN and HMC capture two modes in this marginal distribution, but A-SVGD fails. (b): Clus- tering result of 2500 samples. Samples of MsIGN are more balanced between two modes. The similarity of the cluster means of MsIGN and HMC implies that they both are likely to capture the correct modes. For more experimental results, please refer to Appendix !
Multi-mode capture. Due to lack of true samples, check the marginal distribution of the pos- terior along eigen-vectors of the prior, and pick a particular one to demonstrate that we can capture double modes in Figure a). We also confirm the capture of multiple modes by embedding samples
7

---

Under review as a conference paper at ICLR 2021
by Principle Component Analysis (PCA) to 2-D space. We report the clustering (by K- K-means) result and means of each cluster in Figure B B)), where we can see that A-SVGD failed to capture the two symmetric modes, while MsIGN has more balanced capture of the symmetric posterior.
4.1.3 ABLATION STUDY OF ARCHITECTURE DESIGN AND TRAINING STRATEGY
We run extensive experiments to study the effectiveness of the network architecture and training stragtegy of MsIGN, see Figure We refer Appendix G for details in setting and more results.
Network architecture. We replace the prior conditioning layer by two direct alternatives: a stochas- tic nearest-neighbor upsampling layer (model denoted as "MMIGN-SNN"), or the split and squeeze layer in Glow design (now the model is essentially Glow, so we also denote it as "Glow").
Figure de (a) shows that the prior conditioning layer design is crucial to the performance of MsIGN on both problems, because neither "MsIGN-SNN" nor "Glow" has a successful mode capture.
Training strategy. We study the effectiveness of the Jeffreys divergence objective and multi-stage training. We try substituting the Jeffreys divergence objective (no extra marks) with the KL diver- gence (model denoted with a string "-L"") or kernelized Stein discrepancy (which resumes A-SVGD algorithm, model denoted with string "-AS"), and switching between multi-stage (no extra marks) or single-stage training (model denoted with a string "-S"). We remark that single-stage training using Jeffreys divergence is infeasible because of the difficulty to estimate DRL (gllpe).
Figure (b) and (c) show that, all models trained in the single-stage manner ("MSIGN-KL-S". "MsIGN-AS-S") will face mode collapse. We also observe that our multi-stage training strategy can benefit training with other objectives, see "MsIGN-KL" and "MsIGN-AS"
We also notice that the Jeffreys divergence leads to more balanced samples for these symmetric problems, especially for the complicated elliptic BIP in Section 122
Network Architecture
Objective: KL Divergance
Objective: Kerrelized Stein Discreparcy
1.50
Crounskäruth MJ/Ch
Eround-trum
MalCNe-L KL-S
Groundersth Maiih
Glive
MS-UNAS-S
Synthell 3.P
0.75
0.50
-2
-1
-2
MaCh
MaIGHNEL MckiM KL
Mh
MalS McNGNASS 5
3.00
Slowe
Ellock BPP
2.20
1.20
0.50
-0.6
-0.2
0.4
0.6
-0.6 -0.4
-02
022
0.4
0.
-0. -0.6
-0.4
-3.2
5.2
0.4
Figure 4: Ablation study of the network architecture and training strategy. "MsIGN" means default setting: training MsIGN network with Jeffreys divergence and multi-stage strategy. Other models are named by base model (MsIGN or Glow), followed by strings indicating its variance from the default setting. For example, "MsIGN-KL." refers to training MsIGN network with single KL divergence in multi-stage way, while "MsIGN-KL-S" means traininng in a single-stage way.
4.2 IMAGE SYNTHESIS TASK
We train our MsIGN architecture with maximum likelihood estimation to benchmark with other flow-based generative models. The prior conditional distribution p(x[x,) is modeled by simple Gaussian with a scalar matrix as its covariance and is learned from a training set. We refer readers to Appendix H for more experimental details, and to Appendix for additional results.
We report the diim-eerddi value with our baseline models of flw-based generative networks in Table 2 Our MsIGN is superior in number and also is more efficient in parameter size: for examplc, MsIGN uses 24.4% fewer parameters than Glow for CelebA 64, and uses 37.4% fewer parameters than Residual Flow for ImageNet 64.
8

---

Under review as a conference paper at ICLR 2021
In Figure show synthesized images of MsIGN from CelebA 64 dataset, and linear interpolation of real images in the latent feature space. Figure & of the invertible flow at different scales which demonstrates the interpret-ability of MsIGN.
we visualize internal activations at checkpoints
Table 2: Bits-per-dimension value comparison with baseline models of flow-based generative net-
works. All models in this table do not use "variational dequantization" in Ho al. (0199) *; Score
obtained by our own reproducing experiment.
Model
MNIST
CIFAR-10
CelebA 64
ImageNet 32
ImageNet 64
Real NVP el.
1.06
3.49
3.02
4.28
3.98
Kingma GlowaKnggma & Dhariwal 2018
1.05
3,35
2.20
4.09
3.81
et
201
0.99
3.40
Flow+ Ho 2019
3.29
i-ResNNet Behrmann et
1.05
3.45
Residual Flow Cheo ct
2019b
0.97
3.28
4.01
3.76
MsIGN (Ours)
0.93
3.28
2.15
4.03
3.73
7
Figure 5: Left: Synthesized CelebA 64 images with temperature 0.9. Right: Linear interpolation in
latent space shows MsIGN's parameterization of natural image manifold is semantically meaningful.
Figure 6: Visualization of internal activation shows the interpret-ability of MsIGN hidden neurons. From left right, show how MsIGN progressively generates new samples in high resolution by taking snapshots at internal checkpoints. See Appendix []] for details.
CONCLUSION
For dith-dimensional Bayesian inference problems with multiscale structure, we propose Multi- scale Invertible Generative Networks (MsIGN) and associated training algorithms to approximate the high-dimensional posterior. this paper, we demonstrate the capability of this approach high- dimensional (up to 4096 dimensions) Bayesian inference problems with spatial multiscale structure, leaving several important directions as future work. The network architecture also achieves the state-of-the-art performance in various image synthesis tasks. We plan to apply this methodology to other Bayesian inference problems, for example, Bayesian deep learning with multiscale structure in model width depth (c.g.. Chang al. (201)) Haber et al. (2018) and data assimilation prob- lem with multiscale structure in the temporal variation (e.g., Giles (008)). We also plan to develop some theoretical guarantee of the posterior approximation performance for MsIGN.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint 1808::8808.0477 2018.
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks. arXiv preprint arXiv: 1907.02.97 2019.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573-582, 2019.
Alexandros Beskos, Gareth Roberts, Andrew Stuart, and Jochen Voss. Mcme methods for diffusion bridges. Stochastics and Dynamicy, 19-35,, 2008.
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual net- works from dynamical systems view. arXiv preprint 1010. 2017.
Changyou Chen, Nan Ding, and Lawrence Carin. On the convergence stochastic gradient meme algorithms with high-order integrators. In Advances in Neural Information Processing Systems, pp. 2278-2286, 2015.
Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. A unified particle- optimization framework for scalable bayesian sampling. arXiv preprint arXiv::8055 /159, 2018.
Peng Chen, Keyi Wu, Joshua Chen, Tom O'Leary-Roseberry, and Omar Ghattas. Projected stein variational newton: A fast and scalable bayesian inference method in high dimensions. In Ad- vances Neural Information Processing Systems, pp. 15104-15111 2019a.
Tian Qi Chen, Jens Behrmann, David Duvenaud, and Born-Herrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp- 9913-9923, 2019b.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pp. 1683-1691, 2014.
Alexandre J Chorin and Xuemin Tu. Implicit sampling for particle filters. Proceedings of the 2009.
National Academy of Sciences, (06(41):17249-17))
Tiangang Cui, Kody JH Law, and Youssef Marzouk. Dimension-independeer likelibood-informeed meme. Journal of Computational Physics, 304:109-11:3 2016.
Gustavo Deco and Wilfried Brauer. Nonlinear higher-order statistical decorrelation by volume- conserving neural architectures. Neural Networks, 8(4):525-535, 1995.
Emily Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances neural information processing systems, pp. 1486-1494, 2015.
Laurent Dinh, Jascha Sohl- Dick-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv::600: 160500803,, 08803, 2016.
Tarek A El Moselhy and Youssef M Marzouk. Bayesian inference with optimal maps. Journal of Computational Physics, 231(23):8815-788 2012.
Yihao Feng, Dilin Wang, and Qiang Liu. Learning draw samples with amortized stein variational gradient descent. arXiv preprint arXiv:1707.06626, 2017.
Michael B Giles. Multilevel monte carlo path simulation. Operations research, 56(3):607-617, 2008.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.012 2018.
10

---

Under review as a conference paper at ICLR 2021
Eldad Haber, Lars Ruthotto, Elliot Holtham, and Seong- Hwan Jun. Learning across scales multiscale methods for convolution neural networks. In Thirty-Second AAAI Conference on Arti- ficial Intelligence, 2018.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.000255 2019.
Thomas Y Hou, Ka Chun Lam, Pengchuan Zhang, and Shumao Zhang. Solving bayesian inverse problems from the perspective of deep generative networks. Computational Mechanics, 64(2): 395-408, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lchtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 1710. 10/96, 2017.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Jakob Kruse, Lynton Ardizzone, Carsten Rother, and Ullrich Köthe. Benchmarking invertible archi- tectures on inverse problems. ICML, 2019.
Chang Liu and Jun Zhu. Riemannian stein variational gradient descent for bayesian inference. In Thioty- aaai conference on artificial intelligence, 2018.
Qiang Liu, Stein variational gradient descent as gradient flow. In Advances neural information processing systems, pp. 3115-3123, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances In Neural Information Processing Systems, pp. 2378-2386, 2016.
Hermann Matthies, Elmar Zander, Bojana Rosić, Alexander Litvinenko, and Oliver Pajonk. Inverse problems in bayesian setting. In Computational Methods for Solids and Fluids, pp- 245-286. Springer, 2016.
Matthias Morzfeld, Xuemin Tu, Ethan Atkins, and Alexandre Chorin. A random map implemen- tation of implicit filters. Journal of Computational Physies, 231(4):004-200) 2066, 2012.
Radford M Neal al. Meme using hamiltonian dynamies. Handbook of Markov Chain Monte Carlo, 2(11), 2011.
Frank Nielsen and Richard Nock. Sided and symmetrized bregman centroids. IEEE transactions on Information Theory. 55(6):2882-2900 29044, 2009.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil-
iary classifier gans. In International conference on machine learning, pp.
2017.
Matthew Parno, Tarek Moselhy, and Youssef Marzouk. A multiscale strategy for bayesian inference using transport maps. SIAM/ASA Journal on Uncertainty Quantification, 4(1):1160-1199: 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2015.
Alessio Spantini. On the low-dimensional structure of Bayesian inference. PhD thesis, Mas- sachusetts Institute of Technology, 2017.
Alessio Spantini, Daniele Bigoni, and Youssef Marzouk. Inference via low-dimensional couplings. The Journal of Machine Learning Research, (0(1):639-2700) 2018.
Max Welling and Yee W Teh. Bayesian learning stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-II), pp. 681-688, 2011.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1316-1324, 2018.
11


Reviews:
[{"review_id": "y-15D5Yklf0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a multi-scale method to design invertible network and apply it to bayesian inverse problems.\n\nThe method itself is quite concise and beautiful in its simplicity (a nice case of “obvious once you see it”), and looks rather simple to implement. The results are also rather promising.\n\nWith that said, there are some steps that are lacking. Notably the results leave some things open (e.g. the SotA-ness is highly debatable given more recent methods such as routing transformers) and for the Bayesian inverse problems it’s hard to judge given that the only baselines are “in house” rather than externally defined. I also find the theoretical exposition to be a bit hard to follow, especially given the generous use of appendices.\n\nI have some further comments:\n\n* The existence of a prior conditioning operator in the general case seems somewhat debatable (consider odd cases like singular distributions, distributions with weird topologies and discrete distributions), I’m quite sure you need at least some further conditions. I’d also be fine with a “if it exists” style statement.\n* The optimisation described in appendix D is quite critical to the method as is (especially since using Jeffreys divergence is even mentioned in the abstract as a main point) and should be in the body of the article.\n* The statement “the subsurface flow simulation at the scale of meters approximates well the simulation at the scale of centimeters” is more or less repeated twice verbatim. It’s also not the most accessible example.\n* Spelling errors like dimebsion, groundtruth. Overall the language could use some polishing throughout the article.\n* All figures should be in vector format\n* The statement “ q(x(s1, s2)) = q(x(s1, 1 − s2))” does not afaik imply more than one mode, consider e.g. a mode centered at s2=0.5.\n* Showing the “worst of 3” seems rather dishonest. Why was this done?\n* Table 2 is quite lacking, several SotA methods are not in there (which makes this method not SotA).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "rDTTm8hs41d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Paper Summary\nThis paper presents a model and a corresponding training approach for multi-scale invertible models. The presented model is defined on multiple scales with information on finer scales being conditioned on coarser scales. Data generation is hence done sequentially from a coarser to finer scale. The authors argue that this multi-scale sampling helps in addressing the curse of dimensionality problem by allowing to sample from high density regions more efficiently.\n\nAs a core part of the model, the authors present a linear layer that combines information from a coarse scale and a noise vector to predict the data on a finer scale. This prior conditioning layer, as the authors call it, exploits the fact that the downsampling operation from a finer scale to a coarser scale is a linear operation. Under the assumption that the fine scale data follows a multivariate normal distribution, the downsampling operation can be analytically inverted.\n\nTraining is done by minimizing Jeffreys divergence, i.e. the loss is calculated bi-directinally in the forward and the backward direction of the model, akin to Ardizzone et al. (2019). The authors argue that Jeffreys divergence is less prone to the mode dropping that can happen when training with KL divergence. The authors justify this argument by citing Nielsen & Nock (2009), by doing ablation studies on toy data, and by giving a toy example in Appendix C.\n\nDue to the multi-scale structure and the fact that the coarse scale data can be generated from a fixed corruption process (the downsampling model), it is possible to train this model by decoupling the different scales. The authors suggest to train the model from a coarser to finer scale in a multi-stage process. The authors claim the superiority of this approach to regular end-to-end training.\n\nIn an extensive experimental section, the authors evaluate their approach on two 'Bayesian Inverse Problems', as well as on generative modeling of the Celeb A 64 data set. In further experiments, the authors perform ablation studies on a synthetic data set in order to justify their presented approach. In all experiments, the presented approach yields the highest performance.\n\n# Assessment Summary\n\nThe presented model and training approach exploit the hierarchical nature of data in a smart way. This in turn results in good performance on a number of data sets. The experimental section is rigorous and convincing. These aspects of the work should put me in favor of acceptance if it weren't for the lack of clarity in places (see below). In its current state the paper does not explain the method well enough in order to get a full understanding of the approach. I therefore tend to reject the paper at this stage. With appropriate improvements in the writing I would be happy to improve my score.\n\n# Positives\n- A multi-scale approach is a reasonable choice in many real world applications. The presented model exploits our world knowledge by utilizing a fixed downsampling operator to generate views of the data on different scales. This kind of inductive bias is simple to interpret and it appears to be a promising aid in model training.\n- The experimental section is extensive. The authors evaulate their approach on multiple problems, synthetic and real world. Especially the ablation study should be pointed out.\n- The ablation study aims to confirm the authors claims that the presented approach results in better mode capture. To do that the authors compare different model architectures, different training losses, and the multi-stage training to end-to-end training. Their results suggest that the multi-scale model trained in multiple stages using Jeffreys divergence indeed performs best.\n\n# Negatives\n- One of the biggest issues in this paper is the lack of clarity in places, which in turn makes the work hard to digest. This is often owed either to a poor choice of notation or to abbreviated explanations. Especially the methods section should be pointed out in this regard. I will give a few examples below.\n\t- In equation (1) the authors surrogate $q(x|y)$ with $q(x)$. This choice makes it difficult to follow equations (5) and (6) as it is not clear anymore where we are dealing with marginal or conditional probabilities.\n\t- The use of $\\rho(x)$ and $L(y|x)$ does not follow the ICLR style guide for probabilities.\n\t- Section 2.1 is not clear. Here, the choice of abbreviations in (5) and (6) hinders an understanding. It is not made clear which role variable y plays in the network. It should be made more clear by the authors that the forward process of the multiscale model is sampling from the posterior distribution.\n\t- Figure 1 only shows the sampling process, but does not help to understand the training. $\\tilde{x}_l$ is represented with smaller images than $x_l$ although both have the same dimensionality.\n\t- Section 2.3 discusses biefly the use of Jeffreys divergence with importance sampling for training the model. The authors do not make clear what $p(x)$ and $q(x)$ stand for in this context. As such it is not possible to make out how to train the model with the suggested loss.\n- The prior conditioning layer (upsampling layer) exploits the fact that the downsampling operation is linear and the prior over fine-scale data is assumed to be a multivariate normal distribution. As such, the upsampling can be implemented with a linear function of the coarse data and a noise vector. It is well known that this kind of inversion, although analytically correct, is computationally infeasible in high dimensional data. The authors do not address this issue in their text.\n- With respect to the multi-scale structure of the model, the paper does not position itself well in the literature. Invertible models such as in Dinh et al. (2016) already use a multi-scale structure. The authors should point out the difference of their approach more explicitly. Using explicit downsampling as an inductive bias was used successfully before for training GANs in Denton et al. (2015). This work might be a good starting point to discover other related multi-scale approaches.\n\n\n# Recommendations\n- Figure 1 could be improved to visualise the overall approach better. It would be worth visualising the sampling process for estimating Jeffreys divergence. Further, I would suggest to use an example from the Celeb A data set in order to get a better understanding of the intermediate scales. In addition, the downsampling operation $\\mathcal{A}$ could be emphasided in this figure.\n- Notation should be improved and made more consistent. Is there a way to reduce the amount of variables used? Is it possible to use consistent notation to denote probabilities?\n- Section 2.1 should be clarified. Is there a simpler way to explain this? For example by exploiting the Markov property of the model.\n\n# Questions\n- In Lemma 2.1 it is assumed that the fine-scale signal follows a multivariate normal distribution. Is this a reasonable assumption? Does this not contradict the idea of multiple modes in the data?\n- Figure 6: Why do the interemediate coarser scale samples not look like downsampled versions of the high resolution image?\n- What happens if the models on each scale are trained completely decoupled? Would it be fair to call the model a stacked super-resolution model?\n\n# References\n\nArdizzone, L., Kruse, J., Wirkert, S., Rahner, D., Pellegrini, E. W., Klessen, R. S., & Köthe, U. (2018). Analyzing inverse problems with invertible neural networks.\n \nDenton, E. L., Chintala, S., & Fergus, R. (2015). Deep generative image models using a￼ laplacian pyramid of adversarial networks. \n\nDinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using real nvp.\n\nNielsen, F., & Nock, R. (2009). Sided and symmetrized Bregman centroids.\n\n---------------------------------------------------------\n\n# Review Update\nBased on the response and update from the authors I will improve my score from '5: Marginally below acceptance threshold' to '6: Marginally above acceptance threshold'. I generally believe that the presented approach merits publication. The methodology is sound and the experiments are convincing. I appreciate the fact that the authors made an effort to improve the writing.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U1xFEv5p8T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the use of an hierarchical flow-based model to obtain samples of finer scales seems interesting, I have to admit that I did not understand much of the setup the authors propose. The methodology section is too dense and many of their claims are not really motivated at all. \n\nI do not see this paper accepted the way it is, but it might be true that a person more expert in flow-based models and inference than me thinks completely the opposite.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers sample generation in high-dimensional bayesian inference and proposes a multi scale procedure that performs coarse-to-fine multi-stage training and enables interpretability of intermediate activations at coarse scales.  The method is simple, elegant and addresses a very important bottleneck of high-dimensional bayesian inference. The clarity of the paper has been greatly improved based on the reviewers suggestions. 

However some concerns remain regarding the evaluation that are needed to clearly demonstrate the value of the approach. In particular, it would be important to assess the impact of the number of levels, and how quickly the dimensions grow from one level to the next. The number of forward simulations does not provide a sufficient picture of the computational cost of the approaches. It would be also important to provide wall-clock time.  Figure 2a should also provide the best of 3  independent experiments,  or better, more experiments should be run, and curves with shaded areas should be provided so one could visualize variability w.r.t runs. 
Decision: Reject