Projected Latent Markov Chain Monte Carlo: Conditional Sampling of Normalizing Flows
Authors: Chris Cannella
Mohammadreza Soltani
Vahid Tarokh
Venue: ICLR 2021 Poster (2021)
Keywords: Conditional Sampling
Normalizing Flows
Markov Chain Monte Carlo
Missing Data Inference

Abstract: We introduce Projected Latent Markov Chain Monte Carlo (PL-MCMC), a technique for sampling from the exact conditional distributions learned by normalizing flows. As a conditional sampling method, PL-MCMC enables Monte Carlo Expectation Maximization (MC-EM) training of normalizing flows from incomplete data. Through experimental tests applying normalizing flows to missing data tasks for a variety of data sets, we demonstrate the efficacy of PL-MCMC for conditional sampling from normalizing flows.

Published as a conference paper at ICLR 2021
PROJECTED LATENT MARKOV CHAIN MONTE CARLO: CONDITIONAL SAMPLING OF NORMALIZING FLOWS
Chris Cannella, Mohammadreza Soltani & Vahid Tarokh Department of Electrical and Computer Engineering
Duke University
ABSTRACT
We introduce Projected Latent Markov Chain Monte Carlo (PL-MCMC), a tech- nique for sampling from the exact conditional distributions learned by normaliz- ing flows. As a conditional sampling method, PL-MCMC enables Monte Carlo Expectation Maximization (MC-EM) training of normalizing flows from incom- plete data. Through experimental tests applying normalizing flows to missing data tasks for variety of data sets, we demonstrate the efficacy of PL-MCMC for con- ditional sampling from normalizing flows.
INTRODUCTION
Conditional sampling from modeled joint probability distributions offers statistical framework for approaching tasks involving missing and incomplete data. Deep generative models have demon- strated an exceptional capability for approximating the distributions governing complex data. Brief analysis illustrates a fundamental guarantee for generative models: the inaccuracy (i.e. divergence from ground truth) of a generative model's approximated joint distribution upper bounds the ex- pected inaccuracies of the conditional distributions known by the model, as shown in Appendix [A] Although this guarantee holds for all generative models, specialized variants are typically used to approach tasks involving the conditional distributions among modeled variables, due the diffi culty in accessing the conditional distributions known by unspecialized generative models. Quite often, otherwise well trained generative models possess capability for conditional inference that regrettably locked away from our access.
Normalizing flow architectures like RealNVP (Dinh al. 2014) and GLOW (Kingma & Dhariwal 2018) have demonstrated accurate and expressive generative performance, showing great promise for application missing data tasks. Additionally, by enabling the calculation of exact likelihoods, normalizing flows offer convenient mathematical properties for approaching exact conditional sam- pling. We are therefore motivated to develop techniques for sampling from the exact conditional distributions known by normalizing flows. In this paper, propose Projected Latent Markov Chain Monte Carlo (PL-MCMC), a conditional sampling technique that takes advantage of the convenient mathematical structure of normalizing flows by defining Markov Chain within flow's latent space and accepting proposed transitions based on the likelihood of the resulting imputation. In principle, PL-MCMC enables exact conditional sampling without requiring specialized architecture, training history, or external inference machinery.
Our Contributions: We prove that a Metropolis- Hatsons implementation of our proposed PL- MCMC technique is asymptotically guaranteed sample from the exact conditional distributions known by any normalizing flow satisfying very mild positivity and smoothness requirements. We then describe how to use PL-MCMC to perform Monte Carlo Expectation Maximization (MC-EM) training of normalizing flows from incomplete training data. To illustrate and demonstrate aspects of the technique, we perform a series of experiments utilizing PL-MCMC to complete CIFAR-10 10 images, CelebA images, and MNIST digits affected by missing data. Finally, perform series of experiments training non-specialized normalizing flows to model MNIST digits and continuous UCI datasets from incomplete training data to verify the performance of the proposed method. Through these experimental results, we find that PL-MCMC holds great practical promise for tasks requiring conditional sampling from normalizing flows.

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
A conditional variant of normalizing flows has been introduced by Lu & Huang (2020) to model a single conditional distribution between architecturally fixed sets of conditioned and conditioning variables. While guite capable of learning individual conditional distributions, conditional variants do not enable arbitrary conditional sampling from a joint model. [Richardson et al. (2020) concur- rently train deterministic inference network alongside normalizing flow for inferring missing data. Although such an inference network can produce deterministic imputations consistent with the distributions learned by a normalizing flow, it cannot stochastically sample from the conditional distributions known by the flow. Li et al. (2019) introduce shared parameter approximations that allow the derivation of approximate conditional normalizing flows, though these approximations do not guarantee exact sampling from the conditional distributions of particular joint model. Similar techniques for approaching missing data with other generative models, such as generative adver- sarial networks (GANs) and variational auto-encoders (VAEs), have been introduced with similar limitations [Ivanov et al. 2018 Yoon al. 2018: Li et al. 2018).
A MCMC procedure for sampling from the conditional distributions of VAEs has been introduced by Rezende et al.] (2014) and refined by Mattei Frellsen (2018). This procedure fundamentally relies on the many-to-many relationship between the latent and modeled data spaces of VAEs, and cannot be directly applied to normalizing flows, wherein the latent state uniquely determines (and is uniquely determined by) the modeled data state. By following an unconstrained Markov Chain within the latent space, PL-MCMC mirrors this VAE conditional sampling procedure within the context of normalizing flows.
PL-MCMC leverages the probabilistic structure learned by a normalizing flow to produce efficient Markov Chains. The utility of the mathematical structure of normalizing flows for approaching Monte Carlo estimation via independence sampling has been demonstrated by Müller et al. (2019). The probabilistic structure of normalizing flows has also been shown improve unconditional sam- pling from externally defined distributions by Hoffman et al. (2019). In using this learned structure, we believe that PL-MCMC receives many of the benefits of Adaptive Monte Carlo methods (Haario
et al. 2001 Foreman Mackkey et al. 2013 3:
Zhu 2019), as explained in Appendix B
PL-MCMC's unconstrained Markov Chain through the latent space not the only conceivable op- tion for sampling from the conditional distributions described by normalizing flows. As normalizing flows enable exact joint likelihood calculations, we could employ MCMC methods through the mod- eled data space. Dinh al. (2014) demonstrate stochastic conditional MAP inference that can be adapted to implement the unadjusted Langevin algorithm (Fredrickson et al. 2006 Durmus et al. 2019) the Metropolis adjusted Langevin algorithm (Grenander & Miller 1994). A constrained Hamiltonian Monte Carlo approach has also been introduced in the context of conditional sampling from generative models by Graham et al. (017). MCMC methods restricted to the modeled data space approach the normalizing flow as a sort of blackbox oracle to be used only for calculations regarding data likelihood. By design, PL-MCMC leverages the flow's one-to-one mapping between latent and modeled data spaces, thereby taking better ad advantage of the probabilistic structure learned by our normalizing flows to perform conditional sampling.
3 THE PL-MCMC APPROACH
We consider normalizing flow between latent space E and modeled data space X, defining the mappings fo E + X' and fo : X' + E. This normalizing flow imposes the probability density Px,((x) onto all modeled data values E X. By the pairing (XM;xo), and observed portion of modeled data value with joint density (x.0(XM: xo) under our normal- izing flow. Our goal is to sample from the conditional density described by the normalizing flow, (xM)o.
denote the missing
3.1 THE PROJECTED LATENT TARGET DISTRIBUTION
Rather than targeting the conditional distribution of missing values directly. PL-MCMC targets distribution of latent variables that, after mapping through the flow's transformation, marginalizes to the desired conditional distribution. Let the Markov Chain be composed of latent state & E E,
2

---

Published as a conference paper at ICLR 2021
mapping to the modeled data pair fe) (ym:yo). Let q be arbitrary smooth density observed variables, Yo. PL-MCMC targets the distribution whose (unnormalized) density within the modeled data space is q(yo) o (x)). Fundamentally, PL-MCMC is marginal MCMC method (Van Dyk 2010) that uses the otherwise observed attributes, yo. as auxiliary working vari- ables to take full advantage of the probabilistic structure learned by the normalizing flow.
3.2 DESCRIPTION OF METROPOLIS S PL-MCMC ALGORITHM
For Netropolis-Hastingg implementation of PL-MCMC, we introduce transition kernel g ()) for generating proposal latent states. We sample new proposal latent vector 5 ~ (("(()) mapping to the modeled data pair fo(E') (y' Myy' o')). An illustrative diagram of the production of PL-
MCMC proposals provided in Appendix This proposal is then accepted with probability:
q o))p,ooy Mi Mixo)g(ElEE) det ag'
f
min(1,
Mox)) det
a Ø
Algorithm 1: PL--MMMC Metropolis-Hastings Update
Input:
Observed data xo, normalizing flow fe, modeled joint density Ps (xe(XMiXo)) Latent transition kernel g(E () and auxiliary density 4(yo). Initial latent state E
Sample 5' gE
YM + fo(E); y' M'''' fe(E);
c min(1, <(y' (xxoo(0))))))) det 2fe
De
ol (()))) det 37g
Sample u ~ Uniform (0, 1]: u then E';
3.3 THEORETICAL JUSTIFICATION OF THE ALGORITHM
Proposition. For a given Xo. if E() yo), and q(yo) are positive for any choice of (y);yo) E X and E' PL-MCMC update procedure listed in Algorithm| 7 yields Markov Chain of latent states whose corresponding modeled data pairs, fo(E) (yM:Yo). converge to a distribution with y M having marginal density pralyy (oyaa(o)))))
and are the densities for absolutely continuous distributions, the
Proof. Under these assumptions, the diffeomorphism (i.e, an invertible and differentiable mapping) interpret the latent transition kernel g(E'lE) as the transition kernel " "y"'l'" det 0fa² by- within the modeled data space that is positive for all y, y' X' and the density for an absolutely continuous distribution. Additionally, we note:
provided by the flow fe allows
aly' ooorsaly'mixo (ix)) aly' oooor,lly' m q(yo)p,.0yM::30)
((vo)p,,
The diffeomorphism provided by the flow fo also guarantees that (( is positive for all (y yo) X' and is the density for absolutely continuous distribution. The procedure listed in Algorithm [T] therefore describes a Metropolis-HHatingg update satisfying the conditions described by Tsvetkov et al. (2013). The paired values fe(E) (ymiyo) obtained through iter- ated application of Algorithm thus converge to a target distribution with (unnormalized) density (( (xo).
The requirements for convergence very mild and are satisfied by the most common choices for latent, transition proposal, and auxiliary distributions (e.g. multivariate normal distributions). We note that the eventual convergence of the PL-MCMC update towards the desired conditional
3

---

Published as a conference paper at ICLR 2021
distribution is not influenced by our choice of the auxiliary distribution However, the choice of this auxiliary distribution can affect the rate of convergence. We have found agreeable performance by selecting q be independent normal distribution centered on the conditioning values xo- This guides the Markov Chain towards reasonable samples more quickly by leveraging learned dependencies between the observed and missing components of the modeled data.
TRAINING NORMALIZING FLOWS FROM MISSING DATA
With PL-MCMC providing samples from the conditional distributions of normalizing flows, natu-
ral application of the technique is in MC-EM training (Dempster et al. 1977. Wei & Tanner 1990] Neath et al. 2013) of normalizing flows from incomplete data. MC-EM training involves imputing missing values within the training set via conditional sampling of our current model, and then up- dating the parameters of our model best fit the newly imputed training set. As described within Appendix C this leads to Algorithm 2 with PL.MMMMCXXO.ss Pr,0, %)) denoting the distribution ob- tained by following an implementation of PL-MCMC with auxiliary density qi (defined 3.T) and tra in being any training procedure that returns flow parameters 0 approximately maximizing the likelihood of a complete data training set. For our experimental tests, PL-MCMC is obtained through iterated application of Algorithm
Algorithm 2: Monte Carlo Expectation Maximization Training of Normalizing Flow
Input: Incomplete training data Xtrain {xo.1, Xo.,.... Xo,r). Auxiliary densities li- Normalizing flow training procedure triin. Parameterized flow architecture fo.
while training do
for < 1 to 7 do Sample MM ~ PL-MCMC(XO..IP.. end
X (irain = ((MM.:.o.)) (MM2: (yM.T;X0.))) 0 t train(f, X(rain)
end
Intuitively, this procedure relies on conditional inference "boost" the accuracy of our current model for the joint distribution governing the training data. At each step of Algorithm 2 X train
represents samples from an approximation of the modeled data's 's ground truth distribution. We fit
to model this approximate joint distribution. After conditional inference with the new normalizing flow using PL-MCMC, the next iteration jraan represents samples from distribution with smaller divergence from the ground truth distribution, discussed in Appendix Importantly, this MC-EM training procedure assumes that data is missing at random (Little & Rubin 2019).
QUALITATIVE EXPERIMENTAL RESULTS
For a qualitative examination of the performance of PL-MCMC, we focus on conditionally sampling missing data using normalizing flows that have been trained from complete data. We must note that the the purpose of PL-MCMC is to sample from a model's conditional distributions, which may not coincide with accurately replicating the ground truth values of missing data. These qualitative experiments are therefore intended to illustrate aspects of the operation of PL-MCMC and to provide visual verification of the method's performance. Further details of these experiments and examples of unconditioned samples from the normalizing flows are provided in Appendix D
5.1 CONDITIONAL INFERENCE WITH CIFAR- 10 IMAGES
We first consider sampling a missing central quarter of CIFAR-10 (Krizhevsky et al. 2009) images (32 x 32 full color images) using a normalizing flow following the GLOW architecture (Kingma & Dhariwal 2018). To bolster our claim that PL-MCMC does not require specially trained models, utilize publicly available pre-trained model (van Amersfoort 2019) for this experiment. Initial and final completions provided by the Markov Chain are illustrated in Figure
4

---

Published as a conference paper at ICLR 2021
(a) Initial Completions
(b) Final Completions
(c) Ground Truth
Figure 1: Conditional inference of CIFAR-10 images with normalizing flow trained complete data.
The initial state of the Markov Chain is constructed by filling pixels with RGB values randomly selected from the observed subset. Latent space transitions are generated via small perturbations within the absolute coordinates of the latent space. PL-MCMC is carried out for 25,000 proposals. Example progressions of completions are provided in Figure 2 In comparison with unconditioned samples, the PL-MCMC completions appear reasonable, given the capabilities of the underlying model, and highlight the perceptual benefit provided by conditioned sampling.
Figure 2: Progression of CIFAR-10 10 completions intervals of 3,000 PL-MCMC proposals.
5.2 CONDITIONAL INFERENCE WITH CELEBA IMAGES
Next we consider sampling missing right half of CelebA (Liu et al. 2015) images (aligned, cropped, and resized to 64 x 64 full color images) using a normalizing flow following the GLOW architecture (Kingma & Dhariwal 2018). To bolster our claim that PL-MCMC does not require spe- cially trained models, we utilize a publicly available pre-trained model (Yuki-Chai) 2019) for this experiment. Initial and final completions provided by the Markov Chain are illustrated in Figure
Initial Completions
(b) Final Completions
(c) Ground Truth
Figure 3: Conditional inference of CelebA images with normalizing flow trained on complete data.
The initial state of the Markov Chain is constructed by sampling from the normalizing flow at re- duced variance. Latent space transitions are generated via small perturbations within relative co- ordinates of the latent space. PL-MCMC carried out for 25,000 proposals. Example progres- sions of completions are provided in Figure 4 The progression of PL-MCMC completions clearly demonstrates how defining a Markov Chain through the flow's 's latent space encourages proposing alterations to semantically meaningful attributes.
5

---

Published as a conference paper at ICLR 2021
Figure 4: Progression of CelebA completions over intervals of 1000 PL-MCMC proposals.
5.3 CONDITIONAL. INFERENCE WITH MNIST DIGITS
Finally, consider sampling missing portions of MNIST (LeCun et al., 1998) digits (28 28 monochrome images) using a normalizing flow following a variant of the NICE architecture (Dinh et al. 2014) under variety of data missingness mechanisms. The missingness mechanisms consid- ered are independent missingness (I.M.,, patch missingness (P.M.), and square observation (S.O.), at missingness rates of 0.6, 0.6, and 0.8 respectively. Final completions and conditional expectations as obtained by averaging the final completions of 20 independent PL-MCMC chains are illustrated in Figure
9
5
9
5
9
A
5
6
6
3
6
6
3
6
6
3
5 2 6 0 9
1
8
5
8
5
1 2 5
8 b 8
2
6
2
2
6
2
6 0
9
3
2
9
2
9
2
9
(a) Masked Inputs
(b) Final Completions
(c) Conditional Means
Ground Truth
Figure 5: Conditional inference of MNIST digits with normalizing flow trained on complete data.
The initial state of the Markov Chain is constructed by sampling from the normalizing flow duced variance. Latent space transitions are generated by a mixture of small perturbations within the absolute coordinates of the latent space and resampling reduced variance. PL-MCMC is per- formed over 2,000 proposals. Example progressions of completions are provided in Figure 6
6
6
6
6
6
6
6
6
6
0
2
2
7
7
7
2
2
2
2
2
2
3
3
6
6
6
6
6
6
6
6
6
Figure 6: Progression of MNIST completions over intervals of 200
PL-MCMC proposals.
QUANTITATIVE EXPERIMENTAL RESULTS
As an analytical description the conditional distributions of non-specialized normalizing flows is infeasible, it is difficult to quantify how well PL-MCMC succeeds in sampling from its intended dis- tributions. Given the extreme dependence of Algorithm@ accurate conditional sampling from PL- MCMC for effective training, we therefore quantify the performance of normalizing flows trained from incomplete data as an indication for whether PL-MCMC produces sufficiently accurate and efficient samples to remain useful for real-world missing data tasks. We also test the sampling effi- ciency of PL-MCMC independently of considerations regarding sampling accuracy. Further details of these experiments are provided in Appendix

---

Published as a conference paper at ICLR 2021
6.1 TRAINING FROM INCOMPLETE MNIST DIGITS
In this experiment, we consider training models of MNIST digits from training sets affected by a variety of data missingness mechanisms and imputing test sets affected by the same missingness mechanisms. The data missingness mechanisms used are are independent missingness (I.M.), patch missingness (P.M.), and square observation (S.O.), with missingess rates of 0.3, 0.6, and 0.9. As imputation performance measures, we consider per-pixel reconstruction RMSE and Fréchet Incep- tion Distance (Heusel et al. 2017). As comparison, we include results for imputing using pixel wise observed means and using the convolutional variant of MisGAN (Li et al. 2018). Our normalizing flow is a variant the NICE architecture. We performed MC-EM training of the normalizing flow for total of 1,000 epochs. Inference with normalizing flows is performed using PL-MCMC chain of 2, 000 proposals. Our reported results within Table | reffect performance across fifteen distinct pairings of training and test sets (models trained, where applicable, from three distinct training sets and each tested on five distinct test sets). For PL-MCMC, our results reflect imputation performance using individual conditional samples (Ind.) and using the average of 10 conditional samples (Avg.) for test set completion.
Table 1: Comparison of imputation performance for reconstructing MNIST digits. Value means are reported most the first significant digit of standard error.
Reconstruction RMSE
FID
PL-MCMC PL-MCMC
PL-MCMC Ind.
PL-MCMC
Rate
Mean
MisGAN
Mean
MisGAN
Ind.
Avg.
Avg.
0.3 0.2570(1)
0.153(1)
0.130(2)
0.1277(4)
23.5(1)
1.56(7)
1.58(8)
0.17(1)
P.M. S.O. L.M.
0.6 0.2573(1)
0.1585(6)
0.1456(1)
0.167(2)
72.2(1)
5.7(5)
6.1(5)
0,78(2)
0.9 0.2574(0)
0.261(2)
0.256(1)
0.326(4)
114.7(1)
87(2)
90(2)
I1(1)
0.3 0.0577(3) 0.0410(3)
0.0371(3)
0.0439(6)
0.1(0)
0,075(1)
0.076(1)
0.006(1)
0.6 0. 1688(2)
0.152(3)
0.137(2)
0.159(2)
5.8(1)
1,4(2)
1.7(2)
0.6(1)
0.9 0.2467(1)
0.2595(8)
0,2535(7)
0.322(1)
68.7(2)
50(2)
54(1)
4(1)
0.3 0.2629(3)
0.1795(8)
0.1565(5)
0.1956(8)
17.0(1)
1.6(1)
1.8(1)
0.8(1)
0.6 0.2641(1)
0.221(4)
0.205(3)
0,247(()
57.6(1)
15(1)
16(1)
2.9(2)
0.9 0.2622(0) 0,2675(8)
0.2648(9)
0.3693(9)
110.5(1)
89(2)
92(2)
16(2)
As RMSE and FID score are measures of distortion and divergence. respectively, single imputation estimate cannot simultaneously optimize both (Blau & Michaeli 2011) MisGAN primarily focuses on minimizing imputation FID, while our MC-EM training favors reducing reconstruction RMSE. Our results highlight potential advantage of performing imputation via sampling from conditional distributions. With its deterministic imputation procedure, MisGAN is dedicated to minimizing FID and cannot reduce reconstruction RMSE by averaging multiple reconstructions, With PL-MCMC sampling, we can choose, to some degree, whether to minimize FID by imputing with a single sample from the flow's conditional distribution samples. These results demonstrate that PL-MCMC is able to sample from the conditional distri- butions of normalizing flows sufficiently well acceptably train normalizing flows from MNIST digits affected by a variety of data missingness mechanisms and rates.
minimize RMSE by averaging across multiple
6.2 TRAINING FROM INCOMPLETE UCI DATASETS
In this experiment, we consider training models of various continuous UCI datasets (Bache Lich- man (203)) affected by 50% uniformly missing values. As performance measure, we consider nor- malized MSE of imputing missing values within the training set. As comparison, we include results for imputing using variable-wise observed means, using the missForest (Stekhoven & Bühlmann, 2012) R package with default settings, and using VAES via MIWAE (Mattei & Frellsen (019)). Our normalizing flow is variant of the NICE architecture. We performed MC-EM training of the normalizing flow for a total of 1,000 epochs. For inference, the PL-MCMC chain is run for 1,000 proposals. Our reported results within ( performance across five distinct training sets. For PL-MCMC, our results reflect imputation performance using individual conditional samples (Ind.) and using the average of 25 conditional samples (Avg.) for test set completion.
7

---

Published as a conference paper at ICLR 2021
Table 2:
Comparison of imputation NMSE results for continuous UCI datasets affected by 50%
uniform missingness. Value means are reported to at most the first significant digit of standard emror.
banknote
breast
concrete
red-wine white-wine yeast
PL-MCMC Ind.
1.12(5)
0.46(2)
1.22(4)
1,22(3)
1,45(3)
1.67(5)
PL-MCMC Avg.
0.58(3)
0.31(2)
0.67())
0.69(3)
0.76(1)
0,96(6)
MIWAE
0.56(4)
0.29(1)
0.63(3)
0.66(2)
0.73(3)
0.95(5)
missForest
0.74(3)
0.31(1)
0.67(2)
0.74(3)
0.81(1)
1.18(3)
Mean
0.99(1)
1.00(3)
1.00(1)
1.00(2)
1.01(1)
0.96(6)
In all cases, the MC-EM trained normalizing flows perform at least as well as missForest and closely match MIWAE for estimating conditional expectations. We can conclude that, while there some potential room for improvement in capturing the exact ground truth conditional distributions, MC- EM training of normalizing flows with PL-MCMC produces imputations comparable to those from current methods for this particular task.
6.3 SAMPLING EFFICIENCY FOR INFERENCE OF MNIST DIGIT
Here we consider the task of estimating the conditional expectation for the missing region of a single MNIST digit using the average of 100 independent Markov Chains. We also use this experiment as an opportunity to explore the effect on conditional sampling performance produced by different choices for PL-MCMC's auxiliary distribution and the transition proposal distribution. The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL.-MCMC with varying auxiliary distributions are compared Figure 7 Statistics are gathered from 10 distinct replications of the experiment.
Gibbs Sampl ng PL-MCMC,
0.225
PL-MCMC, 6,
0.20
Conditional Mean RMSF
0.200
PL-MCMC, o, 10-15
PL-MCMC, d 10
Acceptance Rate
0.175
PL-MOMC, O0 = 10
0.15
0.150
0.125
0.10-
0.100
0.075
0.05
0
10000
20000
10000
20000
Sitropolis-Hasting Proposals
Hasting Proposals
Figure 7: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by choice of auxiliary density.
These results demonstrate that PL-MCMC can offer significant performance gains over comparable MCMC methods confined to the modeled data space. Even when using improper uniform distri- bution as the auxiliary density (effectively omitting q from the acceptance probability calculation in Algorithm l.. PL-MCMC can accelerate conditional sampling by leveraging the flow's 's latent space to propose more effective proposal transitions. Depending on the characteristics of the normalizing flow's 's conditional distribution, selecting more restrictive auxiliary distribution erate sampling even further. As the results with auxiliary distributions with standard deviations of a. 10-3 and da 10-6 closely overlap, there may be some concern that the auxiliary distri-
greatly accel-
8

---

Published as a conference paper at ICLR 2021
bution might dominate PL-MCMC's behavior and reduce the procedure to simple search in the latent space to best rebuild the observed data, starting around og = 10-3. -3 While this concern may be warranted when using exceedingly strong choices for the auxiliary distribution, analysis demon- strates (Appendix F_4) that this is not the case for our results with oa = 10-3 The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL-MCMC with varying transition proposal distributions are compared in Figure
Gibbs Samp ing PLMMCMC, d3 (1.001
0.25-
0.225
PL-MCMC, o2 0.01
Conditional Mean RMSF
0.200
PL-MCMC. oF
0.05
0.20-
Acceptance Rate
0.175
0.15
0.150
0.10
0.125
0.100
0.05
0.075
0.00
0
10000
20000
10000
20000
Metropolis- Hastings Proposals
Metropolis-Haste Propesals
Figure 8: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by scale of perturbations used in their transition proposals.
From these experiments, we offer few preliminary conclusions regarding the effect of the aux- iliary and transition proposal distributions on conditional sampling performance with PL-MCMC. In the Metropolis-Hastings implementation of PL-MCMC, the transition proposal distribution be- haves much as one would expect for the transition proposal distributions of any Metropolis-Hastingg procedures. Increasing the proposal distribution's 's scale will accelerate initial convergence, but may encounter problems traversing concentrated regions of the target distributions. To some target dis- tribution dependent point (around o, 10-1.5 1.5 in Figure 7. strengthening the auxiliary distribution will continue to accelerate initial sampling. Beyond this point, further strengthening of the auxil- iary distribution can be detrimental to sampling performance, as the auxiliary distribution becomes mismatched to the intrinsic coupling between missing and observed values modeled by the flow's 's conditional distribution. Additional comparisons, including comparisons with respect to approxi- mate computational cost, are provided within Appendix
CONCLUSION AND FUTURE WORK
The mathematical structure of normalizing flows is exceptionally convenient for approaching con- ditional sampling via MCMC. By leveraging this mathematical structure, our proposed PL-MCMC technique enables asymptotically exact conditional inference with normalizing flows, without re- quiring specialized architecture, training history, or external inference machinery. The particular implementations used in our experiments are primarily intended to serve as proof-of-concept illus- trations of the PL-MCMC technique. Further research would be necessary to determine optimal choices of auxiliary distributions. transition proposal distributions, and MC-EM training procedures. Sampling performance may be improved by replacing Metropolis-Hastingg proposals with a more sophisticated technique, such as Hamiltonian Monte Carlo. Our experimental results demonstrate that, even when implemented with a naive Metropolis-Hasting procedure, PL-MCMC enables ef- fective sampling from its intended distributions under practical settings. We believe that, with the PL PL-MCMC technique, normalizing flows hold great promise for approaching missing data tasks.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
This work was supported by the Office of Naval Research Grant No. N00014-18-1122444.
REFERENCES
Kevin Bache and Moshe Lichman. UCI Machine Learning Repository, 2013.
Yochai Blau and Tomer Michaeli. The Anceptiion-Distortion Tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228-6237 2018.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series (Methodological), 39(1):122,, 1977.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. arXiv preprint arXiv:1410.8516, 2014.
Alain Durmus, Eric Moulines, al. High-dimensional Bayesian Inference Via the Unadjusted Langevin Algorithm. Bernoulli, 25(4A):2854- 2882, 2019.
Daniel Foreman-Mackey. Mackey, David W Hogg, Dustin Lang, and Jonathan Goodman. EMCEE: The
MCMC Hammer. Publications of the Astronomical Society of the Pacifie, 125(925):306, 2013.
Glenn Fredrickson et al. The Equilibrium Theory of Inhomogeneous Polymers, volume 134. Oxford University Press on Demand, 2006.
Matthew M Graham, Amos Storkey, al. Asymptotically Exact Inference in Differentiable Gen- erative Models. Electronic Journal of Statistics, 11 11(2)55105-5144, 2017.
Ulf Grenander and Michael Miller. Representations of Knowledge in Complex Systems. Journal of the Royal Statistical Society: Series B (Methodological)) 56(4))549-581 1994.
Heikki Haario, Eero Saksman, Johanna Tamminen, et al. An Adaptive Metropolis Algorithm. Bernoulli, 7(2):223-242, 2001.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, pp. 1626-6637,55 2017.
Matthew Hoffman, Pavel Sountsov, Joshua V Dillon, Ian Langmore, Dustin Tran, and Srinivas Vasudevan. Neutra-lizing Bad Geometry in Hamiltonian Monte Carlo using Neural Transport. arXiv preprint arXiv: 1903.03704 2019.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational Autoencoder with Arbitrary Condi- tioning, arXiv preprint (T06:0880002388 2018.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible Ix1 Convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Alex Krizhevsky et al. Learning Multiple Layers of Features from Tiny Images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11))2278-2324, 1998.
Steven Cheng Xian Li. misgan. httpps///ggthub..co///ttte//////
2019.
Steven Cheng-Xiae Xian Li, Bo Jiang, and Benjamin Marlin. MisGAN; Learning from Incomplete Data with Generative Adversarial Networks, In International Conference on Learning Representationss 2018.
Yang Li, Shoaib Akbar, and Junier B Oliva. Flow Models for Arbitrary Conditional Likelihoods. arXiv preprint arXiv: 1909.0639999 2019.
10

---

Published as a conference paper at ICLR 2021
Roderick JA Little and Donald B Rubin. Statistical Analysis with Missing Data, volume 793, John Wiley & Sons, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
You Lu and Bert Huang. Structured Output Learning with Conditional Generative Flows. In AAAI, pp. 5005-5012, 2020.
P.A. Mattei. miwae. https:
2019.
Pierre-Alexandre Mattei and Jes Frellsen. Leveraging the Exact Likelihood of Deep Latent Variable Models. In Advances in Neural Information Processing Systems, pp. 1855-3866, 2018.
Pierre-Alexanrre Mattei and Jes Frellsen. MIWAE: Deep Generative Modelling and Imputation of
Incomplete Data Sets. In International Conference on Machine Learning, pp. 4413-4423, 2019.
Fengzhou Mu. Nice. https:://githbbb.mmmmmmitt htt.:/ocoommmmm////m 2019.
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural Im- portance Sampling. ACM Transactions Graphics (TOG), 38(5):1-19, 2019.
Ronald Neath et Convergence Properties of Monte Carlo EM Algorithm. In Advances in Modern Statistical Theory and Applications: Festschrift in Honor of Morris L. Eaton, pp- 63-62. Institute of Mathematical Statistics, 2013.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference Deep Generative Models. In International Conference on Machine Learning. pp. 2778-1286, 2014.
Trevor Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A Bernal. MCFlow: Monte Carlo Flow Models for Data Imputation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 14205-14214, 2020.
Daniel Stekhoven and Peter Bühlmann. for Mixed-type Data. Bioinformutics, 28(1):112-118, 2012.
-ateiaeest--oo--aaaaaaaaaa Missing Value Imputation
Dimiter Tsvetkov, Lyubomir Hristov, and Ralitsa Angelova-Slavova. On the Convergence of the Metropolis-Hastingg Markov Chains. arXiv preprint 445440000004 2013.
Joost van Amersfoort. Glow-pytorch. 2019.
David A Van Dyk. Marginal Markov Chain Monte Carlo Methods. Statistica Sinica, pp. 1423-1454, 2010.
Greg CG Wei and Martin Tanner. A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms. Journal of the American Statistical Association, 85 (41 (411))699-704, 1990.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets. arXiv preprint 1rXiv:1806.02922 2018.
Yuki-Chai. glow-pytorch. jttpps///gtttabb..mm//c/ass////.
2019.
Michael Zhu. Sample Adaptive MCMC. In Advances in Neural Information Processing Systems, pp. 98066-90777 2019.
11

---

Published as a conference paper at ICLR 2021
A KNOWING THE JOINT DISTRIBUTION IMPLIES KNOWING THE CONDITIONAL DISTRIBUTIONS
In this section, establish fundamental guarantec regarding the accuracy of the conditional dis- tributions known by generative models. Consider two random variables, a and y, with ground truth joint distribution Pr,y that have approximated by generative model with joint distribution (x.y- In the discrete case, we may expand through the definition of the Kullback-Liebler divergence be- tween our generative model and the ground truth find that:
DRI.Paaylleen (ll = E ple, v) log g(0,,)) + Σ p(a, v) log p(a,y)
-B(l- Σ p(ylz)) log ())1))((x) + E ((ylz)) p(vl=)p(x)]
Eg (Ee(((Py))))) log ((x) log p(z)]
E.g (Pe DKL
From the non-negativity of the Kullback- divergence, we are then guaranteed:
DEL L 19x.0) (P
With equality only when our generative model perfectly models the marginal distribution of x. When considering the task of inferring y from observed values of x, the expected performance of our generative model in approximating the conditional distribution of y given x is no worse than its performance approximating the full joint distribution between I and y. Therefore, if we know that generative model is good approximation of the joint distribution governing some set of random variables, then it must also know good approximations of the conditional distributions among those random variables.
This inequality also serves as a justification for Monte Carlo Expectation Maximization training. When using our modeled distribution Jr.y to impute incomplete training set, the newly imputed training set is sampled from the distribution qytaP.. We can easily see that:
DKL (Pa. (P = (,
S DKL (Paylllll)
In the asymptotic limit of dataset size, conditionally inferring missing values within the dataset results in samples from a distribution whose divergence from ground truth no worse than that of the original model. Assuming that the original model describes the distribution of previously imputed version of the training set, this implies that our newly training set at least as reflective of the ground truth distribution as the previous training set. In practice, we find that conditional imputation tends to improve divergence the training set, which in turn allows MC-EM training to improve our model of the joint distribution.
B THE ADVANTAGE OF LATENT SPACE PROPOSALS
Here, we relay our intuition regarding the advantages of defining a Markov Chain within the latent space of normalizing flow. This section provides a heuristic argument and therefore utilizes infor- mal terminology to convey our current understanding. Take a normalizing flow between latent space and modeled data space X, defining the mappings fo E + X' and fø L X' -> E. This normal- izing flow imposes the probability density py,e(x) onto all modeled data values applications normalizing flows primarily involve data embedded within a euclidean space, we will confine this discussion scenarios where latent and modeled data values are both points in I for some n. In these cases, it is straightforward to discuss neighborhoods of fixed radius around points within both the latent and modeled data spaces.
X. As practical
For now, let us consider the task of forming a Markov Chain for unconditionally sampling from the density Pf ps.0 (x). For simplicity, let only consider proposal perturbations within some fixed
12

---

Published as a conference paper at ICLR 2021
radius of the Markov Chain's 's current state. With the one-to-one mapping provided by the flow, we have the option of tracking and perturbing the current state within either the latent space or the modeled data space. When considering the neighborhood of data points, probability mass within the modeled data space is often non-isotropic for highly structured data. However, probability mass nearly isotropic within the latent space in the neighborhood of the latent representations of data points, assuming that the distribution on latent space states has been appropriately chosen (as is the case for the commonly used multivariate normal or logistic distributions). As result, performing an isotropic perturbation within the latent space results in proposals that are about as likely as the starting state. Within the modeled data space, even a very small isotropic perturbation can produce proposals that are far more unlikely than the starting state. As an example, suppose our normalizing flow was well trained to model a set of high-fidelity images. If our proposals within the modeled data space were created by adding independent Gaussian perturbations to pixel values, we would almost always inject noise into the image and proposals within the modeled data space would be tend to be unlikely, low-fidelity images. With the assumption that transitions between equally likely states are usually accepted and transitions to much more unlikely states are usually rejected, we should expect latent space proposals to be accepted more frequently compared to modeled data space proposals.
As an intuition, we could say that perturbations within the flow's latent space are semantically mean- ingful for the modeled data As demonstrated by Hoffman al. (2019), the normalizing flow inherently transforms the modeled probability distribution in a manner that is well suited to explo- ration using naive, isotropic proposals. This is related to Adaptive Monte Carlo methods (Haario itt al. 2000 Foreman- Mackey et al. 2013 Zhu 2019). which attempt to transform the proposal density to most effectively explore fixed distribution. With latent space transitions in normalizing flows, it is as though the modeled data distribution has been transformed so as to be best explored by a fixed proposal density.
With PL-MCMC we are concerned with making effective proposals with respect to a conditional distribution. Even when attempting to sample conditional distribution, utilizing latent space pro- posals remains beneficial. Define a projection operator via projxo Yo) y y Xo. y which simply replaces the observed component of a E X' with the conditioning values xo. The elements of proposed transition within Hattopolis-Hastingg implementation of PL-MCMC are illustrated in Figure
PL MCMC Propssa
8(5'15)
fe
fe
3
YMiYO
Y'iY'
proj
projso
Observed Data
xO
NMiXo Xo
Y' Xo
Pr.)(MwiXo)
P Xo)
Figure 9: A Catropolis-Hasttngg PL-MCMC proposal when inferring from observed data xo-
Averaging over possible pieces of observed data Xo, we expect to find that the set of likely com- pletions, XM, under the conditional density Pf.0 (xM (x)) remains essentially subset of the set
13

---

Published as a conference paper at ICLR 2021
of likely values under the marginalized density Py,0(XM). If proposed XM is unlikely under Pr,a(xM) we expect it to be unlikely under Py.0(XM (xo). Hence, with PL-MCMC, the advantages of latent space proposals carry through to the conditional inference setting, as the resulting proposed completions can remain likely under (x.0
Returning the example of modeling set of high-fidelity images, suppose observe the left half of these images. In general, we would believe that the set of likely right half completions conditioned on the observed left half well covered by of likely right halves that we see across the entire distribution of images. Perturbing the pixel values of the right half injects noise, tending to produce a low-fidelity right half, which results an unlikely, noisy image when combined with the observed left half. Following the PL-MCMC latent perturbations, proposed right halves may be more able to least remain the high-fidelity right halves of high-fide nigh-fideliyy images.
By employing latent space proposals to sample from (x. (x)). (xo). PL-MCMC can more easily propose completions XM that could plausibly have been taken from likely members of the modeled data distribution. Of course, for conditional inference, we also need to produce samples that are well matched to the observed data. While latent space proposals assist in making meaningful and efficient transitions within Markov Chain, PL-MCMC ultimately relies on the auxiliary distribution, g. and guaranteed convergence to the correct conditional distribution to effectively sample from typical completions of the observed data.
C DETAILS OF MC-EM TRAINING
In this section we review the derivation of Monte Carlo Expectation maximization (Dempster et al., 1977 WWi Tanner 1990; Neath al. 2013) in the context its use with PL-MCMC. Suppose we presented with training set of I observed values (not all missing the same entries), Xtrain [xo.1, 1.0.1.X0,2-0 X0.2. Xo.T) J. Ideally, under the assumption that data values are missing at random (Little) & Rubin 2019 we'd d wish to find the flow parameters & that maximize the oog-liielihood of Xtrain
T
(f
log P).e(Xtraii = Σ log E=
Yet the complexity of the normalizing flow makes analytical computation of the marginal like- lihoods of observed data entirely impractical. We therefore utilize the Expecation-Maximmzatioo (EM) algorithm (Dempster et al. 1977) approach this optimization. Following Dempster et al. 1977 we define 100'00) to be:
Q(0'00) = Σ Eps.o [log( Pro (XM. xo,].
PL-MCMC can be immediately applied to approximate these expectations. Let the set Y be created by sampling each YM, ~ PP,ooyM,,/Xox using a PL-MCMC chain described previously. We may now the approximation:
(yM,1,3 M,,, ..t....yM.r.. }
Q(010) ~
log( Pf.0r (VM. )
In principle, we would then update / following;
+ argmaa Q(0')0).
In practice, it is more feasible to continue to train the flow on the conditionally imputed version of Xtrain- With X/ran = ((,1.) (y xo.2).... (y..r: xo.)) denoting our newly im- puted training set and train being any training procedure that returns flow parameters 0 approx- that:
imately maximizing the likelihood of complete data training set, we rely on the approximation
14

---

Published as a conference paper at ICLR 2021
argmax Q(010) = train(f, Xtrain).
6'
This approximation immediately leads our described algorithm for the MC-EM training of nor- malizing flows using PL-MCMC.
D DETAILS REGARDING QUALITATIVE EXPERIMENTS
D.1 DETAILS REGARDING CONDITIONAL INFERENCE WITH CIFAR- 10 IMAGES
In this experiment, infer missing central quarter (an 8 x 8 pixel square) of CIFAR-10 (Krizhessky et al. 2009) images. The CIFAR-10 dataset composed of 60, 000 full color 32 x 32 images of 10 distinet classes of objects, with 6, 000 images provided for each class. The standard training and test set split for the CIFAR-I0 dataset is 50, 000 and 10, 000 images, respectively.
Our chosen normalizing flow is variant of the GLOW (Kingma & Dhariwal, 2018) architecture. We utilized a publicly available, pre-trained model (van Amersfoort 2019) for this experiment. In the terminology of Kingma & Dhariwal (2188). the model has depth of flow (K) of and a total of 3 levels (L) and flow layers utilize 512 hidden channels. The model was reportedly trained for a total of 1, 500 epochs using Adamax with learning rate of 5 x 10 and a batchsize of 64. We presume, but cannot guarantee, that the model was trained on the standard 50, 000 example CIFAR- 10 training set. Examples unconditioned samples from this model are provided within Figure T0] as obtained with the standard sampling variance, = 1.0 (temperature T 1.0, in the terminology of Kingma Dhariwal (2018)). From these unconditioned samples, it is clear that the model has not collapsed memorizing the training set.
Figure 10: Unconditioned samples standard variance (a 1.0) from CIFAR-10 model.
The particular implementation of the normalizing flow most easily provided access to a coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space. Fundamentally, the Markov Chain within PL-MCMC may utilize any convenient representation of the latent space, long data. For this CIFAR-10 10 experiment, we chose to employ a Markov Chain within the architecture's absolute coordinates. As general note, the qualifier "absolute" merely refers to the representation favored by the flow's 's implementation, while the qualifier "relative" refers to the representation best coinciding with the chosen prior distribution for the flow. The terms only reflect aspects of our prac-
diffeomorphism still maps that representation back to the modeled
15

---

Published as a conference paper at ICLR 2021
tical usage of the representations, there is theoretically favored diffeomorphic representation of the latent space.
For inference, PL-MCMC, latent space transitions are generated within the above mentioned absolute coordinates of the flow's latent space by small perturbations from the current latent state. When perturbing the latent state, proposals are generated following a perturbation kernel, (p(E)))) such that g' ~ (S I)... The auxiliary distribution, q, is chosen to target yo ~ [xx, o . For this experiment, we select op = 0.01 and 0, x 10-3 PL-MCMC is carried out over 25,000 proposals.
selected images from the standard CIFAR-10 test set. During inference with
D.2 DETAILS REGARDING CONDITIONAL INFERENCE WITH CELEBA IMAGES
In this experiment, we infer missing right half 64 x pixel rectangle) of CelebA (Liu al. 2013) images. The CelebA dataset is composed of 202, 599 full color images of celebrity faces. We utilize the aligned and cropped version the CelebA dataset, resized to size of 64 64.
Our chosen normalizing flow is variant of the GLOW (Kingma & Dhariwal 2018) architecture. We utilized a publicly available, pre-trained model (Yuki-Chal 2019) for this experiment. In terminology of Kingma & Dhariwal (2018), the model has depth of flow (K) of 32 and total of levels (L) and flow layers utilize 512 hidden channels. The model was reportedly trained for total of 1, 500 epochs using Adamax with learning rate of 1 10-3 and batchsize of 12. We believe that this model was trained on the entirety of the CelebA dataset, with no withheld test or unconditioned samples from this model provided within Figures [[] and 12, as obtained with reduced and standard sampling variance, a = 0.5 and a = 1.0 respectively (temperatures T 0.5 and T 1.0, in the terminology of Kingga & Dhariwal (018)). From these unconditioned samples, is clear that the model has not collapsed to memorizing the training set.
validation set. Examples
Figure 11: Unconditioned samples at reduced variance (a 0.5) from CelebA model.
16

---

Published as a conference paper at ICLR 2021
Figure 12: Unconditioned samples at standard variance (a 1.0) from CelebA model.
As in the CIFAR-I0 experiment, the particular implementation the normalizing flow most eas- ily provided access coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space. For this CelebA experiment, we chose to employ a Markov Chain within what we call relative coordinates of the latent space. For this architecture, we have convenient access to three subsets of latent variables, Eι, E2, and Es within absolute coor- dinates. Fixing ε (resp. fixing & and &2) there is invertible transformation h2($2; (resp- ha(Es;E &1, {2)) such that h2 E1) ~ (O, !) (resp. ha(E3; E2) ~ N I)) under our flow's prior. The Markov Chain in relative coordinates simply follows and proposes transitions for the triplet h2(2; E1), hs(E:: E2).
As it appears that no test set had been withheld during training of the model, we selected images at random from the full dataset for our experiment. During inference with PL-MCMC, latent space transitions are generated within relative coordinates of the flow's latent space by small perturbations from the current latent state. When perturbing the latent state, proposals are generated following perturbation kernel, 9p (E))),) such that g o The auxiliary distribution, q. is chosen to
target yo ~ (xxo, I). For this experiment, we select op = 0.01 and 0r = x 10 PL-MCMC carried out over 25,000 proposals.
D.3 DETAILS REGARDING CONDITIONAL INFERENCE WITH MNIST DIGITS
In this experiment, we infer missing portions of MNIST (LeCun et al. 1998) digits. The MNIST dataset composed of 70, 000 monochrome 28 28 images of handwritten digits. The standard training and test set split for the MNIST dataset is 60, 000 and 10, 000 images, respectively. Our data missingness mechanisms are independent missingness, where pixels are lost uniformly random, patch missingness, where randomly located contiguous rectangular blocks are missing, and square observation, where only a randomly located contiguous square is observed.
Our chosen normalizing flow is variant of the NICE (Dinh et al., 2014) architecture. Our im- plementation a modification of that by Mu (2019). the terminology of Kingma & Dhariwal 2018), the model has depth of flow (K) of 5 and total of 4 levels (L) and intermediate flow layers have a dimension of 1000. The flow utilizes an independent logistic prior distribution. Rather than splitting
and odd pixels within coupling layers, we split between partitions that are chosen at the time of the flow's initialization and remain fixed for all layers of the flow. Of course, better performance would be expected by selecting flow architecture that best
randomly selected
17

---

Published as a conference paper at ICLR 2021
suits the spatial organization of image data. However, random partitioning ensures that the expected performance of the flow remains independent of the spatial structuring of the data.
The normalizing flow is trained for 1000 epochs over the standard 60, 000 element MNIST training set using RMSprop with a learning rate of x 10-S and a momentum 0.9 and batch size of 200. The data pre-processed by performing pixel-wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation). The training procedure is intended follow that used later for training from incomplete MNIST digits to serve baseline for comparison and is not intended to produce the best possible generative model of MNIST digits. Examples of unconditioned samples from this model are provided within Figure as obtained with the standard sampling variance (temperature T 1.0, in the terminology of Kingma & Dhariwal 2018
3
3
8
A
5
9
8
3
3
2
3
4
Figure 13: Unconditioned samples at standard variance from MNIST model.
During inference with PL-MCMC, latent space transitions are generated within the absolute co- ordinates of the flow's ' latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at reduced standard deviation. When perturbing the latent state, proposals are generated following a perturbation kernel, (E' such that 5' ~ ((E,o)1) When completely resampling the latent state, proposals are generated fol- lowing a resampling kernel, (), such that g' ~ (O, T) The auxiliary distribution, g, is chosen to target yo N(xo,02)) For this experiment, we select d, 0.05, o, 0.5, and acceptance probabilities, em- the latent space result from (p' while large
Oa = 10-3, To simplify calculation of Metropolis- play the assumption that small displacements displacements result from 9r (E) which is valid in the limit of op << dr- Therefore, rather than utilize the true transition kernel ( we simply assume that g(")E)) x (5'(() following a perturbation and s(' {) ()'((( following resample. PL-MCMC is carried out over 2,000 proposals. To determine conditional expectations, the results of 20 independent PL-MCMC chains are averaged together.
E DETAILS REGARDING QUANTITATIVE EXPERIMENTS
E. DETAILS REGARDING TRAINING FROM INCOMPLETE MNIST DIGITS
In this experiment, we train normalizing flows model the distribution of MNIST digits from incomplete training data. Our data missingness mechanisms are independent missingness, where pixels are lost uniformly at random, patch missingness, where randomly located contiguous rectan- gular blocks are missing, and square observation, where only randomly located contiguous square observed. For each missingness mechanism, we consider missingness rates of 0.3, 0.6, and 0.9. For training, apply the missingness mechanism to the standard MNIST training set, resulting in a training set of 60, 000 incomplete digits. For testing, we apply the missingness mechanism to the standard MNIST test set, resulting in test set of 10, 000 incomplete digits.
Our chosen normalizing flow is a variant the NICE DDin et al., 2014) architecture. In the ter- minology of Kingma & Dhariwal 2018), the model has a depth of flow (K) of 5 and total of levels (L) and intermediate flow layers have a dimension of 1000. The flow utilizes an independent logistic prior distribution. Rather than splitting even and odd pixels within coupling layers, split
18

---

Published as a conference paper at ICLR 2021
between two randomly selected partitions that chosen at the time of the flow's initialization and remain fixed for all layers of the flow. The flow architecture and implementation is the same as used above for training from complete MNIST digits. Better performance could be obtained by selecting an architecture that best suits the spatial organization of image data or by scaling model complexity along with the missingness rate (at missingness rate of 0.9, have tenth much observed data available for training as in the complete data case).
In all cases, the normalizing flow is trained for 1000 epochs using RMSprop with a learning rate of 1 x 10-5 and a momentum of 0.9 and a batch size of 200. The data is pre-processed by per- forming pixel- wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation). At each of the first 50 epochs of training, missing pixels are resampled following an independent normal distribution, such that XM NO0 7) (in whitened coordinates). Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the flow being trained. During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the flow's latent space, half of the time at random by small per- turbations from the current latent state and half of the time entirely resampled at a reduced standard deviation. When perturbing the latent state, proposals are generated following a perturbation ker- nel, (E)) such that g' ~ NN{, I) When completely resampling the latent state, proposals are generated following resampling kernel, gr such that g (O I) The auxiliary distribution, q. is chosen target yo (xxo TaT). For computation of Metropolis- Hitropoolis-astinggs probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments. Throughout training, we use op 0.05 and o, first 500 epochs, we utilize o 1.814. After the first 500 epochs, resample at reduced variance with Op = 0.5. These parameters and this training procedure were chosen because they provided acceptable performance when applied to training data with the moderate missingness rate of 0.6. would certainly be beneficial to determine a more principled approach to their selection. After PL-MCMC sampling,
10 Within the
clamp values between pixel-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training. Figure 14 below illustrates the progression of how the training is imputed over training.
19

---

Published as a conference paper at ICLR 2021
2
3
3
2
3
3
5
/
9 1 9
2 2
4 1
3 3 8
5
3
3
4
2
50 4 4
3 3
8 3 / 4 1
30 4 &
3
3 3 8
3
50 4 9 2 $
3
3
1 1 2 0 4 9
2 50 9
7 0
9
3
2 30 9
3 33 3
7 6 4
9
3
3
2 0 4
/
2 1
0 4 /
1 1 1 1 1 1 3
2 0 4
/ 9
3
3 0
4 1
3
7 0 4
/ 9
2 2
9
0 4 1
2
3
3 6 4
/
9
I
3
5 0
4 9
2
3
y 6 4
/
9
1 7 7
7
3
5 0 9
2
3
7 8
/
9
3
0
2
7
0 1
/
9
1
9
5 5
9
2
1
3
7 4
4
/
9 7
3
Independent Missingness
(b) Patch Missingness
Figure 14: Example completions of MNIST digit training set by PL-MCMC during MC-EM training missingness rates are 0.6. Initial completions are shown in the top row and completions at epoch 950 are shown in the bottom row.
For testing. we utilize the normalizing flows to reconstruct the 10, 000 element incomplete test sets. For this reconstruction, utilize PL-MCMC chains over 2, 000 proposals following the same pro- cedure as training. During testing, we use statisties for performance measures, we train three normalizing flows on distinctly prepared (i.e. resampled missingness patterns) training sets, each of which is tested on five distinctly prepared test sets. We consider ing data. The first method is imputation with single sample from a PL-MCMC chain (imputation with sample from the conditional distribution) and the second method is imputation with the aver- age across multiple samples from independent PL-MCMC chains (imputation with the conditional mean). we average across the samples from independent PL-MCMC chains for the second method, our reported statistics for the first method encompass these additional 10 replications for individual sample imputation performance.
0.05, op = 0.5, and aa = 1 x 10 To collect
methods of employing PL-MCMC produce reconstructions of miss-
As imputation performance measures, consider per-pixel reconstruction RMSE and Fréchet In- ception Distance (Heusel et al. 2017). To compute Fréchet Inception Distance, we use the imple- mentation provided along with that of MisGAN 2019). Reconstruction RMSE is recorded within the original, unwhitened representation of pixel data. When imputing test examples with M, de- noting the set of pixel indices that missing for the i-th example and 6;.5 denoting imputed
20

---

Published as a conference paper at ICLR 2021
estimate for the j-th pixel of the i-th example (having ground truth value Fizy). our reconstruction RMSE is calculated as:
Reconstruction RMSE
1 T
Σ JEM,
(يn - o.,))²
MM
For comparison, we consider imputation using pixel-wise observed means and imputation using the convolutional variant of MisGAN (Li et al. 2018). We use the implementation of MisGAN provided by [i (209)). The MisGAN models are trained following the provided default parameters (500 epochs with batch size of 64 with т 0, 0.1, 8 0.1, O, maskgen fusion, gp-lambda = 10, n.critic = 5, and n_latent 128, with three layer fully connected imputer network with 784 units in each layer).
E.2 DETAILS REGARDING TRAINING FROM INCOMPLETE UCI DATASETS
In this experiment, we train normalizing flows to model the distributions of various continuous UCI datasets (Bache & Lichman| 2013) from incomplete training data. In all cases, data missingness mechanism is independent missingness with a missingness rate of 0.5. A summary of the UCI datasets used in this experiment provided below in Table For training, we apply the missingness mechanism to the entirety of a single copy of the dataset. For testing, we attempt to reconstruct the missing portions of the training set.
Table 3: Summary of continuous UCI datasets used.
Dataset Num. Instances Num. Attributes
banknote
1372
4
breast
569
30
concrete
1030
9
red-wine
1599
12
white-wine 4898
12
yeast 1483
8
Our chosen normalizing flows are variants of the NICE (Dinh et al. 2014) architecture. In the terminology of Kingma & Dhariwal (2018), all models have a depth of flow (K) of 5 and total of levels (L. and intermediate flow layers have dimension of 120. The flows utilize an independent normal prior distribution. Rather than splitting even and odd pixels within coupling layers, we split between two randomily selected partitions that are chosen the time of the flow's initialization and remain fixed for all layers of the flow. As our implementation works most easily with an even number attributes, we copy the concrete attributes and data missingness to double the number of attributes. By copying data missingness patterns, ensure that the doubling does not introduce additional information for training. A summary of input attribute dimensions and training batch sizes is provided within Table 4
Table 4: Summary of continuous UCI datasets used.
Dataset Input Dimensions
Batch Size
banknote
4
3000
breast
30
1500
concrete
18
2000
red-wine
12
3000
white-wil
12
10000
yeast
8
3000
In all cases, the normalizing flow is trained for 1000 epochs using Adamax with a learning rate of 0.002, 31 = 0.9, and B3 0.999. We duplicate the training sets (data missingness patterns included) 10 times to form a larger training set without introducing additional information beyond
21

---

Published as a conference paper at ICLR 2021
that present in the original incomplete training set. The batch sizes used are listed in Table 4 The data is pre-pne pre-processed by performing attribute-wise whitening of the dataset (subtracting attribute- wise observed means and dividing by attribute-wise observed standard deviation). At each of the first 50 epochs of training, missing attribute are resampled following an independent normal distribution, such that XM ~ (O, 7) (in whitened coordinates). Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the flow being trained. During inference with PL- MCMC, latent space transitions are generated within the absolute coordinates of the flow's latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at reduced standard deviation. When perturbing the latent state, proposals are generated following a perturbation kernel, (pp(5)'5555 ()), such that 5' ~ (NE, o2)) When completely resampling the latent state, proposals are generated following a resampling kemell, gr (5)()))), such that e ~ NO, T). The auxiliary distribution, g. is chosen to target yo [xxo a. For computation of Hasttng probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments. Throughout training, we use op- = 0.01,
or 1.0, da 1 x 10-3 These parameters and this training procedure were chosen because they provided acceptable performance across the UCI datasets considered. It would certainly be beneficial to determine a more principled approach to their selection. After PL-MCMC sampling. clamp values between attribute-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training.
For testing, we utilize the normalizing flows to reconstruct the missing values from their training sets. For this reconstruction, we utilize PL-MCMC chains over 2, 000 proposals following the same procedure in training. During testing, we use op 0.01, op = 1.0, and da 1 x 10-3 To collect statistics for performance measures, we train five normalizing flows on distinctly prepared methods of employing PL.-
(i.e. resampled missingness patterns) training sets. We consider MCMC to produce reconstructions of missing data. The first method is imputation with a single sample from PL-MCMC chain (imputation with sample from the conditional distribution) and the second method is imputation with the average across multiple samples from independent PL- MCMC chains (imputation with the conditional mean). As we average across the samples from 25 independent PL-MCMC chains for the second method, our reported statistics for the first method en- compass these additional 25 replications for individual sample imputation performance. In the case of the concrete dataset, we consider the copied attribute values as an additional single sample from the conditional distribution that is also incorporated into averaging.
As an imputation performance measure, we consider per-attribute normalized MSE. When imputing T test examples with M₁ denoting the set of attribute indices that are missing for the i-th example and Gi,; denoting our imputed estimate for the j-th attribute of the i-th example (having ground truth value Zi,)), ), our normalized MSE is calculated as:
1 T
Figj - Lig
NMSE
M
Jj
JEM;
where oy denotes the ground truth standard deviation of the j-th attribute.
For comparison, we consider imputation using attribute-wise observed means, imputation using the missForest (Stekhoven & Bühlmann 2012) R package with default parameters, and imputation with VAEs using MIWAE (Mattei & Frellsen| 2019). For imputation with missForrest, no data prepro- cessing is employed. "We use the implementation of MIWAE provided by Mattei (2019). In all cases, the VAE architecture employed has an intrinsic dimension d of 10, an encoder and decoder comprised of 3 layers each with 128 hidden units with ReLU activation functions, an independent normal prior, and a Student's t distribution observation model. In all cases, we utilize zero impu- tation as the MIWAE imputation function. For training, we use 20 importance weights while for inference we use 10, 000 importance weights. We train the models using the provided default pa- rameters (2, 000 epochs using Adam with a learning rate of 0.001 and a batch size of 64). In all cases, the data is pre- processed by performing atribute-wise whitening of the dataset (subtracting sttribute-wise observed means and dividing by attribute- wise observed standard deviation).
22

---

Published as a conference paper at ICLR 2021
E.3 DETAILS REGARDING SAMPLING EFFICIENCY FOR INFERENCE OF MNIST DIGIT
In these experiments, we use MCMC to estimate the conditional expectation for the missing portion of single MNIST digit. In this case, the data missingness mechanism checkerboard pattern masking half of the digit, as shown in Figure 15
7
Figure 15: Masked digit used for sampling efficiency experiments.
To perform this inference, we use the same normalizing flow as used the qualitative experiments in Section 5.3 detailed within Appendix D.3 During inference with PL-MCMC, latent space tran- sitions are generated within the absolute coordinates the flow's latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation. When perturbing the latent state, proposals are generated following perturbation kemnel, (p(5' such that ç ~ (E When completely resampling the latent state, proposals are generated following a resampling kernel, gr-(E(()) such that g (O, oz.). The auxiliary distribution, is chosen to target yo N(xo, I).). For this experiment, un- less otherwise specified, we select o, 0.01, o, 1.0, and 0, 10-3 For computation of Metropolis-Hasting probabilities, we employ the same approximation used when inferring MNIST digits within the qualitative experiments.
Ideally, for comparison with PL-MCMC, we would consider employing a Metropolis MCMC through the modeled data space proposing X ~ (x IM for some appropri- ately chosen иM.. However, we found that naive Metropolis-Hastings through the modeled data space required such a small OMH that the Markov Chain made no discernible progress. We there- fore resorted to per-missing-pixe Gibbs sampling, with missing pixel proposals generated following ThM ~ ((,,, o?), with 14 and a; denoting the i-th missing pixel's observed mean and standard deviation throughout the training set.
For both techniques, the initial state Markov Chain (latent state for PL-MCMC and missing pixel values for Gibbs sampling) is determined by unconditional sampling from the model at standard variance (temperature T = 1.0, in the terminology of [Kingma & Dhariwal (2018)). In each repli- cation, the conditional mean is estimated from the average of 100 independent Markov Chains. To gather statistics regarding the variance of estimated conditional mean RMSE, we perform 10 sepa- rate replications of the experiment.
23

---

Published as a conference paper at ICLR 2021
Gibbs Sampling ng
0.225-
PL-MCMC, 6o- -
PL-MCMC, 0a - 1
Conditional Mean RMSF
0.200-
PL-MCMC, 6 10-25 25
0.20-
PL-MCMC, 6o 10-1
Acceptance Rate
0.175
PL-MCMC, Oa 10-4
0.15-
0.150
0.125
0
0.100
0.015
0.05
0
20000
40000
20000
40000
Flow Transformations
Flow Transformations
Figure 16: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by choice of auxiliary density.
Gibts Samp ing
0.25
0.225
PL-MCMC, o2 0.001
PL-MCMC, op 0.01
Conditional Mean RMSF
0.200-
PL-MCMC, of = 0.05
0.20
Acceptance Rate
0.175
0.15-
0.150
0.125
0.10
0.100
0.05
0.075
0.00
20000
10000
20000
10000
How Iranstormations
How transformations
Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by the scale of perturbations used in their transition proposals.
Figure 17:
As the evaluation of PL-MCMC Metropolis- Heattpolis-Haatinnss proposal involved two transformations through the normalizing flow while the evaluation of Gibbs sampling proposal involves only one transformation, it can be argued that each PL-MCMC proposal was twice as costly each Gibbs sample. For this reason, Figures [6 and 17 perform the same comparisons as Figures 7 and g but with respect to the number of flow transformations utilized in the Markov Chains. We can see that, even accounting for the relative computational costs of the two methods, PL-MCMC offers significantly improved sampling performance.
Figures 18 and 19 compare the effect of altering proposal distribution scale when the transition proposals generated only by the perturbation kernel. For reference, the results from default op = 0.01, or = 1.0, and 0a = x 10-3 PL-MCMC implementation are also included in red.
24

---

Published as a conference paper at ICLR 2021
Gibb: Sampling
0.5
0.225-
PL-MCMC, op- 121011
PL-MCMC, o, D.01
Conditional Mean RMSF
0.200
PL-MCMC, o, D.05
0.4
PL-MCMC, op D.01,
Acceplance Rate
0.175
0.3-
0.150
0.2
0.125
0.100
0.1
0.073
0.0
0.050
0
5000
10000
15000
20000
25000
0
5000 10000
15000
20000 25000
Netrooolis-Hastingg Proposals
Metropolis- Proposals
Figure 18: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. Unless otherwise stated, PL- MCMC implementations only utilize a perturbation transition kernel and only differ by the scale of perturbations used in their transition proposals
Gibbs Sampling
0.5
0.225-
PL-MCMC, o 13001
PL-MCMC, op = D.01
Conditional Mean RMSF
0.200
PL-MCMC, op = D.0G
0.1-
PL-MCMC, of 2,0,
Acceptance Rate
0.175-
0.3
0.150
0.2
0.5
0.100-
0.1
0.075
0.0
0.050
0
10000
20000
30000
10000
50000
0
10000
20000
30000
40000 50000
How Iransformations
How Transformations
Figure 19: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. Unless otherwise stated, PL- MCMC implementations only utilize a perturbation transition kernel and only differ by the scale of perturbations used in their transition proposals.
Figures 20 and 21 compare the effect of altering the scale of the transition proposal's 's resampling kernel.
25

---

Published as a conference paper at ICLR 2021
0.25
Gibas Sam olnn
PL-MCMC, or
PL-MCMC, o, 1.D
0.20
Conditional Mean RMSE
PL-MCMC, a. 1.81
0.20
Acceptance Rate
0.15-
0.15
0
0.10
0.05
0.05
0
10000
20000
10000
20000
Metropolis- Proposals
Metropolis- Hastingg Proposals
Figure 20: Single standard deviation envelopes of estimated conditional per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by the scale of the resampling kernel for their transition proposals.
0.25
Gibas Samoling
PL-MCMC, a,
PL-MCMC a 1.0
0.20
Conditional Mean RMSE
PL-MCMC, 1.81
0.20
Acceptanca Rate
0.15
0.15
0.10
0.10
0.05
0.05
20000
10000
20000
10000
How Iranstormations
How rransformations
Figure 21: Single standard deviation envelopes of estimated conditional mean per-pixel RMSE and proposal acceptance rate for conditional sampling of MNIST digit. PL-MCMC implementations only differ by the scale of the resampling kernel for their transition proposals.
E.4 RESTRICTIVE AUXILIARY DISTRIBUTIONS Do NoT NECESSARILY REDUCE PL-MCMC To STOCHASTIC SEARCH
Given that our results from auxiliary distributions with standard deviations of a. 10-3 and G_ 10-6 closely overlap, we may be concerned that the auxiliary distribution might dominate PL-MCMC's behavior and reduce the procedure a simple search in the latent space best rebuild the observed data. To determine whether an auxiliary distribution with ingly dominates the conditional sampling process, we follow PL-MCMC implementation with
10 overwhelm-
0, = 10-3 and determine how often the its decisions regarding proposal acceptance would be contradicted by an implementation with o. = 2c. These results are provided within Figure 22
26

---

Published as a conference paper at ICLR 2021
Probability of Decision Change
0.40
0.35
All Preposals
Accepted Proposals
0.30
Rejected Proposals
0.25
0.20-
0
10000
20000
Metropolis-Haatingg Proposals
Probabilities that a PL-MCMC implementation with da 10-3 makes proposal deci- sions that would be contradicted by an implementation with da co. "Accepted" and "Rejected" proposals refer to the decisions made by the oa 10-3 implementation.
Figure 22:
Fundamentally, we that, at least with o, 10-3 in this particular task, the two implementations are likely to agree in their decisions to accept or reject particular proposal transitions. Choosing an auxiliary distribution with aa 10-3 does not overwhelmingly "bully" the Markov Chain into mindlessly reconstructing observed data. Still, there is a substantial probability that this choice of auxiliary distribution will alter the decision, which is the mechanism by which the auxiliary distribution helps to guide the Markov Chain and improve conditional sampling performance. We suspect that these decision change probability computations could useful for tuning the choice of auxiliary distribution.
27


Reviews:
[{"review_id": "y_YRoMR2zpN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission introduces an approach for conditional sampling by running a Random walk Metropolis algorithm on the latent space as a pullback of normalizing flows. The normalizing flows are learnt via a MC-EM approach for incomplete training data. The method is illustrated on numerous experiments sampling missing parts for image and UCI datasets. \n\nPositives:\nThe combination of normalizing flows and MCMC sampling for conditional sampling is new as far as I am aware and is an interesting approach. Qualitative experimental results on different data sets seem promising with quantitative results indicating that it performs competitively (better reconstruction RMSE compared to MisGAN, closely matching the performance of MIWAE). More advanced MCMC samplers (such as gradient-based ones) could be used in the proposed framework, potentially yielding better experimental results.\n\nNegatives:\nThe approach relies on hyperparameters such as the choice of the auxiliary data density q and the MCMC proposal density g, and I feel that choices for these should be analyzed in greater detail in the paper. For example, the choice of both q and g affect the exploration of the target and so do they affect the sampling quality in practice for a reasonable number MCMC steps? Likewise, it is well documented that adaptive MCMC methods can converge to a different target and I am not yet convinced that the MC-EM approach approximately maximizes the complete training data likelihood without some restrictions on the MCMC adaptation strategy.\n\nRecommendation:\nI am recommending a weak accept, since the approach is interesting and is supported by many experiments, however, the hyperparameter choices for the adaptation should be better analyzed.\n\nComments:\nHow do you choose the perturbation scales for the random walk proposals?\nCould it be that the target has different scales across different dimensions, so that the MCMC kernel explores only some dimensions well and does this affect the learning process?\nDo the acceptance rates have a large influence on the MC-EM training? My intuition would be that for high acceptance rates, the learning process would be more volatile (the training samples X’_train would have many same entries and some very different ones)?\nWhy do the acceptance rate go to zero in Figure 7?\nWhy do you make the approximation of the Metropolis-Hastings ratio in the experiments? Is the proposal not just a simple 2-mixture density which should not incur a much higher computational cost?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oWQ1rg9bfsJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I enjoyed reading this paper and the idea of combining normalizing flow density models with conditional sampling seems natural, useful and has interesting potential applications to missing data problems.\n\nThings that were not clear to me are the following:\n\n1. I don't understand what this \"pre-trained\" model being referred to is. Is this a generative model for the entire data set? Do you mean that you get the model $p_{f, \\theta}$ from somewhere already and not learn it yourself?\n\n2. The choice of $q$ is one of the more interesting parts of the proposed algorithm that deserves to be studied in greater detail! If we don't use $q$, wouldn't the algorithm still work? You should try a version of your algorithm without assuming any $q$ on $y_O$, then checking how much does it affect performance.\n\n3. I am not convinced that it is fair to set \"q to be an independent normal distribution centered on the conditioning values $x_{O}$\". Clearly, with sufficiently low variance for $q$ it is possible to force the chain to sample values of $y_{O}$ that are very close to $x_{O}$. Since the dependence between $y_{O}$ and $y_{M}$ is generally strong, this will have you perform good reconstruction of $x_{M}$. If you do use a $q$ you should try variances other than the (1e-03)^2 which you have used as mentioned in the Appendix.\n\n4. Is it expensive to compute the probability density for the Metropolis correction? If you don't do a Metropolis correction, do the conditional samples still look reasonably good?\n\n5. Somewhat related to 2). In section 6.3 figure 7 the acceptance rate for PL-MCMC looks really low and is continuing to drop. You should probably run the chain for even longer until the acceptance rate plateaus completely and see what the samples at this point look like. Or perhaps you should decrease your proposal scaling.\n\nI am willing to upgrade my review if particularly point 2,3 is addressed.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ka9z27ytVg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a stochastic expectation--maximisation (EM) algorithm. The main idea is that the target distribution is specified as a deterministic mapping, a.k.a. a normalising flow, from some simple \"base\" distribution.\n\n\nStrengths\n\nThe algorithm appears to be formally correct (in the sense that it is a standard stochastic EM algorithm). The method is demonstrated on a large number of examples.\n\nWeaknesses\n\nThe proposed algorithm is just a standard Metropolis--Hastings (MH) update interspersed with a stochastic EM update for the parameters of the target distribution. This does not seem novel; such algorithms have been around for decades.\n\nThe authors should explain why they need for extending the space to include $y_O$ instead of just mapping $\\xi$ to $y_M$.\n\n\nMinor comments\n\n- There are a number of typos in the bibliography mostly related to inconsistent use of capital letters in article titles and journal/conference names.\n\n- What does the semi-colon in $p_{f,\\theta}(x_M; x_O)$ mean? Why not use a comma if this is meant to be a joint density?\n\n- In Section 3, it would be helpful to write the (extended) target distribution of the Metropolis--Hastings algorithm down explicitly and formally.\n\n- Within LaTeX's maths mode, you cannot just write operators $min$ and $Uniform$ (LaTeX treats this, e.g., as multiplying $m$ by $i$ by $n$ which leads to the wrong spacing).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work combines normalizing flows with conditional sampling. While there are connections to other works, the paper seems novel and applicable, and has nice experimental results. The authors did a good job clarifying the reviewers questions, and have addressed their major concerns. We appreciate the additional analyses added to the paper.
Decision: Accept (Poster)