VISCOS Flows: Variational Schur Conditional Sampling with Normalizing Flows
Authors: Vincent Moens
Aivar Sootla
Haitham Bou Ammar
Jun Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Normalizing Flows
Conditional Sampling
Implicit Methods

Abstract: We present a method for conditional sampling for pre-trained normalizing flows when only part of an observation is available. We derive a lower bound to the conditioning variable log-probability using Schur complement properties in the spirit of Gaussian conditional sampling. Our derivation relies on partitioning flow's domain in such a way that the flow restrictions to subdomains remain bijective, which is crucial for the Schur complement application. Simulation from the variational conditional flow then amends to solving an equality constraint. Our contribution is three-fold: a) we provide detailed insights on the choice of variational distributions; b) we discuss how to partition the input space of the flow to preserve bijectivity property; c) we propose a set of methods to optimise the variational distribution. Our numerical results indicate that our sampling method can be successfully applied to invertible residual networks for inference and classification.

Under review as a conference paper at ICLR 2022
VARIATIONAL SCHUR CONDI- TIONAL SAMPLING WITH NORMALIZING FLOWS
VISCOS FLOWS:
Anonymous authors Paper under double-blind review
ABSTRACT
We present method for conditional sampling for (possibly pre- (re-trained) normaliz- ing flows when only part of an observation is available. We derive a lower bound to the conditioning variable rog-probability using Schur complement properties in the spirit of Gaussian conditional sampling. Our derivation relies on partitioning flow's domain in such way that the flow restrictions to subdomains remain bijec- tive, which is crucial for the Schur complement application. Simulation from the variational conditional flow then amends to solving equality constraint. Our contribution is three-fold: a) we provide detailed insights on the choice vari- ational distributions; b) discuss how to partition the input space of the flow to preserve bijectivity property; c) we propose a set of methods to optimise the variational distribution. Our numerical results indicate that our sampling method can be successfully applied to invertible residual networks for inference and clas- sification.
INTRODUCTION
Conditional data generation is ubiquitous and challenging problem, more when the data is high dimensional. If the partitioning of the conditioned data and conditioning itself can be estab- lished in advance, a wide variety of inference tools can be used ranging from Bayesian inference
(Gelman 2013) and its approximations (e.g. variational inference (Klys et al. 2018)) to Gaussian (Williams& & Rasmussen 2006) or Neural Processes (Garnelo al.) 2018). The task is considerably more challenging when the partitioning cannot be anticipated. This is met in wide range of appli-
cations, from few shots learning (Wang et al., 2020) to super resolution image generation (Bashir
et al. 2021 high-dimensional Bayesian optimisation (Moriconi et al., 2020) GGomez-Bobaate
ctt al. 2018 learning with incomplete datasets YYon et al. 2018 Li et al. 2019 Richardson et al. 2020 image inpainting (Elharrouss et al. 2020) and many more.
If we can assume that the data is normally distributed, the conditional distribution can be computed in closed form using Gaussian elimination and the Schur complemens. This observation paved the way for Gaussian Processes (GPs) solutions (Williams Rasmussen] (006) and alike. Although they constitute solid component of the modern machine learning toolbox, the Gaussian assumption is quite restrictive in nature. Indeed: GPs in their vanilla form are used when the observations uni-dimensional and when inference has be performed on subset of these variables. As consequence, alternative tools may need to be used when dealing with even moderately dimensional data, or when part of the input data can be missing.
is well-established that normalizing flows (NFs) (Rezende & Mohamed] 2015) can model arbitrar- ily complex joint distributions. Furthermore, they naturally embed together the joint probability of any partition of data that could occur during training or at test time. Hence, it is tempting to make use of NFs to perform conditional sampling, but only few works have provided solutions each with some limitations. (Cannella et al., 2020) proposed Projected Latent Monte Carlo Markov Chains (PL-MCMC) to generate samples whose probability is guaranteed to converge to the desired condi- tional distribution as the chain gets longer. While such convergence is certainly advantage, one can identify several important drawbacks: first, as for every other Monte Carlo sampling method, the mixing time cannot be known in advance, and hence may wonder whether the samples that are gathered truly belong to the conditional distribution. Second, although the observed part of the imputed data converges towards the true values, difference between the may persist. Third,

---

Under review as a conference paper at ICLR 2022
training model comprising of missing data with PL-MCMC is achieved through a Monte Carlo Expectation Maximisation (MCEM) scheme (Dempster et al. 1977 Neath, 2013). Under appro- priate assumptions, MCEM is guaranteed to converge to local maximiser of the log-likelihood function, but this is conditioned on the quality of the data generated by the Monte Carlo algorithm. Consequently, as the optimisation progresses towards the optimum, longer chains may be required to ensure convergence or even to obtain convincing results (Neath) 2013). MCFlow (Richardson al., 2020) relies on an auxiliary feedforward neural network whose role is to produce latent em- beddings with maximum posteriori likelihood values. Those values are constrained to lie on the manifold of latent vectors whose mapping to the observed space match the observations. MCFlow produces state-of-the-arr results in terms of image quality or classification accuracy when compared to GAN-based methods such as (Yoon et al. 2018 Li et al. 2019). However, this method requires a set of adjustments to the model and needs retraining for any additional incomplete data. On the other hand, ACFlow Li et al. (2020) learns all conditional distributions for all possible masking op- erations, which can be quite computationally expensive. As such, both MCFlow and ACFlow cannot be applied to post-training data completion as PL.MCMC.
Several conditioning techniques have been used with NFs in contexts where the joint probability distribution of the conditioning and conditioned random variable is of no interest. For instance, (Rezende & Mohamed| 2015: van den Berg al., 2018) extended the amortisation technique used in Variational Auco-encodeers (VAHs) (Kingma & Welling, 2013) to the parameters of the flow. WWinkler et al.) 2019) extended this use to the scenario of conditional likelihood estimation, while Trippe & Turner 2018) provided a Variational Bayes view on this problem by using a Bayesian Neural Network for conditioning feature embedding. On different line of work, (Kingma & Dhan- wal 2018) use heuristic form of posteriori conditioning. The generative flow is trained with a classifier over the latent space, which forces the latent representation location to be indicative of the class they belong to. At test time, some parametric distribution is fitted on the latent representations to account for the attributes of the images that are to be generated. (Nguyen et al. 2019) used hy- brid approach, whereby the conditioning and conditioned variables joint distribution modelled by a normalizing flow in such a way that conditional sampling of one given the other is straightforward to apply. Still, either with this approach or the others above, the knowledge of what part of the data constitutes the conditioning random variable is required beforehand.
We propose conditional NF sampling method in the following setting: a) conditional data generation is performed after a model has been trained without taking conditional sampling into account; b) training data may be itself incomplete, in the sense that some training features might be missing from examples in the training dataset; c) the subset of missing data may not be known in advance, and it could also be randomly distributed across input data. Importantly, we are interested in deriving a method whereby the distribution of the generated data faithfully reflects the Bayesian perspective. Our derivations heavily rely on the Schur complement properties in the spirit of the conditional Gaussian distributions. To highlight this feature we call our approach VISCOS Flows: Varlational Schur COnditional Sampling with normalizing Flows. The use of a variational posterior brings some advantages: for example, with single fitted posterior, multiple samples can quickly be recovered. We also show how to amortise the cost of inference across multiple partially observed items by using inference networks (Kingma & Welling 2013).
Our Contribution:
2 VISCOS FLOWS: VARIATIONAL SCHUR CONDITIONAL SAMPLING WITH NORMALIZING FLOWS
2.1 PRELIMINARIES AND PROBLEM FORMULATION
Preliminaries: Consider C¹ c'ddiffeomorphism f(X) x- where X' C Rd. y C Rd are open sets and define the inverse of f to be g(y) = f- + X. Consider the case where the random variable Y =(X) while X is distributed according to the base distribution Po(X) with density Po(x). According to the change of variable rule Y has a log-density given by the following formula:
log p(y) = log (o()()))
+ log (det Vyg(y))
= log logpo(#)
log |det (=)(
We will use two set of complementary indexes covering [d]: observable O C [d] with cardinality #0 d0, and hidden H. C [d] with #H d and O H [d]. In what follows, without loss of
2

---

Under review as a conference paper at ICLR 2022
generality we assume O= =(0...., d - -1) and H = ........) and use the following notation:
f(x) (fr (0(20) ; wtt (wott ;000
= (2))())
)OH(a)) ( JH (z))
and we will use a similar partition for ( and its Jacobian G(y). We will use the notation (J) = Joo to denote sub-matrix masking. The central to approach is the Schur comple- ments and its properties and, in particular, the following well-known identities:
GHH(S(z)) (JH -H00 (z) (1000) (OH()))
det(J) = det (JOO) det (JHH -HO (JOO)-) jon) det (JOO)
(3)
del (G H)
We denote by A the "detached" version of matrix- valued function A(x), i.e. a matrix whose values match those of but have a null gradient.
Problem formulation: Consider set Y containing some samples of interest, for example, images. Assume also a pre-trained normalizing flow, i.e,, map f X' - y with X, C RR. In what follows we relax the assumption the pre-trained flow, but for streamlining the presentation assume that the normalizing flow is given at this point. Let us now assume that the observation sample y is partially missing or masked (for instance, photographer's finger is covering part of the image). In particular, the observation is split into observed y = fv y E y} and hidden y² "yy y } regions, where O, H are observed and hidden indexes, respectively. Our goal is to sample y from the conditional distribution over a partial observation Yo E yoo While this distribution has a density given by " ))
ply) ply computing p() p(y", " ydy in general intractable. Since it is often easier sample from and optimise in the latent space will aim to express all distributions in the space X. First, we introduce partition into observed and hidden indexes in the latent space X.
Assumption AO. Both spaces X' and y are partitioned 1} and H dJ..
as O- {0,... .., d0 ,o
We stress that Assumption AO is made only to sim- plify the exposition. In general, we can have arbitrary partitions of the observed space. which will not affect our algorithm. While this partition seems somewhat artificial, it enables the use of the Schur complement in our derivations. Choosing the same partition in X' and ) is natural for the invertible residual networks (iRes- Net) (Behrmann et al. 22199) structure of the flow. This observation also holds for any flows sharing a residual structure + h(())
((.)
X
f())
[019: CCen et al. [201)) due the
Figure 1: Example of partitioning.
with an activation h (e.g. ReLU), such as Planar and Radial flows (Rezende & Mohamed 2015
Sylvester flows (van den Berg et al. 2018), continuous flows (Chen et al. [212200110aaa Onken al. 2021
Mathieu & Nickel 2020)) or to some extent Implicit NFs (Lu et al. 2021). where at each layer / of the flow, the derivative of the output o'41 with respect the input layer m, will be dominated by the 7tth component of this input. We note that the partition in latent space X' can be made different from the partition in the observed y for numerical purposes. For example, picking a latent space partition- ing with maximal value of log det(J will ensure that the matrix Joo is not ill-conditioned. Indeed, for any matrix A we have log det(A)) log(det are the singular values of A. Thus promoting larger singular values will improve conditioning. We empirically found that across several architectures (iResNet, Planar, Radial, Sylvester and Implicit flows), this approach leads to a well-behaving algorithm. Maximising log det(J to choose partitioning can also potentially be used for coupling flow architectures (e.g. Glow (Kingma & Dhariwal 2018) or RealNVPs (Dinh et al. (0166), however, this requires additional research. While Assumption AO is made for convenience of notation, the following assumption on is necessary.
Σ, log(a;(A) where a;
Assumption A1 The map f X + y and its restriction (xx;x"") x + yo for all r E H are Ci-diffeomorshsssss.
3

---

Under review as a conference paper at ICLR 2022
Let us now formally define our reparametrisation from ) space to X' as follows:
x (y yy) := := (y" (x H .)) such that fo (xx0x=") 30 =0
(4)
(5)
Our reparametrisation well-posed as stated in the following lemma with the proof in Appendix
[A]
Lemma 1 If Assumption Al holds then the reparametrisation in Equations + Lnnd is well-posed, i.e, g" (y" )) is diffeomorphism, while the map x x0 y)) exists and it is differentiable.
Now we can rewrite Equation | uuinn Equation 3 in simpler terms:
logp(z) log Pa (x log det G H| + log log log det J
(6)
Here we used the fact that the base distribution (o(x) can be factorised as a product of independent univariate distributions, and hence pu(z:) po )po(= Now we can define the variational H; distribution in terms of the latent samples as follows: "(y"; : = ((==)) ddet G H Combining Equation 6 with our variational distributions we obtain the evidence lower bound:
LELBO = Eg(y":00) [logp(y", ))
+ HHI [q(3"; y)
Po (a
= E ((xl1)
log
+ log Po px log det Jo (x0 x)
(x"")
where H[-] is the entropy and we have taken advantage of the fact that the log-absolute determinant (LAD) of the Jacobian belonging to the approximate posterior o(y"::" (0) cancels with the one from the joint log-probabilin given by the model (i.e., the terms with G cancel each other out). Furthermore, we have eliminated the dependence on the variable y" completely instead we treat x " as variational variable. We note here that our, at the first sight, artificial partition of the latent space into observable x and hidden a: parts was the key allowing for the reparametrisation from y space X' space. In particular,
used the invertibility of sub-jacobians GH и and Joo allowing to map between observable and hidden samples in the X' and ) spaces. Our approach is in the spirit of Gaussian conditional sampling and Schur complement, which justifies the method's name. Note that this ELBO loss can be added to other losses, e.g., to classifier loss as we do in what follows. Note that this ELBO can be used in combination with other losses. Thus the normalizing flow can be trained simultaneously while performing conditional sampling of missing data. This way we can treat masked data during training as well as after training.
There are still two technical issues that we cover in the following subsections before presenting our algorithm. First, we need to get x0 by solving Equation 5 Second, we need to discuss the computation of the ELBO gradient with respect to .. Specifically, need reparametrise 2 and compute the LAD gradient. As the following two sections are quite technical the reader interested in the algorithm itself can skip to Subsection 2.4
2.2 SOLVING THE EQUALITY CONSTRAINT
We consider two root-finding approaches: a new fixed-point iterative algorithm and Jewton-Krylon methods. In practice, we used the latter as fallback option when the former did not converge below the desired convergence threshold.
Fixed-point iterative algorithm As our method heavily relies on solving equality constraints, one may wish to reduce the computational burden of this operation. We therefore seek for an efficient gradient-free method that can achieve this in some settings. In particular,
solve simultancously for x: and y the equations x = ( o H) and n = f (x. æ:²), instead of solving y fo .H ) for 2: This allows to consider fixed-point iterative approach in Algorithm whose convergence properties are analysed in Theorem I proved in Appendix B:
Theorem Consider Algorithm assume that there exists unique solution x= x0= g و "y around which gd and fd are chitz constants of f and oo are equal to La and Ly, respectively. Then the algorithm converges to the unique solution for any a E (0, 1), 3 E (0, 1), if T.g Lo < 1.
y f (x0 x
and the global Lips-
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Iterative algorithm solving for x:
Input: Invertible flow f - g partial (masked) observation y m0 (y). conditional latent
hidden sub-vector _ mixing coefficients {a, 3 (0. Output: y and x0 satisfying
x. S (a: x )= y for given a and y
Initialisation: x² 0:
7 f ,:
x = x (y y));
while has not converged do
y" fd (x x= " );
y = ay" " (1 a)y':
x = - (yO: .y")
2 - Bx + (1 B) ;
end
Newton-Krylov methods (Knoll & Keyes iterative manner for given step size
2004): In this approach, the value ax is updated in an
x++1 x (JOO)-+ (=)
which is derived using the first order expansion of the flow restriction. The reliance of this method on the inverse Jacobian-vector product (J00)-'u requires us to invert a sub-matrix (defined by the input and output masks) that, in most cases, is only accessible via left or right vector-matrix product, due to the constraints imposed by the use of backward propagation algorithms. Moreover, explicit computation of this sub-Jacobian is also usually prohibitively expensive to retrieve in closed form. Therefore, to compute this product involving an inverse sub-Jacobian, we rely on the generalized minimal residual method (GMRES) (Saad & Schultz 1986), a Krylov subspace method (Simoncini & Szyld 2007). At their core, Newton-Krylov methods (Knoll & Keyes) 2004) rely on the compu- tational amortisation of the GMRES- based inversion step by caching intermediate values between successive iterations of the solver (Baker et al., 2005) accelcrating the algorithm convergence.
Still, computing single Jacobian vector product can be quite computationally expensive, for in-
stance when using invertible residual networks (Chen et al. 2019 Behrmann et al. 2019). To this GOO
end we employ an identity reciprocal Equation 2 i.e. (J 100-1- - GOOlH (C
giving an alternative representation of (J0O)-1 First, if do
>> dd then inverting the sub-Jacobian GHH is more computationally efficient, e.g., using GMRES or similar technique. In this case, the inverse of the sub-Jacobian can be obtained or approximated only by querying Jacobian-vector products involving G, and not J. Otherwise, when computing the plain inverse of Joo using GM- RES, we used GOO preconditioner. We found this to speed up the computation by factor of to 4, depending on the size of the problem and the architecture used.
2.3 ELBO GRADIENT
Taking the gradient of ELBO relies solving two technical issues: computing the gradient of o and the gradient of LAD with respect to variational posterior parameters 0.
Reparametrising œ0, According to the implicit function theorem and Equation we can reparam- eterise d0 as a function of t y and 0 to:
===z0=-(J00) -J
The inverse of the sub-Jacobian can be obtained via GMRES algorithm with similar considera- tions regarding methods to speed up this computation to the Newton-Krylow method in Section 2.2 Reparameterising function of the variational posterior parameters 0 is then trivially achieved via pathwise or implicit gradient calculation.
Log-Absolute Determinant gradient Estimators. First recall that the gradient of the log- determinant can be represented using trace as follows (Petersen & Pedersen, 2008):
V a log det A(x) Tr A- Ax:).
5

---

Under review as a conference paper at ICLR 2022
Algorithm 2: VISCOS Flows: Variational Schur Conditional Sampling with Normalizing Flows
Input: Base distribution po invertible flow f = X + y" partial (masked)
observation yd = m m
Output: Conditional sample y
~p(- g)0)
Initialization: set qe (0(xx" (x")., select partitioning function x0 - m (x);
while has not converged do sample x ~ (x"));
compute x as in Section 2.2
reparameterise (x" y 0);
compute stochastic gradient estimate VeC(g) and update variational posterior:
end
which allows us to use built-in functions if the inverse of A is readily available. Hence, although the value of log det A(x) cannot be estimated in closed form, its gradient only requires us to dif- ferentiate the trace of the product of A and A(x), where only the latter differentiated through.
In many cases, one learns the map g(-) and hence J is only implicitly defined by its inverse G (e.g. is obtained through Neumann series or Krylov methods). In such cases, higher order derivatives (i.e.. (z)) cannot be accessed readily by backpropagating through the graph. Therefore, it is necessary to express J (x) through G(y). In particular, if 11 o < d we can obtain the Natural LAD gradient Estimator (NLADE) of the sub-Jacobian log det Joo (0)())))
Va log det (x)) = V Tr [J00)-1000
V, (1000-1m mo 00 (G- y=1(x)
((x,, (8)
where used (x)) V VG G(y) (x) Petersen & Pedersen 2008 Using NLADE can be inefficient dH < do In this case, one could express (J (JOO)- GOO GO (G H but this approach log |det Gllll (f(x)) log ddet G(f(x))) in the first place. This leads to the Corrected LAD gradient Estimator (CLADE):
Vx(G(f(w) 1-
Va G(
be executed more efficiently using the identity log det Joo
Vn log dett joo (x)) = V, (Tr
(y) - Tr -TrGG-'G(i)) y=f(x)
J(x).
(9)
Both expressions have their advantages and possible drawbacks: if do < d then NLADE would be more efficient. On the other hand, if d < do, then CLADE appears to be a better option. Fur- thermore, as CLADE does not use implicit matrix value (JOO)-1 (defined through G), its variance is potentially lower than NLADE's. In other cases, the choice would depend on the efficiency of matrix inversions of J, G, Joo, H.
2.4 ALGORITIM
We suggest a two-pass iterative algorithm aimed at sampling from a conditional normalizing flow using a parametric posterior go (y ()) first, a set of conditional samples drawn by solving the equations ( " x 0, I (a: .x y 0 or (0 (æ= == o = 0. Next, this
sample reparameterised given the variational posterior parameters, and lastly an optimisation step
made given the stochastic estimate of the ELBO gradient. The summary is given in Algorithm [
We used two distinct designs of approximate posterior depending on the task at hand. When possible, each data to be completed was treated separately, with single variational posterior configuration being fitted to each item in the dataset. When the amount of incomplete items was too large to be completed one-to-one basis, amortised the cost of inference by acquiring the variational posterior parameters through a inference network that was trained to minimize the KL divergence in Equation 7 More details be found in Appendix

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
Experimental setup We tested our variational conditional sampling algorithm on a classification and data completion task when the training and test data were partially observed, and on a post- training data completion task. Due to the restriction imposed by Assumption A1, we only consid- ered the iResNet (Behrmann al., 2019 Chen al., 2019) architecture to test our approach on in the main text. Appendix D provides results with Implicit NFs, showing that our approach can be extended to other types of architectures with possibly higher Lipschítz constants. Several steps of our gradient computation rely on computing the inverse of the Jacobian of the transform. This quantity was queried using Neumann series: at each layer i [L] of a L-deep network, we locally computed the product of vector with the inverse Jacobian
uJ-1= = u Σ (Vah(x)))
1=
In practice, the sum was truncated when the difference between two successive iterations below a predefined threshold (10-5 for single and 10 10 for double precision). Jacobian-vector products were computed using the identity Jax) uV. (v.((x)) for some As this formula requires graph to be built on top of another one, and to avoid memory overflow, we computed this quantity locally at each layer the network. The convergence threshold for Algorithms aadd was 10-3
At early stages of training, Algorithm S quickly converged with a = 3 = 1. However, lower values needed later on, when using fully trained model (in the post-training data imputation setting). Therefore, we adopted the following schedule throughout the training process: an initial mixing rate was set a =
0.5, and at each iteration of Algorithm [ the mixing rate was decayed by factor of (l.5. In less than I 1% of the iterations, our fixed-point algorithm did not reach the convergence threshold, and then we used each optimisation step took approximately 20 seconds, roughly equally split between solving the equality constraint retrieving the gradient estimate.
Newton-Krylov solver. the case of MNIST dataset,
9
/
2
2
3
9
6
1
4
2
2
3
9
}
7
4
2
6
9
7 2 g
9
)
1
4
2
4
9
7
9
4
6
1
0
9
8
0 3 5 0
1
0
9
8
2
0 3
1
2
5
8
)
3
2
5
8
>
Z 8
2
1
7
0
9
0
8
2 1
7
0
n
6
5
2
9
6
9
9
5
3
3
9
1
5
3
6
2
/
2
8
7
7
3
4
2
7
7
3
4
6
8
8
1
5
8
6
8
4
&
7
5
7
8
(a) Original MNIST digits
(b) Masked MNIST digits
(c) Completed MNIST digits
Figure 2: MNIST completion with 90% missingness rate.
Quantitative experiment: Training incomplete MNIST data To test the capability of Vari- ational Schur Conditional Sampling (VISCOS Flows) to learn from incomplete data, we used the MNIST dataset (LeCun & Cortes 2010) where percentage of the pixels of each digit image was missing. We considered only the challenging setting where the missing pixels were randomly spread across the image. The pre-trained NF component was similar to (Chen al., 2019) and consisted of an iResNet model with 73 convolutional layers with 128 channels each, followed by four fully con- nected residual layers. The trained inference network consisted of a 5-layer convolutional network with 3 3 kernel, SAME-padding strategy and with [4, 8, 16. 32, 8] channels at each layer, respec- tively. For this network, the ReLU activation function (Nair Hinton 2010) was used. Finally, we used a simple classifier consisting of the sequence of squeezing layer, a I-dimensional batch-norm layer (loffe & Szegedy 2013) and a bias-free linear layer. We used the Adam optimiser (Kingma & Ba 2015) with learning rate of 10 decaying by factor of 0.5 at every epoch. The batch size was set to 64 digits, and the components were trained simultancously for a total of 5 epochs.
The NLADE gradient estimator used for these experiments with the Hutchinson stochastic trace

---

Under review as a conference paper at ICLR 2022
Missing rate +
0.5
0.6
0.7
0.8
0.9
MisGAN
0.968
0.945
0.872
0.690
0.334
PL-MCMC
MCFlow
0.985
0.979
0.963
0.905
0.705
VISCOS Flow (ours)
0.9616
0.9588
0.9485
0.9355
0.886
Table 1: Top-1 classification accuracy on incomplete MNIST dataset (higher is better). We highlight in bold the best performance, while the symbol "-" represents missing experiment.
Missing rate
0.5
0.6
0.7
0.8
0.9
MisGAN
0.3634
0.8870
1.324
2.334
6.325
PL-MCMC
5.7
87
MCFlow
0.8366
0.9082
1.951
6.765
15.11
VISCOS Flow (ours)
0.2692
0.4398
0.7823
1.5491
7.315
Table 2: FID on incomplete MNIST dataset (lower better). mance, while the symbol "_" represents a missing experiment.
We highlight in bold the best perfor-
Missing rate +
0.5
0.6
0.7
0.8
0.9
MisGAN
0.12174
0.13393
0.15445
0.19455
0.27806
PL-MCMC
0.1585
0.261
MCFlow
0.10045
0.11255
0.12996
0,15806
0,20801
VISCOS Flow (ours)
0.1127
0.1221
0.1340
0.1470
0.1924
Table 3: RMSE on incomplete MNIST dataset (lower is better). performance, while the symbol "-" represents a missing experiment.
We highlight in bold the best
estimator (Hutchinson) 1990), taking a single sample for each gradient computation. We compared our results to the ones presented in PL-MCCC (Cannella et al., (020). MCFlow (Richardson et al.) 2020j and MisGAN (Li et al. 2019). Due to the limits of our computational resources, we resort to the performances displayed (Richardson et al.) 2020 Cannella et al. 2020). All the presented results are computed on MNIST official test dataset, which was not used for training. Figure 2 gives visual account of the performance of the VISCOS Flows data completion capability. In all cases, VISCOS Flows was either first or second ranked in terms of Top-1 classification accuracy (Table [. Fréchet Inception Score (FID (Heusel et al. (017), Table(2) or Seat--Men-Squureee (RMSE, Table 3) It is important to note that, although MCFlow outperformed our approach on some occasions, this algorithm relies on the assumption that one can train separate classifier on complete data, and then use this classifier on the completed dataset. Our classifier was, instead, trained on-the-fi fly on the incomplete dataset only.
Qualitative experiment: post-training data imputation We now turn to the problem of completing a partially observed sample using the VISCOS Flow technique. We trained a standard iResNet- architecture on the CIFAR-10 10 dataset (Krizhevsky 2009) for 250 epochs. VISCOS Flow completion capability was compared to PL-MCMC, as MCFlow cannot perform post-training data imputation. The task was similar to the one presented in (Cannella et al. 2020): masked centered square of size x 8 had to be completed. PL-MCMC completion was achieved using the same hyperparameters provided by the authors. For VISCOS Flows, we used Gaussian approximate posterior with sparse approximation to the covariance matrix encoded by 50 Householder layers. The parameters were trained using the Adam optimiser with learning rate of 10-2, and at each step, batch of 8 completed images were generated to obtain a gradient estimate. A total of 500 steps showed to be sufficient reach convergence. Figure 3 shows how our algorithm compared with PL-MCMC for a set of test images. We also included the first solution obtained by the initial standard Gaussian distribution show that, although training did make the images look more convineing, the initial guess was already somewhat well fitted to the rest of the image. As qualitative measure of fitness, we measured the RMSE between generated and true images for both algorithms, and obtained a similar value for both (VISCOS Flow (NLADE): 0.2291 + 0.1570, PL-MCMC: 0.2287 + 0.1961, where the confidence interval is 0.95% of standard deviation). By comparison, at the first iteration of the VISCOS Flow algorithm, RMSE of 0.2802 0.1188 was measured. As the gradient of the variational posterior parameters can be retrieved with two different methods (CLADE or NLADE), compared one against the other, but observed little qualitative or quantitative difference between the two (RMSE for VISCOS Flow (CLADE): 0, 0.1793. A noticeable advantage of VISCOS Flow is that with single fit of the variational posterior can generate wide variety of candidate
8

---

Under review as a conference paper at ICLR 2022
Figure 3: Upper Original images (left panel) and PL-MCMC reconstruction (right panel).
Lower row: VISCOS Flow after the first iteration (left panel), VISCOS Flow after the final iteration
Figure 4: Random samples obtained from a VISCOS Flow trained variational posterior.
images with little effort, as only few seconds are needed to solve the equality constraint depicted above. Figure giies some flavour of this capability.
4 CONCLUSION
This article presents VISCOS Flows, a new method for data completion with NFs. This technique can be successfully applied to architectures when the units of the flow are made of instances of resid- ual layers. Unlike other approaches, such as MCFlow (Richardson et al., 2020) and ACFlow (Li al. 2020). we do not need to include the conditioning in the training process. This feature im- proves algorithm's modularity by allowing to decouple NF training and conditional sampling. While PL-MCMC (Cannella et al., 2020) also enables such modularity, it relies on MCMC algorithms, which are arguably harder to use than gradient descent based algorithm. For some important classes of NFs (e.g. composed of coupling flows, such as Glow (Kingma & Dhariwal, 2018) or RealNVP Dinh et al. 2016)) the partitioning choice and solving for " may be more involved than for NEs with residual architectures. Future work should focus on finding alternative ways to fit low-dimensional variational posteriors for data completion in those cases.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Allison H Baker, Elizabeth R Jessup, and Thomas Manteuffel. A technique for accelerating the convergence of restarted gmres. SIAM Journal on Matrix Analysis and Applications, 26(4):962- 984, 2005.
Syed Muhammad Arsalan Bashir, Yi Wang, Mahrukh Khan, and Yilong Niu. A comprehensive review of deep learning-basee based single image super-resolution. PeerJ Computer Science, 7:6621, 2021.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pp. 573-582. PMLR, 2019.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American statistical Association, 112(518):859-93) 877, 2017.
Chris Cannella, Mohammadreza Soltani, and Vahid Tarokh. Projected latent markov chain monte carlo: Conditional sampling of normalizing flows. In International Conference on Learning Rep- resentations, 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Proceedings the 32nd International Conference Neural Information Processing Systems, pp. 6583,6583, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and fbem--Herike Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations, 2016.
Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, and Younes Akbari. Image inpainting: A review. Neural Processing Letters, (1(2:)207-20225 December 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Jimenez Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, 2018.
Andrew Gelman. Bayesian Data Analysis, Third Edition. Chapman and Hal//CRC, 3rd edition, nov 2013. WRKLBtttss///ww.ceet wwanciirfrriiissso.. bwok/bbokk///// / 9781 4394 433498999820
Rafael BombarBoo Jennifer N. Wei, David Duvenaud, Jose Miguel Hernández-Lobate Benjamín Sånchez-Lengeling, Dennis Sheberla, Jorge Eguilera-Iparraguirg Timothy D. Hirzel, Ryan P. Adams, and Alan Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS Central Science, 4(2):268 276, 2018
Louis Guttman. Enlargement Methods for Computing the Inverse Matrix. The Annals of Mathemat- ical Statisties, 17(3):336 343, 1946.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. Journal of Machine Learning Research, 14(5), 2013.
M.F. Hutchinson. A stochastic estimator the trace the influence matrix for laplacian smoothing splines. Communicatiions in Statistics Simulation and Computation, 19(2):433-45 1990.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 1448-56- PMLR. 2015.
Tommi S. Jaakkola and Michael I. Jordan. Computing Upper and Lower Bounds on Likelihoods in Intractable Networks. Proceedings of the Twelfth international conference on Uncertainty in artificial intelligence, pp. 340-348, 1996.
Michael Karow, Diederich Hinrichsen, and Anthony Pritchard. Interconnected systems with un- certain couplings: Explicit formulae for mu-values, spectral value sets, and stability radii. SIAM journal on control and optimization, (5(3):856-884) 2006.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes, 2013.
URL http:
//arxiv. //arxiv.orr//aao abs/ 1112. 6114
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 10215-10224. Curran As- sociates, Inc., 2018.
Jack Klys. Jake Snell, and Richard Zemel. Learning Latent Subspaces in Variational Autoencoders. In 32nd Conference on Neural Information Processing Systems, Montréal, Canada., 2018.
D.A. Knoll and D.E. Keyes. Jacobian-free newton-krylov methods: a survey of approaches and applications. Journal of Computational Physics, 193(2):357-397, January 2004.
David A. Knowles and Thomas P. Minka. Non-conjugate Variational Message Passing for multino- mial and binary regression. Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 201 pp. 1-9, 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Depart- ment of Computer Science, University of Toronto, pp. 32-33, 2009.
cs.
lecun. haaibbo.edd//kkiii/eeeaaaa features- 2:00 20000TR..p0
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010. URLhtt/://yann.
Steven Cheng- Xian Li, Bo Jiang, and Benjamin Marlin. Learning from incomplete data with gener- ative adversarial networks. Intermational Conference on Learning Representations, 2019.
Yang Li, Shoaib Akbar, and Junier Oliva. Acflow: Flow models for arbitrary conditional likelihoods.
In International Conference on Machine Learning, pp. 5831-5841. PMLR, 2020.
Winfried Lohmiller and Jacques E Slotine. On contraction analysis for non-linear systems. Automatica, 34(6)6683-696, 1998.
Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows. In International Conference on Learning Representations, 2021.
Emile Mathieu and Maximilian Nickel. Riemannian Continuous Normalizing Flows. In Advances in Neural Information Processing Systems, 2020.
Riccardo Moriconi, Mare Peter Deisenroth, and KS Sesh Kumar. High-dimensional bayesian opti- mization using fow-dimensional feature spaces. Machine Learning, 109(9): 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 807-814 2010.
Ronald C. Neath. On convergence properties of the monte carlo EM algorithm. In Advances in Modern Statistical Theory and Applications: A Festschrift honor of Morris L. Eaton, pp. 43- 62. Institute of Mathematical Statisties, 2013.
11

---

Under review as a conference paper at ICLR 2022
Tan M. Nguyen, Animesh Garg, Richard G. Baraniuk, and Anima Anandkumar. Infocnf: An efficient conditional continuous normalizing flow with adaptive solvers. arXiv preprint arXiv:1912.03376. 2019.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. OT-Flow: Fast and Accurate Con- tinuous Normalizing Flows via Optimal Transport. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9223-92222222 2021.
K. B. Petersen and M. Pedersen. The matrix cookbook, October 2008. URL URL htttp://www2
imm. dtu wwtppkk/ppppppppp php332274 Version 20081110.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In 32nd International Conference on Machine Learning, ICML 2015, volume 2, pp. 1530-1538, 2015.
Trevor W Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A Bernal. Mcflow: Monte carlo flow models for data imputation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14205-14214, 2020.
Youcef Saad and Martin H. Schultz. GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems. SIAM Journal Scientific and Statistical Computing, (3):856-869, July 1986.
Valeria Simoncini and Daniel B. Szyld. Recent computational developments in Krylov subspace methods for linear systems. Numerical Linear Algebra with Applications, 14(1):1-59, 2007.
ISSN 10705325.
Eduardo D Sontag. Mathematical control theory: deterministic finite dimensional systems, vol- ume 6. Springer Science Business Media, 2013.
Jakub M. Tomezak and Max Welling. Improving Variational Auto-Encoders using Householder 09630
Flow, 2016.
Duc N Tran, Björn S Rüffer, and Christopher M Kellett. Incremental stability properties for discrete- time systems. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 477-482. IEEE, 2016.
Brian Trippe and Richard Turner. Conditional density estimation with bayesian normalising flows. arXiv preprint DeXXiv/802..04900 2018.
Rianne van den Berg, Leonard Hasenclever, Jakub Tomezak, and Max Welling. Sylvester normaliz- ing flows for variational inference. In proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3);1-34, 2020.
Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows. arXiv preprint 1rX1v:1912.00042, 2019.
Jinsung Yoon, James Jordon, and Mihacla van der Schaar. GAIN: Missing data imputation using generative adversarial nets. In Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 5689-5698. PMLR, 10-15 Jul 2018.
12

---

Under review as a conference paper at ICLR 2022
A DIFFERENTIABILITY OF THE REPARAMETRISATION
For convenience we state the lemma
Lemma If Assumption Al holds then the reparametrisation in Equations Aand S is well well-posed, ie, gl is a diffeomorphism, while the map x (x y exists and is differentiable.
First, let us show that nاd is C diffeormorphism. Recall that Va J invertible since f(+) and (= for all x are C' c'ddifffomorphiiss. Now, according to the Guttman rank additivity formula (Guttman 1946 if Joo is an invertible submatrix of an invertible matrix J, then
the notation JoO V 20 v p Note that the matrices Joo, J are
JoH , T0 JH
rank(J) = rank(Joo) + rank (J j (JOO) jou)
It is easy to see that in this case, the block JHH JHO (JOO) joH is also full-rank, and its inverse is given by GHH which is the Jacobian matrix V Vyng" (y" As these identities are valid for all æ, the Jacobian matrix of g (y y0) for yd is always invertible and hence g is C dd-ifffeomorphimm as well.
Now let us show the second part of the statement. Since (xx a diffcomorphism, due to implicit function theorem there exists locally differentiable map x (x )) for all w y)
B THEOREM 1: A CONVERGENCE RESULT
For convenience we restate the theorem.
Theorem Consider Algorithm| assume that there exists unique solution - y f . around which go and f are muniimously-differentihhh and the global Lips- chitz constants of f and do are equal to La and Lo, respectively. Then the algorithm converges to the unique solution for any a E (0, 1}, (0, 1), if La
x0= = )
We argue convergence of the iterative procedure using dynamical systems theory and the notion of stability (cf., (Sontag 2013)). In general, stability is stronger property than convergence, as stability requires convergence under perturbations in the initial conditions of an iterative procedure. We will prove the theorem Proposition T what follows, but first we need to develop theoretical notions of stability.
Recall that the updates in the algorithm are as follows:
y = f M (x) x "
0)y
0 = 3g 0 (1
-3x
By letting 7), & stand for y x0, respectively, and al-) (: and b(.) (3O :), our iterative procedure can be written without loss of generality as the following dynamical system:
k+1 (1- =a)n** + axa(e))
(10)
c++1 = * + *b())+
We now will review stability results, which are presented for the systems ck+1 = C(C*) with the initial state Ç = v, we will denote systems' trajectories as (²()). In our bounds, we will make use of the class of ICC functions. The function 7 E KLL if y R200 x Rzo + Rzo. 7 is continuous and strictly increasing the first argument, continuous and strictly decreasing the second argument, with lim,- r(C,k) k)) = 0 for every Ç. (0, k) for all à: 0.
Definition A system is called globally usymptotically stable at S if there exists a function 7 E K.L. such that for all and all k:
(x)
13

---

Under review as a conference paper at ICLR 2022
Stability theory offers many additional tools, for instance, the ability of studying how trajectories
behave in comparison to each other, which can be done using contraction theory (Lohmiller & Slotine 1998) or using similar concept of incremental stability Tran et al. 2016).
Definition 2 A system is called globally asymptotically incrementally stable if there exists function Y such that for all vs, V2 and all
(v2) -0 S 1 -2111,
In our derivation, we will also employ local convergence concepts and results
Definition 3 A system is called locally asymptotically stable at Ç if there exist a function and > 0 such that for all | 5- and all k:
(v) - -- S
Checking local stability is rather straightforward given some regularity conditions. In particular, need to check the magnitude of the spectral radius of (C((). Recall that spectral radius p (D) is defined as p max; (^x where X are the eigenvalues of D.
Lemma 2 Let A be continuously-die differrsstyydieeeettabblele around C- If the spectral radius p (OC((*)) is strictly smaller than one, then the system is locally asymptotically stable around Ç" If C(() is linear function, then the system is also globally stable.
We also have this lemma linking incremental stability and stability properties, which can be shown in straightforward manner.
Lemma If the system gh+1 = C(C²) is locally asymptotically stable around S* and globally asymptotically incrementally stable, then it is also globally usymptotically stable.
Finally, we need the following lemma, which is combination of two results discussed in (Karow et al., 2006): properties of nonnegative matrices (the property p2) and Lemma 4.1., which follows the introduction of these properties.
Lemma 4 Let CjR E Rbxl,, ||- is the singular value matrix norm, and || ||Cr S Cjk. then
C
Cim
||Cnll
|
Clm
Sp
SPP
Cml
Cmmm
[
||
Cml
Crnm
Now we are ready to proceed with the proof.
Proposition 1 The system equation 10 is globally asymptotically incrementally stable for any a E 10,1) if La < 1. Furthermore, if there exists E", 11" such that E" a(n²*), ), n" b(g*), then the system is globally asymptotically stable.
Proof: First let us show incremental stability. Let dny = nf nb. sm
k+1
772 k+1
se* E thi and oeh+1 El+t1 ek+1 then we have
||
++
|m
k+1
n2
*+1|| = (1 a) n + a(a(E)) - a($)))) 5
(1 a)ll - n ll + alla(Ef)) a(<))) s a0)l + aT. 11
Similarly
|| 1| = ||5 - 1 [1(1 (1
3)(F - + ((b((((( *+1 b(n + +)1+ S
(1 (1-99)1st 15t - <till *|b()t -b(n-bb()))))) [|]] s (1 11 BLg 18n -
((1 (1- 3) a.LL, (1- -))Iollōn"11
14

---

Under review as a conference paper at ICLR 2022
Summarising have the following bounds
|!ó + 2 (1 a))l aLا ||
(11)
|| I ao) + ((1 - 5) + aBLsLa))(8
which we can equivalently write in the matrix form:
(10s++11)
aLa = (1 a)3L) 5) a3 a LoLaa
1 1-a
(12)
As these bounds are valid for every step k, we can study convergence of [|877]], || using the system:
k+1
1-a a (1 a3L
ala
k+1
1 1- 1-3+
a3
Indeed, if 12 + |2 converges to zero k + co, then ||6 + || also converges to
as k + oc. According to Lemma 2 as long as the matrix C' has the largest absolute value of the eigenvalues smaller than one, the sequence (1=|12 + [²) converges initialization z = ||,9"|], = || ||. This would imply that || + |6 converges to zero k xo and hence the system equation 10 global asymptotic incremental stability.
k for any
Now all we need to show is that the spectral radius is strictly smaller than one. First, we make (I-a) 2n resulting in the aku
simple transformation obtained by change of variables 22 following dynamical system
a)BLobb aBBoo (a(L-a)BLook aBLbLa
1 a Va(1
Since the system matrix Ć' is symmetric, all we have do is derive the conditions when
(Va(l (Va(1-a)BLa) aLBLoLo
1-a
(((1-a)BLal) 1 3 aBLoLa/ (o 9).
which is equivalent to
Va a)
(vall ALL
x +
8 aBLoLb
aB a² BI Top al1 a3 To > aB a.L.T - 1 >> I Lsb
This proves that 21 + | and consequently ||6m 1|2 |18 converge to zero + co.
Now we need to show that the system is locally asymptotically stable around any solution to E a(n*). π7" b(E*). Let compute the Jacobian of the dynamical system equation I0 Denoting A = Oa(n)) ) B = Ab(E*) we have the following Jacobian
C C= c-(1-
1- (1-033B (1- + 3BA
AA
Due to the Lipschitz constraint on the function and b, we have |||]| S L, and According to Lemma this leads to:
|B] S Lb-
p(C) (p
1 1-a a) (8B
al A||A|
BAI) ))
[1(1 |11-
((1-710
aT (1- 3) (
and finally to p(C) < 1, local asymptotic stability around the fixed point, as well as global asymp- totic stability according to Lemmas and
15

---

Under review as a conference paper at ICLR 2022
C ON THE FORM OF THE VARIATIONAL POSTERIOR
Post-training conditional inference broad range of variational algorithms impose a fully factoris- able form for the approximate posterior, which usually referred as the mean-field assumption (Hoffmman et al. 2013 Blei et al., 2017; Kingma & Welling 2013 Jaakkola & Jordan 1996; Wain- wright & Jordan 2008 Knowles & Minka 2011). In this context, one would usually opt in favour of diagonal Gaussian distribution for ge base distribution. To relax this assumption and bring flex-
ibility to the variational posterior, we relied on Householder flows (Tomezak & Welling 2016) as
sparse approximation to a full covariance matrix:
μz II H.D(O)E
(13)
where the function D maps the vector a a diagonal matrix. Householder flows consist of series of n orthogonal transformation H; of a base vector e ~ [00 I). In principle, such maps can model any orthogonal matrix when n sufficiently large. The main advantage of this approximate posterior formulation is its time and memory computational cost, as it only requires inner vector-vector and vector-scalar products.
Learning from incomplete data A common usage in VAEs is to amortise the construction of para- metric variational posteriors by building a so-called inference network that maps each datapoint to its correspondent parameter configuration. In the case of incomplete data, one usually has to deal with missing values that are randomly spread in the data space and whose dimensionality can be hard to foreseen. Inputting such sparse data in neural network can be challenging. To solve this, filled every missing value with the median value of the data tensor, thereby creating workable in- put for the inference network. In turn, this inference network produced fully-factorised variational posterior which was indexed according to the latent space partition.
D MNIST DATA COMPLETION WITH IMPLICIT NORMALIZING FLOWS
We tested the VISCOS Flow algorithm on a data completion task with Implicit Normalizing Flows (INF) (Lu et al. 2021 show that our method can be extended beyond iResNet architectures. Unlike IResNet, the Lipschitz constant of a single block of INF is unbounded, making it potentially considerably more expressive. To efficiently train INE, we note that the original problem that is to be solved for one block, which reads
F(x, 2) x z + f(x)) - g(2) =0
equivalent to solving
h + f(x))
where l(2) = = ((z) is iResNet block. We know how to invert (Behrmann et al. 2019) using fixed-point iterations, and have already presented how propagate gradients through this trans- formation using Neumann series (see Section B). These two solutions showed to be considerably faster and more memory efficient than the joint use of Brodyen method and linear system solving algorithms presented in the original work.
We trained a convolutional INF network that was defined in a similar manner as the iResNet model designed for the CIFAR task in the main text, with the following notable amendments. First, to match the INF structure, the direction of one every two iResNet blocks was flipped (fi + + such that plain and inverted residual blocks alternated. ActNorm layers (Kingma & Dhariwal 2018) were interleaved
between iResNet blocks. To compute the higher order gradients required in Equation [ and since the gradient of inverted iResNet block was computed using Neumann series, we relied once more on the following identity:
VyG(v) (-( G)
where G J-1 (=)( the inverse Jacobian obtained through the truncated series 2j=o (V (x))" and the dependence of G (or æ) on y is made explicit where needed. We trained this INF network on the complete MNIST training dataset using the Adam optimiser for 20 epochs with learning rate of 10-3 We used the VISCOS Flow algorithm to complete digits
16

---

Under review as a conference paper at ICLR 2022
7
G
Figure 5: Implicit Normalizing Flow data completion results. Incomplete and completed digits are shown on the first and second row, respectively.
where 90% of the pixels were missing. The CLADE gradient estimator used to approximate the gradient of the LAD of the partial Jacobian. Results of this task are displayed in Figure We noticed that the fixed-point algorithm was slightly less effective with INF networks than with iResNet: for small proportion of the iterations during the optimisation (< 30%), our algorithm did not converge iterations (< 5) complete the process. It is
the solution and required few Newton-Krylan worth emphasising that, in all cases, the fixed-point algorithm always played major role in finding the solution of the equality constraint, reducing the distance between the imposed values yo x! and their estimate (x). gd (ÿ) by more than 90% with respect to the original values. Also, using the same partition of the observed and latent space led us to find a workable solution in every single completion case.
17


Reviews:
[{"review_id": "xDzQcaLkDtf", "reviewer": "Reviewer_urKg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is relatively well written although I believe that parts of writing in the paper can be made more precise that will help the overall quality of the paper. The problem formulation is clear and the proposed solution is interesting as well as explored in detail. \n\nHowever, the paper suffers from several weaknesses:\n\n1) Pre-trained flows: The paper does not train a normalizing flow for data that may itself have missing features/data but assumes access to a pre-trained flow and uses it for sampling when some features are missing. I find this set-up a bit unnatural since I'd expect that for application where such a conditional sampling might be required, the training data for learning any probability model will itself have missing features.\n\n2) Restricted flow architectures: Secondly, the proposed solution works for only a restricted class of flow architectures ie those that have residual flow (or iResNet) like structures. I think this is a major limitation of the method and there is no clear discussion or solution of how the proposed method can be extended for a general class of flow models.\n\n3) Comparison to ACFlow and MCFlow: The authors in the introduction and as part of the motivation highlight why these models are not well suited for the problem they consider. They mainly highlighted computational cost etc as a major roadblock to deploying such models. However, unfortunately, this claim has not been substantiated with any justification. I believe clear empirical analysis is required to include such a claim and to highlight why VISCOS flows a re a better alrenative to the already proposed models in the literature.\n\n4) Experiments: I also believe that the present experiments are insufficient to evaluate the performance of VISCOS flows. Particularly, condiering the restrictive nature of the flow architectures that VISCOS flows use, I think the present experiments do not fully reveal the advantages of different methods. Precisely, since the experimental results obtained for methods like PL-MCMC, MFlow use a different flow architecture than for the results for VISCOS flows, I believe the experimental comparisons are not fair and sufficient enough to draw any meaningful conclusions. Furthermore, the results themselves are mostly mixed and due to the lack of any error bars for these results it is hard to conclude that nay method may be better than the others. I'd recommend the authors to atleast perform experiments that uses the same flow architecture to make the comparisons fair and if possible include error bars for the results.", "summary_of_the_paper": "The paper proposes a method to sample from conditional distributions using a pre-trained flow based method for applications in problems that have missing features. The proposed method utilizes a lower bound on the log-probability of the conditional distribution (conditioned on observed features) using the Schur complement. ", "summary_of_the_review": "The paper studies an interesting problem for flows and proposes a nice solution. However, in the current form, the paper suffers from many weaknesses surrounding the experimental setup and being valid for only a restrictive class of flow models.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "p8DwjCFEdL-", "reviewer": "Reviewer_7gsd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. \nThis paper provides an interesting utilization of the Schur complement to model the dimensions of observed and unobserved features.\nI cannot understand the notation of $G^{HH}(x)$ in Eq 2 because $G$ is defined on $y$. \n\n2.\nThis paper derives the gradient of the suggested ELBO formulation. This derivation results in some theoretic suggestions, such as the corrected gradient estimators for the log-absolute determinants, i.e. CLADE.\n\n3.\nIt would have been better to show an illustrative figure on the suggested Flow structure. Particularly, the structure is decomposed into the observed and the unobserved variables; and this decomposition leads to the link to the Schur decomposition. It is confusing which Jacobian sub-matrix is related to which variable.\n\n4.\nThe in-painting task is compared from the perspective of classification accuracy, FID, and RMSE. There are some other performance standards, i.e. SSIM, PSNR. I wonder whether these metrics could be used or not.[1]\n\n5.\nCurrently, only MNIST is being used for the test. Previous works utilize more diverse datasets, such as places2, celeba, streetview, etc.[2]\n\n6.\nBenchmark baseline models are too few. if you search inpainting flow models, there will be plenty of recent flow models for the same task.\n\n\n[1] Xu, Rui, et al. \"Deep flow-guided video inpainting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n[2] Ren, Yurui, et al. \"Structureflow: Image inpainting via structure-aware appearance flow.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n\n\n", "summary_of_the_paper": "This paper presents a variant of flow, VISCOS flow, utilizing the Schur complement. This flow model is applied to the in-painting task, which estimates the unobserved feature variables given the observed features. Given this in-painting task, the inference requires the modeling on the latent variable, which results in the ELBO derivation on such latent variables as variational variables.", "summary_of_the_review": "This paper provides an interesting utilization of Schur complement in the in-painting task. However, the evaluation is very weak. \n\n////////////////////\n\nI read the rebuttal from the authors, and I maintain my evaluation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Z2EMpqg1Rvl", "reviewer": "Reviewer_puL5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n- AFAIK this is a novel approach to parameterising a variational distribution for inference within a normalising flow.\n- I think that parameterising variational distributions for normalizing flows in this way is an interesting avenue to explore, and the derivation of this method and its gradient estimators is clearly non-trivial.\n\n### Weaknesses:\n- More comparisons to (or at least discussion of) related work should be included. [1] also perform imputation using variational inference in normalizing flows (although with a relaxed likelihood), and present (at least visually) better results than this paper. [2] present a conditional VAE that can be trained with incomplete training data.\n- This approach seems to be fundamentally difficult to scale to higher-dimensional datasets (due to the cost of solving the equality constraint and estimating the gradient). Even for the moderately high-dimensional CIFAR10 and MNIST datasets experimented on, I suspect that alternative techniques (like [1,2]) work better. A much better case needs to be made for why this method should be preferred.\n\n### Minor\n- last 3 lines of first paragraph of Section 2.4 - This sentence is confusing. An ELBO of zero is not meaningful for continuous data, since the likelihood may be above or below zero. And it is not clear what is meant by \"whenever $x^O$ is independent of $x^H$\": should they be independent under the prior, or under the posterior conditioned on observations?\n- I think $\\widetilde{\\mathbf{y}^H}$ is used before being defined in Algorithm 1: it is used on the RHS of line 8 but has no assigned value until after line 8 is executed.\n\n[1] Whang, Jay, Erik Lindgren, and Alex Dimakis. \"Composing Normalizing Flows for Inverse Problems.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Ma, Chao, et al. \"Eddi: Efficient dynamic discovery of high-value information with partial vae.\" arXiv preprint arXiv:1809.11142 (2018).", "summary_of_the_paper": "This paper proposes a technique for performing data imputation using pre-trained normalizing flows. They do so by fitting a variational distribution over a subset of the 'base' variables of the normalizing flow. Along with the observations, a sample from this variational distribution is sufficient to fully specify a sample from the normalizing flow model. The authors present techniques for constructing samples in this way, and computing the derivatives required to optimise the variational distribution.", "summary_of_the_review": "While I appreciate the novel idea  and derivations, it is not clear to me that there are any use cases where this method should be preferred over the alternatives. Given the lack of a convincing case for its usefulness, I am recommending reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique to perform data imputation with normalizing flow defining a joint density between observed and unobserved variables. This is achieved by introducing a variational posterior over the missing variables which is parametrized in terms of the original model by using the Schur complement of the model's Jacobian over the hidden variables.
The idea is interesting, but the proposed setup is quite complex and the experimental results are not conclusive. The quality of the results shown can likely be matched or surpassed with much simpler techniques. The paper would substantially benefit from more detailed experiments.
Decision: Reject