AmbientFlow: Invertible generative models from incomplete, noisy measurements
Authors: Varun A. Kelkar
Rucha Deshpande
Arindam Banerjee
Mark Anastasio
Venue: Accepted by TMLR ()

Abstract: Generative models have gained popularity for their potential applications in imaging science, such as image reconstruction, posterior sampling and data sharing. Flow-based generative models are particularly attractive due to their ability to tractably provide exact density estimates along with fast, inexpensive and diverse samples. Training such models, however, requires a large, high quality dataset of objects. In  applications such as computed imaging, it is often difficult to acquire such data due to requirements such as long acquisition time or high radiation dose, while acquiring noisy or partially observed measurements of these objects is more feasible. In this work, we propose AmbientFlow, a framework for learning flow-based generative models directly from noisy and incomplete data. Using variational Bayesian methods, a novel framework for establishing flow-based generative models from noisy, incomplete data is proposed. Extensive numerical studies demonstrate the effectiveness of AmbientFlow in learning the object distribution. The utility of AmbientFlow in a downstream inference task of image reconstruction is demonstrated.

Published in Transactions on Machine Learning Research (01/2024)
AmbientFlow: Invertible generative models from incomplete, noisy measurements
Varun A. Kelkar
wwrunkellarrannllog.omm com
University of Blinois at Urbona- Caann--aaap,in,, Urbana, IL 61801 Currently Anolog Devices, Inc., Boston, MA 02110
Rucha M. Deshpande
v. r.deshpande@oute Cwnustl edu
Washington University in St. Louis, St. Louis, MO 63130
Arindam Banerjee
orindamb@rllinois.com edu
University of Ilinois at Urbone- Champaign, Urbana, IL 61801
Mark A. Anastasio
maa@tilkiois.c edu
University of Ilimnois at Urbana- Champap,, Urbana, IL 61801
Reviewed on OpenReview: https: openreview. net/ forum? id- tap YITRoa
Abstract
Generative models have gained popularity for their potential applications in imaging science, such as image reconstruction, posterior sampling and data sharing. Flow-based generative models are particularly attractive due to their ability to tractably provide exact density estimates along with fast, inexpensive and diverse samples. Training such models, however, requires large, high quality dataset of objects. it is often difficult to acquire such data due to requirements such as long acquisition time or high radiation dose, while acquiring noisy partially observed measurements of these objects is more feasible. In this work, we propose AmbientFlow, a framework for learning flow-based generative models directly from noisy and incomplete data. Using variational Bayesian methods, a novel framework for establishing flow-based generative models from noisy, incomplete data proposed. Extensive numerical studies demonstrate the ellectiveness of AmbientFlen in learning the object distribution. The utility of AmbientFlow in downstream inference task of image reconstruction is demonstrated.
applications such as computed imaging,
1 Introduction
Generative models have become prominent focus of machine learning research in recent years. Modern generative models are neural network-based models of unknown data distributions learned from a large number of samples drawn from the distribution. They ideally provide an accurate representation of the distribution of interest and enable efficient, high-quality sampling and inference. Most modern generative models are implicit, Lee. they map a sample from a simple, tractable distribution such as the standard normal, sample from the distribution of interest. Popular generative models for image data distributions include variational autoencoders (VABs), generative adversarial networks (GANs), normalizing flows and diffusion probabilistic models, among others (Kingma et al. 2019 Goodfellow et al. 2020 Kingma & Dhariwal 2018 Ho et al. 2020). Recently, many of these models have been successful in synthesizing high-quality perceptually realistic images from the underlying distribution.
Generative models have also been investigated for applications in imaging science. For example, computed imaging systems such as computed tomography (CT) or magnetic resonance imaging (MRT) rely on computa- tional reconstruction to obtain an estimate of an object from noisy or incomplete imaging measurements. Generative models have been investigated for their priors in image reconstruction in order to mitigate

---

Published in Transactions on Machine Learning Research (01/2024)
the effects of data-Incompletennss noise in the measurements. While GANs have been explored for this purpose (Bora et al.
2017 Menon et al. 2020 Kelkar & Anastasio 2021 they suffer from shortcomings
such as inadequate mode coverage (Thanh- Tung & 'Tan (020), insufficient representation capacity (Karras 2017) and misrepresentation of important domain-specific statistics (Kelkar et al, 2023a On the other hand, invertible generative models (IGMs), or normalizing flows offer exact density estimates, tractable Logllikllhoods and useful representations of individual images (Dinh et al. 2016 2014
et al. 2020; Bora et al.
Kingma& & Dhariwal 2018 Kelkar al. 2021), ) making them more reliable for downstream inference tasks in imaging science 4Asmm et al.) 2020 Jalal et al. 2021a). These models have shown potential for use in tasks
such as image reconstruction, posterior sampling, uncertainty quantification and anomaly detection (Kelaar
et al.
2021 Asim et al. 2020 Khorashadizadeh et al. 2023 Jalal et al. 2021a Zhao et al.).
Although IGMs are attractive for imaging applications, training them requires a large dataset of objects or high-quality image estimates as a proxy for the objects. Building such dataset challenging since acquiring complete set of mcasurements to uniquely spocify each object can be infcasible. Therefore, it of interest to learn an IGM of objects directly from dataset of noisy, incomplete imaging measurements. This problem previously addressed in the context of GANs via the Ambient GAN framework, which augments conventional GAN with the mcasurement operator (Bora et al., 2018 Zhou et al. 2022). It consists of generator and discriminator networks that are jointly optimized via an adversarial training strategy. Here, the generator attempts to synthesize synthetic objects that produce realistic measurements. The real and syuthetic object distributions are then compared indirectly via a discriminator that distinguishes between real measurements and simulated measurements of the synthetic objects. This is fundamentally different from the training procedure of IGMs, which directly computes and maximizes the log- pg-probabbility of the training data samples. Therefore, ideas from the AmbientGAN framework cannot be casily adapted to train an IGM of objects when only incomplete measurements is available.
The key contributions of this work are as follows. First a new framework named Ambient is developed for training IGMs using noisy, incomplete measurements. Sccond, the accuracy of the object distribution recovered via AmbientFlow is theoretically analyzed under prescribed ideal conditions using compressed sensing. Next, numerical studies are presented to demonstrate the effectiveness of the proposed method on several different datasets and measurement operators. Finally, the utility of Ambient for downstream Bayesian inference task is illustrated using case study of image reconstruction from simulated stylized MRI measurements.
The remainder of this manuscript is organized as follows. Section 2 describes the background of invertible generative models, computed imaging systems and image reconstruction from incomplete measurements. Section 3 describes the notation used and the proposed approach. [Section 4 describes the setup for the numerical studies, with the results being presented in [Section Finally, the discussion and conclusion is presented in |Section 6
2 Background
Invertible generative models. Invertible generative models (IGMs) or flow-based generative models, are class of generative models that employ invertible neural network (INN) to learn an implicit mapping from simple distribution such as an independent and identically distributed (iid) Gaussian distribution
to the data distribution of interest. The INN is bijective mapping Go : R + R" constructed using a composition of L simpler bijective functions 9: I + TR", with
( = (h(( (gi o -:-1 o . (z), 0 OciSL, 2 ER",
(1)
and x= Go(z) h (*), As consequence of bijectivity, the probability distribution function (PDF) pe of x represented by the IGM can be related to the PDF da of as:
Po det V=Go(z) = (z).
(2)
In order the establish the IGM, a dataset D = (x(0) 1 is used, where ( are assumed to be independent draws from the unknown true distribution 9x- The INN is then trained by minimizing the following negative
2

---

Published in Transactions on Machine Learning Research (01/2024)
log- regllikelhhodd objective over the training dataset D:
D
(()) - Σ log pa (x())
Σ log Oz (z(2))) - log jdet Va z - GG (x()).
(3)
Minimizing the above loss function equivalent to minimizing the Kullback Leibler (KL) divergence Dxl(Ee ==
log pe pepp)) (x) between pe and the true data distribution lx. Most invertible
generative models utilize a specialized architecture in order to guarantee invertibility of the constituent. functions 93 and tractable computation of the Jacobian loss term log det V=Go(z)]. Popular building blocks
of such an architecture affine coupling layers (Dinh et al. 2016), random permutations Dinh et al. 2014 and invertible 1 x convolutions (Kingma & Dhariwal 2018
Image reconstruction from incomplete imaging mcasurements. Many computed imaging systems can be modeled using an imaging equation described by the following linear system of equations:
g Hf n,
(4)
where E R" approximates the object to-be-imaged, H E mR known as the forward model, a linear operator that models the physics of the imaging process, n E R , ~ In models the measurement noise, and g R are the measurements of the object. Often, H is ill-conditioned or rank deficient, in which case the measurements g are not sufficient to form unique and stable estimate f of the object f, and prior knowledge about the nature of f is needed. Traditionally, one way to incorporate this prior information is to constrain the domain of H. For example, compressed sensing stipulates that if the true object is k-sparse after a full-rank linear transformation
R 1 2 n, then the object can be stably estimated for all vectors v E R" that are k-sparse in the transform domain Φ, H satisfies the restricted isometry property (RIP) (Candes et al. 2006):
Definition 2.1 (Restricted isometry property). For s e N, define the restricted isometry constant (RIC) 6, the smallest, constant. that satisfies
(1 5.) v < Hv < (1 + &
(5)
for all v such that | S 8. H is said to satisfy the restricted isometry property for all v such that 1 I k if & + d2x + das < 1 (Candes et al. 2006
3 Approach
In this section, an AmbientFlow method is proposed for obtaining an IGM of objects from a dataset of measurements. The following preliminary notation will be used in the remainder of this paper.
Notation. Let gr. 9g and In denote the unknown true object distribution to-be-rocovered, the truc measurement distribution and the known measurement noise distribution, respectively. Let D = (g(")) 1=1 be dataset of independent and identically distributed (iid) measurements drawn from 9g- Let Go : R² + R be an INN. Let pe be the distribution represented by Gg, i.e. given a latent distribution dz = (' Ln), Go(z) ~ po for z ~ 92- Also, let dhe be the distribution of synthetic measurements, i.c. for f po: Hf ~ tha- Let (f g) xx (n(g Hf) (f() denote the posterior induced by the learned object distribution represented by Go. Let E sparsifying transform). Also, let S {v I s.l. |[øv]lo S k} be the set of vectors k-sparse with respect to Φ. Since & is full-rank, throughout this work we assume without the loss of generality, that | S 1, where t is the Moore-Penrose pseudoinverse of Φ. Throughout the manuscript, we also assume that or is absolutely continuous with respect to Po, and 0g is absolutely continuous with respect to tig.
Rlxn_ 1 2 n be full-rank lincar transformation (henceforth referred to as a
Conventionally, according to the discussion below Eq. (3) (gellp) would have to be minimized in order train the IGM to estimate gr. However, in the present scenario, only samples from 9g are available. Therefore, attempt to minimize the divergence DKI(1g [bba); and show that for certain scenarios of interest, this
3

---

Published in Transactions on Machine Learning Research (01/2024)
is formally equivalent to approximately minimizing distance between Ir and pe. However, computing PKI(ggllvo) is non-trivial because a direct representation of (o(g) that could enable the computation of log (e() is not available. Fortunately, used to compute log for a given f E IR, using Eq. (2) Therefore, an additional INN, known as the posterior network ha(l ;g) IR" + R" is introduced that represents the model posterior ( g) designed to approximate gg) when jointly trained along with Gg. The posterior network ha is designed take two inputs a new latent vector { q2 = (O, In); and an auxiliary conditioning input g ~ 9g from the training dataset, to produce h2(2; g) ~ P& (f g). The following theorem establishes loss function that minimizes (KL(ggli (1vo)) using the posterior network, circumventing the need for direct access to the (g). samples of true objects from qr-
direct representation of pa is available via Gg, which can be
Theorem 3.1. Let he be such that po (f g) > over R" Minimizing DKL (1o)) is equivalent to mazimizing the objective function C(B,0) Φ) over 0, Φ, where [(0, 6) is defined
(holo(Cig)) (n (Cigg po (ho((ig))
}
[(8,0) Eg log Bogge
(6)
The proof of Theorem 3.1 is provided in the appendix. A variational lower bound of L is employed, which promoles consistency belween the modeled posterior ps(. g) and the posterior induced by the learned object distribution, pa(-1g)
LM Φ) = Eg,., logavgexp [log po g)) + log qu (g (ho(C:g) g)) - log Pa (ho g) g)ggg)]] g)]
(7)
0<3
where 2, gc, O<iS M, and logavgexposig = log
I MM M exp(x:))
Intuitively, the three terms inside logavgexp in Eq. (7) can be interpreted as follows. The first term implies that Gg is trained samples produced by the posterior network hg- The second lerm is a data-fidelity term that makes sure that he produces objects consistent with the measurement model. The third term penalizes degencrate hy for which Vcha(C: g) is ill-conditioned, for example when he produces no variation due to & and only depends on g. Note that the first and third terms are directly accessible via the INNs Go and hg- Also, for noise models commonly used in modeling computed imaging systems, such as the Gaussian noise model (Barrett & Myers 2013), In can be explicitly computed.
For sufficiently expressive parametrizations for Pe and h&; the maximum possible value of L is Eg log 4g (g), which corresponds to the scenario where the learned posteriors consistent, i.e. pø (f g) po (f g), and the learned distribution of measurements matches the true measurement distribution, i.e. tha = 9g. It can be shown that for class of forward operators, matching the measurement distribution is equivalent to matching the object distribution:
Lemma 3.1. If H is a square matrie (n = m) with full-rank, if the noise n independent of the object, and if the characteristic function of the noise X En-an exp(ev T n) has full support over I (t the square-root of -1), then 2he 9g = Pa qr-
The proof of Lemma 3.1 is provided in Appendix A Lemma 3.1 can be extended to a certain class random. rank-deficient. forward operators that nevertheless provide an invertible push-forward operator (Bora el al. 2018). However, in computed imaging, forward models are often deterministic with a fixed null-space, where it is typically not possible to design the hardware ensure the invertibility of the push-forward operator Graff & Sidky 2015 Lustig et al., 2208B.. In such a setting, it not possible to uniquely relate the learned object distribution po to the learned measurement. distríbution the without additional information about gr- Nevertheless, if the objects of interest are known to be compressible with respect to sparsifying transform Φ Pe can be constrained to the set of distributions concentrated on these compressible objects. In order to recover a distribution pe concentrated on objects that are compressible with respect to & the following optimization problem is proposed:
0, l = arg min - LL (0, subject to Eggng B--po(- 18) 06 Eprojs, (f)]li <e,
(8)
0,0
4

---

Published in Transactions on Machine Learning Research (01/2024)
where projs, (f) denotes the orthogonal projection of f E R" onto the set Sk of objects for which of is k-sparse. It can be shown that if H and f ~ Of satisly the conditions of compressed sensing and the Flo is trained sufficiently well using Eq. (8) then the error between the true and recovered object distributions can be bounded. This is formalized as follows.
Theorem 3.2. For PDF R" + R, let you denole the distribulion of projs, (x), for x q. Also, for distributions q1, 92. let (1 = infger with r being the set of all joint distributions q Ruxx - R with marginals qı 92, i.c. [()xxx22)xx; = qi (x1), I q((x1,,)) x
- X2ll2, denote the Wasserstein 1-distance,
If the following hold:
1. Wilar |l oi S (the true object distribulion is concentraled on k-sporse objects under Φ),
2. Н satisfies the RIP for objects k-sparse w.r.t. Φ with isometry constant òk,
3. the characteristic function of noise Xn(v) has full support over C" and
4. (0. 6) satisfying Po = and Pol- (g)gg = Po is feasible solution to Eq. (8) (Go and h& have sufficient capacity),
then the distribution p6 recovered via Eq. (8) is close to the true object distribution, in terms of the Wosserstein distance i.e.
Wilps ll (F) S 1+
|1 (e - E)).
(9)
VI - 6R
The proof of Theorem 3.2 is deferred to Appendix A In practice, Eq. (8) is reformulated in its Lagrangian form, and a regularization parameter ja is used to con- trol the strength of the promttoigg constraint. Also, inspired by the V-VAF framework (Higgins et al.) 2017), an additional regularization parameter À was used to control the strength of the likelihood term log ln - (4: This modifies the problem maximizing the following objective function, which was optimized using gradient- based methods.
log 9m (g (
Training data
log (Ca(gh)) g
log Po (ho (C:g))
<~N(0.1)
Posterior flow
hig (S g
Main flow
2
Figure 1: A schematic of the AmbientFlow framework
(m Φ) Ex Er
logavgexp (log (ho(Z:: g)) + A log in -hh - log (hs g)} 0<IEM
-u| -1l(hh(((:g) prois, (Th (2: (chs(4:)))l1) (10)
The proposed additive sparsifying penalty is computed by hard-threshokding the output of (ha(C:; g) to project it onto the space of k-sparse signals, and computing the é norm of the residual, with & and being treated as tunable hyperparameters. However, note that consistent with Eq. (10) the loss terms for both the INNs correspond to the original (un-thresholde)) outputs of the posterior. This ensures that invertibility is maintained, and the loss terms from both the INNs are well-defined. Empirically, we observe that the proposed é, penalty also promotes sparse deviation of the output hg from SN, which improves the quality of the images generated by AmbientFlow.
4 Numerical Studies
This section describes the numerical studics used to demonstrate the utility of AmbicntFlow for learning object distributions from noisy and incomplete imaging measurements. The studies include toy problems in two dimensions, low-dimensional problems involving a distribution of handwritten digits from the MNIST dataset, problems involving face images from the CelebA-HQ dataset as well as the problem of recovering the object distribution from stylized magnetic resonance imaging measurements. A case study that demonstrates the utility of Ambient in the downstream tasks of image reconstruction and posterior sampling also
5

---

Published in Transactions on Machine Learning Research (01/2024)
described. Additional numerical studies, including evaluation of the posterior network, additional ablation studies, and face image inpainting case study are included in Appendix B
Datasets and imaging models.
1) Toy problems: First, a two-dimensional object distribution Or R² + R was considered, which was created as a sum of eight Gaussian distribu- tions Nic,, oī12), S < 8, with centers c, cated at the vertices of a regular octagon centered the origin, and standard deviation or 0.15, shown in Fig. 2/. The forward operator was the identity operator, and the noise n was dis- tributed as a zero-mean Gaussian with covariance 22 with on = 0.45. The distribution of the measurements g= shown in Fig. 2p. A training dataset of size 5 x 107 was used.
(b)
(c)
(d)
Figure 2: (a) True distribution gr, (b) distribution 4g of measurements, (c) distribution learned by a flow model trained on true objects, and (d) distribution learned by Ambient trained measurements.
Next, problem of recovering the distribution of MNIST digits
from noisy and/or blurred
images of MNIST digits was considered (LeCun iet al. 1998). For this problem, three different forward models were considered, namely the identity operator, and two Gaussian blurring operators Halurl and Hatur2 with root-mean- squared (RMS) width values σ = 1.5 and 3.0 pixels. The measurement noise was distributed as n n N Im), with on = 0.3.
True objects
g
H
g Hawe +
7
2 4
/
Measurements
O 4
/
0 4
0 4 1
/
4 5
9 2
3
2
1
3
9
4
3
5
4 3
4 106
3
7
3
FID
203.8
FID 269.3
FID 301.4
Flow trained on true objects
Ambient low trained on measurements
4
3 8
9
5
0
3
9
6
0
6
1X
2) Face image study: For the face image study, images of size n 64x64x3 from the CelebA-HQ dalaset were considered Karras et al. 2017). Two forward models were considered, namely the identity operator and the blurring operator with RMS width Th 1.5, and the measurement noise was distributed as n (10 Im)), on = 0.2. A discrete gradient operator was used as the sparsifying transform 4.
5 8 9 4 7 6
6
2
6
9
2
6
9
0
6
5
9
3 4
0
3
3
1
6 3
(
6
2
4
6
2
7
3
FID = 146
FID = 146.9
FID
= 187.4
FID
= 244.1
Figure 3: Samples from the flow model trained on true objects and AmbientFlow trained measurements shown alongside samples of true objects and measurements for the MNIST dataset.
3) Stylized MRI study; In this study, the problem of recovering the distribution of objects from simulated, stylized MRI measurements was considered. Te-weihhted brain images of size n 128 128 from the FastMRI initiative database were considered (Zbontar al. 2018) as samples from the object distribution. Stylized MRT measurements with undersampling ratio n/m 1 (fully sampled) and n/m the fast Fourier transform (FFT). Complex valued iid Gaussian measurement noise with standard deviation 0.1 times the total range of ground truth gray values was considered, A discrete gradient operator was used as the sparsifying transform +
simulated using
Network architecture and training. The architecture of the main flow model Go was adapted from the Glow architecture (Kingma Dhariwal 2018). The posterior network was adapted from the conditional INN architecture proposed by Ardizzone, et. al (Ardizzone al. 2222I). AmbientFlow was trained using Py Torch using NVIDIA Quadro RTX 8000 GPU. All hyperparamelers for the main INN were fixed based PyTorch implementation of the Glow architecture (Seonghyeon), except the mumber of blocks, which was set to scale logarithmically by the image dimension.
'Qur Torch implementation of AmbientFlow can found http:///httbbcccccccmmmmmaaaggao

---

Published in Transactions on Machine Learning Research (01/2024)
Baselines and evaluation metrics. For each dataset, an INN was trained on the ground-truth objects. The architecture and hyperparameters used for this INN for a particular dataset were edentical to the ones used for main flow Gg within the AmbientFlow framework trained on that dataset. Besides, for each forward model for the face image and stylized MRI study, a non-data-driven image restoration restoration/rcconstneene rrcconsrructoinn algorithm was used to generate a dataset of individual estimates of the object. For H = Im with and 3D filtering (BM3D) denoising algorithm was used to perform
n (O Im)) the block-matching image restoration (Dabov et al. 2007). For the blurring operator with Gaussian noise, Wiener deconvolution was used for this purpose (Blahut) 2004). For the stylized MRI study, penalized least-squares with TV regularization (PLS-TV) algorithm was used for image reconstruction (Beck & Teboulle, 2009). The regularization parameter for the image reconstruction method was tuned to give the lowest RMS error (RMSE) for every individual reconstructed image. Although this method of luning the parameters is not feasible in real systems, it gives the best set of reconstructed images in terms of the RMSE, thus providing a strong basclinc.
The Frechet Inception distance (FID) score, computed using the Clean-FID package (Parmar et al. 2022
was used to compare a dataset of 5,000 true objects with an equivalent number of images synthesized using (1) the INN trained on the true objects and (2) the AmbientFlow trained on the measurements, and (3) images individually reconstructed from their corresponding measurements. Additionally, for the stylized MRI study, radiomic features meaningful to medical imaging were computed on the true objects, generated objects, and reconstructed images [Van Griethuysen et al. 2017).
Case study. Next, the utility of the AmbientFlow in downstream Bayesian inference lask was examined. For this purpose, a case study of image reconstruction from incomplete measurements was considered, where the AmbientFlow was used as a prior. Importantly, we consider the scenario where the forward model used for simulating the measurements is different from the one associated with the AmbientFlow training. Preliminaries of generative priors for image two image reconstruction tasks are considered (1) approximate maximum a posteriori (MAP) estimation, i.c. approximating the mode of the posterior p. g), and (2) approximate sampling from the posterior po(: [g).
ruction are discussed in (Dimakis el al. 2022). In this study, the following
For both the tasks, an AmbientF Fow trained on the fully sampled, noisy simulated MRI measurements, as well as the flow trained on the true objects were considered. For the first task, the compressed sensing using generative models (CSGM) formalism was used to obtained approximate MAP estimates from measurements, for held-out for a test dataset of size 45 Asim al., 20200
MAP Go(2MAr), with MAAP = arg min lg Hz) A//2112
(11)
z
For the second task, approximate posterior sampling was performed with the flow models as priors using annealed Langevin dynamics (ALD) iterations proposed in Jalal, el al. (Jalal el al., 2021a). For each true object, the minimum mean-squared error (MMSE) estimate fMMSE and the pixelwise standard deviation map ô were computed cmpirically using 40 samples obtained via ALD iterations. These image estimates were compared with reconstructed images obtained using the PLS-TV method. The regularization parameters for each image reconstruction method were tuned to achieve the best RMSE on single validation image, and then kept constant for the entire test dataset.
5 Results
Figure 2 shows the truc object distribution, the distribution learned by flow model trained on objects, the measurement distribution, and the object distribution recovered by Ambient trained using the measurements. It can be seen that AmbientFlow successful in generating nearly noiseless samples that belong to one of the eight Gaussian blobs, although a small number of generated samples lie in the connecting region between the blobs.
7

---

Published in Transactions on Machine Learning Research (01/2024)
True objects
Flow trained on true objects (FID 57.8)
Noisy images with o, 0.2 (FID-256.4)
Blurred noisy img. with o 1.5.0, 0.2 (FID=310)
BM3D recons from noisy images (FID=139.1)
Wiener deconv. from noisy, blurred img. (FID=287.9
AmbientFlow trained on noisy img. (FID=135.5) AmbientFlow trained on noisy blurred img. (FID=227.8)
Figure 4: True objects, noisy/ poisy/blune blurred image measurements, reconstructed images and images synthesized by the flow model trained on the true objects, as well as the Ambient trained on the measurements for the CelebA-HQ datasct.
For the MNIST dataset, Fig. 3 shows samples from the flow model trained on true objects and Ambient trained on measurements alongside samples of true objects and measurements, for different measurement models. It can be seen that when the degradation process is a simple noise addition, the AmbicntFlow produces samples that visually appear denoised. When the degradation process consists of blurring and noise addition, the AmbientFlow produces images that do not contain blur or noise, although the visual image quality degrades as the blur increases. The visual findings are corroborated with quantitative results in terms of the FID score, shown in Fig. 3
Figure shows the results of the face image study for the two different measurement processes considered (1) additive noise, and (2) Gaussian blur followed by additive noise. It can be seen that both visually and in terms of the FTD score, the quality of images generated by the Ambient Flow models was second only to the flow trained directly on the true objects, for both the forward models considered. The images synthesized by the AmbientFlow models had better visual quality and better fidelity in distribution with the true objects with respect to FID than the ones produced by individually performing image restoration using BM3D and Wiener deconvolution for the two forward models respectively. This suggests that an AmbientFlow trained directly on the measurements would give a better approximation to the object distribution in terms of the considered metrics as compared to a regular flow model trained on the image datasets individually reconstructed via BM3D/Want /Winner deconvolution.
The results of the stylized MRI study are shown in Fig. The visual and FID-based quality of images synthesized by the imbientFlow models was inferior only to the images synthesized by the flow trained
8

---

Published in Transactions on Machine Learning Research (01/2024)
True objects
Flow trained on true objects (FID 45.6)
IFFT recons from fully sampled meas. (FID 94.2)
IFFT recons from 4x undersampled meas (FID=162.0) 162.0)
PLS-TV recons from fully sampled meas. (FID=94.4) PLS-TV recons from undersamp. meas (FID=181.7)
AmbientFlow trained fully sampled meas (FID=72.4) AmbientFlow trained 4x undersamp. (FID=88.
Figure 5: True objects, IFFT-based image estimates, PLS-TV TV based image estimates and images synthesized by the flow model trained on the true objects, well as the Ambient trained on the measurements for the stylized MRT study.
Table 1: The mean standard deviation of the RMSE and SSTM values computed over the following test. image datasets (1) the images reconstructed using the PLS-TV method, (2) the MAP and MMSE estimates using the flow prior trained on true objects, and (3) the MAP and MMSE estimates using the AmbientFlon prior trained fully sampled noisy measurements.
Method
RMSE
SSIM
PLS-TV
0.038 + 0.007
0.806
0.019
MAP Estimate
0.025 + 0.005
0.922 + 0.013
Flow prior trained on true objects
MMSE Estimate
0.022 + 0.003
0.940 + 0.006
MAP Estimate
0.025 + 0.004
0.925 + 0.012
AmbientFlen prior
MMSE Estimate
0.022 + 0.004
0.936 + 0.008

---

Published in Transactions on Machine Learning Research (01/2024)
directly objects, and was superior to the images reconstructed individually from the measurements using the PLS-TV method. Since the underlying Inception network used to compute the FID score is not directly related to medical images, additional evaluation was performed in terms of radiomic features relevant to medical image assessments.
PLS-TV from noisy
(a) True objects
(c) n/m
n/ m
Figure G plots the empirical PDF over the first two
principal components of the radiomic features ex- tracted from each of the MR image sets shown in Fig. 5) except the IFFT image estimates. It can be seen that there is a significant disparity between the principal radiomic feature PDFs of the true objects and the images reconstructed individually using PLS- TV. On the other hand, the Ambient AmbientFlow-geene Flow-geneatteed images have a radiomic feature distribution closer to the true objects for both the fully sampled and 4- fold undersampled cases. This implics that, training an Ambient Flow on the noisy/ /incomplete measure- ments yielded an estimate of the object distribution that was more accurate in terms of relevant radiomic features, than the one defined by images individu- ally reconstructed from the measurements using the PLS-TV method.
(b) Flow trained objects
AmbientFlow trained on noisy measurements
(d) n/m
(f) n/m
Figure 6: Empirical PDF over the first two principal components of the radiomic features extracted from MRI images. For each plot, the bold contour encloses the region containing 80% the probability mass. For (b-f), the dotted contour encloses the region containing 80% of the probability mass of the true objects.
Next, the utility of the AmbientFlow for Bayesian inference demonstratod with the help of an image reconstruction study. Figure 7 shows true object alongside images recon- structed from stylized 4-fold undersampled MR measurements simulated from the object, using the reconstruction methods described in [Section_ 4 Recall that for both the flow-based priors shown, the MAP estimate was obtained using the CSGM framework (Bora et al., 2017), and the MMSE estimate and the pixelwise standard deviation maps
True object (f) Zero-filled (H g)
PLS-TV
Flow prior trained on true objects
MAP Estimate
MMSE Estimate
Std. Deviation
computed empiri-
All
cally from samples from the posterior (f (f] g) obtained using annealed Langevin dynamics (Jalal et al. 2021a) Visually, it
0
can be seen that the images reconstructed using the Ambient- Flow prior were comparable to the images reconstructed using the flow prior trained on the true objects, and better than the image reconstructed using the PLS-TV method. [abll 1 shows the rool-nean--quared error (RMSE) and structural similarity (SSIM) (Wang et al. 2004] of the reconstructed images with respect to the true object, averaged over datasct of 45 test images. It can be seen that in terms of RMSE and SSIM, both the MAP and the MMSE image estimates obtained using the AmbientFlow prior are comparable to those obtained using the flow prior trained on true objects, despite the AmbientFlow being trained only using noisy stylized MR measurements.
A.11
AmbientFlow prior
Figure 7: Image estimates and pixelwise standard deviation maps from the image reconstruction case study.
6 Discussion and conclusion
An estimate of the distribution of objects known to be important for applications imaging seience. This is because an unconditional generative model of objects can potentially be used in image reconstruction tasks without the need for paired data of images and measurements, and in way that accommodates
10

---

Published in Transactions on Machine Learning Research (01/2024)
wide variety of relevant forward models (Bora et al. 2017 Asim et al. 2020 Additionally, unconditional generative models can be used to approximate ideal Bayesian classifiers (Zhou et al. 2023), or for anomaly detection (Zhao et al.), or image manipulation (Torres & Brefeld). However, obtaining such an estimate in terms of a generative model that is useful for downstream tasks has remained challenging, especially when only noisy and incomplete measurements of the objects are available. In this work, a framework for learning flow-based generative models of objects from noisy/incompll presented numerical studies show that the proposed AmbientFlow framework was able to mitigate the effects of data incompleteness and noise present in the measurements in order to build accurate estimate of the object distribution in terms of the considered evaluation metrics, and generate visually appealing images. In terms of perceptual measures such as the FID score as well as domain specific radiomic features, the images synt henizes by AmbientFlow maintained higher distributional fidelity with the true objects than the images individually reconstructed from the measurements. Furthermore, when the AmbientFlow trained on noisy mcasurements was employed as a prior in an image reconstruction task, the image estimates obtained were as accurate in termss of RMSE and SSIM as those obtained when a flow model trained directly on the true objects was employed.
mcasurements was developed. The
Some of the above observations also apply to the AmbicntGAN framework developed by Bora, et al. (Bora et al. 2018 Although the current state-of state-of-the-art the- GANs may lead to images with better perceptual quality than IGMs, GANs trained on medical images have been shown to misrepresent medically relevant statisties despite producing visually appealing images (Kelkar et al.) 2023a). Also, an important drawback of GANs with in the context of imaging science its unreliability in downstream Bayesian inference tasks. For example, GAN-constrained image reconstruction methods are known to be prone to hallucinations caused by dataset. bias, distribution shifts and representation error, whereas IGMs used for the same purpose have been shown to be comparatively robust (Bhadra et al. 2021 Asim el al. 2020 Menon et al. 20200 Kelkar el al. 2021 Jalal et al. 2021b a Zhao et al. (022). These drawbacks of GANs are in part due to the insufficient representation capacity and inability to access the log-probability, both of which are applicable for GANs learned in the ambient setting as well. IGMs, on the other hand, due to their ability to accurately represent images and compute fast, exact density estimates, are more suitable for some rownstreea inference tasks such as image reconstruction, posterior sampling and uncertainty quantification as compared to GANs. Further similarities and differences between AmbientGAN and AmbientFlow as follows. In some sccnarios, AmbientGAN can be more flexible since it can easily accommodate arbitrary differentiable random forward models, whereas AmbientF Flow is limited by the capabilities of the posterior network. However, in the presence of a fixed null space, some studies have shown that Ambient performance can degrade and the images produced by it can display aliasing artifacts characteristic of the measurement operator [Zhou et al. 2022). To the best of our knowledge, incorporating additional priors into the AmbientGAN loss function has not been rigorously studied. Another key difference between the two approaches is that unlike AmbientGAN, AmbientFlow also provides a posterior network which can be useful by itself for certain image reconstruction tasks, shown in Appendix B
The AmbientFlow framework bears some similarity with variational autocncoders (VAEs). In both cases, a variational lower bound of the evidence of the data is minimized. The object distribution in AmbientFlow is analogous to the latent variable distribution in VAEs. However, in VAEs, the latent distribution is typically simple and non-unique, and its desirable properties include disentanglement, tractable sampling, and accurate representation of the data via a trained decoder. However, in the AmbientFlow framework, the object distríbution is a complex high dimensional distribution that is of primary interest and needs to be recovered accurately possible. It is related to the measured data via physical measurement process, and may have known structure such as transform compressibility. Therefore, the aims and objectives of the two frameworks that guide their design are radically different.
The posterior network in our work also has interesting connections to deep probabilistic imaging (DPI) (Sun & Bouman 2021), which also uses an invertible network to model the posterior, and trains it in a way that is rained by prior. However, unlike DPI, the posterior model in our work is an end-to-end conditional gencrative model that, when trained, can dircctly produce posterior samples by taking in the measurement vector as one of the inputs.
11

---

Published in Transactions on Machine Learning Research (01/2024)
The presented framework can be adopted to other generative models that utilize log-Hikelihodd-base training objective, such as denoising diffusion probabilistie models (DDPMs) which enable high-quality Bayesian inference in imaging (Song et al. 2021). Recent examples of learning diffusson models from uoisy/incompp data include (Aali et ol. 2023) and (Daras et al. 2023). However, the approach by Aali et al. applics only to measurements with while Gaussian noise and an identity forward operator. In contrast, the approach by Daras, et al. can incorporate a wide class of forward operators, but does not account for measurement noise. In the future, an extension of the Ambient. framework to diffusion models could account for both, forward operators with a mull space, as well measurement noise.
A key limitation of the proposed framework is that its performance depends heavily on the design of the posterior network. The posterior network architecture currently employed is inspired by Ardizzone, et. al [Ardizzone et al. 2021). It displays favourable inductive biases for images due to several design choices such as wavelet-based downsampling layers, and Laplacian pyramid feature extractor for the conditioning input. However, this architecture may not be able to properly model the posterior when H depends on a random parameter, such random view angle relevant. for eryo-electron microscopy (Zhong el al.) 2021]. In theory, it is straightforward to modify the loss function so that in addition to g, random forward model И ~ 9H is also conditioning input to the posterior network. However, in practice, designing a posterior network architecture that can successfully account for the random forward operator is nontrivial, and could be an interesting avenue for future work. Also, although this work involves preliminary assessments of AmbientFlow using the FID score and radiomic features, a proper evaluation of generative models for imaging applications is still an open problem (Sankaranarayanan et al. 2023). Thorough evaluation of such models would involve assessing whether they reproduce image statistics that are relevant to wide variety of downstream tasks [Kelarr et al. 2023a b
Acknowledgements
This work was supported by the National Institute of Health (NIH) award EB034249. We acknowledge Carl Edwards for useful discussions.
References
Asad Aali, Marius Arvinte, Sidharth Kumar, and Jonathan I Tamir. Solving inverse problems with score-based generative priors learned from noisy data, orXiv preprint or exXi:2205.001666 2023.
Lynton Ardizzone, Jakob Kruse, Carsten Lüth, Niels Bracher, Carsten Rother, and Ulirich Köthe. Conditional invertible neural networks for diverse image-to-image translation. In Pattern Recognition: 42nd DAGM German Conference, DAGM GCPR 2020, Tübingen, Germany, September 28 October 1, 2020, Proceedings 42, pp. 373 387. Springer, 2021.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. International Conference on Machine Learning, pp. 399 409. PMLR, 2020.
Harrison H Barrett and Kyle J Myers. Foundations of image science. John Wiley & Sons, 2013.
Amir Beck and Mare Teboulle. A fast iterative shrinkagg- algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-200 202, 2009.
Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. Navier-stokes, fluid dynamics, and image and video inpainting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pottern Recogmition. CVPR 2001, volume 1, pp. 1-1. IEEE, 2001.
Sayantan Bhadra, Varun A Kelkar, Frank J Brooks, and Mark A Anastasio. On ballucinations in tomographic image reconstruction. IEEE transactions on medical imaging, 40(11):3249-32)) -2260, 2021.
Richard E Blahut. Theory of remote image formation. Cambridge University Press, 2004.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Ashish Bora, Eric Price, and Alexandros G Dimakis. Ambientgan: Generative models from lossy meusurements. International conference on learving representations, 2018.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. or preprint arXi::1509.000 00519, 2015.
Emmanuel Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8):1207 1223, 2006.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d wranform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080-2099) 2007.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alexandros Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. arXiv preprint arXie:2905, 19256, 2023.
Alexandros G Dimakis, Ashish Bora, Dave Van Veen, Ajil Jalal, Sriram Vishwanath, and Eric Price. Deep generative models and inverse problems. Mathematical Aspects of Deep Learning, 400, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 1 2014.
Laurent Dinh, Jascha focl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1oX::1605.08803, 2016.
Ian Goodfellow, Jean Ponget-Ahadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Christian G Graff and Emil Y Sidky. Compressive sensing in medical imaging. Applied optics, 54(8):C23 C44, 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Terchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffuusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-685) 2020.
Ajil Jalal, Sushrut Karnalkar, Alexandros G Dimakis, and Eric Price. Instance-optimmal compressed sensing via posterior sampling, arXiv preprind ar arXiv:2106.11 11438, 2021a.
Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmann, Alex Dimakis, and Eric Price. Fairness for image generation with uncertain sensitive attributes. In International Conference on Machine Learning, pp. 4721 4732. PMLR, 2021b.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. Xiv preprint arxiv: 1019611996,, 2017.
Tero Karras, Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVE conference computer vision and pattern recognation, pp. 8110 8119, 2020.
Varun A Kelkar and Mark Anastasio. Prior image-constrained reconstruction using style-based generative models. In International Conference on Machine Learning, pp. 5367-556 5377. PMLR, 2021.
Varun Kelkar, Sayantan Bhadra, and Mark A. Anastasio. Compressible latent-space invertible networks for generative model-constraiied image reconstruction. IEEE transactions on computational imaging,
2021.
Varun A Kelkar, Dimitrios S Gotsis, Frank J Brooks, KC Prabhat, Kyle J Myers, Rongping Zeng, and Mark A Anastasio. Assessing the ability of generative adversarial networks to learn canonical medical image statistics. JEEE trunsactions on medical imaging, 2023a.
Varun A Kelkar, Dimitrios S Gotsis, Rucha Deshpande, Frank J Brooks, KC Prabhat, Kyle J Myers, Rongping Zeng, and Mark A Auastasio. Evaluating generative stochastic image models using lask- image quality measures. In Medical Imaging 2023: Image Perceplion, Observer Performance, and Technolooy Assessment, volume 12467, pp. 304-310. SPIF, 2023b.
13

---

Published in Transactions on Machine Learning Research (01/2024)
AmirEhsan Khorashadizadeh, Konik Kothari, Leonardo Salsi, Ali Aghababaei Harandi, Maarlen de Hoop, and Ivan Dokmanić, Conditional injective flows for bayesian imaging. IEEE Transactions on Computational Imaging, 9: 224-237, 2023.
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends& in Machine Learning, 3774)33073392, 2019.
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances neural information processing systems, 31, 2018.
Bhagwandas Pannalal Lathi and Roger A Green. Linear systems and signals, volume 2. Oxford University Press New York, 2005.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document. recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Michael Lustig, David L Donoho, Juan M Santos, and John M Pauly, Compressed sensing mri. IEEE signal processing magazine, 25(2):72-82, 2008.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/eof conference on computer vision and pattern recognation, pp. 2437 2445, 2020.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu, On aliased resizing and surprising subtleties gan evaluation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 11410-11420, 2022.
Swami Sankaranarayanan, Thomas Hartvigsen, Camille Bilodeau, Ryutaro Tanno, Cheng Zhang. Florian Tramer, and Phillip Isola. Challenges in deployable generative AI, ICML 2023 workshop. httpe:///eplooiiggeenrtttaaa
github.1o//ndee. 2023.
Kim Seonghyeon. Pylorch Implementation of Glow: Generative Flow with Invertible 1x1 Convolutions. URL
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems medical imaging with score-based generative models. arxi preprint arXic:2111.08000 2021.
He Sun and Katherine L Bouman. Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 2628 2637, 2021.
Hoang T'anh- Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans, In 2020 international joint conference on neural networks (ijenn), pp. 1-10. 1EEE, 2020.
Ricardo da S Torres and Ulf Brefeld. Principled interpolation in normalizing flows.
Joost JM Van Griethuysen, Andriy Fedorov, Chintan Parmar, Ahmed Hosny, Nicole Aucoin, Vivek Narayan, Regina GH Beets- Tan, Jean- Christophe Robin, Steve Pieper, and Hugo JWL Aerts. Computational radiomics system to decode the radiographic phenotype. Cancer research, :704 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simonoolli. Image quality assessment: from error visibility to structural similarity. JEFE transactions image processing, 13(4):600-61) 662, 2004.
Jure Zbontar, Florian Knoll, Amuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, Mare Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal, Adriana Romero, Michael Rabbat, Pascal Vincent, Nafissa Yakubova, James Pinkerton, Duo Wang, Frich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: An open dataset and benchmarks for accelerated MRI, 2018.
Xuebin Zhao, Andrew Curtis, and Xin Zhang. Bayesian seismic tomography using normalizing flows. Geophysical Journal International, 228(1):213-22)) 2399, 2022.
Yuzhong Zhao, Qiaoqiao Ding, and Xisoqun Zhang, Ac-flow: Autoencoders with normalizing flows for medical images anomaly detection. In The Fleventh International Conference on Learning Representations.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph H Davis. Cryodrgu: reconstruction of heterogeneous cryo-em structures using neural networks. Nature methods, 18(2):176-15) 185, 2021.
Weimin Zhou, Sayantan Bhadra, Frank J Brooks, Hua Li, and Mark A Anastasio. Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks. Journal of Medical Imaging, 911:005508-0115033 2022.
Weimin Zhou, Umberto Villa, and Mark A Anastasio. Ideal observer computation by use of markov-chain monte carlo with generative adversarial networks. Xre preprint urXiv:2304.00433 2023.
15

---

Published in Transactions on Machine Learning Research (01/2024)
A Theoretical analysis for Section 3
First, the notation defined in Section 3 is rehashed here for convenience.
Notation. Let gr. 9g and denote the unknown true object distribution to-be-recovered, the true measurement distribution and the known measurement noise distribution, respectively. Let D = {g {g Ji=1 be
a dataset of independent and identically distributed (iid) measurements drawn from 9g. Let Ge R" + R" be an INN. Tet po be the distribution represented by Gg, i.e. given a latent distribution le = N/O, I,), Gæ(z) Pe for qa. Also, let the be the distribution of fake measurements, i.e. for f ~~ pe, Hf Hf) po (f) denote the posterior induced by the learned object distribution represented sparsifying
thg.
Let Po (f) g) Qn (g by Gg. Let 4 R I 2 n be full-rank linear transformation (henceforth referred to transform). Also, let S = {v E IR" s.t. ||ōv|lo s k} be the set of vectors k-sparse with respect to $. Since & is full-rank, throughout this chapter we assume without the loss of generality, that || 12 S 1, where Ø is the Moore-Penrost pseudoinverse of Φ. Throughout the manuscript, we also assume that or is absolutely continuous with respect to pa, and 9g is absolutely continuous with respect to thg.
A.1 Proof of [Theorem 3.1
Let he be such that (f g) > O over R Minimizing DKI(gellvo) is equivalent to mazimizing the objective function L(0, Φ) over 8, Φ, where L(B, 6) is defined as
Theorem 3.1
po (ho((C())) qn(g Hha g)) po (ho((ig) ())
((0,0) = Egguue log Benge
(12)
Proof. From the definition of KT, divergence, we have
log 2g (g)
DKL (gglloo))) Eggug
(13)
v (g)
Egnag log qg (g) Eggag log she (g).
(14)
Now, the (g) can be written as
(o(g) 9gl (g)
(15)
/ (f(gg)
(n(g - Hf) (f(
df
(16)
(f(g)
9n Hlf) (f(
Er-po(-ls
(17)
(f(g)
Therefore,
po 9n - -Hf)
(f g)) }
E Eg log (g)( = Ex-ax
log Be-pa(lg)
(18)
Po (Go(())))
On (m (ho((:g)
Egngx
log BE
(19)
Po
(ho (h
g) g)
= C(0,
(20)
Therefore,
DKL (gellvb) Egguge log Ez (g) L(0,)).
(21)
Since Egzng log Uz (g) is constant, minimizing (kl(ggllvv)) is equivalent to maximizing L(), Φ).
16

---

Published in Transactions on Machine Learning Research (01/2024)
Remark. As described in Section 3
variational lower bound of C is optimized in this work:
LM( ④) - Eg,,, logavgexp 0C1.
+ log yn (g Hha(C; B)) - log (ho(C g)]
(22)
where ~~
qc, qc, 0 O<is M, and (x)) (2;) log M En exp(xx))
For sufficiently expressive parametrizations for Pe and he. the maximum possible value of LM is Bg log 9g (g),
which corresponds to the scenario where the learned posteriors are consistent, i.e. Pe (f g) = Po ( g), and the learned distribution of measurements matches the true measurement distribution, i.e. Vya Og. This can be shown as lollows.
For M 1, LM (0, 6) reduces to the evidence lower bound (ELBO):
(f, = E Eg <~gc
+ log On (g Hha((:g))
-lgg (hg((ig) g)
(23)
= E Eg log (o(g) DKL(P (- (po (o((lg))
(24)
Similar to importance--eiighee autoencoders (IWAE), (Brdda et al. 2015 it can shown that
CELBO o) S (mr(0, ox log (g) S g log (g(g),
(25)
with equality occurring when the 9g and ps(-1g) po(-)g). Thus the maximum value that can be achieved by L (0, Φ) is Eg-ag log (g))
Lemma 3.1, H is a square matrie (n= m) with full-rank, if the noise n is independent of the object, and if the characleristic function of the noise Xn = En exp(iv' n) has full support over Thm (t the square-root of -1), then the 9g + pø dr.
Proof. This proof as been adapted from the AmbientGAN work (Bora al. 2018 Let y Hf represent the noiscless measurements. Therefore,
g = y +
(26)
* On.
(27)
where represents a convolution (in the sense of linear systems theory) Lathi & Green 2005). Therefore,
Xx(v) (v)x(v()))
(28)
Since Xn has full support over R", Xx uniquely determines Xy. Therefore, gg uniquely determines Qy-
Also, since H is bijective, dy uniquely determines dr- Therefore, the = 9g - Pe = Gr-
A.2 Proof of Theorem 3.2
In order to prove Theorem 3.2 we first establish essential notation and intermediate results needed. Specifically, in Lemma A.1 we derive an expression for the wasserstein distance between a distribution of a random variable, and the distribution of projection onto set. We then procced to prove Thoorem 3.2
Notation. For a closed set C R", let projs(f) denote the orthogonal projection of f onto S, defined
projs(f) = min || f'es
fl]2
(29)
For a PDF q: R. R, let σ denote the distribution of (x)js(x), for x q. Also, for distributions qı, Q2, let
Wi(gill92) = inf (e((a))))
E ((xxx)))~ Xxll2
(30)
17

---

Published in Transactions on Machine Learning Research (01/2024)
denote the Wasserstein 1-distance, with [(q1, 92) being the set of all joint distributions 7 Rax* - R with marginals Q1, 92, L.e.
/ Xxxxx qi(x)), / Xx),, 92(xx2)
(31)
Lemma A.1. Let E IR be rundom vector with distribution q- Then, with the above notalion,
Wilalla") Ex-g |x prois
(32)
Proof. Let 7o _uxx + R be degenerate joint distribution given by 70(x, w) = ((x))(( px)js(x)); x, X,WER.
(33)
where, 6(w) denotes the Dirac delta. Therefore, by definition of the Wasscrstein distance,
Wi(alla S B(xxw)) ||x ||x-
(34)
/ ( (x-oosxs(()) ||x - w/1p dxdw,
(35)
/ ((x) ||x projs (x) dx.
(36)
Ex ||x projs (x)112-
(37)
On the other hand, by definition of orthogonal projection,
||x - pr)js(x)]12 s ||x wll2, V w E supp(q
(38)
Therefore,
E Exsg ||x - px)js(x)]l S ((x,w) 1 E I(g.9") (E.
(39)
E Exng ||x - --roosss)sll S inf E(x ||x wll2
(40)
7Er(100))
=Wi(allg))
(41)
Equations
37
and 41 imply
Wi(all Exnu || projs(x))l2
(42)
With all the tools in place, we now proceed to prove Theorem 3.2 Consider the optimization problem stated
in Eq. (8)
0,
mig (0r subject to Egog Ee-ps(1 *prois, (f) <
0,6
Theorem 3.2.
If the following hold:
1. Wilge ll qe") S é' (the true object distribution is concentrated on k-sparse objects under b),
2. H satisfies the RIP for objects k-sparse w.r.t. Φ, with isometry constant ò,,
the characteristic, function of noise Xn(v) has full support over C" and
4. (0,0) satisfying po q and pol. g) = po(l)))) is fensible solution to Eg. (8) (Go and h& have sufficient capacity),
then the distrubution Pş recovered via Eq. (8) is close to the true object distribution, in terms of the Wasserstein distance i.e.
Wi Wi |l gr) S (1+
1
VI Ox 12
(c (c+ c)-
(43)
18

---

Published in Transactions on Machine Learning Research (01/2024)
The intuitive idea behind the proof of this theorem is as follows. Compressed sensing stipulates that under precribed conditions, the forward operator is injective a set of sparse vectors. Thus, if an object distribution is sparse, then the distribution of its measurements should be uniquely linked to it. If the object distribution oe is compressible, and if is ensured that a compressible distribution both dr and Pg will be concentrated on the sparse vectors, and will associated with the same measurement distribution lg- Since the sparse vectors are uniquely determined by the measurements, Pa and Of must. themselves be close.
recovered via Eq. (8) then
Proof. Since (0, 6) satisfying pe = qr, and po(-lg) - po(-lg) is a feasible solution Eq. (8) the maximum value of LM under Eq. (8) is Eg log (g). Therefore, according to Eq. (25) for the estimated and a C(0. 3) Eg log (g)
V8 4g and p&(+1g) pg(-1g).
(44)
Let f1, fo E IR.. Therefore, by triangle inequality,
ff f2l12 = [f f
+ +
f2 + f
-/ 1/
(45)
I [ 1l + ||E5 fzll2 + | 1 11
(46)
where g is shorthand for projs, (f) for f E RR.
fi,fz can be represented in terms of the spasifying transform Φ. Let c, of, and cf = #ff for i 1,2.
Therefore,
f 2212 < f ff + || 112 ||o2 l6 + || 11:
(47)
where xx is the Mooro-Penrose pscudoinverse of Φ. Also, by definition, recall that of and cf have at most k non-zero values.
Now, let y. Hff, for i 1,2. Therefore,
||yī vēll2 = llyī - yi + Y2 - y² -1 -22l22
(48)
I ly yfll2 + ||y2 y
2 - |ly, -y2llg
(49)
< ||ll f f 111 |H
112 c2 c l1z lly: y2ll2.
(50)
Now, the restricted isometry property (RIP) on H defined in Definition 2.1 implies
112
I /1
11yf y
(51)
6%
Therefore, Equations
47
50
and 51 give
f fa
f
11112 + +|00
||o - -
+ VT dz
| 22 [ 1/111
+ ||
11 | cf 011g + |ly y2112
(52)
sa(11e
+ ||0 |1 |/22 cg (
+ v1 6x
|yy - y2ll2
(53)
where
1
12
(54)
/1 6x
19

---

Published in Transactions on Machine Learning Research (01/2024)
Now, let -(...Pup Pa), i.c. the set of all joint distributions RR R that have marginals de and Po- Also, let Pá be the distribution of y Hf for f ~ Pg. i.e. the noiseless version of thg. Therefore, for B E B,
Ba) [f fell2 I a wer fi --ffl 1 |10 12Epspp
[f2 $5. l12
B ||f
Hf2 122
(55)
v1 6x
From Lemma A.1
we have
Bgg [|fi
Wi f1112
W1 Su
(56)
Also, from Eq. (6)
E Pa-pa B8f2 of. ll2
E Ex Ep mp (1R)) [ff; $b 11
(57)
= Eg Er -p (1)
of - Pprojs. (f)
(58)
Eg (11))
| pprois. (f)
(59)
S c.
(60)
Taking the infimum of Eq. (55) over B E B, we get
1
inf (F |f f2ll2 2 I a(c' + || (12c) + VI- 0g BEB
inf (f f2)0 [Hf Hf2ll2
(61)
BEB
Note that the left-hand side of the above equation is precisely Wi (Pollar). Also, note that the rightmost term in Eq. (61) is Wi(qyllpg):
(iggyllpg) = inf fa)m Hf Hfall2 JEB
(62)
From Eq. (44) since 4g Via Lemma 3.1
implies qy = Pg
WW lpg) = 0.
(63)
Combining with Eq. (61) and setting | and according to Eq. (54) get
Wi(pollue) < (1 vI -1 H112 (e (e)
1
(64)
B Additional numerical experiments
B.1 Evaluation of the posterior network
In order to evaluate the posterior network hg, the following experiments were designed using the setup of the stylized MRI study. First, the consistency of the modeled posterior Pa(: and the posterior induced by the main flow model p g) x 9n(g- Hff) pa (f) was examined. For fixed measurement vector g, 50 images were sampled from p g) using hg- For the 50 samples, log Pa gg) and log 9n (g Hf) log Pa (f) were computed and plotted against each other. This was repeated was different measurement vectors g. A scatter plot of (f g) versus log (g Hf) + log pf()) is shown in Fig. & for the AmbientFlow trained on two different measurement configurations (1) fully sampled noisy measurements, and (2) 4x undersampled noisy measurements. The scatter plots for each individual g are plotted different colors. It can be seen that the slope of linear fit of the scattered points is close for all three measurement vectors considcred. This indicates that po(. g) & qn(g Hf) pf()), i.c. the modeled posterior and the posterior induced by the learned prior consistent.
Po(()
20

---

Published in Transactions on Machine Learning Research (01/2024)
9 (bits/dim)
log gst i (bits/dimm)
Slope 1.1
Slope 1.0
7.4
Slope 1.1
8.1
Slope 1.1
Slope 1.0
Slope 0.9
8.0
se
7.2
log 9pt
7.9
7.0
7.8
jo (f)
Iognu (
6.8
7.4
7.8 loge; (f g) (bits/dim)
7.6
7.7
7.9
8.0
8.1 logp; (f) g) (bits/dim)
(a) Fully sampled noisy measurements
(b) 4x undersampled noisy measurements
Figure 8: Scatter plot of log (f g) versus log (n Hf) + log pf( for three different measurement vectors g, for Flo trained using two different measurement configurations from the stylized MRI study. The scatter plots for each individual g are plotted in different colors.
Table 2: The mean + standard deviation of the RMSE and SSIM values computed over the following test. image datasets - (1) the MMSE estimates using the flow prior trained on true objects, and (3) the MMSE estimates using the AmbientFlow prior trained on measurements, and (3) the MMSE estimates using samples from h- g). The forward and noise models in the image reconstruction task were the same the ones used to train the AmbientFlow.
n/m= =
n/m =
Method
RMSE
SSIM
RMSE
SSIM
Flow prior trained on true objects
0.017 + 0.002
0.95 + 0.01
0.022 + 0.003
0.94 + 0.01
AmbientFlow prior
0.016 = 0.002
0.96
0.01
0.025 + 0.004
0.92
0.01
Posterior network ho(: g)
0.016 0.002
0.96
0.01
0.026 + 0.004
0.91 0.01
(a) Flow prior from objects Estimate
(b) AmbientFlow prior
(c) Posterior network
True object
Zero filled (H"g)
Deviation
MMSE Estimate
Deviation 0.04
MMSE Estimate
Deviation
0.04
0104
0.06
0.06
0.0G
+
Il
E
Figure 9: MMSE estimates obtained by an empirical average over images obtained from (a) Langevin dynamics- based posterior sampling that employs the flow prior trained on true objects, (b) Langevin dynamics- basod posterior sampling that employs the AmbientFlow prior, and (c) the posterior network ho(: g).
21

---

Published in Transactions on Machine Learning Research (01/2024)
AmbientFlow
300
PLS-TV individual recon.
Flow trained on true objects
250
FID score
200
150
100
50
1
2
8
Subsampling ratio n/m for AmbientFlow training data
Figure 10: FID score as a function of the undersampling ratio used to simulate the training data for AmbientFlow for the stylized MRI study.
Table 3: MSE error in the mean of the images generated by the unconditional model relative to the true mean image; and relative squared- error in the rank-1000 approximation Σp of the learned covariance matrix relative to the rank-1000 approximation Eg of the true covariance matrix.
Conventional flow n/m n/m 2 n/m n/m 8
Pp f Eag f1 Er 0.3%
0.8%
0.7%
0.4%
1.4%
EE 1/ F |1 1
14%
16%
15%
15%
35%
Next, for the two measurement configurations within the stylized MRI study and for a test dataset of size 20, the outputs of the posterior model h g) for a measurement input g was compared with the images reconstructed using the corresponding flow prior trained the ground truths, as well those reconstructed using the corresponding AmbientFlow prior. The empirical MMSE estimates well as pixelwise standard deviation maps obtained from these posterior samples are shown in Fgg. 9 The RMSE and SSIM of the MMSE estimates are shown in Table 2 It can be seen that the MMSE estimates computed using samples from ha( g) closely resemble those computed via Langevin dynamics-based posterior sampling employing the AmbientFlow prior, both visually, as well as in terms of the RMSE and SSIM.
B.2 Additional evaluation of unconditional models
In this section, we present additional evaluation and ablation studies on the learned unconditional models for the stylized MRI study. Figure I0 shows the FID score as a function of the undersampling ratio used to simulate the incomplete, noisy stylized MRI measurements used to train FumbientFlow Next, we evaluate the ability of our models to learn first- and second-order image statistics. For this experiment, a conventional flow model trained on the true objects, as well as AmbientFlow models trained on noisy, undersampled 2, 4,8 were considered. Figure II shows the empirical mean,
stylized MRI measurements with n/m autocorrelation, and radial profile of the autocorrelation of images generated from the true unconditional distribution gr, and learned unconditional distribution ps for each of the flow models trained. The MSE error in the empirical mean image relative to the squared & norm of the empirical mean of the true distribution is shown in Table 3 Next, a rank-1000 approximation of the sample covariance matrix was computed for all the models using randomized SVD of the data. Since the first and the 1000th singular values of this matrix differed by almost orders of magnitude, a low-rank approximation to the covariance matrix differs minimally from the full covariance matrix in terms of the Frobenius norm, while also ensuring numerical stability in computation. The error in the empirical mean image relative to the squared la norm of the empirical mean of
22

---

Published in Transactions on Machine Learning Research (01/2024)
True
Conventional flow n/
n/m
n/m m 4
n/m
Mean image
Autocorrelation & its radial profile
Real Synthetic
Figure 11: The first shows the empirical mean of the images generated from the true unconditional distribution ge, and learned unconditional distribution for four different undersampling ratios considered in the sylized MRI study. The second row shows the image autocorrelations, and the third row shows the radial profile of the autocorrelation along the dotted red line.
Table 4: The mean + standard deviation of the RMSE and SSTM values for the CelebA face image inpainting study computed over the following test image datasets (1) the images reconstructed using the PLS-TV method, (2) the MAP and MMSE estimates using the flow prior trained on true objects, and (3) the MAP and MMSE estimates using the Ambient Flow prior trained on fully sampled noisy measurements.
Method
RMSE
SSIM
Navier-Stokes, based inpainting (Bertalmio et al. 2001
24.3 + 6.0
0.81 + 0.03
MAP Estimate
17.8 + 4.5
0.81 + 0.05
Flow prior trained on true objects
MMSE Estimate
15.1 = 3.7
0.88 + 0.03
MAP Fstimate
17.7 + 4.4
0.80 + 0.03
AmbientF Flow prior
MMSE Estimate
15.0 + 4.2
0.88 + 0.03
The squared-Frobenius error between the rank-1000 approximation Ep and the learned covariance matrix relative to the rank-1000 approximation Mg of the true covariance matrix relative to || is shown in Table
the true distribution is shown in Table 3
B.3 Additional case study: face image inpainting using Ambient prior
Here, the results of a case study of face image inpainting is presented. In particular, we consider the case where a trained uncomditional AmbientFlow is used as a prior in an image reconstruction task, where the forward operator applicable the task different from the forward operator used to train the AmbientFlow. The following two image roconstruction tasks are considerod - (1) approximate maximum posteriori (MAP) estimation, i.e. approximating the mode of the posterior pa(: (g), and (2) approximate sampling from the posterior pe(: gg).
For both the lasks, an AmbientFlow trained on noisy face images from the CelebA dataset, as well as a conventional flow model trained on the uncorrupted CelebA images were considered. For a held-out dataset of size 20, the first task was performed using the compressed sensing using generative models (CSGM) formalism [ssm al. 2020). For the second task, approximate posterior sampling was performed with the flow models priors using annealed Langevin dynamics (ALD) iterations proposed in Jalal, et al. (Jalal et al. 2021a).
23

---

Published in Transactions on Machine Learning Research (01/2024)
Flow prior trained on uncorrupted images MMSE Estimate
AmbientFlow prior trained on noisy images MMSE Estimate
True object
Masked image
Navier-Stokes
MAP Estimate
Std. Deviation
MAP Estimate
Std. Deviation
Figure 12: Image estimates and pixelwise standard deviation maps from the image inpainting case study,
For each true object, the MMSE estimate and the pixelwise standard deviation map Ô were computed empirically using 36 samples obtained via ALD iterations. A baseline Navier-Stokes--ann based inpainting method was also compared [Bertalmio et al. 2001]. Table shows the RMSE and SSIM of the roconstructed images with respect lo the true object, averaged over a dataset of 20 Lest images. can seen that terms of RMSE and SSIM, both the MAP and the MMSE image estimates obtained using the AmbientFlow prior are comparable to those obtained using the flow prior trained uncorrupted images, despite the AmbientFlow being trained only using noisy CelebA images. Figure 12 shows the true and masked images along with the images inpainted by the algorithms described above. It can be seen that although the AmbientFlow prior provides comparable RMSE and SSIM estimates to the flow prior trained on uncorrupted images, it still retains some smoothing artifacts characteristic the training.
penalty imposed on during
24


Reviews:
[{"review_id": "FTXKa7OA5t", "reviewer": "Reviewer_JKdo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThis is a very nice paper which proposes a creative adaptation of AmbientGAN to normalizing flows. The structure and the prose is clear. The experiments convincingly illustrate the ideas and the empirics are supported by theoretical results. The addressed problem---estimating a distribution from indirect observations---is relevant in a number of domains.\n\n### Weaknesses\n\n- (minor) The number of training samples for a 2D toy problem in Figure X seems excessive. It would be interesting to investigate how the quality of the result scales with the amount of training data.\n\n- It is unclear how well in practice the prior and the posterior are modeled. One way to check this would be to use images that are Gaussian random fields for which all the distributions can be explicitly characterized if the forward operator is linear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a normalizing flow-based framework to train a generative model for a distribution observed through a known linear  operator and corrupted with known noise. The framework resembles AmbientGAN but adapts it to the setting of flows trained by maximum likelihood. This adaptation entails training a separate posterior network which is a conditional normalizing flow (and which is useful in its own right) and a bespoke loss to facilitate the minimization of the Kullback–Leibler divergence between the measurement distribution and the variational approximation. The authors provide an analysis of the distributional error for low-dimensional (k-sparse) objects under idealized training and apply their framework to MAP estimation and posterior sampling in several stylized problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The idea to train a posterior network reminds me a bit of what Sun and Bouman do in \"Deep Probabilistic Imaging\" (AAAI 2021) to sample from the posterior given a known forward model and a generative prior. It is not the same but the connection is worth mentioning.\n\n- KL divergence makes sense for distributions with common support. For low-dimensional inputs (e.g. k-sparse), a perfectly trained model would have to become non-injective when the latent distribution is a white Gaussian, and in particular the volume change formulas would collapse. Does this cause any issues in theory or practice? It would be nice to see a comment.\n\n- In AmbientGAN the forward operators can also be random. Although you do not do this, it seems that your framework should accommodate it (except for some parts of analysis). Perhaps a toy example along these lines would further strengthen the manuscript.\n\n- projection on k-sparse signals looks like hard thresholding; how is this implemented in the learning pipeline?\n\n- Please carefully check the writing (\"is approximates the object\", \"remainder of the _chapter_\", \"is introduced that is represents\", \"Wienier deconvolution\", ..., I would also suggest to replace \"fake measurements\" by something like \"synthetic measurements\" since at least to my ear fake has a pejorative connotation)", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yScCSEmvjA", "reviewer": "Reviewer_8cDe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe this is a strong submission and of interest to the TMLR community.\n\nStrengths:\n\n- Importance of the problem: learning from corrupted data is a very interesting research direction with many scientific applications.\n- Relevance: this is the first framework for learning generative models from corrupted data. It extends a line of recent works on learning generative models from corrupted data and it proposes the first solution for learning flow models from measurements.\n- Presentation: The paper is clearly written and easy to follow.\n- Novelty of the method: the proposed method is interesting. The authors circumvent the difficulty of modeling the likelihood of fake measurements by introducing an auxiliary network that produces clean samples from the posterior distribution given the lossy observations. The objective penalizes the auxiliary network if the measured generations drift away from the distribution of observed measurements. The authors use the fact that for certain corruptions, if the measured distributions match, then it must be that the distributions before the measurements match. This ensures that the auxiliary network produces samples from the right distribution and with those, Flow Models can be trained to model the true likelihood.\n- Correctness and novelty of theoretical results: I read the theoretical results and proofs of the paper and seem novel and correct to me. The authors establish the correctness of the training objective, analyze cases in which you can learn perfectly from measurements and finally obtain approximate recovery guarantees in the case of matrices that have a null-space.\n\n\nWeaknesses:\n\n- I think the condition that H is full-rank in Lemma 3.1 can be relaxed in certain cases and still get optimal reconstruction results. I refer the authors to results from the Ambient GAN paper, e.g. for the random inpainting case.\n-  The experimental evaluation is not very thorough. Comparisons with baselines are missing. Ablation studies regarding how performance changes with increased corruption are limited.\n- The authors missed some recent work in learning diffusion models from corrupted data. I refer the authors to the following works:\nSolving Inverse Problems with Score-Based Generative Priors Learned from Noisy Data, Ambient Diffusion: Learning Clean Distributions from Corrupted Data. \n- The proposed method bears some resemblances to AmbientGAN. Particularly, both works look at how far measured generated images are from the real measurements. This is the critical component for the proposed objective to work and I think a more thorough discussion of the similarities and differences to AmbientGAN would have been helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a framework for learning flow models from linearly corrupted, noisy data. The problem of learning from corrupted data is important for scientific applications where access to clean data is expensive or impossible. Prior to this work, the problem of learning generative models from corrupted data has been explored in the context of GANs and diffusion models. The authors extend this line of work to Flow Models that have the advantage that model the likelihood function directly. The proposed method is based on a novel training objective that uses two networks: a model that generates clean samples conditioned on measurements and a model that gets the generated samples and maximizes their likelihood. The authors demonstrate the empirical performance of their method in toy datasets, CelebA-64x64 and a stylized MRI application.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper would improve by strengthening the experimental evaluation. It would be nice to compare with Ambient GAN / Diffusion on Celeb-A 64x64 or some other image dataset. It would also be nice to include plots of how the performance (in FID) changes as we increase the noise or the blurring. I would also encourage the authors to experiment with different types of linear corruptions, e.g. random inpainting or block inpainting as in prior work.", "broader_impact_concerns": "I do not have broader impact concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ozUhSle9uv", "reviewer": "Reviewer_gPnD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The paper is very well written. Notation is clear and concise. The level of abstraction is appropriate.\n- The paper has both theoretical and numerical results.\n- The numerical results are presented well.\n\n# Weaknesses\n- The conditional posterior was not evaluated, and hence, the comparisons in the paper are a bit confusing to me. In Fig. 4, for instance, the authors show qualitative results of applying wiener deconvolution and BM3D to measurements. However, the appropriate comparison for BM3D and Wiener deconvolution would be the output of the trained posterior on the same images, or at least a reconstruction algorithm that uses the unconditional prior. The unconditional samples do not allow for a principled qualitative comparison.\n- The authors only evaluate the unconditional generator numerically, not the conditional posterior. \n- The authors claim \"An estimate of the distribution of objects is known to be important for applications in imaging science.\". I agree, but I think the authors would do well to motivate why exactly that is the case. Specifically, the question the authors should answer: why do we care about the unconditional model when we perform reconstruction? The only reason that I can see is that the measurement forward model is *different* at test time than it was at training time, such that the posterior network would be out-of-distribution. In this case, the unconditional prior enables us to solve the inverse problem still.\n  - I would hence expect that the authors evaluate exactly this case. I.e., run an experiment where you show that you can reconstruct measurements subject to a forward model that was *different* than the one the model was trained on. For instance, a different class of blur kernels, or different noise levels.\n  - This is also extremely important b/c otherwise, I would expect the authors to benchmark with a method that is trained to directly invert the measurement model, i.e., a state-of-the-art image-to-image translation model that is trained to directly reconstruct a noisy & blurry image.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper's goal is to recover a data distribution where observations are not direct samples from the distribution, but are rather measured through a known linear forward model with additive noise sampled from a known noise distribution.\n\nThe authors propose to tackle this problem in a probabilistic framework. Specifically, they build an unconditional generator as well as a measurement-conditional posterior network, both parameterized as an invertible neural network. They then leverage the known forward model and noise distribution to train the posterior network to inver the forward model and generate \"clean\" data samples $\\mathbf{f}$ from noisy measurements $\\mathbf{g}$. They then train the generator to match the distribution of the posterior network. \n\nThe authors now prove that under certain assumptions that are of practical interest, optimizing their objective minimizes the KL-divergence of $p_\\theta$ and the underlying, true data distribution, thus successfully learning a generative model of the true data distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor:\n- I have never heard \"computed imaging\" before. We usually call it \"Computational Imaging\", see wikipedia article [here](https://en.wikipedia.org/wiki/Computational_imaging)\n- \"that is represents\" --> \"that represents\" in paragraph 3 of section 3.\n- Wienier --> Wiener in Baselines and evaluation metrics", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is clear, well structured, and support the claims that the authors made. The authors build on the previous AmbientGAN work to provide a substantial contribution, especially in the medical imaging domain.
Decision: Accept (poster)