Invertible generative models for  inverse problems: mitigating representation error and dataset bias
Authors: Muhammad Asim
Ali Ahmed
Paul Hand
Venue:  (2020)
Keywords: Invertible generative models
inverse problems
generative prior
Glow
compressed sensing
denoising
inpainting.

Abstract: Trained generative models have shown remarkable performance as priors for inverse problems in imaging.  For example, Generative Adversarial Network priors permit recovery of test images from 5-10x fewer measurements than sparsity priors.  Unfortunately, these models may be unable to represent any particular image because of architectural choices, mode collapse, and bias in the training dataset. In this paper, we demonstrate that invertible neural networks, which have zero representation error by design, can be effective natural signal priors at inverse problems such as denoising, compressive sensing, and inpainting.   Our formulation is an empirical risk minimization that does not directly optimize the likelihood of images, as one would expect.  Instead we optimize the likelihood of the latent representation of images as a proxy, as this is empirically easier.
For compressive sensing, our formulation can yield higher accuracy than sparsity priors across almost all undersampling ratios.  For the same accuracy on test images, they can use 10-20x fewer measurements.  We demonstrate that invertible priors can yield better reconstructions than sparsity priors for images that have rare features of variation within the biased training set, including out-of-distribution natural images.  

Under review as a conference paper at ICLR 2020
INVERTIBLE GENERATIVE MODELS FOR INVERSE PROBLEMS: MITIGATING REPRESENTATION ERROR AND DATASET BIAS
Anonymous authors Paper under double-blind review
ABSTRACT
Trained generative models have shown remarkable performance as priors for inverse problems imaging. For example, Generative Adversarial Network priors permit recovery of test images from 5- 5-10x fewer measurements than sparsity priors. Unfortunately, these models may be unable represent any particular image because of architectural choices, mode collapse, and bias in the training dataset. this paper, we demonstrate that invertible neural networks, which have zero representation error by design, can be effective natural signal priors at inverse problems such as denoising, compressive sensing, and inpainting. Our formulation is empirical risk minimization that does not directly optimize the likelihood images, as one would expect. Instead we optimize the likelihood of the latent representation of images proxy, this is empirically easier. For compressive sensing, our formulation can yield higher accuracy than sparsity priors across almost all undersampling ratios. For the same accuracy test images, they 10-20x fewer measurements. We demonstrate that invertible priors can yield better reconstructions than sparsity priors for images that have rare features of variation within the biased training set, including out- oiff---iistrbutioon natural images.
INTRODUCTION
Training
Truth
Lasso
DCGAN
Ours
Figure 1: We train invertible generative model with CelebA images (including those at left). When used as a prior for compressed sensing, it can yield higher quality image reconstructions than Lasso and trained DCGAN, even -iitt-idiitiittiion images. Note that the DCGAN reflects biases of the training set by removing the man's 's glasses and beard, whereas our invertible prior does not.
Generative deep neural networks have shown remarkable performance as natural signal priors in imaging inverse problems, such as denoising, inpainting, compressed sensing, blind deconvolution. and phase retrieval. These generative models can be trained from datasets consisting of images of particular natural signal classes, such as faces, fingerprints, MRIs. and more (Karras et al. 2017 Minace and Abdolrashidi 2018 Shin et al. 2018 Chen al. 2018). Some such models, including variational autoencoders (VAEs) and generative adversarial networks (GANs), learn an explicit low-dimensional manifold that approximates natural signal class (Goodfellow al. 2014 Kingma and Welling 2013, Rezende et al.) 2014). We will refer to such models as GAN priors. With an explicit parameterization of the natural signal manifold by low dimensional latent representation,

---

Under review as a conference paper at ICLR 2020
these generative models allow for direct optimization over natural signal class. Consequently, they can obtain significant performance improvements over non-learning based methods. For example, GAN priors have been shown to outperform sparsity priors compressed sensing with 5-10x fewer measurements. Additionally, GAN priors have led to theory for signal recovery in the linear compressive sensing and nonlinear phase retrieval problems (Bora al. 2017 Hand and Voroninski] 2017 Hand et al. 2018), and they have also shown promising results for the nonlinear blind image deblurring problem (Asim et al. 2018).
A significant drawback of GAN priors for solving inverse problems is that they can have representation error or bias due to architecture and training. This can happen for many reasons, including because the generator only approximates the natural signal manifold, because the natural signal manifold of higher dimensionality than modeled, because of mode collapse, or because of bias in the training dataset itself. As many aspects of generator architecture and training lack clear principles, representation error of GANs may continue to be a challenge even after substantial hand crafting and engineering, Additionally, tearning-basel methods are particularly vulnerable to the biases of their training data, and training data, no matter how carefully collected, will always contain degrees of bias. As example, the CelebA dataset (Liu et al. 2015) is biased toward people who are young, who do not have facial hair or glasses, and who have a light skin tone. As we will see, a GAN prior trained on this dataset learns these biases and exhibits image recovery failures because of them.
In contrast, invertible neural networks can be trained as generators with zero representation error. These networks are invertible (one-to-one and onto) by architectural design (Dinh et al. 2016) Gomez et al.] 2017 Jacobsen et al. 2018 Kingma and Dhariwal (018). Consequently, they are capable of recovering any image, including those significantly training set; see Figure Ι We call the domain of an invertible generator the latent space, and we call the range of the generator the signal space. These must have equal dimensionality. Flow-based invertible generative models are composed of sequence of learned invertible transformations. Their strengths include: their architecture allows exact and efficient latent-variable inference, direct log- likelihood evaluation, and efficient image synthesis; they have the potential for significant memory savings in gradient computations; and they can be trained by directly optimizing the likelihood of training images. This paper emphasizes an additional strength: because they lack representation error, invertible models can mitigate dataset bias and improve performance on inverse problems with out-of-distributte data.
relative to a biased
In this paper, we study generative invertible neural network priors for imaging inverse problems. We will specifically use the Glow architecture, though our framework could be used with other architectures. A Glow-based model is composed of a sequence of invertible affine coupling layers, 1x1 convolutional layers, and normalization layers. Glow models have been successfully trained to generate high resolution photorealistic images of human faces (Kingma and Dhariwal 2018).
We present a method for using pretrained generative invertible neural networks as priors for imaging inverse problems. The invertible generator, once trained, can be used for a wide variety of inverse problems, with no specific knowledge of those problems used during the training process. Our method is an empirical risk formulation based on the following proxy: we penalize the likelihood of an image's 's latent representation instead of the image's likelihood itself. While this may be couterintuitive, it admits optimization problems that are easier to solve empirically. In the case of compressive sensing, our formulation succeeds even without direct penalization of this proxy likelihood, with regularization occuring through initialization of gradient descent in latent space.
We train a generative invertible model using the CelebA dataset. With this fixed model as signal prior, we study its performance at denoising, compressive sensing, and inpainting. For denoising, it can outperform BM3D (Dabov et al. 2007). For compressive sensing on test images, it can obtain higher quality reconstructions than Lasso across almost all subsampling ratios, and at similar reconstruction errors can succeed with 10-20x fewer measurements than Lasso. It provides an improvement of about 2x fewer linear measurements when compared to Bora et al. (2017). Despite being trained on the CelebA dataset, our generative invertible prior can give higher quality reconstructions than Lasso on out-o--distribttio images of faces, and, a lesser extent, unrelated natural images. Our invertible prior outperforms pretrained DCGAN (Radford et al.) 2013) at face inpainting and exhibits qualitatively reasonable results on out-of -fi-oiiiittiuttionn human faces, We provide additional experiments in the appendix, including for training on other datasets.
2

---

Under review as a conference paper at ICLR 2020
2 METHOD AND MOTIVATION
We assume that have access pretrained generative invertible neural network G: R IR" We write 2 = G(2) and = (z), where E R" is an image that corresponds to the latent representation 2 R We will consider a G that has the Glow architecture introduced in Kingmal and Dhariwal (0018). It can be trained by direct optimization of the likelihood of a collection of training images of a natural signal class, under a standard Gaussian distribution over the latent space. We consider recovering an image x from possibly-noisy linear measurements given by A E R _
y = Ax + 17:
where 17
In models noise. Given a pretrained invertible generator G. we have access likelihood estimates for all images z E R". Hence, it is natural to attempt to solve the above inverse problem by a maximum likelihood formulation given by
min | y1 7 pc(x),
(I)
where PC the likelihood function over induced by G, and y a hyperparameter. We have found this formulation to be empirically challenging to optimize; hence we study the following proxy:
min ZER
|[AG(()) - yll² + >||=||
Unless otherwise stated, we initialize 22 at 20 = 0.
The motivation for formulation (2) is follows. As a proxy for the likelihood of an image x E R" we will use the likelihood of its latent representation = (z)) Because the invertible network G was trained map standard normal in Rn to a distribution over images, the log-likelihood of point 2 is proportional to ||=]|² Instead of penalizing || we alternatively penalize the unsquared || In Appendix B
we show comparable performance for both the squared and unsquared formulations.
In principle, our formulation has an inherent flaw: some high- high-likelihood latent representations correspond low-likelihood images x. Mathematically, this comes from the Jacobian term that relates the likelihood in to the likelihood in x upon application of the map G. For multimodel distributions, such images must exist, which we will illustrate in the discussion. This proxy formulation relies on the fact that the set of such images has low probability and that they are inconsistent with enough provided measurements. Surprisingly, despite this potential weakness, we will observe image reconstructions that are superior to BM3D and GAN-based methods at denoising, and superior to GAN-based and Lasso-based methods at compressive sensing.
In the case of compressive sensing and inpainting, we take y = 0 in formulation The motivation for such a formulation initialized at 20 = 0 is as follows. There is a manifold of images that are consistent with the provided measurements. We want to find the image of highest likelihood on this manifold. Our proxy turns the likelihood maximization task over an affine space in x into the geometric task of finding the point on manifold in z-space that is closest the origin with respect to the Euclidean norm. In order to approximate that point, we run a gradient descent in 2 down the data misfit term starting at 2g
In the case of GAN priors for G : IR* + R", we will use the formulation from Bora et al. 2017). which is the formulation above in the case where the optimization performed over RM 7 0, and initialization is selected randomly.
All the experiments that follow will be for an invertible model we trained on the CelebA dataset of celebrity faces, as in Kingma and Dhariwal (2018). Similar results for models trained on birds
and flowers (Wah et al., 201 Nilsback and Zisserman 2008) can found in the appendíx. Due to computational considerations, we run experiments on 64 x 64 color images with the pixel values scaled between [0, 1]. The train and test sets contain a total of 27, 7,000 and 3,000 images, respectively. We trained Glow architecture (Kingma and Dhariwal 2018); see Appendix A for details. Once trained, the Glow prior is fixed for use in each of the inverse problems below. We also trained a DCGAN for the same dataset. We solve (2) using LBFGS, which was found to outperform Adam (Kingma and Ba 2014). DCGAN results are reported for an average of runs because we observed some variance due random initialization.
3

---

Under review as a conference paper at ICLR 2020
3 APPLICATIONS
3.1 DENOISING
We consider the denoising problem with A = and v ~ (V(0, o2)). for images x in the CelebA test dataset. We evaluate the performance of Glow prior, DCGAN prior, and BM3D for two different noise levels. Figure 2 shows the recovered PSNR values as a function of for denoising by the Glow and DCGAN priors, along with the PSNR by BM3D. The figure shows that the performance of the regularized Glow prior increases with 7, and then decreases. If 7 is too low, then the network fits the noise in the image. left panel reveals that an appropriately regularized Glow prior can outperform BM3D by almost dB. The experiments also reveal that appropriately regularized Glow priors outperform the DCGAN prior, which suffers from representation error and is not aided by the regularization. The right panel confirms that with smaller noise levels, less regularization is needed for optimal performance. A
too high. then data fit is not enforced strongly enough. The
0.20
0.05
45
45
40
40
35
Noise Level
35
Noise Level
PSNR
CLOW
PSNR
CLOW
DCCAN
DXCAAN
25
20
20
15
15
10
40
10
40
Figure 2: Recovered PSNR values function of y for denoising by the Glow and DCGAN priors. All the results are averaged over test set images. For reference, we show the average PSNRs the original noisy images, after applyig BM3D, and under the Glow prior in the noiseless case (a 0).
visual comparison of the recoveries at the noise level 0.1 using Glow, DCGAN priors, and BM3D can be seen in Figure Note that the recoveries with Glow are sharper than BM3D. See Appendix for more quantitative and qualitative results.
Truth
1.S.
Noisy
DCGAN
D
BM3D
12..
GLOW
Figure 3: Denoising results using the Glow prior, the DCGAN prior, and BM3D at noise level 0.1. Note that the Glow prior gives sharper image than BM3D.
3.2 COMPRESSED SENSING
In compressed sensing, one is given undersampled linear measurements of an image, and the goal recover the image from those measurements. In notation, A Ruxn with As the
4

---

Under review as a conference paper at ICLR 2020
image is undersampled, there is an affine space of images consistent with the measurements, and an algorithm must select which is most 'natural.' A common proxy for naturalness in the literature has been sparsity with respect to the DCT or wavelet bases. With GAN prior, an image is considered natural if it lies in or near the range of the GAN. For an invertible prior under our proxy for likelihood, consider an image be natural if it has latent representation of small norm.
We study compressed sensing in the case that an m x n matrix of i.i.d N(0, 1/m) entries, and is an image from the CelebA test set. Here, n 64 64 x 12288. We consider the case where yy is standard iid Gaussian random noise normalized such that v / 0.1. We compare Glow, DCGAN, and Lasse with respect the DCT and wavelet bases.
0 outperforms both DCGAN and Lasso in reconstruction quality over all undersampling ratios, as shown in the left panel of Figure Surprisingly, in the case of extreme undersampling, Glow substantially outperforms these methods even though it does not maintain direct -in-dimeniional parameterization of the signal manifold. The Glow prior (1) can result in 15 dB higher PSNRs than DCGAN, and (2) can give comparable recovery errors with 2-3x fewer measurements at high undersampling ratios. This difference is explained by the representation error of DCGAN, which has been shown to be the dominant source of error in DCGAN by Bora et al. (2017). Additional plots and visual comparisons, available in Appendix C show notable improvements in quality of in- and out-or-distributtin images using invertible prior relative to DCGAN and Lasso.
Our main result is that the Glow prior with 0 and initialization
10
34
PSNR
PSNR
13
CLOW
30
and 0.1
DCGAN
LASSO-DCT
LASSO-WVT Kull Space
LASSO-WVT
10
2000
4000
8000
10000
12000
0.0
16:05
1c:05
0.0001
0.001
0.01
0.1
messurements (m)
Figure 4: The left panel shows recovered PSNRs averaged over 12 test set images under the Glow, and DCGAN prior with 0; and the Lasso with respect to the DCT and Wavelet Transform. We initialize with 20 = O. See Appendix for zoom-in of the case of small m. The right panel shows the resulting PSNR when m 5000 with Glow prior after different initialization strategies, as described in the text. The highest PSNR was recovered with initialization 2g O and y = 0.
We conducted several additional experiments to understand the regularizing effects of y and the initialization zg- The right panel of Figure 4 shows the PSNRs under multiple initialization strategies: 20 0, 2g ~ (0, 0.1 0.127), 2g ~ (N 0.777), 20 G=1(z)) with xo given by the solution to Lasso with respect the wavelet basis, and 2g G (xo) where 2o is perturbed by random point in the null space A. The best performance was observed with initialization 2 = 0. The hyperparameter y can be taken be zero, which is surprising because then there is no direct penalization of likelihood for this noisy compressive sensing problem. In the case o = 0, we observe that larger initializations result in recovered images of lower PSNR. See Appendix C for additional experiments that show this effect. We observe that initialization strategy can have a strong qualitative effect on the recovery formulation. For example, if the optimization is initialized by the solution the Lasso, then directly penalizing the likelihood of can improve reconstruction PSNR. though those reconstruction are still worse than with initialization 2g 0 and ? = O. Suboptimal initialization procedures apparently benefit from direct penalization of likelihood, whereas the = O initialization apparently does not.
Finally, we observe that the Glow prior is much more robust to out-of of--stirbbuttionn examples than the GAN Prior. Figure 5 shows recovered images using (2) for compressive sensing for images not belonging to the CelebA dlttaset. DCGAN's performance reveals biases of the underlying dataset and limitations of low-dimensional modeling. For example, projecting onto the CelebA--rainee DCGAN can cause incorrect skin tone, gender, and age. It's performance on distrii-iitiitttion images is poor.
TTe inverse problems with I Lasso were solved by min; A vl13 + [|[[] using coordinate descent.
5

---

Under review as a conference paper at ICLR 2020
In contrast, the Glow prior mitigates this bias, even demonstrating image recovery for natural images that are not representative the CelebA training set, including people who are older, have darker skin tones, wear glasses, have beard, or have unusual makeup. The Glow prior's performance also extends to significantly out-of-distribution images, such as animated characters and natural images unrelated faces. See Appendix C.2 for additional experiments.
Trum
DCT
WVV
DCGAN
GLOW
Figure 5: Compressed sensing (CS) with number 500 (>= 20%) of measurements of out- of-distribution images. Visual comparisons: CS under the Glow prior, DCGAN prior, Lasso-WVT, and Lasso-DCT at a noise level > = 0.1. In each case, we choose values of the penalization parameter 7 yield the best performance. We use 7 for both DCGAN and Glow priors and 0.01 for Lasso-WVT, and Lasso-DCTS respectively.
3.3 INPAINTING
In inpainting, one is given masked image of the form M @ where M is masking matrix with binary entries and R. an n-pixel image. The goal is to find x. We could rewrite (2) with y 0
min 1ly M a G(=)]l2
=ER
Within Distrib
Out Distrib
There is an affine space of images consistent with the mea- surements, and algorithm must select which is most ural. As before, using the minimizer 2, the estimated image given by G()). Our experiments reveal the same story as for compressed sensing. If initialized at 2g = O, then the empirical risk formulation with 7 exhibits high PSNRs on test images. Algorithmic regularization is again occurring due initialization. In contrast, DCGAN is limited by its representation error. See Figure 6 and Appendix D for more results, including visually reasonable face inpainting, even for out-of-diitributtio human faces.
Truhh
Masked
DCGAN
GLOW
DISCUSSION
Figure 6: Inpainting: Recoverics under DCGAN and Glow, both with 7 0.
have demonstrated that pretrained generative invertible models can be used natural signal priors in imaging in- verse problems. Their strength is that every desired image is in the range of an invertible model, and the challenge that they overcome is that every undesired image is also in the range of the model and no explicit low- dimensional representation is kept. We study a regularization for empirical loss minimization that promotes recovery of images that have high value of proxy for image likelihood

---

Under review as a conference paper at ICLR 2020
under the generative model. We demonstrate that this formulation quantitatively and qualitatively outperform BM3D at denoising. Additionally, it has lower recovery errors than Lasso across all levels of undersampling, and it can get comparable errors from 10-20x fewer measurements, which is 2x reduction from |Bora et al. 2017). The superior recovery performance of the invertible prior at very extreme undersampling ratios is particularly surprising given that invertible nets do not maintain explicit low dimensional representations, as GANs do. Additionally, our trained invertible model yields significantly better reconstructions than Lasso even on out-of-distribution images, including images with rare features of variation, and on unrelated natural images.
The idea of analyzing inverse problems with invertible neural networks has appeared in Ardizzone et al. (2018). The authors study estimation of the complete posterior parameter distribution under a forward process, conditioned on observed measurements. Specifically, the authors approximate particular forward process by training an invertible neural network. The inverse map is then directly available. order to cope with information loss, the authors augment the measurements with additional variables. This work differs from ours because it involves training separate net for every particular inverse problem. In contrast, our work studies how to use a pretrained invertible generator for variety of inverse problems not known training time. Training invertible networks is challenging and computationally expensive; hence, it is desirable to separate the training of off-the-shelf invertible models from potential applications in variety of scientific domains.
Why optimize a proxy for image likelihood instead of optimizing image likelihood directly?
As noted in Section 2 the immediate formulation one would write down for inverse problems under an invertible prior is to optimize a data misfit term together with an image log- ligliikl term. Unfortunately, we found it difficult to get this optimization converge in practice. The likelihood term can exhibit rapid variation due to the Jacobian of the transformation z + x = G(z); additionally the likelihood term may in principle even contain local minima or other geometric properties that make gradient descent difficult. Figure [coopares the loss landscapes in æ and 2, illustrating that the learned likelihood function in x may lead difficulty in choosing appropriate step sizes for gradient descent algorithms.
(a) lly - Axll² 200 log pa (x)
(b) log (c))
(c) 1ly AG(2) 50||
Figure 7: Landscapes of (a) the loss surface in æ-space, (b) just the image likelihood in x-space, and (c) the loss surface in z-space, as functions of two random directions in either x or z, as appropriate.
In contrast, there are nice geometric properties that appear in latent space from an invertible model. As an illustration, consider the compressive sensing problem with noiseless measurements. Here, the formulation corresponds to gradient descent down the data misfit term |AA((z) y starting at 2n = 0. This data misht term has a favorable geometry for optimization in that all local minima are global minima. This is because the level sets in of ||a))()) yll are given by G-1 applied to the level sets in æ of Ax - y/ which have a simple structure because of the linearity of the measurements in There may be additional benefits due to optimizing in because the invertible net learns representations that permit interpolation between images and semantically meaningful arithmetic, reported in Kingma and Dhariwal (2018).
Why is the likelihood of an image's 's latent representation reasonable proxy for the image's 's likelihood?
The training process for an invertible generative model attempts to learn target distribution in images space by directly maximizing the likelihood of provided samples from that distribution, given standard Gaussian prior in latent space. High probability regions in latent space map regions in image space of equal probability. Hence, broadly speaking, regions small values of ||=]]| are expected to map regions of large likelihoods in image space. There will be exceptions this
7

---

Under review as a conference paper at ICLR 2020
log p(z)
log p(rp
1.
1.0
4
0
0.5
0.5
-
10
2
5
0.0
0.0
0-
20
10
0.5
0.5
2
1.0
1.0
30
15
1.0
0.5
0.0
0.5
1.0
1.0-0.5
0.0
0.5
1.0
-2
0
2
4
21
Figure 8: An invertible net was trained on the data points in æ-space (left), resulting in the given plots of latent z-likelihood versus z (middle), and z-likelihood versus latent representation 2 (right).
property. For example, natural image distributions have multimodal character. The preimage of high probability modes in image space will correspond to high likelihood regions in latent space. Because the generator G is invertible and continuous, interpolation latent space of these modes will provide images of high likelihood in but low likelihood in the target distribution. To illustrate this point, we trained a Real-NVP (Dinh et al. (016)) invertible neural network on the two dimensional set of points depicted in Figure 8 (left panel). The middle and right panels show that high likelihood regions in latent space generally correspond to higher likelihood regions in image space, but that there are some regions of high likelihood in latent space that map to points of low likelihood in image space and in the target distribution. We see that the spurious regions are of low total probability and would be unlikely to be the desired outcomes of an inverse problem arising from the target distribution.
How can solving compressive inverse problems be successful without direct penalization of the proxy image likelihood?
there are fewer linear measurements than the dimensionality of the desired signal, an affine space of images is consistent with the measurements. In our formulation, regularization does not occur by direct penalization of our proxy for image likelihood; instead, it occurs implicitly by performing z-space with an initialization of 2u O. The set of latent representations that are consistent with the compressive measurements define me-dimensional nonlinear manifold. As per the likelihood proxy mentioned above, the spirit of our formulation is to find the point on this manifold that is closest to the origin with respect to the Euclidean norm. Our specific way of estimating this point is to perform a gradient descent down a data misfit term in z-space, starting at the origin. While gradient flow typically will not find the closest point on the manifold, it empirically finds a reasonable approximation refine the output of this gradient flow, but elect not to do for the sake of simplicity.
the optimization
that point. In practice, one could further do a local search to
Why does the invertible prior do so well, especially on out-of-distribution images?
One reason that the invertible prior performs so well is because it has representation error. The lack of representation eттor of invertible nets presents a significant opportunity for imaging with learned prior. Any image is potentially recoverable, even if the image is significantly outside of the training distribution. In contrast, methods based on projecting onto an explicit low-dimensional representation of a natural signal manifold will have representation error, perhaps due to modeling assumptions, mode collapse, or bias in training set. Such methods will see performance prematurely saturate as the number of measurements increases. In contrast, an invertible prior would not see performance saturate. In the extreme case of having full set of exact measurements, an invertible prior could in principle recover any image exactly.
It is natural to wonder which images can be effectively recovered using an invertible prior trained on particular signal class. As expected, we see the best reconstruction errors on in-distribution images and performance degrades as images get further aut-of-distributin. Nonetheless, we observe that reconstruction errors of unrelated natural images are still of higher quality than with the Lasso. It appears that the invertible generator learns some general attributes of natural images. This leads to several questions: when a generative invertible net is trained, how far duitr--distribution can an image be while maintaining a high likelihood? How do invertible nets learn useful statistics of natural images? Is that due primarily to training, or is there architectural bias toward natural images, as with the Deep Image Prior and Deep Decoder (Ulyanov et al. 2018 Heckel and Hand 2018)?
8

---

Under review as a conference paper at ICLR 2020
The results of this paper provide further evidence that reducing representational error of generators can significantly enhance the performance generative models for inverse problems in imaging. This idea was also recently explored in Athha al. (2018), where the authors trained GAN-like prior with high-dimensional latent space. The high dimensionality of this space lowers representational error, though it is not zero. In their work, the high-dimensional latent space had structure that was difficult to directly optimize, so the authors successfully modeled latent representations as the output of untrained convolutional neural network whose parameters are estimated
time. Their paper and ours raises several questions: Which generator architectures provide good balance between low representation error, ease of training, and ease of inversion? Should generative model be capable of producing all images in order to perform well on out-of-distribution images of interest? Are there cheaper architectures that perform comparably? These questions are quite important, as solving equation 2 in our 64x64 pixel color images experiments took 15 GPU-minutes. New developments are needed on architectures and frameworks in between low-dimensional generative priors and fully invertible generative priors. Such methods could leverage the strengths of invertible models while being much cheaper to train and use.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. 2017.
Shervin Minace and Amirali Abdolrashidi. Finger-gan: Generating realistic fingerprint images using connectivity imposed gan. arXiv preprint 1TXiv:881.110422 2018.
Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey Gunter, Katherine Andriole, and Mark Michalski. Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In International Workshop on Simulation and Synthesis in Medical Imaging, pages 1-11. Springer, 2018.
Yuhua Chen, Feng Shi, Anthony G Christodoulou, Yibin Xie, Zhengwei Zhou, and Debiao Li. Efficient and accurate mri super-resolution using generative adversarial network and 3d multi- level densely connected network. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 91-99. Springer, 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Bing Mirza, Mehdi; Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv:1406.2661, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114, 2013.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4000 2014.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 537-546. JMLR. org, 2017.
Paul Hand and Vladislav Voroninski. Global guarantees for enforcing deep generative priors by empirical risk. arXiv preprint
2017.
Paul Hand, Oscar Leong, and Vlad Voroninski. Phase retrieval under generative prior. In Advances in Neural Information Processing Systems, pages 9136-9146, 2018.
Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Blind image deconvolution using deep generative priors. arXiv preprint 18Xi:: 00000.00473.... 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738. 2015.
Laurent Dinh, Jascha Sohh-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0888 2016.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in neural information processing systems, pages 2214-2224, 2017.
Jörn-Henrik Jacobsen, Amoll Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprini arXiv: 1802.07088, 2018.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
Kostadin Dabov, Alessandro Foi, and Karen Egiazarian. Video denoising by sparse 3d transform- domain collaborative filtering. 2007 15th European Signal Processing Conference, pages 145-149, IEBE, 2007.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rXiv: 2015.
C. Wah. S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds- 2011 Dataset. (CNS-TR-2011-00)) 1-001), 2011.
10

---

Under review as a conference paper at ICLR 2020
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf's S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.047300 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446-9454, 2018.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks. arXiv preprint UrXiv:1810.03982. 2018.
ShahRukh Athar, Evgeniy Burnaev, and Victor Lempitsky. Latent convolutional models. arXiv preprint arXiv: 06284, 2018

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL SETUP
Simulations were completed mainly on CelebA-HQ dataset, used Kingma and Dhariwal (2018): has 30,000 color images that were resized to 64 x 64 for computational reasons, and were split into 27,000 training and 3000 test images. We also provide some additional experiments on the Flowers Nilsback and Zisserman (2008), and Birds Wah et al. (201T) datasets. Flowers dataset contains 8189 color images resized to 64 x 64 out of which 500 images are spared for testing. Birds dataset contains total of 11,788 images, which were center aligned and resized to 64 x 64 out of which 5794 images are set aside for testing.
We specifically model our invertible networks after the recently proposed Glow Kingma and Dhariwal (2018) architecture, which consists a multiple flow steps. Each flow step comprises of an activation normalization layer, a invertible. Let be the number of steps of flow before a splitring layer, and L. be the numher of times the splitting performed. To train over CelebA, choose the network to have K = 48, L. 4 and affine coupling, and train it with a learning rate 0.0001, and a batch size 6 at resolution 64 x 64 x 3. The model was trained over bit images with 10,000 warmup iterations as in Kingma and Dhariwal (2018), but when solving inverse problems using Glow original 8 bit images were used. We refer the reader to Kingma and Dhariwal (2018) for specific details on the operations performed in each of the network layer.
convolutional layer, and an affine coupling layer, each of which is
We use LBFGS solve the inverse problem. For best performance, we set the number of iterations and learning rate for denoising, compressed sensing, and inpainting to be 20, 1; 30, 0.1; and 20, 1 respectively. we use Pytorch implement Glow network training and solve the inverse problem. Glow training was conducted on single Titan Xp GPU using maximum allowable (under given computational constraints) batch size of 6. In case CS, recovering a single image on Titan Xp using LBFGS solver with 30 steps takes 889. 125 seconds (14.82 minutes). However, inverse problems in parallel on the given hardware platform.
solve 6
Unless specified otherwise, inverse problem under Glow prior is always initialized with 20 0. Whereas under DCGAN prior, we initialize with 2o ~ NO0 0.127) and report average over three random restarts. In all the quantitative experiments over, the reported quality metrics such PSNR, and reconstruction errors are averaged over 12 randomly drawn test set images.
Figure 9: Samples from training set of CelebA downsampled 64 64 x 3.
B DENOISING: ADDITIONAL EXPERIMENTS
We present additional quantitative experiments on image denoising here. Complete set of experiments on average PSNR over 12 CelebA (within distributior test set images versus penalization parameter y under noise levels message is that Glow prior outperforms DCGAN prior uniformly across all y due to the representation limit of DCGAN. In addition, striking the right balance between the misfit term and the penalization term by appropriately choosing 7 improves the performance of Glow, and it also approaches state- of-the-art BM3D algorithm at low noise levels, and clearly visible in higher noise, for example, at a noise level of 0.2, the Glow prior improves upon BM3D by 2dB. Visually the results of Glow prior are clearly even superior to BM3D recoveries that are generally blurry and over smoothed as can be spotted in the qualitative results below. To avoid fitting the noisy image using the Glow model, we force the recoveries to be natural by choosing large enough
0.01, 0.05, 1. and 0.2 are presented in Figure I0 below. The central
"The redundant 'within distribution' phrase is added to emphasize that the test set images are drawn from the same distribution as the train set. We do this to avoid confusion with the out-of-distribution recoveries also presented in this paper.
12

---

Under review as a conference paper at ICLR 2020
0.01
= 0.05
45
45
GLOW (a 0)
GLOW (o = 03
40
40
BM3D
BM3D
35
Noise Level
35
Noise Level
PSNR
GLOW
PSNR
GLOW
30
30
DCGAN
DCGAN
25
25
20
20
15
15
0
10
20
30
40
50
10
20
30
40
50
7
7
0.10
0.20
45
45
GLOW (o 0)
GLOW (a
40
40
BM3D
BM3D
35
Noise Level
35
Noise Level
PSNR
GLOW
PSNR
GLOW
30
30
DCGAN
DCGAN
25
25
20
20
15
15
0
10
20
30
40
50
20
30
40
50
7
Figure 10: Image Denoising - Recovered PSNR values as function of under Glow prior, and DCGAN prior on wiithin-distribution test set CelebA images. For reference, we show the average PSNRs of the original noisy images, and under the Glow prior in the noiseless case (a 0) in both panels. The average PSNR after applying BM3D, and the average PSNR under the Glow prior at noise levels 0.01, 0.05, 0.10, 0.20 reported.
Recall that we are solving regularized empirical risk minimization program
arg min Domainn(G)
where H(.) is monotonically |y +
In general, one can instead solve arg min
=EDomain(G))
increasing function. Figure shows the comparison of most common choices of linear (already used the rest of the paper), and quadratic in the context densoing. We find that the highest achievable PSNR remains the same in both the cases, however, the penalization parameter y has to be adjusted accordingly.
We train Glow and DCGAN on CelebA. Additional qualitative image denosing results under higher noise level σ 0.1 and 0.2 comparing Glow prior against DCGAN prior, and BM3D are presented below in Figure and 13
We also trained Glow model on Flowers dataset. Below we present its qualitative denoising perfor- mance against BM3D the test set Flowers images. We also show the effect of varying smaller leads to overfitting and vice versa.
13

---

Under review as a conference paper at ICLR 2020
BM3D
GLOW (II=I)
GLOW (II=11)
PSNR
0.0
0.5
2.0
2.5
3.0
Figure 11: Image Denoising Recovered PSNR values function of under Glow prior with
||=|| and ||2 penalization on (within-distribure test set CelebA images. Comparison is provided with BM3D denoising at noise level 0.1
DOGAN Noisy
n.S.
BADD
GLOW
0
GLOW GLOW
7
Figure 12: Image Denoising - Visual comparisons under the Glow prior, the DCGAN prior, and set images. Under DCGAN prior, we only show the case of y = 0 as this consistently gave the best performance for DCGAN, Under Glow prior, the best performance over is achieved with ? 1, overfitting of the image occurs with 0 and underfitting occurs at = 5. Note that the Glow prior with = 1 also gives a sharper image than BM3D.
BM3D at noise level 0.1 on CelebA wwithin-distribution
14

---

Under review as a conference paper at ICLR 2020
Trosh
n..
BMBD DCGAN
0
GLOW
0
GLOW GLOW
7 7= 2.5
5
Figure 13: Image Denoising Visual comparisons under the Glow prior, the DCGAN prior, and BM3D at noise level a = 0.2 on CelebA (within-d distribution) test set images. Under DCGAN prior, only show the case of y 0 this consistently gives the best performance. Under Glow prior, the best performance achieved with y 2.5, overfitting of the image occurs with y = 0 and underfitting occurs with 5. Note that the Glow prior with y 2.5 also gives sharper image than BM3D.
Truth
BMDD
GLOW GLOW GLOW
2 =
Figure 14: Image Denoising Visual comparisons under the Glow prior, and BM3D noise level 0.1 on (wittin-diistrbbutio)) test set Flowers images. Under Glow prior, the best performance is obtained with 7 = 1. Note that the Glow prior with 7 = also gives a sharper image than BM3D.
15

---

Under review as a conference paper at ICLR 2020
C COMPRESSED SENISNG: ADDITIONAL EXPERIMENTS
Some additional quantitative image recovery results on test set of CelebA dataset are presented in Figure [5] depicts the comparison Glow prior, DCGAN prior, LASSO-DCT, and LAS at compressed sensing. We plot the reconstruction error: = J +| - E where 2 is the recovered image and n 12288 is the number of pixels in the x x 3 CelebA images. Glow uniformly outperforms DCGAN, and LASSO across entire range of the number of measuremnts. LASSO- DCT and LASSO-WVY eventually catch up Glow but only when observed measurements are significant fraction of the total number of pixels. On the other hand, DCGAN is initially better than LASSO prematurely saturates due limited representation capacity.
0.10
DCGAN
reconstruction error
0.08
LASSO DCT
LASSO-WVT
0.06
0.02
0.00
0
2000 4000
6000
8000
10000
12000
no. of measurements (m)
Figure 15: Compressed sensing Reconstruction enror vs. number of measurements under Glow prior, DCGAN prior, LASSO-DCT and LASSO-W WVT on CelebA (within-distribution) test set images. Noise 7) is scaled such that E||n|| 2 = 0.01 and the penalization parameter = 0 for Glow, and DCGAN; and 7 0.01 for LASSO-DCT, and LASSO-WVE
40
GLOW
DCGAN
LASSO-DCT
LASSO-WVT
PSNR
25
20
10
200
400
600
800
1000
1200
1400
of measurements (m)
Figure 16: Compressed sensing - Zoomed-in version of the left panel of Figure in the main paper in the low measurement regime for CelebA. PSNR vs. number of measurements under Glow prior, DCGAN prior, LASSO-DCT and LASSO-WVE on the CelebA (within distribution) test set images. Noise 7/ is scaled such that v / 0.1 and the penalization parameter 7 0 for Glow and DCGAN; and 7 = 0.01 for LASSO-DCT, and LASSO-WVT.
16

---

Under review as a conference paper at ICLR 2020
35
30
PSNR
PSNR
GL LW (LBFGS)
20
Ca OW (Adam)
DCGAN
20
13
LASSO-DCT
GLOW (LBFGS)
LASSO-WVT
GLOW (Adam)
15
0
2000
4000
6000
8000
10000
12000
2000
4000
6000
8000
10000 12000
no. measurements (m)
no. of measurements (m)
Figure 17: Compressed sensing under Glow prior. Performance comparison between LBFGS and Adam solver for the inverse problem. For Adam solver, 2000 gradient steps were taken with learning rate chosen to be 0.01. The rest of the parameters were fixed to be the same as with LBFGS.
GLOW
lbfgs
2WV
800
DCGAN
adam
residual error
800
residual arrer
E00
100
800
200
200
8
10
12
iterations (t)
Iterations
Figure 18: Residual error vs. number of iterations. Left panel compares DCGAN and Glow priors. Both converge roughly the same rate to their respective saturation levels. The right panel compares LBFGS and Adam solvers for compressed sensing under Glow prior. LBFGS tends to converge far more quickly than Adam. We choose 7 0 in both the experiments.
17

---

Under review as a conference paper at ICLR 2020
Surprisingly, observe that explicit penalization of likelihood is necessary for compressive
sensing with an invertible generative prior under formulation equation 2 That is, we can take = when the optimization is initialized at 2p 0. This indicates that algorithmic regularization is occurring and that initialization plays role. We performed some additional experiments to study the role of initialization. The left panel in F shows that the norm of the latent initialization increases, the norm of the recovered latent representation increases and the PSNR of the recovered image decreases. Moreover, the right panel Figure T9 shows the norm of the estimated latent representation at each iteration of the optimization. In all our experiments, it monotonically grows versus iteration number. These experiments provide further evidence that smaller latent initializations lead to outputs that are more natural and have smaller latent representations.
110
-30.0
100
33.5
60
-35.0
50
||
31.5
||
60
34.0
20
|
40
|
33.5
20
20
80
100
120
10
25
||=0||
iterations
Figure 19: The left panel shows the average PSNR 12 test set images and norm of the optimizer as a function of the norm of the initialization for the LBFGS solver to equation 2 for Compressed sensing under Glow prior with y The initialization 20 was chosen randomly and rescaled the desired norm. The right panel shows the norm of the estimated latent representation as function of iteration number for multiple initializations. The Adam solver behaves similarly.
Recall that the natural face images correspond to smaller 2u. Figure 20 we plot the norm of the latent codes of the iterates of each algorithm the number of iterations. The central message is that initializing with smaller norm 2n tends to yield natural (smaller latent representations) recoveries. This is one explanation the affine space of solutions without penalizing the unnaturalness of the recoveries.
why in compressed sensing, is able to obtain the true solution out of
80
120
100
60
40
||a0||
S0
20
80
20-
|
zol
20
2
20l
10
15
25
30
200
400
600
800
1000
iterations (t)
iterations (t)
Figure 20: Compressed sensing Norm of the latent codes with iterations. Left panel shows how the norm of the latent codes evolves over iterations of the LBFGS solver under different size initializations. Right panel shows the same experiment for the Adam solver (although over much larger number of iterations
Adam requires comparatively more iterations to converge). Each point averaged over 12 test set images under random rescaled initializations 2μj We set the penalization parameter 7 0 in both experiments.
18

---

Under review as a conference paper at ICLR 2020
We now present visual recovery results test images from the CelebA dataset under varying number of measurements in compressed sesing. We compare recoveries under Glow prior, DCGAN prior, LASSO-DCT, and LASSO-WVT.
Truth
DCT
WVT
GLOW DCGAN
Figure 21: Compressed sensing visual comparisons - Recoveries
test set images with a number m = 200 (== 1.5%) of measurements under the Glow prior, the DCGAN prior, LASSO-WYL WVT, and LASSO-DCT at a noise level / / 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We for both DCGAN, and Glow prior and yy 0.01 for LASSO-WVI, and LASSO-DCT, respectively.
Truih
DCT
WVT
GLOW DCGAN
test set images with a number m = 300 (== 2%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level /E|in/ 0.1. In each case, we choose values of the penalization parameter " to yield the best performance among the tested values. We use y for both DCGAN, and Glow prior and y 0.01 for LASSO- W.V., and LASSO-DCT, respectively.
Figure 22: Compressed sensing visual comparisons Recoveries the
19

---

Under review as a conference paper at ICLR 2020
Truth
DCT
WVT
GLOW DOGAN
Figure 23: Compressed sensing visual comparisons Recoveries crithin-distributionn test set images with a number m = 400 (= 3%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level y / 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We y for both DCGAN, and Glow prior and 0.01 for AASO-WVT, and LASSO-DCT, respectively.
Truhh
DCT
WVT
GLOW DCGAN
test set images with a number m = 500 (== 4%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level / / 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use y for both DCGAN, and Glow prior and 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Figure 24: Compressed sensing visual comparisons Recoveries
20

---

Under review as a conference paper at ICLR 2020
Truth
DCT
WVT
GLOW DOGAN
Figure 25: Compressed sensing visual comparisons Recoveries frithin-distributionn test set images with a number m = 750 (= 6%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level y / 0.1. In each case, we choose values of the penalization parameter " to yield the best performance among the tested values. We y for both DCGAN, and Glow prior and 0.01 for AASO--VT, and LASSO-DCT, respectively.
Truhh
DCT
WVT
GLOW DCGAN
test set images with a number m 1000 (+s 8%) of measurements under the Glow prior, the DCGAN prior, AASSO-WVT, and LASSO-DCT at a noise level / / 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use for both DCGAN, and Glow prior and 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Figure 26: Compressed sensing visual comparisons Recoveries
21

---

Under review as a conference paper at ICLR 2020
Truth
DCT
WVT
GLOW DOGAN
Figure 27: Compressed sensing visual comparisons Recoveries writhin-distribution) test set images with a number m = 2500 (== 20%) of measurements under the Glow prior, the DCGAN prior, AVS, and LASSO-DCT at a noise level y /Eilzi 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use y for both DCGAN, and Glow prior and 0.01 for ASSS--VV,, and LASSO-DCT, respectively.
Truhh
DCT
WVT
GLOW DCGAN
test set images with a number m = 5000 (se 41%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level / / = 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use y for both DCGAN, and Glow prior and 0.01 for LASSO-WYT, and LASSO-DCT, respectively.
Figure 28: Compressed sensing visual comparisons Recoveries
22

---

Under review as a conference paper at ICLR 2020
Truth
DCT
WVT
GLOW DOGAN
Figure 29: Compressed sensing visual comparisons Recoveries frithin-distributionn test set images with a number m = 7500 (== 61%) of measurements under the Glow prior, the DCGAN prior, LASSO-WVT, and LASSO-DCT at a noise level / / 0.1. In each case, we choose values of the penalization parameter " to yield the best performance among the tested values. We y for both DCGAN, and Glow prior and 0.01 for ASSO-WVT, and LASSO-DCT, respectively.
Truhh
DCT
WVT
GLOW DCGAN
test set images with a number m 10, 000 (== 81%) of measurements under the Glow prior, the DCGAN prior, MAASS--VVI and LASSO-DCT at noise level / = 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use = 0 for both DCGAN, and Glow prior and 7 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Figure 30: Compressed sensing visual comparisons Recoveries
23

---

Under review as a conference paper at ICLR 2020
C.1 COMPRESSED SENSING ON FLOWER AND BIRD DATASET
We also performed compressed sensing experiments similar to those on CelebA dataset above on Birds dataset, and Flowers dataset. We trained Glow invertible network for each dataset, and present below the quantitative and qualitative recoveries for each dataset.
33
35
30
30
PSNR
PSNR
25
15
CLOW
15
GLOW
LASSO-WVT
LASSO-WVT
10
10
0
2000
4000
6000
8000
10000
12000
0
2000
4000
6000
5000
10000
12000
of measurements (m)
of measurements (m)
Figure 31: PSNR number of measurements in compressed sensing under Glow prior, LASSO- DCT and LASSO-V WVT on Birds dataset (left panel) and Flowers dataset (right panel). Noise is scaled such that E | 2 = 0.1 and the penalization parameter ? = 0 for Glow, and = 0.01 for
LASSO-DCT, and LASSO-WVT.
Truth
DCT
GLOWW WVT
Figure 32: Compressed sensing Visual comparisons (within- -iihin-distribution)) test set images from
Birds and Flowers dataset with a number m 200 (< 1.5%) measurements under Glow prior,
LASSO- WVT, and LASSO-DCT at noise level / E 1 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use 0 for Glow prior and 7 = 0.01 for LASSO-W- WVV., and LASSO-DCE respectively.
24

---

Under review as a conference paper at ICLR 2020
Truth
DCT
GLOW WVT
Figure 33: Compressed sensing Visual comparisons on (within- (rrhin-distribution) test set images from Birds and Flowers dataset with a number m 300 (== 2%) of measurements under the Glow prior,
LASSO-WVE and LASSO-DCA DCT at noise level v E 2 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use = for Glow prior and 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Truhh
DCT
GLOW WVT
Figure 34: Compressed sensing Visual comparisons on wwithin-distributinn test set images from Birds and Flowers dataset with number 400 (z< 3%) of measurements under the Glow prior, LASSO-WVY WV,, and LASSO-DCT at noise level E 1 2 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use = for Glow prior and 0.01 for LASSO-WVT, and LASSO-DCT. respectively.
Truth
DCT
GLOW WVT
Figure 35: Compressed sensing Visual comparisons on (within-diittriuutin)) test set images from Birds and Flowers dataset with number 500 (z 4%) of measurements under the Glow prior, 0.1. In each case, choose values of the penalization parameter y to yield the best performance among the tested values. We use = for Glow prior and y 0.01 for LASSO-WVT, and LASSO-DCT. respectively.
LASSO-WVE and LASSO-DCT DCT at noise level v E
25

---

Under review as a conference paper at ICLR 2020
Truth
DCT
WVT
GLOW
Figure Compressed sensing Visual comparisons on the test set images from Birds and Flowers dataset with number m 750 (== 6%) of measurements under the Glow prior, LASSO-WVT, and 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among tested values. We use 7 = 0 for Glow prior and ? 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
LASSO-DCT at noise level I
Truhh
DCT
GLOW WVT
Figure 37: Compressed sensing Visual comparisons on the test set images from Birds and Flowers dataset with number 1, 000 (z 8%) of measurements under the Glow prior, LASSO-WVT, and LASSO-DCT at noise level E | 2 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use y = 0 for Glow prior and 7 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Truth
DCT
GLOW WVT
Figure 38: Compressed sensing Visual comparisons on test set images from Birds and Flowers dataset with number 7 2, 500 (Re 20%) of measurements under the Glow prior, LASSO-WVT, and LASSO-DCC DCT at noise level / E 2 0.1. In each case, choose values of the penalization parameter 7 to yield the best performance among the tested values. We use 7 = 0 for Glow prior and y 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
26

---

Under review as a conference paper at ICLR 2020
Truth
DCT
GLOW WVT
Figure Compressed sensing Visual comparisons on the test set images from Birds and Flowers dataset with number m 5, 000 (=s %)) of measurements under the Glow prior, LASSO-W -VTT, and LASSO-DCT at noise level v E 2 0.1. In each case, we choose values of the penalization parameter y to yield the best performance among the tested values. We use 7 = 0 for Glow prior and 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Truth
DCT
WVT
GLOW
Figure 40: Visual comparisons of compressed sensing of the test set images from Birds and Flowers dataset with number 7, 500 (=s 61% 61% of measurements under the Glow prior, LASSO-WVT, and LASSO-DCT at noise level / /E|n 2 0.1. In each case, we choose values of the penalization parameter to yield the best performance among the tested values. We use 7 = 0 for Glow prior and 7 0.01 for LASSO-WVT, and LASSO-DCT, respectively.
Trush
DCT
GLOW WVT
Figure 41: Visual comparisons of compressed sensing of the test set images from Birds and Flowers dataset with number m 10, 000 (= 81%) measurements under the Glow prior, LASSO-W -VV, and LASSO-DCT at noise level E / 2 0.1. each case, we choose values of the penalization parameter to yield the best performance among the tested values. We use 7 = 0 for Glow prior and = 0.01 for LASSO-WVY WVT, and LASSO-DCT, respectively.
27

---

Under review as a conference paper at ICLR 2020
C.2 COMPRESSED SENSING ON OUT OF DISTRIBUTION IMAGES
Lack of representation error in invertible nets leads us to an important and interesting question: does the trained network fit related natural images that underrepresented even unrepresented in the training dataset? Specifically, can a Glow network trained on CelebA faces be a good prior on other faces; for example, those with dark-skin tone. faces with glasses or facial hair, or even animated faces? In general, our experiments show that Glow prior has an excellent performance on such out-of-distribution images that are semantically similar to celebrity faces but not representative the CelebA dataset. In particular, we have been able to recover faces of darker skin tone, older people with beards, eastern women, men with hats, and animated characters such as Shrek, from compressed measurements under the Glow prior. Recoveries under the Glow prior convincingly beat the DCGAN prior, which shows a definite bias due to training. Not only that, the Glow prior also outperforms unbiased methods such LASSO-DCTS and LASSO-WVI -VV.
Can we expect the Glow prior to continue to be an effective proxy for arbitrarily distiiibiittittioonn images? To answer this question, tested arbitrary natural images such car, house door, and butterfly wings that are semantically unrelated to CelebA images. In general, we found that Glow is effective prior compressed sensing of out-of-distribution natural images, which are assigned high likelihood score (small normed latent representations). On these images, Glow also outperforms LASSO.
Recoveries of natural images that are assigned very low-likelihood scores by the Glow model generally run into instability issues. During training, invertible nets learn assign high likelihood scores to the training images. All the network parameters such as scaling in the coupling layers of Glow network are learned behave stably with such high likelihood representations. However, on very low-likelihood representations, unseen during the training process, the networks becomes unstable and outputs of network begin diverge very large values; this may be due to several reasons, such as normalization (scaling) layers not being tuned to the unseen representations. An LBFGS search for the solution of an inverse problem to recover a low-likelihood image leads the iterates into neighborhoods of low- www-lielihooo representations that may lead the network to instability.
We find that Glow network has the tendency to assign higher likelihood scores to arbitrarily out- of-distribution natural images. This means that invertible networks have at least partially learned something more general about natural images from CelebA dataset - may be some high level features that face images share with other natural images such smooth regions followed by discontinuities, etc. This allows Glow prior to extend its effectiveness as prior to other natural images beyond just the training set.
Figure
45
and 46 compare the performance of LASSO-DCT, LASSO- WVT,, DCGAN
prior, and Glow prior on the compressed sensing of out-of-distribution images under varying number of measurements.
28

---

Under review as a conference paper at ICLR 2020
1, 1.000
Truth
DCT
WVT
GLOWW DCGAN
Trush
DCT
WVT
GLOW DOGAN
Figure 42: Compressed sensing (m 1000 ~ 8% of n)) visual comparisons out-of-distribution images. We compare the recoveries under Glow (trained on CelebA) prior, DCGAN (trained on CelebA) prior, LASSO-WVT, and LASSO-DCT at noise level /E||n| = 0.1. In each case, choose values of the penalization parameter y yield the best performance. We 7 for both DCGAN, and Glow prior and and optimize y for each recovery using LASSO-W WV,, and LASSO-DCT.
29

---

Under review as a conference paper at ICLR 2020
2, 5,00
Truth
DCT
WVT
GLOW DCGAN
Trush
DCT
WVT
GLOW DOGAN
Figure 43: Compressed sensing (m 2500 x 20% of n)) visual comparisons outttf-distribution images. We compare the recoveries under Glow (trained on CelebA) prior, DCGAN (trained on CelebA) prior, LASSO-WVT, and LASSO-DCT at noise level /E|nn| = 0.1. In each case, choose values of the penalization parameter y yield the best performance. We 7 for both DCGAN, and Glow prior and and optimize y for each recovery using LASSO-W VT, and LASSO-DCT.
30

---

Under review as a conference paper at ICLR 2020
5, 000
Truth
DCT
WVT
GLOWW DOGAN
Trush
DCT
WVT
GLOW DOGAN
Figure 44: Compressed sensing (m = 5000 x 41% of n)) visual comparisons out-of-distributio images. We compare the recoveries under Glow prior (trained on CelebA), DCGAN prior (trained on CelebA), LASSO- WVT, and LASSO-DCT at a noise level /E||nl? 2 = 0.1. In each case, we choose values of the penalization parameter yield the best performance. We y for both DCGAN, and Glow prior and and optimize for each recovery using LASSO- WVVT, and LASSO-DCT.
31

---

Under review as a conference paper at ICLR 2020
7,500
Truth
DCT
WVT
GLOW DCGAN
Trush
DCT
WVT
GLOW DOGAN
Figure 45: Compressed sensing (m 7500 s 61% of n)) visual comparisons out-of-distributio images. We compare the recoveries under Glow prior (trained on CelebA), DCGAN prior (trained on CelebA), LASSO-WVTS WVT, and LASSO-DCT at a noise level /E||nl? 2 = 0.1. In each case, we choose values of the penalization parameter yield the best performance. We y for both DCGAN, and Glow prior and and optimize for each recovery using LASSO- WVT, and LASSO-DCT.
32

---

Under review as a conference paper at ICLR 2020
10, 000
Truth
DCT
WVT
GLOW DCGAN
Trush
DCT
WVT
GLOWW DOGAN
Figure 46: Compressed sensing (m 10, 000, x 81% of n) visual comparisons out-of-distribution images. We compare the recoveries under Glow prior (trained on CelebA), DCGAN prior (trained on CelebA), LASSO-WVI and LASSO-DCT at a noise level /El|nl? 2 = 0.1. In each case, we choose values of the penalization parameter yield the best performance. We y for both DCGAN, and Glow prior and and optimize for each recovery using LASSO- WVTT, and LASSO-DCT.
33

---

Under review as a conference paper at ICLR 2020
D IMAGE INPAINITING
Our experiments with inpainting reveal similar story as with compressed sensing. Compared to DCGAN, the recovered PSNRs using Glow prior are much higher under appropriate 7y as depicted in the right panel in Figure 47 If improperly initialized, then performance for oy 0 could be poor.
Even if improperly initialized, sufficiently large leads to higher PSNRs.
As with compressive sensing, if the initialization is from small latent variable, then the empirical risk formulation with y exhibits high PSNRs. Algorithmic regularization is again occurring due to the small latent variable initialization.
35.0
36
GLOW
32.5
DCGAN
34
std 0.7
30.0
PSNR
27.5
PSNR
32
25.0
22.5
20.0
17.5
10
20
30
40
50
0.0
0.5
1.0
2.0
2.5
3.0
7
Figure 47: Inpainiting: PSNR (averaged over 12 test images of CelebA) vs. penalization parameter ? under Glow prior and DCGAN prior (left panel) and using different initializations under Glow prior (right panel).
We present here qualitative results image inpainting under the DCGAN prior, and the Glow prior on the CelebA test set. Compared to DCGAN, the reconstructions from Glow are of noticeably higher visual quality.
GLOW DOGAN Misked Truth
7 0
Figure 48: Image inpainiting results on CelebA test set. Masked images recovered under DCGAN prior and Glow prior. Recoveries under DCGAN prior are skewed and blurred whereas Glow prior leads to sharper and coherent inpainted images, For both Glow and DCGAN, we set ? 0.
D.1 IMAGE INPAINTING ON OUT OF DISTRIBUTION IMAGES
We now perform image inpainiting under Glow prior, and DCGAN prior each trained on CelebA. Figure 49 shows the visuals of oul-of-distribution inpainiting. As before, DCGAN continues to suffer due to representation limits and data bias while Glow achieves reasonable reconstructions out-of oft-of-distribuue distribution images semantically similar to CelebA faces. As one deviates to other natural images such houses, doors, and butterfly wings, the inpainting performance deteriorates. At compressed sensing, Glow performed much better on such arbitrarily nut-of-diitribution images as good recoveries there only require the network only assign higher likelihood score the true
34

---

Under review as a conference paper at ICLR 2020
image compared to the all the candidate static images given by the null space of the measurement operator.
Marked
DCGAN
GLOW
Masked
DCGAN
GI. .OW
Figure 49: Image inpainiting results duiti--distribution images. Masked images recovered under DCGAN prior and Glow prior. Recoveries under DCGAN prior are skewed and blurred whereas Glow prior leads sharper and coherent inpainted images. For both Glow and DCGAN, set = 0.
DISCUSSION
Figure 50 confirms the intuition brought up in the Discussion Section of the main paper that trained Glow network assigns lower likelihoods (larger latent representations) to noisy images. Histograms show that noisy images generally occupy the less likelihood regimes equivalently, the larger norm latent representations.
0,10
0.05
0.01
0.12
0:14
clean
0.12
naley
Frequend
0.08
Frequency
0.10
0.08
0.06
0.06
0.04
clean noisy
0.00
clean
0.04
0.02
noity
0.02
0.00
0.00
0.00
100
120
130
100
180
200
220
210
100
120
120
100
180
200
220 210
100
120
140
160
180
200
220
210
[|=|]
||=|]
||=|]]
Figure 50: Histograms the norm of the latent representation, z, over 3000 test images under additive Gaussian noise with 0.1 (left), 0.05 (middle), and 0.01 (right).
35

---

Under review as a conference paper at ICLR 2020
Our experiments verify that natural images have smaller latent representations than unnatural images. Here we also show that adding noise to natural images increases the norm of their latent representa- tions, and that higher noise levels result in larger increases. Additionally random perturbations in image space induce larger changes z than comparable natural perturbations in image space. Figure[51 shows plot of the test images, as function of the size of perturbation in latent space. Natural directions are given by the interpolation between the latent representation of two test images. For the denoising problem, this difference in sensitivity indicates that the optimization algorithm might obtain larger decrease in || by an image modification that reduces unnatural image components than by correspondingly large modification in natural direction.
provide evidence that
of the change in image space, averaged over 100
0.5
random direction
natural-direction
0.4
[Del
0.2
0.1
0.0
0.0
0.1
0.2
0.3
01.
0.5
0.6
0.7
0.8
D=
Figure 51: The magnitude of the change in image space as function of the size of a perturbation in latent space. Solid lines are the mean behavior and shaded region depicts 95% confidence interval.
Loss LANDSCAPE: DCGAN vs. GLOW
In Figure plot |yy AG(== + ad, + [6 versus (a,3) 3) where d, and dw are scaled have the same norm as 2", the latent representation of fixed test image. For DCGAN, we plot the loss landscape versus two pairs of random directions. For Glow, we plot the loss landscape versus a pair of random directions and a pair of directions that linearly interpolate in latent space between 2 and another test image.
DCGAN random dir. (b) DCGAN random dir.
(c) Glow random dir. (d) Glow interpolating dir.
Figure 52: Loss landscapes for ||A((=) y12 + >| with 0 around the latent representation
of fixed image and with respect to either random latent directions or latent directions that interpolate between images.
G IMAGE AND LATENT SPACE FORMULATIONS
As mentioned in the main paper, natural formulation of the inverse problem is
min
| y||² > log Pc(x),
(3)
RER
where (z) is the target density. We instead formulate the inverse problem
min AG 2113 + >||=||2;
(4)
=ER
measurement misfit combined with Gaussian prior the latent space.
36

---

Under review as a conference paper at ICLR 2020
We will denote the target distribution by (z) and the latent Gaussian distribution by p(z). To illustrate the differences between equation 3 and equation 4 we train a Real-NVP model Dinh et al. (2016) synthetic two-dimensional dataset, visualize both the log Pa) and log p(2) in latent and image space, and solve simple compressive sensing recovery problem. Our two dimensional data points generated by sampling the first coordinate 2i from bimodel Gaussian distribution and the second coordinate 22 from uniform distribution as shown in Figure 53
1.0
0.5
0.0
-0.5
1.0
1.0
0.5
0.0
0.5
1.0
D1
Figure 53: A point cloud of the synthetically generated data x E IR2
For comparison, plot the x-likelihood versus (left), latent z-likelihood versus (middle), and a-likelihood versus 2 (right) in Figure [54] These plots illustrate that generally high-likelhhood x points are also given higher latent z-likelihood, however, some low æ-likelihood might be assigned higher Gaussian z-likelihood; these are, for example, the points living on the darker contour spearing through the Gaussian bowl in the right plot. 55gure[55sshir shows some of the points the z-likelihood (left) that map to this contour in the 2-space (right).
log p(x)
log p(z)
logp(x)
1.0
1.0
4-
U
0.5
10
0.5
10
2-
6'
0.0
6
0-
20
10
0.5
20
0.5
-2
-1.0
-10
15
1.0-0.5
0.0
0.5
1.0
1.0-0.3
0.0
0.5
1.0
-2
2
Figure 54: z-likelihood versus (left), z-likelihood versus x (middle), and z-likelihood versus (right).
p(x)
log p(x)
1.0
0.5
10
10
0.0
0.5
20
20
1.0
1.0-0.5
0.0
0.5
1.0
2
0
2
21
Figure 55: Some points (red-crosses) in x-space mapped z-space. The (unwanted, corresponds to low-likelihood points) bridge connecting the models of the learned bimodal distribution is mapped the contour in the s-space.
G.1 COMPRESSIVE SENSING IN 2D
To compare latent-space formulation equation 4 and data-space formulation equation 3 we construct simple compressive sensing recovery problem for this two-dimensional data and illustrate the
37

---

Under review as a conference paper at ICLR 2020
difference under both good and bad initializations. Specifically, want to recover vector [2] x2]" from a single linear measurement y = (a, x)) 22, where a = [0 1]". Figure 56 shows the gradient descent path, and final solution, while solving equation 4 (left column), and equation 3 (right column) from a good and a bad initialization. zz-likelhhod formulation seems more robust to a bad initialization in this case compared to z-likelihood x-likelihood for some points. This bad case is carefully crafted to illustrate the difference between the two formulations, however, in practice, seems unlikely that a low z-likelihood points that somehow achieves higher z- z-likelihood will also obey the measurement constraints.
s-likelihood might not be a good proxy for
optimization in latent space
optimization in data space
1.00
grssierr deowe
1,00
2rsdeer
0.75
Treesss data
0.75
Treis data
exfinoted soleion
entivcted solucion
0.50
adittion
0.50
adutiion
0.25
0.00
0.00
0..55
-0.5
-0.50
0.50
0.755
0.75
-1.00
1.00
-1.00
-0.75
0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75 0.50
0.25 0.00
0.25
0.50
0.75
1.00
21
optimization in latent space
optimization in data space
1.00
arspeer deowe
1.00
gradeer denovee
0.75
0.75
Lrue catten
livo adatkan
0.25
0.25
catten
32
0.00
0.00
0.25
0.50
0.50
1,4
-1.00
-1.00
-1.00
-0.75
00.50
-0.2
0.00
0.25
0.50
0.75
1.00
-1.00-0.0
0..7
75
0.50
0.00
0.25
0.50
0.75
1.00
21
Figure 56: We show gradient descent path from the initialization to the final estimate along with true solution. In the first row, we initialized from (good initialization) and in the second row we used low likelihood data points intializations (bad initialization).
G.2 COMPRESSIVE SENSING FOR CELEBA
In case of CelebA images, we found that optimizing over direct likelihood of images proved very hard tune. To better understand why iquation easier compared to equation we draw the landscape of the loss surfaces of equation 4 versus and equation versus x under different 7 in two random directions around an the ground truth in z, or æ, as appropriate; see Figure 57 In the z-fformultiinn the loss surfaces (first row) have a sharp dip at the ground truths, which comes from - log PG (x) term. We believe that sharp dip in the loss surface makes difficult to tune the ? parameter, the learning rate, and makes the optimization using equation numerically more challenging as observed in our experiments. On the other hand, the loss surfaces equation (second row) appear smoother.
We now show quantitative comparison of the z- z-liielihood formulation in equation 3 and likelihood formulation in equation 4 on compressive sensing for CelebA test images versus m for fixed values of 7; see Figure 58 We initialize with
0, and zo G(zo), appropriate. We simply choose ? = in equation However, we need to choose more carefully in equation 3 and
different values of y appropriate across different undersampling ratios. Even if one ignores the difficulty of choosing the hyperparameter y, the formulation in equation generallly performs much better than equation as evident from the plots.
38

---

Under review as a conference paper at ICLR 2020
Y 50
y 100
y 200
Figure 57: Landscapes of the loss surfaces in the x-space (first row), the loss surfaces of in the 2-space (second row) for various values of y, and loss surface of æ-likelihood -log p(r) (third row).
40
35
30
PSNR
25
20
C.I
image space: 10
image space:
2000
4000
6000
8000
10000
12000
no. of measurements (m)
Figure 58: We report PSNR against number of measurements m when optimizing in the latent space equation 4 with ? = 0 and the image space equation 3 with set to 10, 50 and 100.
To show the effect of noise on recovery compressive sensing under different values and noise levels, we plot PSNR of the iterates when solving equation lggannst iterations in Figure 59 This plot shows, perhaps surprisingly, that even under noisy compressed measurements it good idea to solve the inverse compressed sensing problem equation 4 with 0.
G.3 DENOISING CELEBA
For completeness, we also compare denoising using latent space formulation equation our image space forumation equation 3 under different noise levels o = 0.05 and o 0.10; see Figure 60 and Figure 61 respectively. For both noise levels, observe equal performance (indicated by the highest PSNR) when optimizing in the latent or image space. We do not report results over 0.20 it was hard to tune hyper paremeters for higher noise levels in equation 3
39

---

Under review as a conference paper at ICLR 2020
m
5000
5000
40
35
PSNR
PSNR
30
25
U.D w/o noise
25
0.0 w/o nose
0.0 with
0.0 with moise
20
0.1 w/o naisc
0.1 w/o naise
0.1 with noise
0.1 : with moise
20
40
60
80
0
20
40
60
80
100
iterations
iterations
Figure 59: We plot PSNR against gradient iterations for compressive sensing at m = 5000 on single image under the presence and absence of noise with different values of with noise level / 0.1 (left) and m 1 (right).
optimizing in latent space
optimizing in image space
35
35
34
34
33
32
PSNR
PSNR
31
30
30
29
29
28
28
27
27
26
26
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0
20
40
60
80
100
120
140
160
7
7
Figure 60: Denoising comparision at = 0.05 when optimizing over latent space (left) versus image space (right).
optimizing in latent space
optimizing in image space
35.0
32.5
32.5
30.0
30.0
PSNR
27.5
PSNR
27.5
25.0
25.0
22.5
22.5
20.0
20.0
17.5
17.5
15.0
15.0
3
0
100
200
300
400
500
7
7
Figure 61: Denoising comparision at a = 0.10 when optimizing over latent space (left) versus image space (right).
40


Reviews:
[{"review_id": "rkxWorWy5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to employ the likelihood of the latent representation of images as the optimization target in the Glow (Kingma and Dhariwal, 2018) framework. The authors argue that to optimize the ''proxy for image likelihood'' has two advantages: First, the landscapes of the surface are more smooth; Second, a latent sample point in the regions that have a low likelihood is able to generate desired outcomes. In the experimental analysis, the authors compare their proposed method with several baselines and show prior performance.\n\nThis paper has three major flaws and should be clearly rejected. \nFirst, the novelty of this paper is trivial, in my opinion, the Eq. 2 is the only contribution of this paper.\nSecond, the experimental results are not convincing, almost all the methods proposed after 2015 have better performance compared to these baseline methods.\nThird, there are a lot of claims in this paper have been made without clarification, I have huge troubles in understanding certain sentences.\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeomuT_cB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the performance of invertible generative models for solving inverse problems. They argue that their most significant benefit over GAN priors is the lack of representation error that (1) enables invertible models to perform well on out-of-distribution data and (2) results in a model that does not saturate with increased number of measurements (as observed with GANs). They use a pre-trained Glow invertible network for the generator and solve a proxy for the maximum likelihood formulation of the problem, where the likelihood of an image is replaced by the likelihood of its latent representation. They demonstrate results on problems such as denoising, inpainting and compressed sensing. In all these applications, the invertible network consistently outperforms DCGAN across all noise levels/number of measurements. Furthermore, they demonstrate visually reasonable results on natural images significantly different from those in the training dataset.\n\nThe idea of using invertible networks for estimating a specific forward process is not new, as the authors also pointed out. The contribution of this paper is that they use a pre-trained invertible model as a prior in various tasks not known in training time and support their technique with experimental results and therefore I would recommend accepting this paper. \n\nSince one of the main arguments in the paper is how the lack of representation error benefits the Glow prior compared to DCGAN prior, it would be interesting to see the representation error quantitatively for the DCGAN results and how it contributes to the total error.  Moreover, demonstrating the comparison results in other metrics than PSNR (MSE, SSIM) would be interesting and more comprehensive.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJeePN_AFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I have read the other reviews and the author response and have not changed my evaluation.\n\nRecent work has shown that GANs can be effective for use as priors in inverse problems for images such as compressed sensing, denoising, and inpainting. A drawback is that GANs may have the problem of inexact reconstruction, and strongly reflect the biases in the training set yielding poor performance on out-of-distribution data. This paper shows that the exact inverses available to normalizing flow models and their broad assignment of likelihoods allows for better reconstructions especially on out-of-domain data.\n\nAs far as I know, this is the first work to use normalizing flows for inpainting and compressed sensing. The approach and application is very natural, although it’s a bit surprising that using the likelihood as a prior term directly did not work very well. The results of this work show that invertible generative models have utility for inverse image problems even when the quality of raw samples is substantially below GANs. In my opinion the main advantage in this method is not on having low reconstruction error on observed pixels, which becomes less of a problem for more powerful GAN models, but rather the good performance on out of domain data which is somewhat surprising. The authors are reasonably thorough, testing their model on a variety of problem settings and perform ablation studies on hyperparameters.\n\nAs additional baselines for compressed sensing and denoising, it would be good to compare to the Deep Image Prior since there is effectively no out-of-distribution input for this untrained model and it performs well with moderate image corruption. Additional discussion about the two could be useful, as for the Deep Image Prior a similar patter is observed where denoising requires explicit regularization (early stopping or gradient noise for DIP) but image completion and compressed sensing do not. Also, there have been many improvements to DCGAN over the years that might ameliorate the problems that were observed in reconstruction, but I don’t fault the authors much for this as it can be difficult training models like StyleGAN even at 64x64 sizes.\n\nIt might also be interesting to know whether the good performance on out-of-distribution inputs is due to the exact invertibility or the log-likelihood objective, although I would guess that it is the latter. On way to test this would be training the GLOW model with an adversarial objective instead of NLL as done in [1].\n\nMinor Comments:\n\nFigure 4 would probably be better with a logarithmic scaling for # of measurements\n\nI did not understand the comment about sublevel sets of the data misfit term being inverse images of cylinders, maybe this could use some elaboration.\n\n[1] https://arxiv.org/abs/1705.08868\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bye3icUIqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors extend the invertible generative model of Kingma et. al. to image inverse problems. Specifically, they use Generator trained within the Glow framework as an image prior for de-noising, inpainting and compressed sensing tasks.\nDuring training, a heuristic adjustment to the objective is made allowing optimization of latent variable norm instead of image log likelihood. This seemed critical for convergence of image inverse tasks. The use of Glow prior was shown to be beneficial for all inverse tasks. Experiments were limited to face images from celebA database. While the proposal demonstrates improved empirical performance, it seems to be the only contribution of this paper. Taking an existing model and applying it to a problem where similar extensions have been tried (GAN etc) does not seem quite worthy of a full paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the empirical performance of invertible generative models for compressive sensing, denoising and in painting. One issue in using generative models in this area has been that they hit an error floor in reconstruction due to model collapse etc i.e. one can not achieve zero error in reconstruction. The reviewers raised some concerns about novelty of the approach and thoroughness of the empirical studies. The authors response suggests that they are not claiming novelty w.r.t. to the approach but rather their use in compressive techniques. My own understanding is that this error floor is a major problem and removing its effect is a good contribution even without any novelty in the techniques. However,  I do agree that a more thorough empirical study would be more convincing. While I can not recommend acceptance given the scores I do think this paper has potential and recommend the authors to resubmit to a future venue after a through revision.
Decision: Reject