Removing the Representation Error of GAN Image Priors Using the Deep Decoder
Authors: Max Daniels
Reinhard Heckel
Paul Hand
Venue:  (2020)
Keywords: deep decoder
deep image prior
GAN
inverse problems

Abstract: Generative models, such as GANs, have demonstrated impressive performance as natural image priors for solving inverse problems such as image restoration and compressive sensing. Despite this performance, they can exhibit substantial representation error for both in-distribution and out-of-distribution images, because they maintain explicit low-dimensional learned representations of a natural signal class. In this paper, we demonstrate a method for removing the representation error of a GAN when used as a prior in inverse problems by modeling images as the linear combination of a GAN with a Deep Decoder. The deep decoder is an underparameterized and most importantly unlearned natural signal model similar to the Deep Image Prior.  No knowledge of the specific inverse problem is needed in the training of the GAN underlying our method.  For compressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately, both on in-distribution and out-of-distribution images.  This model provides a method for extensibly and cheaply leveraging both the benefits of learned and unlearned image recovery priors in inverse problems.

Under review as a conference paper at ICLR 2020
REDUCING THE REPRESENTATION ERROR OF GAN IM- AGE PRIORS USING THE DEEP DECODER
Anonymous authors Paper under double-blind review
ABSTRACT
Generative models, such as GANs, learn an explicit low-dimensional represen- tation of particular class of images, and so they may be used natural image priors for solving inverse problems such as image restoration and compressive sensing. GAN priors have demonstrated impressive performance on these tasks, but they can exhibit substantial representation error for both in-distribution and out-of-distribution images, because of the mismatch between the learned, approx- imate image distribution and the data generating distribution. In this paper, we demonstrate a method for reducing the representation error of GAN priors by modeling images the linear combination of GAN prior with Deep Decoder. The deep decoder is an underparameterizee and most importantly unlearned natural signal model similar to the Deep Image Prior. No knowledge of the specific inverse problem is needed in the training of the GAN underlying our method. For com- pressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately, both on mn-distribution and -iitt--diitritttion images. This model provides a method for extensibly and cheaply leveraging both the benefits of learned and unlearned image recovery priors inverse problems.
INTRODUCTION
Generative Adversarial Networks (GANs) show promise priors for solving imaging inverse problems such as inpainting, compressive sensing, super-resolution, and others. For example, they have been shown to perform as well as common sparsity based priors on compressed sensing tasks using 5- 10x fewer measurements, and also perform well in nonlinear blind image deblurring (Bora) al. 2017 Asim et al. 2018). The typical inverse problem in imaging is to reconstruct an image given incomplete or corrupted measurements of that image. Since there may be many potential reconstructions that are consistent with the measurements, this task requires a prior assumption about the structure of the true image. A traditional prior assumption is that the image has sparse representation in some basis. Provided the image is a member of a known class for which many examples are available, a GAN be trained to approximate the distribution of images in the desired class. The generator of the GAN can then be used as prior. by finding the point in the range of the generator that is most consistent with the provided measurements.
We use the term "GAN prior" to refer to generative convolutional neural networks which learn mapping from low dimensional latent code space the image space, for example with the DCGAN, GLO, or VAE architectures (Radford et al. [0115] BBojanoskii et al. 2017 Kingma & Welling 2013). Challenges in the training of GANs involve selecting hyperparameters. like the dimensionality of the model manifold; difficulties in training, such mode collapse; and the fact than GANs are not directly optimizing likelihood. Because of this, their performance as image priors is severely limited by representation eиror (Bora et al. (017). This effect is exaggerated when reconstructing images which are out of the training distribution, in which case the GAN prior typically fails completely to give sensible solution the inverse problem.
In contrast, untrained deep neural networks also show promise in solving imaging inverse problems, by leveraging architectural bias of a convolutional network structural prior instead of learned representation (Ulyanov et al., 2018) Heckel & Hand 20188). These methods are independent of any training data or image distribution, and therefore are robust to shifts in data distribution that

---

Under review as a conference paper at ICLR 2020
True Images
GAN Prior
Hybrid (ours)
Figure 1: Our hybrid model, combination of a GAN prior and Deep Decoder, has significantly less representation error than the GAN Prior alone.
problematic for GAN priors. Recent work by Heckel & Hand (2018) presents untrained decoder-style network architecture, the Deep Decoder, that is an efficient image representation and as consequence works well image prior. In particular, it can represent images more efficiently than with wavelet thresholding. When used for denoising tasks, it outperforms BM3D, considered the state-of-the-arr among untrained denoising methods. The Deep Decoder similar the Deep Image Prior, but it can be underparameterized, having fewer optimizable parameters than the image dimensionality, and consequently does not need any algorithmic regularization, such as early stopping.
In this paper, we propose a simple method to reduce the representation error of a generative prior by studying image models which are linear combinations of trained GAN with an untrained Deep Decoder. We build a method that capitalizes on the strengths of both methods: we want strong performance for all natural images and not just those close training distribution, and we want improved performance when given images are near provided training distribution. The former comes from the Deep Decoder, and the latter comes from the GAN. We demonstrate the performance of this method on compressive sensing tasks using both in-distribution and out-of-distribution images. For in-distribution images, we find that the hybrid model consistently yields higher PSNRs than various GAN priors across wide variety of undersampling ratios (sometimes by 10+ dB), while also consistently outperforming a standalone Deep Decoder (by around dB). Performance improvements over the GAN prior also hold in the case of imaging on far diittr--distiitttion images, where the hybrid model and Deep Decoder model have comparable performance. A major challenge of the field is to build algorithms for solving inverse problems that are at least as good both learned and recently discovered unlearned methods. Any new method should be at least as good as either approach separately. The literature contains multiple answers this question, including invertible neural networks, optimizing over all weights of trained GAN in an image-adaptiv way, and more. This paper provides significantly simpler method to get the benefits of both learned and unlearned methods, surprisingly by simply taking the linear combination of both models.
RELATED WORK
Another approach to reducing the reconstruction еттоr of generative models has been to study invertible generative neural networks. These are networks that are fully invertible maps between latent space an image space by architectural design. The allow for direction calculation and optimization of the likelihood of any image, in particular because all images are the range of such networks. Consequently, they have zero representation error. While such methods have demonstrated strong empirical performance (Asim et al.) 2019), invertible networks are very computationally expensive, as this recent paper used 15 GPU minutes to recover single 64 x 64 color image. Much of their benefit may be obtainable by simpler and cheaper learned models.
Alternatively, representation error of GANs may be reduced through image adaptive process, akin to using the GAN as a warm start to Deep Image Prior. We will make comparisons to one implementation of this idea, IAGAN, in Section 4.1.1 The IAGAN method uses entire GAN as an image model, tuning its parameters to fit a single image. This method will have negligible
2

---

Under review as a conference paper at ICLR 2020
representation enror, and our model achieves comparable performance in low measurement regimes while using a drastically fewer optimizable parameters.
Another approach to reducing the GAN representation error could be to create better GANs. Much progress has been made on this front. Recent theoretical advances in the understanding and design of optimization techniques for GAN priors are driving a new generation of GANs which are stable during training under a wide range of hyperparameters, and which generate highly realistic images. Examples include the Wasserstein GAN, Energy Based GANs, and Boundary Equilibrium GAN (Arjovsky ct al. 2017 Zhao et al. 2016 Berthelot et al. 2017). Other architectures have been proposed which factorize the problem image generation across multiple spatial scales. For example, Style-GAN introduces multiscale latent "style" vectors, and the Progressive Growth of GANs method explicitly separates training into phases, across which the scale of image generation is increased gradually (Karras et al. 22118) 2017). In any of these examples, the demonstration of GAN quality is typically the visual appearance of the result. Visually appealing GAN outputs may still belong GANs with significant representation errors for particular images desired to be recovered by solving an inverse problem.
METHOD
We assume that one observes a set of linear measurements E R" of true image a E R", possibly with additive noise 77:
Ax + 1).
where A E m a known measurement matrix. We introduce an image model of the form (V), where I are the parameters of the image representation under H. The empirical risk formulation of this inverse problem is given by
min (1H -112
In this formulation, one must find an image in the range of the model H that is most consistent with the given measurements by searching over parameters 1. For example, [Bora et al. (2017) propose to use generative image model such as a DCGAN, GLO, or VAE, for which 1/ is a low dimensional latent code. One could also choose i to the coefficients of wavelet decomposition, or the weights of neural network tuned to output single image.
In our model, we represent images as the linear combination of the output of a pretrained GAN (2) and Deep Decoder DD(0):
Hzz, (,,00,,3)) = aGo(z) + (DD(O), 2 = f 0, a.
Here, the ó are the learned weights of the GAN, which are fixed. The variables that are optimized are: z, the GAN latent code; 0, the image-specific weights of the Deep Decoder; and scalars a and
The first part of our image model is a GAN. We demonstrate our model's performance using a BEGAN, and demonstrate the same results generalize to the DCGAN architecture. Our BEGAN has 64-dimensional latent codes sampled uniformly from 1 and we choose a diversity ratio of 0.5. Our DCGAN has 100-dimensional latent codes, sampled from (0, 0.1²1). The BEGAN prior is trained to output 128 x 128 pixel color images and the DCGAN prior is trained to output 64 x 64 color images of celebrity faces taken from the CelebA training set (Liu al. 2015). The GANs are initially pretrained, and the only parameters that are optimized during inversion is the latent code.
The second part of our image model is a Deep Decoder. The Deep Decoder is a convolutional neural network consisting only of the following architectural elements: 1x1 convolutions, relu activations, fixed bilinear upsampling, and channelwise normalization. A final layer uses pixelwise linear combinations to create 3 channel output. In all experiemnts, we consider 4 layer deep decoder with & channels in each layer, where & is chosen so that (0] < m. Thus, our deep decoder, and even entire model, is underparameterized with respect to the image dimensionality. The deep decoder unlearned in that it sees no training data. Its parameters are estimated only at test time.
In experiments, found it beneficial to first partially solve I with G(z) only and separately with (D) only to find approximate minimizers 2", 0* which are then used to initialize H. To maintain fair comparison between H and other image models, we hold the number of global inversion iterations N constant. We use Npre = 500 separate inversion iterations to find =",0", 8", and
3

---

Under review conference paper at ICLR 2020
CAN PMA
DV Noir
Figure The model includes parameters a, 8, 0, and 2, which together comprise the image representation enforced by our Hybrid model. The final output information is a learned linear combination of the two component images.
then initialize 0.5, 3 = 0.5, and continue with = 5000 inversion iterations to optimize the parameters of H. To solve ( with a GAN prior as in (Bora al., 2017), or with proper Deep Decoder as an image prior, we simply run N 5500 inversion steps with no interruptions. We provide details on the hyperparameters used our experiments in Section 6.1 the Supplemental Materials.
Algorithm 1 Inversion Algorithm.
Require:
Plrr. the number of separate preinversion steps for (z and (D( the number of remaining inversion steps for the hybrid model. and 0, random initialization parameters for Gz(2) and DD(
1: for = O..... Ppr do
2:
La + (=) y
3:
+ AdamUpdate V LG)
4:
LDD + |IADD y
5:
At AdamUppdate(( A VOLDD)
6: end for
7: a 0.5, 0.5
8: for = 0, n2 do
9:
H a
(DD(O)
10:
Lt ||H -y1
11: 2,0,a, 3 + Adam ( Van 1nn1 L)
end for
EXPERIMENTS
In this section, show that our hybrid prior consistently outperforms both the individual priors, namely the GAN prior and the Deep Decoder, both for compressive sensing as well as for super- resolution.
4.1 COMPRESSED SENSING
In compressed sensing, one must reconstruct signal given m < n. linear measurements of the image. We study compressed sensing with Gaussian measurement matrices, so that the measurement matrix A E m has i.i.d. Gaussian entries drawn from (0,, 1/m). We choose 7) E R" to be a Gaussian 0.1. Our reported PSNR values averaged over 12
noise vector, normalized that /E[[|]] [ random CelebA test set images.
In Figure 3 we compare the performance of Deep Decoder, GAN prior, and the proposed hybrid model. For our GAN prior, we use the BEGAN architecture, and we demonstrate similar results with the DCGAN architecture in the supplemental materials (Radford et al. 2015: Berthelot et al. (017)) Replicating the results in Bora et al. (2017), we observe the reconstruction quality the GAN prior quickly plateaus
increases, illustrating its performance is limited by representation error. The Deep Decoder yields significantly higher PSNRs (sometimes by 10+ dB) when m not small. We

---

Under review as conference paper at ICLR 2020
BEGAN Low Megsurement Regime
BEGAN - Full Measurement Regime
25.0
Hybrid
35.0
Hybrid
12.9
BEGAN
32.5
BEGAN
Recondtrccion PSNR (05)
(45)
10.0
30.0
27.5
27.5
25.0
22.5
20.0
17.5
2000
4000
COOC
5000
10000
25000
Number messurements
Number
DCGAN Lew Measurement Regime
DCGAN - Full Measurement Regime
Hybrid
Hyybid
35
D0
35
DD
DOGAN
DOGAN
Reconstr actio PSA fo8t
Reconstrattion PSW tuBy
23
25
20
1000
2000
3000
4000
500G
2000
4000
8000
8000
13000
12000
Number of measurements
Number of measurements
Figure 3: Reconstruction PSNRs versus measurement numbers for in-distribution test images for our hybrid model, a Deep Decoder, and a GAN prior. The left panels zoom in to the low measurement regime of the right panels. Our hybrid model is able yield higher PSNRs than both of its components, the Deep Decoder and a GAN, on in-distribution test images, in all but the lowest measurement regime. The effect is replicated both for the BEGAN (top row) and the DCGAN (bottom row).
observe that the hybrid model yields higher PSNRs than the Deep Decoder, usually by around dB, at all subsampling ratios. Further, in all but the smallest m regime, the hybrid model outperforms the GAN models. is able to improve consistently when it has access to more measurements. The hybrid model is the best of all, indicating that despite the limited performance of GAN priors, they are still able to provide useful structure which the Deep Decoder cannot replicate. In particular, the Deep Decoder is biased to low frequency information in the image signal, and has smoothing effect which increasingly prominent with fewer measurements. Because a GAN prior learns an approximation of the true data manifold, is biased to represent semantically relevant features of variation in the images in its training dataset. In the case of celebrity faces, the GAN prior can well represent features like eyes, noses, and mouths, which are especially difficult for a Deep Decoder as they have complicated structure contained in small spatial extent.
We show a sample of the reconstructed images for m 2500 in Figure along with breakdown of the component images from the GAN and Deep Decoder components of the hybrid model for an individual image. More samples are available in the supplemental material.
We next compare the performance of the GAN Prior, Deep Decoder, and hybrid model on images which are outside the training distribution of the GAN prior, shown in Figure 5 We use the same BEGAN trained on CelebA faces, but in this experiment, we reconstruct images of birds from the Caltech-USD Birds 200 dataset (Welinder et al. 2010). The hybrid model's performance no longer significantly better than the Deep Decoder, because the GAN prior is unable to represent useful image features for this new distribution. Additionally, we observe that the cocfficient a the GAN prior in the mixture H is diminished, that the hybrid model "filters out" the irrelevant information from the GAN prior.
5

---

Under review conference paper at ICLR 2020
True Image
BEGAN Compressive Sensing, to 2500
"Tee Imraage
EE-OOL Deep Drecodrr Hjardd
Damp Deundr
Hyleil
0,3988
Figure 4: Left: Samples of reconstructed images for m 2500 measurements, compression ratio of 0.051. The GAN prior has significant representation error, to the point where appears to recover the face of the wrong person. The Deep Decoder has artifacts arising from too much smoothing of facial features. The hybrid model has sharply defined features, as does GAN, without the unnecessary smoothness of the Deep Decoder by itself. Righr: A comparison of output examples for the Deep Decoder and hybrid models, along with the two components underlying the hybrid model. The difference is detail is noticeable between the pure Deep Decoder and the Hybrid model.
BEGAN Out of Distribution
BEGAN Linear Combination Coefficients
Hybrid
32.5
DD
Reconstruction PSNR (65)
30.0
Average Velue on Test Se:
0.8
27.5
0.6
8. DD Weight Out Dist
9. DD Weight hn Dist.
GAN Weight In Dist.
22.5
0.4
GAN Weight Dist.
0.2
17.5
15.0
0.0
0
5000
10000
15000
20000 25000
5000
10000 15000
20000 25000
Number of measurements
measurements
Figure 5: Left: Performance of various image models on compressed sensing of images of birds, using a GAN prior trained to generate images of celebrity faces. Surprisingly, the hybrid model still marginally outperforms the Deep Decoder, as the GAN prior learns some general image statistics which are beneficial. Right: Coefficients of the GAN and Deep Decoder in the hybrid H. Darker colors correspond to images out of the GAN prior training distribution. As one would expect, the coefficient of the GAN prior is diminished when reconstructing images for which the GAN has not learned relevant features.
4.1.1 COMPARISON TO OVERPARAMETRIZED IMAGE MODELS
We compare our model to an overparametrized alternative, drawing from recent work on the Deep Image Prior (DIP) (Ulaanov et al. 2018) and Image Adaptive GAN (IAGAN) (Abu Hussein et al. 2019). The Deep Image Prior is an untrained -ecoddr-decoder architecture with fixed input whose weights are optimized from random initialization to minimize T The IAGAN begins with a GAN prior Gz(=)) and optimizes only over to find the latent code =* so that Gz(z*) minimizes [ Then, they reduce representation error by jointly optimizing over the GAN weights o learned from training data and the latent code, initialized at 2* In contrast the DIP. IAGAN uses only decoder. Both architectures use orders of magnitude more parameters than the dimensionality they vastly overparametrized when fitting single image. Our BEGAN has roughly 18 106 parameters, and we find it can fit measurements in all regimes with near perfect accuracy.
their output, so

---

Under review as a conference paper at ICLR 2020
BEGAN In Distribution
BEGAN- Cut of Distribution
35.0
Hybrid
Hybrid
DD
32.5
BEGAN
BEGAN
GAN az DIP
GAN as DIP
Recostruction PSNR
Reconstruction PSNNR
27.5
25.0
22.5
20.0
17.3
15.0
15.0
0
2000
4000
6000
800O
10000
0
2030
4000
6200
8000
10000
Number mensurements
Number of measurements
Figure 6: Left: Reconstruction quality for images of celebrity faces, using GAN priors trained on celebrity faces. There is range of measurement regimes where the hybrid model outperforms the GAN as DIP model, which has significantly many more parameters. The GAN as DIP method uses significantly more parameters in its image representation, since one must optimize over all weights of a GAN. In the low measurement regime, this has no benefit in comparison to the underparametrized hybrid model. Right: Reconstruction quality for images of birds, using GAN priors trained on celebrity faces. In this case of outttr--iittibuttio images, the range of measurements for which the hybrid model outperforms the GAN as DIP is is even more pronounced, demonstrating the versatility of the hybrid method.
We compare our hybrid model to an overparametrized alternative, which we call "GAN as DIP." The GAN as DIP method is similar to IAGAN in that we begin with a GAN prior and optimize latent code zt which minimizes ( We then optimize over the weights and latent code of the GAN prior to fit an image. However, we break from the IAGAN method in that we choose to omit any post-inversion steps, such as BM3D denoising or backprojection (Abu Hussein et al.) 2019). This method similar to DIP that is an overparametrized CNN, but it is not an encoder-decoder architecture. Hyperparameter details may be found in Section 6. of the Supplemental Materials.
We observe that that our hybrid image model outperforms the GAN as DIP for a certain range of m, excluding the case of extremely few measurements and the case of very many meausrements. In this range, it can outperform GAN as DIP by around 1.5 dB. This performance improvement is notable because the hybrid model has significantly fewer parameters than the GAN as DIP.
5 DISCUSSION
In this paper, we introduced a new hybrid model for natural images, consisting of a trained part and an untrained part; specifically our model linearly combines a GAN and a Deep Decoder. We demonstrate that this hybrid model yields higher PSNRs at compressive sensing than either the underlying GAN model (by 10+ dB) or the underlying Deep Decoder (by 1-2 dB) all but the most extreme levels of undersampling, even when tested on images that in-distribution relative to the distribution the GAN was trained on. Interestingly, we further demonstrate that the hybrid model maintains a slight edge over the Deep Decoder even on out-of-distributin image superresolution.
images. Similar performance is demonstrated for natural point of comparison of this hybrid model is the IAGAN, which uses trained GAN as a warm start to Deep Image Prior. Our model can exhibit improved PSNRs by 2.5 dB over the IAGAN at appropriate undersampling ratios in compressive sensing. This work illustrates that complicated expensive workarounds are not needed in order to build an image model that is as good as trained generators and as good as untrained models. It is unexpected that simply taking the linear combination of these two models would yield benefits over both models separately.
The strength of the proposed hybrid model derives from how the strengths of the unlearned deep decoder compensate for the weaknesses of the GAN and vice versa. The primary weakness of GANs an inherent representation error, particular significant for (slight) out of distribution images, that

---

Under review as a conference paper at ICLR 2020
can be attributed primarily to the fact that the prior pertains to a particular distribution of images. The primary strength of Deep Decoder is that it has minimal representation error for natural images, and no bias towards natural images over the other. The Deep Decoder is underparameterizze neural network that has been empirically shown to be effective at modeling natural images while incapable of fitting noise (unlike the Deep Image Prior, which consequently requires early stopping). However, since the deep decoder does not incorporate any learned information of a particular image class, can't tan't exploit such additional information. Our hybrid model appears to combine the best from the two worlds: it has extremely small representation error, inherited from the deep decoder component, but the same time enables exploiting potential prior knowledge about image class. As the deep decoder is unlearned, it is not tied to any particular training distribution, which is a strength, but it consequently loses out on improvements that should be possible from training. For example, a deep decoder may smooth out regions of an image where there should be edges, as the Deep Decoder does not know from learning that edges may be natural. These improvements are exactly what the GAN provides in the hybrid model.
Philosophically, the hybrid model views the GAN as providing a base image and views the deep de- coder as learning residual between that base image and what is needed to fit provided measurements. As we see, often the GAN outputs a face that is similar to the target face, but with incorrect skin hues, hair details, etc., which the Deep Decoder then corrects. Empirically, we cases where the output of the GAN component of the hybrid model looks like the a different person, but the hybrid model looks like the same person the target image.
Parameter Counts
Hybrid
DD
GAN
GAN as Deep Image Prior
BEGAN
19583
19519
64
2592192
DCGAN
9644
9544
100
3576704
Figure 7: Parameter counts of image representations for image models used in our compressed sensing experiments. We report the maximum parameter count of the Deep Decoder and hybrid models throughout our experiments, but the actual parameter count varies so that the Deep Decoder and hybrid models remain underparametrized with respect to the number of measurements. The BEGAN models represent 128px images, with 49152 degrees of freedom. The DCGAN model represents 64px images, with 12288 degrees of freedom.
One strength of our proposed hybrid model is that it has relatively few parameters. This makes computations particular cheap and would be expected to improve robustness to noise, as was noticed with the plain Deep Decoder. In the experiments we report, the hybrid model is underparameterizzed in that it has fewer parameters to optimized (once its GAN component is fixed) than the number of pixels in the image. See Figure 7 for the number of parameters for each of the approaches studied. The balance between having very small representation error while being underparameterizee primarily stems from the Deep Decoder component of the hybrid model. Now we can compare to alternative approaches to having small or zero representation error. One approach for inverse problems which has zero representation error as been using trained invertible neural networks as image priors. Unfortunately, because such networks are fully invertible, they have an extreme number of parameters to optimize, resulting in very expensive optimization problems. Another approach is to use the trained generative model as a warm start to Deep Image prior (the IAGAN). While cheaper than invertible networks, this model still is significantly overparameterized because all the generator weights can be updated at inversion time. In comparison to both of these perspectives, our proposed hybrid model has significantly fewer parameters to optimize.
This work gets at an important question in the field of inverse problems using generative models: how can we capitalize on the benefits of learning a signal distribution with the benefits of unlearned methods that apply to many signal distributions. The methods we use for inversion in practice should be at least as good as both of these categories of methods across a variety of use cases (low number of measurements training data available). This paper shows that it is possible to combine the benefits of both learned and unlearned methods when solving problems both in-distribution and
high number of measurements, low noise vs. high noise, variations in amount of

---

Under review conference paper at ICLR 2020
REFERENCES
Shady Abu Hussein, Tom Tirer, and Raja Giryes. Image-Adaptive GAN based Reconstruction. arXiv e-prints, art. arXiv: 190..099,,,, Jun 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv e-prints, art. arXiv:1701.07875. Jan 2017.
Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Solving bilinear inverse problems using deep generative priors. CoRR, htss/802.044733 2018. UBLLttt:.//axxx...rrr/bbbb// 04073
Muhammad Asim, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. arXiv e-prints, art. urXiv:1905.11672, May 2019.
David Berthelot, Thomas Schumm, and Luke Metz. BEGAN; Boundary Equilibrium Generative Adversarial Networks. arXiv e-prints, art. arXiv:1703.1077 Mar 2017.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the Latent Space of Generative Networks. arXiv e-prints, art. MtXiv:1107.05777 Jul 2017.
Ashish Bora, Ajil Jalal, Eric Price, and Alexand ros G. Dimakis. Compressed Sensing using Generative Models. arXiv e-prints, art.
Mar 2017.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks. CoRR, abs/18 810.09989, 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv e-prints, art. arXiv:1710.00196, Oct 2017.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. arXiv e-prints, art. arXiv:1812.04948, Dec 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. 1rXiv:1412.660 Dec 2014.
Diederik P Kingma and Max Welling, Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114 Dec 2013.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv e-prints, art. arXiv:1511.06434 Nov 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. pp. 9446-9454, 2018.
Welinder. Branson, Mita, C. Wah, Schroff, Belongic, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001 California Institute of Technology, 2010.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based Generative Adversarial Network. arXiv e-prints, art. arXiv: 1609.1000000 10090031100 03126, 126, Sep 2016.

---

Under review conference paper at ICLR 2020
APPENDIX
HYPERPARAMETERS
We minimize T iteratively using the Adam optimizer (Kingma & Ba 2014). Unless otherwise stated, all experiments use a learning rate a 10- 31 0.9, and 3 = 0.999.
To invert the GAN DIP model, iterations to find 2" and then n 5000 iterations optimizing over the latent code and all weights of the GAN prior. We significantly smaller learning rate, 10-4, optimize the weights of
the conditions the hybrid model, using Mpr 500
the GAN prior.
6.2 SAMPLE SHEETS
BEGAN Compressive Sensing, 24576
True Imaaee
Hybrid
Deep Decoder
Overparametrized BEGAN
10

---

Under review conference paper at ICLR 2020
BEGAN Compressive Sensing, m 7500
True Image
Hybrid
Deep Decoder
Overparamentzeed BEGAN
BEGAN Compressive Sensing, m 5000
True Image
Hybrid
Deep Decoder
Overparamennzee BEGAN
11

---

Under review conference paper at ICLR 2020
BEGAN Compressive Sensing, m 1000
True image
Hybrid
Deep Decoder
Overparamettrzed BEGAN
12


Reviews:
[{"review_id": "rkxEiTqbqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for reducing the representation error generative convolutional neural networks by combining them with untrained deep decoder. The method is evaluated on compressive sensing and super-resolution, where a better performance than the isolated use of Deep Decoders and GAN priors. The main contribution of the paper is not the performance, but the simplicity of this approach.\n\n\n\nFor the title, I would suggest to replace the word Removing with Reducing.\nFurthermore, the clarification of \"GAN prior\" is very nice in the introduction, maybe you could already clarify it in the abstract.\n\nYou should perform a critical grammar check. There are too many commas, for example:\n\"At sufficiently difficult superresolution problems, the Hybrid model outperforms, the Deep\nDecoder, Bicubic upsampling, the BEGAN prior, and the BEGAN as DIP prior.\" -> there should be no comma after \"outperforms\"\nThe sentence from Page 3 to 4 reads strangely, probably a word is missing after \"For our GAN prior, we use the BEGAN architecture, and we demonstrate similar results\"\n\"Philosophically, they hybrid\" -> \"Philosophically, the hybrid\"\n\nFig 6 caption - shouldn't it be 49152 instead of 49512?\n\nYou perform various very good analysis experiments, which is well appreciated. Still, it would be good to think about some more experiments (and include at least one of them in the paper):\n1. You compare to IGAN and show that you achieve similar performance. You describe that a state-of-the-art approach are invertible generative models and that they are very time consuming (e.g., 15 minutes for a 64x64 image). How good would the invertible models be in terms of performance? Could you perform tests as well?\n2. It would be great if you report the runtime of all experiments as well - maybe also the memory usage.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlxZVMlqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to use a combination of a pretrained GAN and an untrained deep decoder as the image prior for image restoration problem. The combined model jointly infers the latent code for the trained GAN and the parameters in the untrained deep decoder. It also jointly infers the mixing coefficient alpha and beta during test time for each image, thus learning how much we should rely on GAN. The proposed hybrid model is helpful on compressed sensing experiments on the CelebA dataset; however, it is only marginally better than deep decoder on image super resolution and out-of-distribution compressed sensing.\n\nDetailed comments:\n-\tThe writing is clear and I was able to understand the model part of the paper. The algorithm box is helpful. However, I would still appreciate if the authors can provide an overall model figure in the model section to help understanding. \n-\tJointly learning the mixing coefficient is an interesting part of the model.\n-\tThe motivation in the abstract and intro could be strengthened. A smaller version of Figure 1 can be probably moved to the beginning of the paper to illustrate the problem of GAN. But even with the help of Figure 1, it is still unclear what is the fundamental problem for GAN. Simply combining a GAN with an untrained decoder model doesn’t help elucidate the source of the problem. \n-\tThe proposed Hybrid model seems to help on compressed sensing experiments on CelebA. However, it doesn’t help much on out-of-distribution experiments. Moreover, in the super-resolution task, as shown in Figure 5, the improvement over deep decoder is also not significant.\n-\tThe out-of-distribution experiments seems lack of thorough study. In particular, the paper only studies the transfer between CelebA -> Caltech-UCSD Bird dataset. It would be better if the paper can study a variety of other image datasets as well. Also some visualization on the Bird dataset should also be included.\n-\tEffect of n_pre needs to be further investigated. Why not directly train both models together? It would be good if the authors could comment on how sensitive the n_pre is and what is the intuition.\n-\tFor figures, I would recommend rename “Hybrid” to “Hybrid (Ours)” to highlight the paper’s contribution, and use a brighter color.\n-\tFigure 5 should be renamed as a Table. \n-\tHyperparameter details should be moved to the Experiment section.\n\nConclusion:\nThe paper proposes a simple combination of a trained GAN and an untrained decoder model for the task of image restoration. Although the method is clear and straightforward, in the experiments, the influence of the new model component seems marginal. Moreover, the motivation is not strong enough. Therefore, I recommend weak reject.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJgwdLAaKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Update after rebuttal period]\nI am sorry that the response cannot address my confusion. I still doubt the motivation of this paper and the actual experimental performance compared with state-of-the-art methods are still ignored. Thus I decrease my score.\n\n[Original reviews]\nThis paper proposed to modeling image as the combination of a GAN with a Deep Decoder, to remove the representation error of a GAN when used as a prior in inverse problems. The proposed methods are evaluated on two image restoration tasks, including compressive sensing and image super-resolution. The effectiveness of the combination is also presented.\n\nAuthors devote themselves to remove the representation error of the GAN image prior. Intuitively, the manner of the proposed linear combination model is rough and less reasonable. In Alg.1, the detailed algorithmic process is presented, it is clear that authors need to pre-train the used GAN and Deep Decoder, then combine them to train one network. If the motivation of this paper is to remove the representation error of GAN, GAN should be viewed as the main body. However, the authors view GAN and Deep Decoder as the same position against the original intention.\n\nAdditionally, in the experimental part, the ablation studies indeed reflect the effectiveness of the proposed algorithm, but it looks like the Deep Decoder plays a key role in all cases. In other words, the GAN image prior just plays a supporting role. This is also far away from motivation.\n\nMore importantly, I cannot see the surprising results because of this work only compare themselves with some basic version or naïve methods. All state-of-the-art approaches to different tasks are ignored, which is the other big disadvantage. \n\nIn Table 2, what is the meaning of ‘CSGM’？ The authors should describe it.\n\nIn a word, from the algorithmic and experimental perspective, this paper cannot achieve satisfying performance.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for removing what they call representation error and apply the method to super resolution and compressive sensing. 

The reviewers have provided constructive feedback. The reviewers like aspects of the paper but are also concerned with various shortcomings. The consensus is that the paper is not ready for publication as it stands.

Rejection is therefore recommended with strong encouragement to keep working on the method and submit elsewhere.
Decision: Reject