Removing the Representation Error of GAN Image Priors Using the Deep Decoder

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) show promise as priors for solving imaging inverse problems such as inpainting , compressive sensing , super-resolution , and others . For example , they have been shown to perform as well as common sparsity based priors on compressed sensing tasks using 5-10x fewer measurements , and also perform well in nonlinear blind image deblurring ( Bora et al. , 2017 ; Asim et al. , 2018 ) . The typical inverse problem in imaging is to reconstruct an image given incomplete or corrupted measurements of that image . Since there may be many potential reconstructions that are consistent with the measurements , this task requires a prior assumption about the structure of the true image . A traditional prior assumption is that the image has a sparse representation in some basis . Provided the image is a member of a known class for which many examples are available , a GAN can be trained to approximate the distribution of images in the desired class . The generator of the GAN can then be used as a prior , by finding the point in the range of the generator that is most consistent with the provided measurements . We use the term `` GAN prior '' to refer to generative convolutional neural networks which learn a mapping from a low dimensional latent code space to the image space , for example with the DCGAN , GLO , or VAE architectures ( Radford et al. , 2015 ; Bojanowski et al. , 2017 ; Kingma & Welling , 2013 ) . Challenges in the training of GANs involve selecting hyperparameters , like the dimensionality of the model manifold ; difficulties in training , such as mode collapse ; and the fact than GANs are not directly optimizing likelihood . Because of this , their performance as image priors is severely limited by representation error ( Bora et al. , 2017 ) . This effect is exaggerated when reconstructing images which are out of the training distribution , in which case the GAN prior typically fails completely to give a sensible solution to the inverse problem . In contrast , untrained deep neural networks also show promise in solving imaging inverse problems , by leveraging architectural bias of a convolutional network as a structural prior instead of a learned representation ( Ulyanov et al. , 2018 ; Heckel & Hand , 2018 ) . These methods are independent of any training data or image distribution , and therefore are robust to shifts in data distribution that are problematic for GAN priors . Recent work by Heckel & Hand ( 2018 ) presents an untrained decoder-style network architecture , the Deep Decoder , that is an efficient image representation and as a consequence works well as an image prior . In particular , it can represent images more efficiently than with wavelet thresholding . When used for denoising tasks , it outperforms BM3D , considered the state-of-the-art among untrained denoising methods . The Deep Decoder is similar to the Deep Image Prior , but it can be underparameterized , having fewer optimizable parameters than the image dimensionality , and consequently does not need any algorithmic regularization , such as early stopping . In this paper , we propose a simple method to reduce the representation error of a generative prior by studying image models which are linear combinations of a trained GAN with an untrained Deep Decoder . We build a method that capitalizes on the strengths of both methods : we want strong performance for all natural images and not just those close to a training distribution , and we want improved performance when given images are near a provided training distribution . The former comes from the Deep Decoder , and the latter comes from the GAN . We demonstrate the performance of this method on compressive sensing tasks using both in-distribution and out-of-distribution images . For in-distribution images , we find that the hybrid model consistently yields higher PSNRs than various GAN priors across a wide variety of undersampling ratios ( sometimes by 10+ dB ) , while also consistently outperforming a standalone Deep Decoder ( by around 1 dB ) . Performance improvements over the GAN prior also hold in the case of imaging on far out-of-distribution images , where the hybrid model and Deep Decoder model have comparable performance . A major challenge of the field is to build algorithms for solving inverse problems that are at least as good as both learned and recently discovered unlearned methods . Any new method should be at least as good as either approach separately . The literature contains multiple answers to this question , including invertible neural networks , optimizing over all weights of a trained GAN in an image-adaptive way , and more . This paper provides a significantly simpler method to get the benefits of both learned and unlearned methods , surprisingly by simply taking the linear combination of both models . 2 RELATED WORK . Another approach to reducing the reconstruction error of generative models has been to study invertible generative neural networks . These are networks that are fully invertible maps between latent space an image space by architectural design . The allow for direction calculation and optimization of the likelihood of any image , in particular because all images are in the range of such networks . Consequently , they have zero representation error . While such methods have demonstrated strong empirical performance ( Asim et al. , 2019 ) , invertible networks are very computationally expensive , as this recent paper used 15 GPU minutes to recover a single 64 × 64 color image . Much of their benefit may be obtainable by simpler and cheaper learned models . Alternatively , representation error of GANs may be reduced through an image adaptive process , akin to using the GAN as a warm start to a Deep Image Prior . We will make comparisons to one implementation of this idea , IAGAN , in Section 4.1.1 . The IAGAN method uses an entire GAN as an image model , tuning its parameters to fit a single image . This method will have negligible representation error , and our model achieves comparable performance in low measurement regimes while using a drastically fewer optimizable parameters . Another approach to reducing the GAN representation error could be to create better GANs . Much progress has been made on this front . Recent theoretical advances in the understanding and design of optimization techniques for GAN priors are driving a new generation of GANs which are stable during training under a wide range of hyperparameters , and which generate highly realistic images . Examples include the Wasserstein GAN , Energy Based GANs , and Boundary Equilibrium GAN ( Arjovsky et al. , 2017 ; Zhao et al. , 2016 ; Berthelot et al. , 2017 ) . Other architectures have been proposed which factorize the problem of image generation across multiple spatial scales . For example , Style-GAN introduces multiscale latent `` style '' vectors , and the Progressive Growth of GANs method explicitly separates training into phases , across which the scale of image generation is increased gradually ( Karras et al. , 2018 ; 2017 ) . In any of these examples , the demonstration of GAN quality is typically the visual appearance of the result . Visually appealing GAN outputs may still belong to GANs with significant representation errors for particular images desired to be recovered by solving an inverse problem . 3 METHOD . We assume that one observes a set of linear measurements y ∈ Rn of a true image x ∈ Rn , possibly with additive noise η : y = Ax+ η , where A ∈ Rm×n is a known measurement matrix . We introduce an image model of the form H ( ϑ ) , where ϑ are the parameters of the image representation under H . The empirical risk formulation of this inverse problem is given by min ϑ ‖AH ( ϑ ) − y‖22 ( 1 ) In this formulation , one must find an image in the range of the model H that is most consistent with the given measurements by searching over parameters ϑ . For example , Bora et al . ( 2017 ) propose to use a generative image model such as a DCGAN , GLO , or VAE , for which ϑ is a low dimensional latent code . One could also choose ϑ to be the coefficients of a wavelet decomposition , or the weights of a neural network tuned to output a single image . In our model , we represent images as the linear combination of the output of a pretrained GAN Gφ ( z ) and a Deep Decoder DD ( θ ) : H ( z , θ , α , β ) = αGφ ( z ) + βDD ( θ ) , ϑ = { z , θ , α , β } Here , the φ are the learned weights of the GAN , which are fixed . The variables that are optimized are : z , the GAN latent code ; θ , the image-specific weights of the Deep Decoder ; and scalars α and β . The first part of our image model is a GAN . We demonstrate our model ’ s performance using a BEGAN , and demonstrate the same results generalize to the DCGAN architecture . Our BEGAN has 64-dimensional latent codes sampled uniformly from [ −1 , 1 ] 64 , and we choose a diversity ratio of 0.5 . Our DCGAN has 100-dimensional latent codes , sampled from N ( 0 , 0.12I ) . The BEGAN prior is trained to output 128× 128 pixel color images and the DCGAN prior is trained to output 64× 64 color images of celebrity faces taken from the CelebA training set ( Liu et al. , 2015 ) . The GANs are initially pretrained , and the only parameters that are optimized during inversion is the latent code . The second part of our image model is a Deep Decoder . The Deep Decoder is a convolutional neural network consisting only of the following architectural elements : 1x1 convolutions , relu activations , fixed bilinear upsampling , and channelwise normalization . A final layer uses pixelwise linear combinations to create a 3 channel output . In all experiemnts , we consider a 4 layer deep decoder with k channels in each layer , where k is chosen so that |θ| < m. Thus , our deep decoder , and even our entire model , is underparameterized with respect to the image dimensionality . The deep decoder is unlearned in that it sees no training data . Its parameters θ are estimated only at test time . In our experiments , we found it beneficial to first partially solve ( 1 ) with G ( z ) only and separately with DD ( θ ) only to find approximate minimizers z∗ , θ∗ which are then used to initialize H . To maintain a fair comparison between H and other image models , we hold the number of global inversion iterations N constant . We use npre = 500 separate inversion iterations to find z∗ , θ∗ , and then initialize α = 0.5 , β = 0.5 , and continue with n = 5000 inversion iterations to optimize the parameters of H . To solve ( 1 ) with a GAN prior as in ( Bora et al. , 2017 ) , or with a proper Deep Decoder as an image prior , we simply run N = 5500 inversion steps with no interruptions . We provide details on the hyperparameters used in our experiments in Section 6.1 of the Supplemental Materials . Algorithm 1 Inversion Algorithm . Require : npre , the number of separate preinversion steps for Gφ ( z ) and DD ( θ ) . n , the number of remaining inversion steps for the hybrid model . z and θ , random initialization parameters for Gφ ( z ) and DD ( θ ) . 1 : for k = 0 , ... , npre do 2 : LG ← ‖AGφ ( z ) − y‖22 3 : z ← AdamUpdate ( z , ∇zLG ) 4 : LDD ← ‖A ( DD ( θ ) ) − y‖22 5 : θ ← AdamUpdate ( θ , ∇θLDD ) 6 : end for 7 : α← 0.5 , β ← 0.5 8 : for t = 0 , ... , n do 9 : H ← αGφ ( z ) + βDD ( θ ) 10 : L← ‖AH − y‖22 11 : z , θ , α , β ← AdamUpdate ( ϑH , ∇ϑHL ) 12 : end for

Summary: This paper proposes to use a combination of a pretrained GAN and an untrained deep decoder as the image prior for image restoration problem. The combined model jointly infers the latent code for the trained GAN and the parameters in the untrained deep decoder. It also jointly infers the mixing coefficient alpha and beta during test time for each image, thus learning how much we should rely on GAN. The proposed hybrid model is helpful on compressed sensing experiments on the CelebA dataset; however, it is only marginally better than deep decoder on image super resolution and out-of-distribution compressed sensing.