Removing the Representation Error of GAN Image Priors Using the Deep Decoder

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) show promise as priors for solving imaging inverse problems such as inpainting , compressive sensing , super-resolution , and others . For example , they have been shown to perform as well as common sparsity based priors on compressed sensing tasks using 5-10x fewer measurements , and also perform well in nonlinear blind image deblurring ( Bora et al. , 2017 ; Asim et al. , 2018 ) . The typical inverse problem in imaging is to reconstruct an image given incomplete or corrupted measurements of that image . Since there may be many potential reconstructions that are consistent with the measurements , this task requires a prior assumption about the structure of the true image . A traditional prior assumption is that the image has a sparse representation in some basis . Provided the image is a member of a known class for which many examples are available , a GAN can be trained to approximate the distribution of images in the desired class . The generator of the GAN can then be used as a prior , by finding the point in the range of the generator that is most consistent with the provided measurements . We use the term `` GAN prior '' to refer to generative convolutional neural networks which learn a mapping from a low dimensional latent code space to the image space , for example with the DCGAN , GLO , or VAE architectures ( Radford et al. , 2015 ; Bojanowski et al. , 2017 ; Kingma & Welling , 2013 ) . Challenges in the training of GANs involve selecting hyperparameters , like the dimensionality of the model manifold ; difficulties in training , such as mode collapse ; and the fact than GANs are not directly optimizing likelihood . Because of this , their performance as image priors is severely limited by representation error ( Bora et al. , 2017 ) . This effect is exaggerated when reconstructing images which are out of the training distribution , in which case the GAN prior typically fails completely to give a sensible solution to the inverse problem . In contrast , untrained deep neural networks also show promise in solving imaging inverse problems , by leveraging architectural bias of a convolutional network as a structural prior instead of a learned representation ( Ulyanov et al. , 2018 ; Heckel & Hand , 2018 ) . These methods are independent of any training data or image distribution , and therefore are robust to shifts in data distribution that are problematic for GAN priors . Recent work by Heckel & Hand ( 2018 ) presents an untrained decoder-style network architecture , the Deep Decoder , that is an efficient image representation and as a consequence works well as an image prior . In particular , it can represent images more efficiently than with wavelet thresholding . When used for denoising tasks , it outperforms BM3D , considered the state-of-the-art among untrained denoising methods . The Deep Decoder is similar to the Deep Image Prior , but it can be underparameterized , having fewer optimizable parameters than the image dimensionality , and consequently does not need any algorithmic regularization , such as early stopping . In this paper , we propose a simple method to reduce the representation error of a generative prior by studying image models which are linear combinations of a trained GAN with an untrained Deep Decoder . We build a method that capitalizes on the strengths of both methods : we want strong performance for all natural images and not just those close to a training distribution , and we want improved performance when given images are near a provided training distribution . The former comes from the Deep Decoder , and the latter comes from the GAN . We demonstrate the performance of this method on compressive sensing tasks using both in-distribution and out-of-distribution images . For in-distribution images , we find that the hybrid model consistently yields higher PSNRs than various GAN priors across a wide variety of undersampling ratios ( sometimes by 10+ dB ) , while also consistently outperforming a standalone Deep Decoder ( by around 1 dB ) . Performance improvements over the GAN prior also hold in the case of imaging on far out-of-distribution images , where the hybrid model and Deep Decoder model have comparable performance . A major challenge of the field is to build algorithms for solving inverse problems that are at least as good as both learned and recently discovered unlearned methods . Any new method should be at least as good as either approach separately . The literature contains multiple answers to this question , including invertible neural networks , optimizing over all weights of a trained GAN in an image-adaptive way , and more . This paper provides a significantly simpler method to get the benefits of both learned and unlearned methods , surprisingly by simply taking the linear combination of both models . 2 RELATED WORK . Another approach to reducing the reconstruction error of generative models has been to study invertible generative neural networks . These are networks that are fully invertible maps between latent space an image space by architectural design . The allow for direction calculation and optimization of the likelihood of any image , in particular because all images are in the range of such networks . Consequently , they have zero representation error . While such methods have demonstrated strong empirical performance ( Asim et al. , 2019 ) , invertible networks are very computationally expensive , as this recent paper used 15 GPU minutes to recover a single 64 × 64 color image . Much of their benefit may be obtainable by simpler and cheaper learned models . Alternatively , representation error of GANs may be reduced through an image adaptive process , akin to using the GAN as a warm start to a Deep Image Prior . We will make comparisons to one implementation of this idea , IAGAN , in Section 4.1.1 . The IAGAN method uses an entire GAN as an image model , tuning its parameters to fit a single image . This method will have negligible representation error , and our model achieves comparable performance in low measurement regimes while using a drastically fewer optimizable parameters . Another approach to reducing the GAN representation error could be to create better GANs . Much progress has been made on this front . Recent theoretical advances in the understanding and design of optimization techniques for GAN priors are driving a new generation of GANs which are stable during training under a wide range of hyperparameters , and which generate highly realistic images . Examples include the Wasserstein GAN , Energy Based GANs , and Boundary Equilibrium GAN ( Arjovsky et al. , 2017 ; Zhao et al. , 2016 ; Berthelot et al. , 2017 ) . Other architectures have been proposed which factorize the problem of image generation across multiple spatial scales . For example , Style-GAN introduces multiscale latent `` style '' vectors , and the Progressive Growth of GANs method explicitly separates training into phases , across which the scale of image generation is increased gradually ( Karras et al. , 2018 ; 2017 ) . In any of these examples , the demonstration of GAN quality is typically the visual appearance of the result . Visually appealing GAN outputs may still belong to GANs with significant representation errors for particular images desired to be recovered by solving an inverse problem . 3 METHOD . We assume that one observes a set of linear measurements y ∈ Rn of a true image x ∈ Rn , possibly with additive noise η : y = Ax+ η , where A ∈ Rm×n is a known measurement matrix . We introduce an image model of the form H ( ϑ ) , where ϑ are the parameters of the image representation under H . The empirical risk formulation of this inverse problem is given by min ϑ ‖AH ( ϑ ) − y‖22 ( 1 ) In this formulation , one must find an image in the range of the model H that is most consistent with the given measurements by searching over parameters ϑ . For example , Bora et al . ( 2017 ) propose to use a generative image model such as a DCGAN , GLO , or VAE , for which ϑ is a low dimensional latent code . One could also choose ϑ to be the coefficients of a wavelet decomposition , or the weights of a neural network tuned to output a single image . In our model , we represent images as the linear combination of the output of a pretrained GAN Gφ ( z ) and a Deep Decoder DD ( θ ) : H ( z , θ , α , β ) = αGφ ( z ) + βDD ( θ ) , ϑ = { z , θ , α , β } Here , the φ are the learned weights of the GAN , which are fixed . The variables that are optimized are : z , the GAN latent code ; θ , the image-specific weights of the Deep Decoder ; and scalars α and β . The first part of our image model is a GAN . We demonstrate our model ’ s performance using a BEGAN , and demonstrate the same results generalize to the DCGAN architecture . Our BEGAN has 64-dimensional latent codes sampled uniformly from [ −1 , 1 ] 64 , and we choose a diversity ratio of 0.5 . Our DCGAN has 100-dimensional latent codes , sampled from N ( 0 , 0.12I ) . The BEGAN prior is trained to output 128× 128 pixel color images and the DCGAN prior is trained to output 64× 64 color images of celebrity faces taken from the CelebA training set ( Liu et al. , 2015 ) . The GANs are initially pretrained , and the only parameters that are optimized during inversion is the latent code . The second part of our image model is a Deep Decoder . The Deep Decoder is a convolutional neural network consisting only of the following architectural elements : 1x1 convolutions , relu activations , fixed bilinear upsampling , and channelwise normalization . A final layer uses pixelwise linear combinations to create a 3 channel output . In all experiemnts , we consider a 4 layer deep decoder with k channels in each layer , where k is chosen so that |θ| < m. Thus , our deep decoder , and even our entire model , is underparameterized with respect to the image dimensionality . The deep decoder is unlearned in that it sees no training data . Its parameters θ are estimated only at test time . In our experiments , we found it beneficial to first partially solve ( 1 ) with G ( z ) only and separately with DD ( θ ) only to find approximate minimizers z∗ , θ∗ which are then used to initialize H . To maintain a fair comparison between H and other image models , we hold the number of global inversion iterations N constant . We use npre = 500 separate inversion iterations to find z∗ , θ∗ , and then initialize α = 0.5 , β = 0.5 , and continue with n = 5000 inversion iterations to optimize the parameters of H . To solve ( 1 ) with a GAN prior as in ( Bora et al. , 2017 ) , or with a proper Deep Decoder as an image prior , we simply run N = 5500 inversion steps with no interruptions . We provide details on the hyperparameters used in our experiments in Section 6.1 of the Supplemental Materials . Algorithm 1 Inversion Algorithm . Require : npre , the number of separate preinversion steps for Gφ ( z ) and DD ( θ ) . n , the number of remaining inversion steps for the hybrid model . z and θ , random initialization parameters for Gφ ( z ) and DD ( θ ) . 1 : for k = 0 , ... , npre do 2 : LG ← ‖AGφ ( z ) − y‖22 3 : z ← AdamUpdate ( z , ∇zLG ) 4 : LDD ← ‖A ( DD ( θ ) ) − y‖22 5 : θ ← AdamUpdate ( θ , ∇θLDD ) 6 : end for 7 : α← 0.5 , β ← 0.5 8 : for t = 0 , ... , n do 9 : H ← αGφ ( z ) + βDD ( θ ) 10 : L← ‖AH − y‖22 11 : z , θ , α , β ← AdamUpdate ( ϑH , ∇ϑHL ) 12 : end for

Summary: This paper presents a method for reducing the representation error generative convolutional neural networks by combining them with untrained deep decoder. The method is evaluated on compressive sensing and super-resolution, where a better performance than the isolated use of Deep Decoders and GAN priors. The main contribution of the paper is not the performance, but the simplicity of this approach.