Improved Image Generation via Sparsity

1 INTRODUCTION . The use of Generative Adversarial Networks ( GANs ) for image synthesis is one of the most fascinating outcomes of the emerging deep learning era , e.g. , Goodfellow et al . ( 2014 ) ; Radford et al . ( 2015 ) ; Zhu et al . ( 2017 ) ; Ledig et al . ( 2017 ) ; Karras et al . ( 2018 ) . GAN is a machine-learning apparatus built of two competing neural networks , a generator and a discriminator . The generator produces samples , while its adversary , the discriminator , attempts to distinguish between them and training ones . The resolution and quality of images produced by these intriguing game-theoretic machines have improved significantly in recent years , e.g. , Karras et al . ( 2019 ) ; Brock et al . ( 2019 ) . Although leading to impressive results , GANs are difficult to train and prone to undesired phenomena , such as mode collapse , failure to converge and vanishing gradients ( Thanh-Tung & Tran , 2020 ) . Much of the research in this field has been focusing on mitigating the above difficulties and on stabilizing the training process . Most of this progress has been achieved by heuristically modifying the architectures of the generator and the discriminator , and by exposing new and better behaved training losses ( Salimans et al. , 2016 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Mao et al. , 2017 ) . As such , while GANs in general have been extensively studied and redesigned , the generator itself still operates as a ” black-box ” of unjustified architecture nor meaning . Image generators within GANs are deep neural networks , comprised of a sequence of convolutionalbased blocks . Distinct from other types of deep learning architectures , generators learn a transfer function from a ( typically simple ) multivariate source distribution to a target one . To do so , each of the generator ’ s blocks performs a transitional step from its input to a latent representation . The desired overall mapping is thus achieved by a chain of these incremental steps . As mentioned above , previous work has considered the generator ’ s mapping as an end-to-end process , with hardly any attempt1 to propose a model that provides a possible explanation to the true nature of it . Motivated by the sparse modeling literature ( Elad , 2010 ) , we aim to propose a novel interpretation that sheds some light on the architecture of image generators and provides a meaningful and effective regularization to such networks . We interpret generators as implicitly relying on sparse models in general , and on the Convolutional Sparse Coding ( CSC ) and its Multi-Layered ( ML-CSC ) version in 1A notable exception to this rule is the work reported in Mahdizadehaghdam et al . ( 2019 ) , which will be described and tied to our contribution in Section 3. particular ( Szlam et al. , 2010 ; Bristow et al. , 2013 ; Chalasani et al. , 2013 ; Grosse et al. , 2012 ; Heide et al. , 2015 ; Papyan et al. , 2017b ; Papyan et al. , 2016 ; Sulam et al. , 2018 ; Sulam et al. , 2018 ) . This observation provides a possible explanation for the generator ’ s intermediate mappings . We harness this insight by proposing a general model-based approach to regularize image generators , which can be easily applied to various architectures . More specifically , we refer to the generator as a machine that synthesizes a sparse representation , followed by a multiplication by a dictionary . Accordingly , we propose several versions of sparsityinsipred regularizations , posed both as constraints or penalties , injected into the GAN training process . Under the CSC interpretation , the last layer of the generator consists of the learned synthesis dictionary , built as a set of filters . ML-CSC further assumes that a sequence of such dictionaries is multiplying the representation vector for generating the output image . We validate our proposed view by conducting extensive experiments on a variety of well-known GAN architectures , from relatively simple to up-to-date ones . Experimenting with a wide range of such networks demonstrates the versatility and generality of our approach . All the conducted tests show a performance improvement ( assessed via Fréchet Inception Distance and Inception Score ) , achieved by our simple yet effective method . We further extend our contribution by demonstrating that the same rationale and improvement are valid for other image generator neural networks . More specifically , we apply the proposed regularizations to the Deep Image Prior ( DIP ) algorithm ( Ulyanov et al. , 2018 ) , which utilizes an image generator as an implicit image prior for solving various inverse problems . We examine the effects of our approach , and show that in restoration tasks as well , performance improvement is attained . Adopting a different perspective towards our contribution , image generators are highly expressive overparametrized neural networks ( Balaji et al. , 2021 ) that solve what seems to be a severely illposed problem . Using overparametrized models with a large capacity might lead to a poor generalization and inferior results , and especially so when trained with relatively small datasets . As such , a properly chosen regularization applied to the generator may lead to benefits in both stability and overall performance . Indeed , driven by similar reasons , regularization has been adopted for the discriminator , being overparametrized as well , e.g. , Wasserstein and Spectral Normalization GANs ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ; Zhang et al. , 2019 ) . Our work offers to close this gap by introducing a meaningful such regularization for generators , thereby further stabilizing and boosting the synthesis process . To summarize , this work focuses on deep neural networks that serve as image generators , and its key contributions are the following : • We propose a novel sparse modeling interpretation of image generators for better understanding them . • We leverage this interpretation for proposing appropriate sparsity-inspired regularizations to generators . • We demonstrate the validity of our view by showing a performance boost in image synthesis with various GAN architectures . • We show that a similar performance improvement is applicable to image generators serving inverse problems , in the context of DIP . 2 BACKGROUND . 2.1 SPARSE MODELING . Sparse Modeling has been proven to be highly effective in signal and image processing applications , e.g. , ( Dabov et al. , 2007 ; Bruckstein et al. , 2009 ; Yang et al. , 2010 ; Elad , 2010 ; Dong et al. , 2011 ; 2013 ; Mairal et al. , 2014 ) . This model assumes an underlying linear generative model , according to which , a signal x ∈ RN can be described as a linear combination of few columns from a dictionary D ∈ RN×M , i.e . x = DΓ , where Γ is a sparse vector . The columns of D are referred to as atoms and they may form an overcomplete set , i.e. , M > N . Retrieval of a sparse vector Γ , corresponding to a given a signal x and a dictionary D , is referred to as sparse coding , formulated as min Γ ‖Γ‖0 s.t . DΓ = x , ( 1 ) where ‖Γ‖0 counts the non-zeros in Γ . Thus , synthesizing a signal according to this model is done by generating a sparse representation vector Γ and multiplying it by D . 2.2 CONVOLUTIONAL SPARSE CODING MODEL . When handling images , sparse modeling has been usually deployed on small overlapping patches due to the complexity of handling a general content dictionary . The Convolutional Sparse Coding ( CSC ) model ( Szlam et al. , 2010 ; Grosse et al. , 2012 ; Bristow et al. , 2013 ; Chalasani et al. , 2013 ; Heide et al. , 2015 ; Papyan et al. , 2017b ) is a global variant of the above model , which bypasses the need to operate on patches . The CSC ’ s dictionary is structured , being a concatenation of banded circulant matrices containing small support filters , each appearing in all possible shifts . Thus , D ∈ RN×mN where N is the size of the signal x and m is the number of filters , each of length n N . According to this model , a signal x can be expressed by x = DΓ , as described above ( see Figure 1 ) . The CSC has demonstrated superb performance in image processing tasks , such as denoising , separation , fusion , and super-resolution ( Gu et al. , 2015 ; Liu et al. , 2016 ; Papyan et al. , 2017a ; Simon & Elad , 2019 ; Zisselman et al. , 2019 ) . A recent work ( Papyan et al. , 2017b ) has proposed a theoretical analysis of this model , exposing the need for a redefinition of the sparsity measure to be used on Γ , in order to account for local use of atoms instead of globally counting non-zeros . We shall get back to this in the next section , when using the CSC model . 2.3 MULTI-LAYER CONVOLUTIONAL SPARSE CODING ( ML-CSC ) . ML-CSC ( Papyan et al. , 2016 ; Sulam et al. , 2018 ; Sulam et al. , 2018 ) is an extension of the CSC , which generates a cascade of sparse representations { Γi } Li=1 , corresponding to CSC dictionaries { Di } Li=1 . This model assumes that a signal x ∈ RN can be represented as x = D1Γ1 s.t . ‖Γ1‖0 ≤ λ1 , Γ1 = D2Γ2 s.t . ‖Γ2‖0 ≤ λ2 , ... ΓL−1 = DLΓL s.t . ‖ΓL‖0 ≤ λL , where { λi } Li=1 are sparsity thresholds . Thus , while the first equation perfectly aligns with the regular CSC model , the additional equations add further structure by suggesting that each sparse representation vector is by itself a CSC signal . Note that by substituting the above equations , a signal x can also be described as x = D1 · · ·DiΓi = DeffΓ , 1 ≤ i ≤ L , with intermediate sparse representations . 3 IMPROVED IMAGE SYNTHESIS VIA GANS . Deep generative models are neural network-based architectures that synthesize signals , such that their output follows the probabilistic distribution of a given data source . In computer vision , these generators are used successfully for synthesizing natural images , by training on massive datasets of photos . To this end , they map a given source distribution Pz ( typically a simple and canonical multivariate Gaussian ) to a data distribution of interest Px , G ( z ) = xgen s.t . z ∼ Pz and xgen ∼ Px . ( 2 ) Due to the complexity of image synthesis , the above mapping function is usually modeled by a highly expressive feed-forward deep convolutional neural network , consisting of several consecutive layers . In its simplest and most common form , an image generator with K layers can be rephrased as a feed-forward CNN of the form GK ( ... ( G1 ( z ) ) ) = xgen , ( 3 ) whereGi represents the ith layer of the generative model , applying convolutions , normalizations and a non-linearity ( typically ReLU ) . Thus , the overall mapping is attained in tandem of K transitional mappings . Despite the incremental nature of the deep image generator , it is treated and trained in an “ end-to-end “ mode , without understanding the purpose of the inner activations , nor enforcing a specific structure or properties on the intermediate activations . In this work , we interpret existing image generators as implicitly relying on sparse modeling , and we propose a novel model-based approach to describe and improve the synthesis process of such architectures . Since these generators are highly expressive and overparametrized , confining them can lead to superior results and facilitate the training process . To do so , we recall the convolutional sparse coding ( CSC ) and its Multi-Layer version ( ML-CSC ) models , as discussed in Section 2 . According to the CSC model , an image x can be represented as a multiplication of a sparse representation vector Γ by a convolutional dictionary D , i.e . x = DΓ . The ML-CSC further assumes that D is achieved by a multiplication of L dictionaries : Deff = D1D2 · · · DL . Note , however , that although both are generative models , there is a substantial gap between their description and the process described in Equation ( 2 ) . Whereas the CSC synthesis starts with a sparse representation vector , the image generation setup in Equation ( 2 ) starts with a dense latent random vector z . To bridge this gap , we propose to interpret image generators as performing two consecutive tasks : 1 . GS : Map the input vector z to a sparse latent vector Γ ( done by the generator ’ s first K − 1 layers ) . 2 . GI : Multiply Γ by a convolutional dictionary D , i.e. , DΓ ∼ Px ( performed by the generator ’ s Kth layer that learns a convolutional dictionary for this purpose ) . The second task is exactly the CSC synthesis process . More specifically , if D is a single convolutional dictionary , the above formulation is equivalent to the CSC , whereas if D is comprised of two or more dictionaries , this becomes the ML version of it . By splitting the generation process into two parts , we identify the role of Γ as the sparse representation in a ( ML- ) CSC-based model . This way , the image synthesis process can be described as GS ( z ) = Γ , GI ( Γ ) = x s.t . z ∼ Pz , Γ is sparse , x ∼ Px , namely , the end-to-end mapping can be separated into two consecutive mappings , where each may be easier to learn . This interpretation is further shown in Figure 3 . In the ML-CSC context , GI is comprised of two or more global convolutional dictionaries . We have analyzed the sparsity of Γ in regular adversarial training and found out that it is not sufficient . For the purpose of designing more compatible generators with both the CSC and the ML-CSC models , we encourage GS to map z to a truly sparse representation Γ . To this end , we propose few explicit techniques , serving as regularization forces applied in the training phase : ( 1 ) L1 regularization : Adding an L1 based penalty on the representations Γ to the overall loss of the image generator . ( 2 ) L0 constraint : Eliminating small non-zero entries in Γ to satisfy a predefined sparsity constraint : ‖Γ‖0 ≤ λ. Namely , the amount of non-zero entries in Γ should be less or equal to λ . This can be viewed as a projection to a constraint-satisfying tensor , obtained by zeroing the smallest absolute values of the representation . ( 3 ) L0 , ∞ inspired constraint : This pseudo-norm is based on a new sparsity measure related to the CSC Papyan et al . ( 2017b ) . ‖Γ‖0 , ∞ is the maximal number of non-zero coefficients affecting any pixel in the image x . Thus , forcing ‖Γ‖0 , ∞ ≤ λ restricts the number of local atoms to λ . While this constraint is theoretically justified in the context of CSC , projection onto it is known to be challenging ( Plaut & Giryes , 2019 ) . To approximate it in a computationally plausible manner , we use a “ needle ” -based sparsity measure ( Papyan et al. , 2017a ; Zisselman et al. , 2019 ) . In the general 2D CSC case , Γ is a 3D tensor , of size H×W×C , where H and W define the image size to be synthesized , and C is the number of filters . We define a needle as 1×1×C tensor , contained in Γ . Hence , Γ contains H×W needles , each contributes to a patch of pixels in a corresponding position in the output image . In this configuration , every pixel is affected by several adjacent needles . This setup is described in Figure 2 . We propose to limit the amount of non-zero entries in every such needle . To this end , for a given representation tensor Γ , we zero the smallest absolute values in each needle of the representation to satisfy the relaxed constraint . Although the above three options all promote sparsity in Γ , they are substantially different , as indeed will become evident in our experiments . While L1 and L0 consider global sparsity , the L0 , ∞ forces a local balance in the use of the atoms , i.e. , limiting the local density in the representation Γ . As for the difference between L0 and L1 , the first is deployed as a constraint , while the latter is used as a penalty . Since both L0 , ∞ and L0 are constraints , we implemented them as projection layers which map an input tensor to a constraint-satisfying one , by zeroing its smallest entries . Figure 4 illustrates the effect of applying the proposed regularization techniques and the differences between them in inducing sparsity . We should note that a connection between sparse modeling and GANs ’ generators has been already proposed in Mahdizadehaghdam et al . ( 2019 ) . In their work , they introduced sparsity in GANs for boosting performance , and achieved their goal by designing a specific architecture that utilizes a patch-based modeling and a pre-trained dictionary . While inspired by their view , our work differs substantially , as we propose a more general strategy that leverages more advanced models ( CSC and ML-CSC ) , does not require a pre-training of any sort , and can be easily applied to various existing GAN architectures . Indeed , as we shall see in the next section , our theme extends to other generators that go beyond GANs . To summarize the proposed theme , we argue that learning a direct mapping from random noise to natural images ’ distribution is an extremely hard task that can be mitigated by enforcing a model that provides a meaningful regularization . As in every model , the CSC introduces specific structured assumptions about the data . Since this model has been shown to be highly compatible with natural images , we believe that these assumptions empirically hold , and hence , the suggested approach will lead to better results . To verify the above statement , we conduct experiments in various types of tasks , compare and dissect the results of regularized and non-regularized generators .

Summary: The paper proposes viewing CNN's through the lens of sparse coding. In practice it mostly boils down to encouraging sparsity in the last layer activations. To this end the paper proposes three different mechanisms that promote sparsity. This is shown to improve the FID scores somewhat in one dataset with a range of different GAN architectures. The ideas are also applied to another generator-like DIP scenario.