GANs Can Play Lottery Tickets Too

1 INTRODUCTION . Generative adversarial networks ( GANs ) have been successfully applied to many fields like image translation ( Jing et al. , 2019 ; Isola et al. , 2017 ; Liu & Tuzel , 2016 ; Shrivastava et al. , 2017 ; Zhu et al. , 2017 ) and image generation ( Miyato et al. , 2018 ; Radford et al. , 2016 ; Gulrajani et al. , 2017 ; Arjovsky et al. , 2017 ) . However , they are often heavily parameterized and often require intensive calculation at the training and inference phase . Network compressing techniques ( LeCun et al. , 1990 ; Wang et al. , 2019 ; 2020b ; Li et al. , 2020 ) can be of help at inference by reducing the number of parameters or usage of memory ; nonetheless , they can not save computational burden at no cost . Although they strive to maintain the performance after compressing the model , a non-negligible drop in generative capacity is usually observed . A question is raised : Is there any way to compress a GAN model while preserving or even improving its performance ? . The lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) provides positive answers with matching subnetworks ( Chen et al. , 2020b ) . It states that there exist matching subnetworks in dense models that can be trained to reach a comparable test accuracy to the full model within similar training iterations . The hypothesis has successfully shown its success in various fields ( Yu et al. , 2020 ; Renda et al. , 2020 ; Chen et al. , 2020b ) , and its property has been studied widely ( Malach et al. , 2020 ; Pensia et al. , 2020 ; Elesedy et al. , 2020 ) . However , it is never introduced to GANs , and therefore the presence of matching subnetworks in generative adversarial networks still remains mysterious . To address this gap in the literature , we investigate the lottery ticket hypothesis in GANs . One most critical challenge of extending LTH in GANs emerges : how to deal with the discriminator while compressing the generator , including ( i ) whether prunes the discriminator simultaneously and ( ii ) what initialization should be adopted by discriminators during the re-training ? Previous GAN compression methods ( Shu et al. , 2019 ; Wang et al. , 2019 ; Li et al. , 2020 ; Wang et al. , 2020b ) prune the generator model only since they aim at reducing parameters in the inference stage . The effect of ∗Equal Contribution . 1 pruning the discriminator has never been studied by these works , which is unnecessary for them but possibly essential in finding matching subnetworks . It is because that finding matching subnetworks involves re-training the whole GAN network , in which an imbalance in generative and discriminative power could result in degraded training results . For the same reason , the disequilibrium between initialization used in generators and discriminators incurs severe training instability and unsatisfactory results . Another attractive property of LTH is the powerful transferability of located matching subnetworks . Although it has been well studied in discriminative models ( Mehta , 2019 ; Morcos et al. , 2019 ; Chen et al. , 2020b ) , an in-depth understanding of transfer learning in GAN tickets is still missing . In this work , we not only show whether the sparse matching subnetworks in GANs can transfer across multiple datasets but also study what initialization benefits more to the transferability . To convert parameter efficiency of LTH into the advantage of computational saving , we also utilize channel pruning ( He et al. , 2017 ) to find the structural matching subnetworks of GANs , which enjoys the bonus of accelerated training and inference . Our contributions can be summarized in the following four aspects : • Using unstructured magnitude pruning , we identify matching subnetworks at 74 % sparsity in SNGAN ( Miyato et al. , 2018 ) and 67 % in CycleGAN ( Zhu et al. , 2017 ) . The matching subnetworks in GANs exist no matter whether pruning discriminators , while the initialization weights used in the discriminator are crucial . • We show that the matching subnetworks found by iterative magnitude pruning outperform subnetworks extracted by randomly pruning and random initialization in terms of extreme sparsity and performance . To fully exploit the trained discriminator , we using the dense discriminator as a distillation source and further improve the quality of winning tickets . • We demonstrate that the found subnetworks in GANs transfer well across diverse generative tasks . • The matching subnetworks found by channel pruning surpass previous state-of-the-art GAN com- pression methods ( i.e. , GAN Slimming ( Wang et al. , 2020b ) ) in both efficiency and performance . 2 RELATED WORK . GAN Compression Generative adversarial networks ( GANs ) have succeeded in computer vision fields , for example , image generation and translation . One significant drawback of the generative models is the high computational cost of the models ’ complex structure . A wide range of neural network compression techniques has been applied to generative models to address this problem . There are several categories of compression techniques , including pruning ( removing some parameters ) , quantization ( reducing the bit width ) , and distillation . Shu et al . ( 2019 ) proposed a channel pruning method for CycleGAN by using a co-evolution algorithm . Wang et al . ( 2019 ) proposed a quantization method for GANs based on the EM algorithm . Li et al . ( 2020 ) used a distillation method to transfer knowledge of the dense to the compressed model . Recently Wang et al . ( 2020b ) proposed a GAN compression framework , GAN slimming , that integrated the above three mainstream compression techniques into a unified form . Previous works on GAN pruning usually aim at finding a sparse structure of the trained generator model for faster inference speed , while we are focusing on finding trainable structures of GANs following the lottery ticket hypothesis . Moreover , in existing GAN compression methods , only the generator is pruned , which could undermine the performance of re-training since the left-out discriminator may have a stronger computational ability than the pruned generator and therefore cause a degraded result due to the imparity of these two models . The Lottery Ticket Hypothesis The lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) claims the existence of sparse , separate trainable sub-networks in a dense network . These subnetworks are capable of reaching comparable or even better performance than full dense model , which has been evidenced in various fields , such as image classification ( Frankle & Carbin , 2019 ; Liu et al. , 2019 ; Wang et al. , 2020a ; Evci et al. , 2019 ; Frankle et al. , 2020 ; Savarese et al. , 2020 ; Yin et al. , 2020 ; You et al. , 2020 ; Ma et al. , 2021 ; Chen et al. , 2020a ) , natural language processing ( Gale et al. , 2019 ; Chen et al. , 2020b ) , reinforcement learning ( Yu et al. , 2020 ) , lifelong learning ( Chen et al. , 2021b ) , graph neural networks ( Chen et al. , 2021a ) , and adversarial robustness ( Cosentino et al. , 2019 ) . Most works of LTH use unstructured weight magnitude pruning ( Han et al. , 2016 ; Frankle & 2 Carbin , 2019 ) to find the matching subnetworks , and the channel pruning is also adopted in a recent work ( You et al. , 2020 ) . In order to scale up LTH to larger networks and datasets , the “ late rewinding ” technique is proposed by Frankle et al . ( 2019 ) ; Renda et al . ( 2020 ) . Mehta ( 2019 ) ; Morcos et al . ( 2019 ) ; Desai et al . ( 2019 ) are the pioneers to study the transferability of found subnetworks . However , all previous works focus on discriminative models . In this paper , we extend LTH to GANs and reveal unique findings of GAN tickets . 3 PRELIMINARIES . In this section , we describe our pruning algorithms and list related experimental settings . Backbone Networks We use two GANs in our experiments in Section 4 : SNGAN ( Miyato et al. , 2018 ) and CycleGAN ( Zhu et al. , 2017 ) ) . SNGAN with ResNet ( He et al. , 2016 ) is one of the most popular noise-to-image GAN network and has strong performance on several datasets like CIFAR10 . CycleGAN is a popular and well-studied image-to-image GAN network that also performs well on several benchmarks . For SNGAN , let g ( z ; θg ) be the output of the generator network G with parameters θg and a latent variable z ∈ R‖z‖0 and d ( x ; θd ) be the output of the discriminator network D with parameters θd and input example x . For CycleGAN which is composed of two generator-discriminator pairs , we use g ( x ; θg ) and θg again to represent the output and the weights of the two generators where x = ( x1 , x2 ) indicates a pair of input examples . The same modification can be done for the two discriminators in CycleGAN . Datasets For image-to-image experiments , we use a widely-used benchmark horse2zebra ( Zhu et al. , 2017 ) for model training . As for noise-to-image experiments , we use CIFAR-10 ( Krizhevsky et al. , 2009 ) as the benchmark . For the transfer study , the experiments are conducted on CIFAR-10 and STL-10 ( Coates et al. , 2011 ) . For better transferring , we resize the image in STL-10 to 32× 32 . Subnetworks For a network f ( · ; θ ) parameterized by θ , a subnetwork is defined as f ( · ; m θ ) , where m ∈ { 0 , 1 } ‖θ‖0 is a pruning mask for θ ∈ R‖θ‖0 and is the element-wise product . For GANs , two separate masks , md and mg , are needed for both the generator and the discriminator . Consequently , a subnetwork of GANs is consistent of : a sparse generator g ( · ; mg θg ) and a sparse discriminator d ( · ; md θd ) . Let θ0 be the initialization weights of model f and θt be the weights at training step t. Following Frankle et al . ( 2019 ) , we define a matching network as a subnetwork f ( · ; m θ ) , where θ is initialized with θt , that can reach the comparable performance to the full network within a similar training iterations when trained in isolation ; a winning ticket is defined as a matching subnetwork where t = 0 , i.e . θ initialized with θ0 . Finding subnetworks Finding GAN subnetworks is to find two masks mg and md for the generator and the discriminator . We use both an unstructured magnitude method , i.e . the iterative magnitude pruning ( IMP ) , and a structured pruning method , i.e . the channel pruning ( He et al. , 2017 ) , to generate the masks . For unstructured pruning , we follow the following steps . After we finish training the full GAN model for N iterations , we prune the weights with the lowest magnitude globally ( Han et al. , 2016 ) to obtain masks m = ( mg , md ) , where the position of a remaining weight in m is marked as one , and the position of a pruned weight is marked as zero . The weights of the sparse generator and the sparse discriminator are then reset to the initial weights of the full network . Previous works have shown that the iterative magnitude pruning ( IMP ) method is better than the one-shot pruning method . So rather than pruning the network only once to reach the desired sparsity , we prune a certain amount of non-zero parameters and re-train the network several times to meet the requirement . Details of this algorithm are in Appendix A1.1 , Algorithm 1 . As for channel pruning , the first step is to train the full model as well . Besides using a normal loss function LGAN , we follow Liu et al . ( 2017 ) to apply a ` 1-norm on the trainable scale parameters γ in the normalization layers to encourage channel-level sparsity : Lcp = ||γ||1 . To prevent the compressed network behave severely differently with the original large network , we introduce a distillation loss as Wang et al . ( 2020b ) did : Ldist = Ez [ dist ( g ( z ; θg ) , g ( z ; mg θg ) ) ] . We train the 3 GAN network with these two additional losses forN1 epochs and get the sparse networks g ( · ; mg θg ) and d ( · ; md θg ) . Details of this algorithm are in Appendix A1.1 , Algorithm 2 . Evaluation of subnetworks After obtaining the subnetworks g ( · ; θg mg ) and d ( · ; θd md ) , we test whether the subnetworks are matching or not . We reset the weights to a specific step i , and train the subnetworks for N iterations and evaluate them using two specific metrics , Inception Score ( Salimans et al. , 2016 ) and Fréchet Inception Distance ( Heusel et al. , 2017 ) . Other Pruning Methods We compare the size and the performance of subnetworks found by IMP with subnetworks found by other techniques that aim at compressing the network after training to reduce computational costs at inference . We use a benchmark pruning approach named Standard Pruning ( Chen et al. , 2020b ; Han et al. , 2016 ) , which iteratively prune the 20 % of lowest magnitude weights , and train the network for another N iterations without any rewinding , and repeat until we have reached the target sparsity . In order to verify that the statement of iterative magnitude pruning is better than one-shot pruning , we compare IMPG and IMPGD with their one-shot counterparts . Additionally , we compare IMP with some randomly pruning techniques to prove the effectiveness of IMP . They are : 1 ) Randomly Pruning : Randomly generate a sparsity mask m′ . 2 ) Random Tickets : Rewinding the weights to another initialization θ′0 .

Summary: The authors study the lottery ticket hypothesis  for generative adversarial networks. Specifically, they attempt to answer the following questions: the existence of winning tickets in GANs; the effect of discriminator pruning in finding such winning tickets; the effect of initialization during the rewinding steps; and finally if the subnetworks found transfer across datasets. They provide extensive empirical evidence using that ```winning' tickets exist in GANs. Further they show that iterative magnitude pruning and channel pruning successfully find such `winning' subnetworks. They analyse the effect of discriminator pruning and find that initialization during the rewind step matters more than the actual pruning of the discriminator. Finally, they show state-of-the-art results on GAN compression through channel pruning.