GANs Can Play Lottery Tickets Too
Authors: Xuxi Chen
Zhenyu Zhang
Yongduo Sui
Tianlong Chen
Venue: ICLR 2021 Poster (2021)
Keywords: lottery tickets
GAN compression
generative adversarial networks

Abstract: Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-constrained real-world applications. However, the compression of GANs has less been explored. A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due to the notorious training instability of GANs. In parallel, the lottery ticket hypothesis shows prevailing success on discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance. In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs. For a range of GANs, we certainly find matching subnetworks at $67\%$-$74\%$ sparsity. We observe that with or without pruning discriminator has a minor effect on the existence and quality of matching subnetworks, while the initialization weights used in the discriminator plays a significant role. We then show the powerful transferability of these subnetworks to unseen tasks. Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previous state-of-the-art GAN compression approaches in both image generation (e.g. SNGAN) and image-to-image translation GANs (e.g. CycleGAN). Codes available at https://github.com/VITA-Group/GAN-LTH.

Published as a conference paper at ICLR 2021
GANS CAN PLAY LOTTERY TICKETS Too
Xuxi Chen",, Zhenyu Zhang", Yongduo Sui', Tianlong Chen²
'University of Science and Technology of China, UUiiversity of Texas at Austin {chanyh, 29619969, syd2019 (aa2019)@maill @emaml...ue........ ustc. enu tianlong, wwwneutexas.ccooooooooooo cdu
ABSTRACT
Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-conse cooutce-conttrained real-world applications. However, the compression of GANs has less been explored. A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due the notorious training instability of GANs. In parallel, the lottery ticket hypothesis shows prevailing suc- cess discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance. In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs. For a range of GANs, we certainly find matching subnetworks at 67%-74% sparsity. We observe that with or without pruning discriminator has a minor effect on the exis- tence and quality of matching subnetworks, while the initialization weights used in the discriminator plays a significant role. We then show the powerful transfer- ability of these subnetworks to unseen tasks. Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previ- ous state-of-the-art GAN compression approaches in both image generation (e.g. SNGAN) and saaagge-t-immage translation GANs (e.g. CycleGAN). Codes avail- able at https: /ttp:///ggttttt com wtt.://wwb.cco//iit/t/ooo////// GAN-LTT
INTRODUCTION
Generative adversarial networks (GANs) have been successfully applied to many fields like image translation (Jing et al. 2019: Isola et al. 2017: Liu Tuzel 2016 Shrivastava al.] 2017 Zhu et al. (017)) and image generation (Myyato et al. 2018 Radford et 2016, Gulrajani et al. 2017 Arjovsky et al., 2017). However, they are often heavily parameterized and often require intensive calculation at the training and inference phase. Network compressing techniques (LeCun et al.) 1990 Wang et al. 2019 2020b Li et al. 2020) can be of help at inference by reducing the number of parameters usage of memory; nonetheless, they can not computational burden no cost. Although they strive to maintain the performance after compressing the model, a non-negligible drop in generative capacity is usually observed. A question is raised:
Is there any way to compress a GAN model while preserving or even improving its performance?
The lottery ticket hypothesis (LTH) (Fankle & Carbin 2019) provides positive answers with match- ing subnetworks (Chen et al.] 2020b). states that there exist matching subnetworks in dense models that can be trained to reach comparable test accuracy the full model within similar training iter- ations. The hypothesis has successfully shown its success in various fields (Yu et al.) 20200 Renda
al. 2020 Chen et al. (020b), and its property has been studied widely (Malach et al. [020 Pen- sia et al. 2020 Elesedy et al. 2020). However, it is never introduced to GANs, and therefore the presence of matching subnetworks in generative adversarial networks still remains mysterious.
To address this gap in the literature, we investigate the lottery ticket hypothesis in GANs. One most critical challenge of extending L.T in GANs emerges: how to deal with the discriminator while compressing the generator, including (i) whether prunes the discriminator simultaneously and (ii) what initialization should be adopted by discriminators during the re-training? Previous GAN com- pression methods (Shu et al. 2019 Wang et al. 2019 Li al., 2020 Wang et al. 2020b) prune the generator model only since they aim at reducing parameters in the inference stage. The effect of
"Equal Contribution.

---

Published as a conference paper at ICLR 2021
pruning the discriminator has never been studied by these works, which is unnecessary for them but possibly essential in finding matching subnetworks. It is because that finding matching sub- networks involves re-training the whole GAN network, in which imbalance in generative and discriminative power could result in degraded training results. For the same reason, the disequilib- rium between initialization used in generators and discriminators incurs severe training instability and unsatisfactory results.
Another attractive property of LTH is the powerful transferability of located matching subnetworks. Although it has been well studied in discriminative models (Mehta 2019; Morcos et al. 2019 Chen al., 2020b), an in-depth understanding of transfer learning in GAN tickets still missing. In this work, we not only show whether the sparse matching subnetworks in GANs can transfer across multiple datasets but also study what initialization benefits more to the transferabilityy
To convert parameter efficiency of LTH into the advantage of computational saving, we also utilize channel pruning (He et al., (017) to find the structural matching subnetworks of GANs, which enjoys the bonus of accelerated training and inference. Our contributions can be summarized in the following four aspects:
Using unstructured magnitude pruning, we identify matching subnetworks at 74% sparsity in SNGAN (Miyato et al. 2018) and 67% in CycleGAN (Zhu al.) (017). The matching subnet- works in GANs exist no matter whether pruning discriminators, while the initialization weights used in the discriminator
crucial.
We show that the matching subnetworks found by iterative magnitude pruning outperform sub- networks extracted by randomly pruning and random initialization in terms of extreme sparsity and performance. To fully exploit the trained discriminator, we using the dense discriminator as a distillation source and further improve the quality of winning tickets.
We demonstrate that the found subnetworks in GANs transfer well across diverse generative tasks.
The matching subnetworks found by channel pruning surpass previous state-of-the-ar GAN com- pression methods (i.e., GAN Slimming (Wang et al. 2020b)) in both efficiency and performance.
2 RELATED WORK
GAN Compression Generative adversarial networks (GANs) have succeeded in computer vision fields, for example, image generation and translation. One significant drawback of the generative models is the high computational cost of the models' complex structure. A wide range of neural net- work compression techniques has been applied to generative models to address this problem. There several categories of compression techniques, including pruning (removing some parameters), quantization (reducing the bit width), and distillation. [Shu et al. (2019) proposed a channel pruning method for CycleGAN by using co-evolution algorithm. Wang et al. (2019) proposed quanti- zation method for GANs based on the EM algorithm. Li et al. (2020) used a distillation method to transfer knowledge of the dense to the compressed model. Recently Wang et al. (2020b) proposed GAN compression framework, GAN slimming, that integrated the above three mainstream com- pression techniques into unified form. Previous works on GAN pruning usually aim at finding a sparse structure of the trained generator model for faster inference speed, while we are focusing on finding trainable structures of GANs following the lottery ticket hypothesis. Moreover, in existing GAN compression methods, only the generator is pruned, which could undermine the performance of re-training since the left-out diseriminator may have a stronger computational ability than the pruned generator and therefore cause degraded result due the imparity of these two models.
The Lottery Ticket Hypothesis The lottery ticket hypothesis (LTH) (Frankle & Carbin 2019) claims the existence of sparse, separate trainable sub-networks in dense network. These subnet- works are capable of reaching comparable or even better performance than full dense model, which
has been evidenced in various fields, such as image classification (Frankle & Carbin 2019 Liu et al.)
2019
Wang et al. 2020a Evci et al. 2019 Frankle et al.) 2020, Savarese et al. 2020 Yin et al.
2020
You et al. 2020 Ma et al. 2021 Chen et al. (020a)), natural language processing Gaale et al,
2019
Chen et al. 2020b). reinforcement learning Yu et al. 2020), lifelong learning (Chen et al.
2021b)
graph neural networks (Chen et al. 202 la). and ad adversarial robustness (Cosentino et al,
2019
Most works of L.H use unstructured weight magnitude pruning (Han et al. 2016 Frankle &
2

---

Published as a conference paper at ICLR 2021
Carbin 2019) to find the matching subnetworks, and the channel pruning is also adopted in a recent work You et al. 2020). In order to scale up LTH to larger networks and datasets, the "late rewind- ing" technique is proposed by Frankle et al. (2019); Renda et al. (2020). MMetta (2019); Morcos et al. 20119); Desai et al. 2019) are the pioneers to study the transferability of found subnetworks. However, all previous works focus on discriminative models. In this paper, extend LTH to GANs
and reveal unique findings of GAN tickets.
3 PRELIMINARIES
In this section, we describe our pruning algorithms and list related experimental settings.
Backbone Networks We two GANs in experiments in Section 4 SNGAN (Miyato al. 2018) and CycleGAN (Zhu et al. 2017)). SNGAN with ResNet (He et al. 2016) is one of the most popular noise-to-image GAN network and has strong performance on several datasets like CIFAR- 10. CycleGAN is popular and well-studied image-to-image GAN network that also performs well on several benchmarks. For SNGAN, let g(z; 0g) be the output of the generator network with parameters 0g and a latent variable z E Bll=llo and d(x; 0a) be the output of the discriminator network 'D with parameters Oa and input example x. For CycleGAN which is composed of two generator- sngeratoo-ddscrmmintoo pairs, we use g(;; 0g) and 9g again to represent the output and the weights of the two generators where X2) indicates pair of input examples. The same modification can be done for the two discriminators in CycleGAN.
Datasets For image-to-ing im-ag experiments, we widely-used benchmark horse2zebra (Zhu et al. 2017) for model training. As for noise-to-image experiments, we use CIFAR-10 (Krizhevsky al. 2009) as the benchmark. For the transfer study, the experiments are conducted CIFAR-10
and STL-I0 (Coates et al. 2011]. For better transferring, we resize the image in STL-10 to 32 x 32.
Subnetworks For a network f(: f) parameterized by 0, a subnetwork is defined as f(-m O 0), where m E {0, 13.11 is pruning mask for 0 e 1B10 and @ the element-wise product. For GANs, two separate masks, ma and mg. are needed for both the generator and the discriminator. Consequently, a subnetwork of GANs consistent of: a sparse generator g(: mg @ 0g) and a sparse discriminator di(; mm @ Oz).
Let 0, be the initialization weights of model f and 0, be the weights at training step Follow- ing Frankle et al. (2019). we define matching network as a subnetwork [(: m @ 0), where 0 is initialized with 0., that can reach the comparable performance to the full network within a similar training iterations when trained in isolation; winning ticket is defined matching subnetwork where f = 0, i.e. 0 initialized with 0o.
Finding subnetworks Finding GAN subnetworks is to find two masks my and ma for the gen- erator and the discriminator. We use both an unstructured magnitude method, i.e. the iterative magnitude pruning (IMP), and structured pruning method, i.e. the channel pruning (He et al., 2017]. to generate the masks.
For unstructured pruning, we follow the following steps. After we finish training the full GAN model for N iterations, we prune the weights with the lowest magnitude globally (Han et al. 2016 obtain masks m (my. ma), where the position of remaining weight in is marked as one, and the position of pruned weight is marked as zero. The weights of the sparse generator and the sparse discriminator are then reset the initial weights of the full network. Previous works have shown that the iterative magnitude pruning (IMP) method is better than the one-shot pruning method. So rather than pruning the network only once to reach the desired sparsity, we prune a certain amount of non-zero parameters and re-train the network several times to meet the requirement. Details of this algorithm are in Appendix AI.1 Algorithm [
As for channel pruning, the first step is to train the full model well. Besides using normal loss function LGAN, we follow Liu et al. (2017) to apply a mn-ormm on the trainable scale parameters y in the normalization layers encourage channel-level sparsity: Lep = 11 To prevent the compressed network behave severely differently with the original large network, we introduce a distillation loss as Wang al. (2020b) did: Laist Ry (dist(ggg() 0g), g(2; my @ fg))) We train the
3

---

Published as a conference paper at ICLR 2021
GAN network with these two additional losses for N1 epochs and get the sparse networks g( mg 0g) and d(:; ma @ 0g). Details of this algorithm are in Appendix A1.1 AL.1
Algorithm
Evaluation of subnetworks After obtaining the subnetworks g(: :; 0g @ mg) and d(.; Oa @ ma), we test whether the subnetworks are matching or not. We reset the weights to a specific step i, and train the subnetworks for N iterations and evaluate them using two specific metries, Inception Score (Salimans et al. 2016) and Fréchet Inception Distance (Heusel et al. 2017).
Other Pruning Methods We compare the size and the performance of subnetworks found by IMP with subnetworks found by other techniques that aim at compressing the network after training to reduce computational costs at inference. We use benchmark pruning approach named Standard Pruning (Chen et al.] [220b] Han et al., (016), which iteratively prune the 20% of lowest magnitude weights, and train the network for another N iterations without any rewinding, and repeat until we have reached the target sparsity.
In order to verify that the statement of iterative magnitude pruning is better than one-shot pruning, we compare 1MPg and IMPcD with their one-shot counterparts. Additionally, we compare IMP with some randomly pruning techniques to prove the effectiveness of IMP. They are: 1) Randomly Pruning: Randomly generate sparsity mask m'. 2) Random Tickets: Rewinding the weights to another initialization 8-
THE EXISTENCE OF WINNING TICKETS IN GAN
In this section, we will validate the existence of winning tickets in GANs with initialization Oo gggo dd,). Specifically, we will empirically prove several important properties of the tickets by authenticating the following four claims:
Claim 1: Iterative Magnitude Pruning (IMP) finds winning tickets in GANs, g(:5 mg 0g0) and 2(:mm @ 0.,). Channel pruning is also able to find winning tickets as well.
Claim 2: Whether pruning the discriminator 'D does not change the existence of winning tickets. It is the initialization used in D that matters. Moreover, pruning the discriminator has slight boost of matching networks in terms of extreme sparsity and performance.
Claim 3: IMP finds winning tickets at sparsity where some other pruning methods (randomly prun- ing, shot magnitude pruning. and random tickets) are not matching.
Claim 4: The late rewinding technique (Frankle et al., 2019) helps. Matching subnetworks that are initialized 0g, i.e., i steps from Oo. can outperform those initialized to Og. Moreover, matching subnetworks that are late rewound can be trained to match the performance of standard pruning.
Are there winning tickets in GANs? To answer this question, we first conduct ex- periments on SNGAN by pruning the generator only in the following steps: 1) Run IMP to get sequential sparsity masks (m&,, myg)) of sparsity s,% remaining weights; 2) Apply the masks to the GAN and reset the weights of the subnetworks to the same random initialization 6o: 3) Train models to evaluate whether they are winning tickets. [ We set x% = (1 0.8*) x 100%, which we use for all the experiments that involve iteratively pruning hereinafter. The number of training epochs for subnetworks is identical to that of training the full models.
Claim 1:
Figure D verifies the existence of winning tickets in SNGAN and CycleGAN. We are able to find winning tickets by iterative pruning the generators at the highest sparsity, around 74% in SNGAN, and around 67% in CycleGAN, where the FID scores of these subnetworks successfully match the FID scores of the full network respectively. The confidence interval also suggests that the winning tickets at some sparsities are statistically significantly better than the full model.
To show that channel pruning can find winning tickets as well, we extract several subnetworks from the trained full SNGAN and CycleGAN by varying p in Algorithm We define the channel-pruned model's 'ss sparsity as the ratio of MFLOPs between the sparse model and the full model.
'The detailed description of algorithms listed in Appendix AL.1
4

---

Published as a conference paper at ICLR 2021
20
125
18
16
CIFAR-10
100
horse2zebra
Zebrra horee
14
Ind
BR0
32.6
28
21.0
1ed
100.0
80.0
6dd
std
320
262 21d
Remaining Weights (%)
Remaining Weights (%)
Figure I: The Fréchet Inception Distance (FID) curve of subnetworks of SNGAN (left) and CycleGAN (right) generated by iterative magnitude pruning (IMP) on CIFAR-10 and horse2zebra. The dashed line indicates the FID score of the full model on CIFAR-10 and horse2zebra. The 95% confidence interval of 5 runs is reported.
We can confirm that winning tickets can also be found by channel pruning (CP), CP is able to find winning tickets in SNGAN at sparsity around 34%. We will analyze it more carefully in Section 7
Full Model (Sparsity: 0%)
Best Winning Tickets (Sparsity: 48.80%)
Matching Subnetworks (Sparsity: 73.79%)
Intoppatatinn Genevation
Figure 2: Visualization by sampling and interpolation of SNGAN Winning Tickets found by IMP. Sparsity of best winning tickets: 88% Extreme sparsity of matching subnetworks: 73.79%.
Suarce Imazs
Best Winning Marching Sabactworka
Suuree Imazz
Full Madel C'puriity:
Bell Wanning Tickets Matching Sutrectworka (Spurdit::
PSparity::
('pariiy
(Sparday:
(Spardiy:
Figure 3: Visualization of CycleGAN Winning Tickets found by IMP. Sparsity of best winning tickets
59.04%. Extreme sparsity of matching subnetworks: 67.24%. Left: visualization results horse2zebra. Right: visualization results on zebraZhorse.
Does the treatment of the discriminator affect the existence of winning tickets? Pre- vious works of GAN pruning did not analyze the effect of pruning the discriminator. To study the effect, we compare two different iterative pruning settings: 1) Prune the generator only (IMPc) and 2) Prune both the generator and the discriminator iteratively (IMPGD). Both the generator and the discriminator are reset the same random initialization Og after the masks are obtained.
Claim 2:
The FID scores of the two experiments are shown in Figure 4 The graph suggests that the two settings share similar patterns: the minimal FID of IMP G 14.19, and the minimal FID of IMP GD is 14.59. The difference between these two best FID is only 0.4, showing slight difference in generative power. The FID curve of IMP lies below that of IMP at low sparsity but lies above at high sparsity, indicating that pruning the discriminator produces slightly better performance when the percent of remaining weights is small. The extreme sparsity where IMPcD can match the performance of the full model is 73.8%. In contrast, IMP can only match sparser than 67.2%, demonstrating that pruning the discriminator can also push the frontier of extreme sparsity where the pruned models are still able to match.
In addition, we study the effects of different initialization for the sparse discriminator. We compare different weights loading methods when applying the iterative magnitude pruning process: 1) Reset the weights of generator to gno and reset the weights of discriminator to 0dn- which is identical IMP Reset the weights of generator 0g and fine-tune the discriminator, which will call IMPE, Figure 4 shows that resetting both the weights to d, produces much better result than
5

---

Published as a conference paper at ICLR 2021
only resetting the generator. The discriminator D without resetting its weights is too strong for the generator with initial weights ggo that will lead to degraded performance. In summary, different initialization of the discriminator will significantly influence the existence and quality of winning tickets in GAN models.
18
IMPc IMPaD
150-
16
IMP IMP
25-
10
1000
20.3
21.0
1000
Remaining Weights (5)
Remaining Weights (%)
Figure 4: The FID of Left: The FID of best subnetworks generated by different pruning
FID score best subnetworks generated by two different pruning settings: TMPc and IMP G. IMPa: iteratively prune and reset the generator. IMPGD: iteratively prune and reset the generator and the discriminator. IMP iteratively prune and reset the generator, and iteratively prune but not reset the discriminator.
settings: IMPg and TMP GD- Right:
follow-up question arises from the previous observations: is there any way to use the weights of the dense discriminator, as the direct usage of the dense weights yields inferior results? One possible way is to use it as a "teacher" and transfer the knowledge to pruned discriminator using a consistency loss. Formally speaking, an additional regularization term used when training the whole network:
LKD 0a ma) Ex [KL (d(x; ma C 0a), d(x; 0a. ))]
where KLDiv denotes the KL-Divergence. We name the iterative pruning method with this additional regularization IMPKD
Figure 5 shows the result of pruning method IMPKD compared the previous two prun- ing methods we proposed, IMPg and IMP GD is capable of finding winning tickets at sparsity around 70%, outperforming IMPg. and showing comparable results to IMP GD re- garding the extreme sparsity. The FID curve of setting IMPKD GD is further mostly located below the curve of IMP demonstrating a stronger generative ability than IMP It suggests transferring knowledge from the full discrimi- nator benefits to find the winning tickets.
IMPa
18
IMPc
IMP
IMPKD
16
300
Remaining Weights (%)
of best subnetworks gen- erated by three different pruning methods: IMPa. IMPGD and IMP IMP LD iteratively prune and reset both the generator and the discriminator, and train them with the KD regularization.
Figure 5: The FID
Claim 3: Can IMP find matching subnetworks sparser than other pruning methods? Pre- vious works claim that both specific sparsity mask and specific initialization are necessary for finding winning tickets (Frankle & Carbin 2019), and iterative magnitude pruning is better than one-shot pruning. To extend such statement in the context of GANs, we compare IMP with several other benchmarks, randomly pruning (RP), one-shot magnitude pruning (OMP), and random tickets (RT), if IMP can find matching networks higher sparsity.
Method
FIDa.e (Sparsity)
FiD (Sparsity)
IMPs
[MP
No Pruning
15.69 (0.0%)
OMP,
(MPin
IMP
14.19 (20.0%)
15.58 (67.2%)
IMP
14.59 (59.0%)
15.53 (73.8%)
OMP
(36.0) 0%6)
15.33 (59.0%)
OMP
14.52 (20.0%)
15.69 (48.8%
14
Table 1:
The extreme sparsity of the matching net- works and the FID score best subnetworks, found by iterative pruning and one-shot pruning. OMP one-shot prune the generator. OMP one-shot prune generatorfdiseriminato.
1000
Remsizing Weights (5)
Figure 6: FID curves: OMP IMPg, OMP
TMP OMPG: one-shot prune generator.
OMP one-shot prune generator/

---

Published as a conference paper at ICLR 2021
Figure Table show that iterative magnitude pruming outperforms one-shot magnitude pruning no matter pruning the discriminator or not. IMP finds winning tickets at higher sparsity (67.23% and 73.79%, respectively) than one-shot pruning (59.00 900% and 48.80%, respectively). The minimal FID score of subnetworks found by IMP is smaller than that of subnetworks found by OMP as well. This observation defends the statement that pruning iteratively is superior compared to one-shot pruning.
We also list the minimal FID scores and extreme sparsity of matching networks for other pruning methods in Figure / and Table 2 It can be seen that IMP finds winning tickets at sparsity where some other pruning methods, randomly pruning and random initialization, cannot match. Since 1MPg shows the best overall result, we authenticate the previous statement that both the specific sparsity mask and the specific initialization are essential for finding winning tickets.
Method
FiD (Sparsity)
FiDestremn (Sparsity)
IMP
No Pruning
15.69 (0.0%
25
RP
IMPa
14.19 (20.0%)
15.58 (67.2%)
Random Pruning
14.57 (20.0%)
15.60 (36.0%)
20
Random Tickets
14.28 (36.0%)
15.33 (59.0%)
15
The FID score of best subnetworks and the extreme sparsity of matching networks found by Ran- dom Pruning, Random Rickets, and iterative magni- tude pruning.
Table 2:
100
8810
Remaining Weights (%)
Figure 7: The FID curve best subnetworks gener- ated by three different pruning settings: TMPG. RP, and RT. RP: iteratively randomly prune the genera- tor. RT: iteratively prune the generator but reset the weights randomly.
Claim 4: Does rewinding improve performance? In previous paragraphs, we show that able to find winning tickets both SNGAN and CycleGAN. However, these subnetworks cannot match the performance of the original network at extremely high sparsity, while the subnetworks found by standard pruning can (Table B To find matching subnetworks at such high sparsity, we adopt the rewinding paradigm: after the masks obtained, the weights of the model rewound to 0g, the weights after i steps of training, rather than reset to the same random initialization On. It was pointed out by Renda al. (2020) that subnetworks found by IMP and rewound early in training can be trained to achieve the same accuracy at the same sparsity as subnetworks found by the standard pruning. providing a possibility that rewinding can also help GAN subnetworks.
We choose different rewinding settings: 5%, 10%, and 20% of the whole training epochs. The results are shown in Ta We observe that rewinding can significantly increase the ex- treme sparsity of matching networks. Rewind- ing to even only 5% of the training process can raise the extreme sparsity from 67.23% to 86.26% and rewinding to 20% can match the performance of standard pruning. We also com- pare the FID score of subnetworks found at 89% sparsity. Rewind to 20% of the training process can match the performance of standard pruning at 89% sparsity, and other late rewinding settings can match the performance of the full model. This suggests that late rewinding techniques can greatly contribute to matching subnetworks with higher sparsity.
Table 3: Rewinding results. SExtreme: Extreme spar- sity where matching subnetworks exist. F The minimal FID score of all subnetworks. FID 89%; The
FID score of subnetworks at 89% sparsity.
Setting of rewinding
S
FID Beet
FIDsos
Rewind 0%
67.23%
14.20
19.60
Rewind 5%
86.26%
13.96
15.82
Rewind 10%
86.26%
14.43
15.63
Rewind 20%
89.26 %
14.82
15.29
Standard Pruning
89.26 %
14.18
15.22
Summary Extensive experiments are conducted to examine the existence of matching subnet- works in generative adversarial models. We confirmed that there were matching subnetworks at high sparsities, and both the sparsity mask and the initialization matter for finding winning tickets. We also studied the effect of pruning the discriminator and demonstrate that pruming the discrimi- nator can slightly boost the performance regarding the extreme sparsity and the minimal FID. We proposed a method to utilize the weights of the dense discriminator model to boost the performance further. We also compare IMP with different pruning methods, showing that IMP is superior to ran- dom tickets and random pruning. In addition, late rewinding can match the performance of standard pruning. which again shows consistency with previous works.
7

---

Published as a conference paper at ICLR 2021
5 THE TRANSFER LEARNING OF GAN MATCHING NETWORKS
In the previous section, confirm the presence of winning tickets in GANs. In this section, will study the transferabilitty of winning tickets. Existing works (Mehta 2019) show that the matching networks in discriminative models can transfer across tasks. Here evaluate this claim in GANs.
To investigate the transferabilityy, we propose three transfer experiments from CIFAR-10 10 to STL-10 on SNGAN. We first identify matching subnetworks g(: my @ 0) and d(; ma @ 0) CIFAR- 10, and then train and evaluate the subnetworks on STI.-10. To assess whether the same random initialization Og is needed for transferring,
test three different weights loading method: 1) reset the weights to 0o: 2) reset the weights to another initialization 06 3) rewind the weights to ÓN. We train the network on STL-10 using the same hyper-parametee as on CIFAR-10. It is noteworthy that the hyper-paramenn parameters setting might not be optimal for the target task, yet it is fair to compare different transferring settings.
Table 4: Results of late rewinding experiments. Oo: train the target model from the random initialization as the source model; 0,: train from random initialization; Best train from the weights trained source
model. Baseline: full model trained on STL-10.
Model
Baseline
IMP (S 67.23%)
IMP (S = 73. 79%)
IMPRD
GD (S = 73.
Metrics
FeD
FID Best
Matching?
FID
Matching?
FID Best
Matching?
0o 0, Best
116.7
V
121.8
x
120.1
115.3
119.2
X
113.1
115.5
V
204.7
X
163.0
X 179.1
X
The FID score of different settings is shown in Table 4 Subnetworks initialized by 6g and using masks generated by IMPa can be trained to achieve comparable results to the baseline model. Sur- prisingly, random re-initialization og shows better transferability than using the same initialization 0, in transfer settings and outperforms the full model trained on STL-10, indicating that the combination of 0o and the mask generated by IMPcD is more focused on the source dataset and consequently has lower transferability.
Summary In this section, tested the transferability of IMP subnetworks. Transferring from Oo and 0, both produce matching results on the target dataset, STL-10. 6, works better with masks generated by IMPg while the masks generated by IMP GD prefer different initialization 0,. Given that IMPGD GD performs better on CIFAR-10, 10, it reasonable that the same initialization 0g has lower transferability when using masks from IMPGD.
EXPERIMENTS ON OTHER GAN MODELS AND OTHER DATASETS
We conducted experiments on DCGAN (Radford et al. 2016), WGAN-GP Gulrajani et al. 2017)
ACGAN (Odena al., (0177). GGAN (Lim & Ye 2017). 7/, DiffAugGAN Zhao et al. 2020a) Pro-
jGAN (Miyato & Koyama 2018), SAGAN (Zhang et al. 2019). as well as a NAS based GAN,
AutoGAN (Gong al. 2019). We use CIFAR-IÓ and Tiny ImageNet (Wu et al.
2017) our
benchmark datasets.
verify that the existence of winning tickets diverse GAN architectures in spite of the different extreme sparsities, showing that the lottery ticket hypothesis can be generalized to various GAN models.
7 EFFICIENCY OF GAN WINNING TICKETS
Unlike the unstructured magnitude pruning method, channel pruning can reduce the number of pa- rameters GANs. Therefore, winning tickets found by channel pruning are more efficient than the original model regarding computational cost. To fully exploit the advantage of subnetworks founded by structural pruning, we further compare our muuee-nd--rriit pipeline with a ofate-of-thh-aa GAN compression framework (Wang et al.) (220bb). The pipeline is described as follows: after extracting the sparse structure generated by channel pruning, we reset the model weights to the same random initialization lg and then train for the same number of epochs as the dense model used.
8

---

Published as a conference paper at ICLR 2021
Table 5: Results other GAN models on CIFAR-10. FuD FID score of the full model. FllB Best The minimal FID score of all subnetworks. FID The FID score of matching networks at extreme sparsity level. AutoGAN-A/B/C are three representative GAN architectures represented in the official repository https::/githuub.omm/II VWTAVVIAA-OOOO Group/AutoGANN
Model
Benchmark
FIDpuu (Sparsity)
Best (Sparsity)
S Sgxtreme (Sparsity)
DCGAN RRdford al.
2016
CIFAR-I0
57.39 (0%)
49.31 (20.0%)
54.48 (67.2%)
WGAN-GP Guirajan
CIFAR-10
19.23 (0%)
16,77 (36.0%)
17.28 (73.8%)
ACGAN Odena et al. 2017
CIFAR-I0
39.26 (0%)
31.45 (36.0%)
38.95 (79.0%)
GGAN Lim & Yo 2017
CIFAR-IO
38,50 (0%)
33.42 (20.0%)
36.67 (48.8%)
ProjGAN (Miyalo & Kovama
2018
CIFAR-I0
31.47 (0%)
28.19 (20.0%)
31.31 (67.2%)
SAGAN Zhang et a. 2019
CIFAR-IO
14.73 (0%)
13.57 (20.0%)
14.68 (48.8%)
AutoGAN(A) Gong al.
2019
CIFAR-I0
14.38 (0%)
14.04 (36.0%)
14.04 (36.0%)
AutoGAN(() Gong al.
2019
CIFAR-I0
14.62 (0%)
13.16 (20.0%)
14,20 (36.0%)
AutoGAN(CC
Gong al.
2019
CIFAR-I0
13.61 (0%)
13.41 (48.8%)
13.41 (48.8%)
DiffAugGAN
Zhao al.
2020b
CIFAR-I0
8.23 (0%)
8.05 (48.8%)
8.05 (48.8%)
Table 6: Results other GAN models on Tiny ImageNet. FIDran: FID score of the full model.
FeD:
The minimal FID score of all subnetworks.
The FID score of matching
networks at extreme sparsity level.
Model
Benchmark
FIDrau (Sparsity)
FID (Sparsity)
Sgstrome (Sparsity)
DCGAN Radford 2016
Tiny ImageNet
121.35 (0%)
78.51 (36.0%)
114.00 (67.25)
WGAN- GP Gutrajmm al 2017
Tiny ImageNet
211.77 (0%)
194.72 (48.8%)
200.22 (67.2%)
We from Figure that subnetworks are founded by the channel pruning method at about 67% sparsity, which provides a new path for winning tickets other than magnitude pruning. The matching networks at 67.7% outperform GS-32 regarding Inception Score by 0.25; the subnetworks at about 29% sparsity can outperform GS-32 by 0.20, setting up a new benchmark for GAN compressions.
CONCLUSION
In this paper, the lottery ticket hypothesis has been extended to GANs. We successfully iden- tify winning tickets in GANs, which are sep- arately trainable to match the full dense GAN performance. Pruning the discriminator, which is rarely studied before, had only slight effects on the ticket finding process, while the initial- ization used in the discriminator is essential. We also demonstrate that the winning tickets found can transfer across diverse tasks. More- over, we provide new way of finding winning tickets that alter the structure of models. Chan- nel pruning is able to extract matching subnet- works from a dense model that can outperform the current state-of-the-an GAN compression after resetting the weights and re-training.
Model
6.25
1 8.00
CIFAR-10
G5-32
GS-12
Remaining Weights (%)
Relationship between the best IS score of SNGAN subnetworks generated by channel pruning and the percent of remaining weights. GS-32: GAN
Figure 8:
Slimming without quantization (Wang al. 2020b). Full Model: Full model trained on CIFAR-IO.
ACKNOWLEDGEMENT
Zhenyu Zhang is supported by the National Natural Science Foundation of China under grand No.U19BB044.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv, abs/2012.06908, 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv, abs/2007.12223, 2020b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks, 2021a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference on Learning Representations, 2021b.
Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. Proceedings of the 14th International Conference on Artificial Intelli- gence and Statistics, 2011.
Justin Cosentino, Federico Zaiter, Dan Pei, and Jun Zhu. The search for sparse, robust neural networks. arXiv,, abs/1912.02386, 2019.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP, 2019.
Bryn Elesedy, Varun Kanade, and Tch. Lottery tickets linear models: An analysis of iterative magnitude pruning. arXiv, abs/2007.08243, 2020.
Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv, ahs//196.1077. 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representationss 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. arXiv, abs/1912.05671,, 2019.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In 8th International Conference on Learning Representations, 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXivv, abs/1902.09574, 2019.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Ishaan Guirajani. Paruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems 30, 2017.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Representations, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2021
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: review. IEEE Transactions on Visualization and Computer Graphics, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems 2, 1990.
Muyang Li, Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han. Gan compression: Effi- cient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Jae Hyun Lim and Jong Chul Ye. Geometric gan. abs/1 0177705029444 2017.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems 29, 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, 2019.
Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good 2021.
students play big lottery better. arXiv,
Eran Malach, Gilad Yehudai, Shai Shalev-Shwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruming is all you need. arXiv, abs/2022.00585,, 2020.
Rahul Mehta. Sparse transfer learning via winning lottery tickets. arXiv, abs/ abs//195.00775, 2019.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In 6th International Conference on Learning Representations, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In 6th International Conference on Learning Representations, 2018.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: gener- alizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Infor- mation Processing Systems 32, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Op- timal lottery tickets via subsetsum: Logarithmic over parameterrtiionnnnnnn sufficient. In Advances in Neural Information Processing Systems 33 pre-proceedings, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learning Rep- resentations, 2016.
11

---

Published as a conference paper at ICLR 2021
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In 8th International Conference on Learning Representations. 2020.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems 31, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances Neural Information Processing Systems 29, 2016.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems 33 pre-proceedings, 2020.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co- evolutionary compression for unpaired image translation. In Proceedings of IEEE International Conference Computer Vision, 2019.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations, 2020a.
Haotao Wang, Shupeng Gui, Haichuan Yang. Ji Liu, and Zhangyang Wang. Gan slimming: All-in- one gan compression by unified optimization framework. In Proceedings of the 16th European Conference on Computer Vision, 2020b.
Peiqi Wang, Dongsheng Wang, Yu Ji. Xinfeng Xie, Haoxuan Song, XuXin Liu, Yonggiang Lyu, and Yuan Xie. Qgan: Quantized generative adversarial networks. abs/1901 000663,,, 2019.
Jiayu WU, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.
Shihui Yin, Kyu-Hyoun Kim, Jinwook Oh, Naigang Wang, Mauricio Serrano, Jae-Sun Sco, and Jungwook Choi. The sooner the better: Investigating structure of early winning lottery tickets, 2020.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In 8th International Conference on Learning Representations, 2020.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nip. In 8th International Conference on Learning Representations, 2020.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In 34th Conference on Neural Information Processing Systems, 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems 33 pre- proceedings, 2020b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2017
A12

---

Published as a conference paper at ICLR 2021
A1
MORE TECHNICAL DETAILS
A1.1
ALGORITHMS
In this section, we describe the details of the algorithm we used in finding lottery tickets. Two distinct pruning methods are used in Algorithm and Algorithm
1.00
Model
Fs
F1/8
Precision
0.75
Full Model 0.971
0.974
Best Winning Tickets
0.977
0.977
0.50
Extreme Winning Tickets
0.974
0.971
baseline
0.25
best
extreme
0.00
0.0
0.2
0.4
0.6
0.8
1.0
Table A7: The Fs and Fizs score of the full net- work, best subnetworks and the matching net- works at extreme sparsity. We used the official codes to calculate recall, precision, Fş and F1/8.
Recall
Figure A9: The of precision and recall of SNGANs under different sparsilies. baseline: Full model. best: Best winning tickets (Sparsity: (8%. extreme: Extreme winning tickets (Sparsity: 73.79%)
A2
MORE EXPERIMENTS RESULTS AND ANALYSIS
We will provide extra experiments results and analysis in this section.
A2.1 MORE VISUALIZATION OF IMP WINNING TICKETS
We also conducted experiments to find winning tickets in CycleGAN dataset win- ter2summer (Zhu et al.) (017). We observed similar patterns and found matching networks at 79.02 99.02% sparsity. We randomly sample four images from the dataset and show the translated im- ages in Figure A10 Figure AIT and Figure A12 The winning tickets CycleGAN can generate comparable visual quality to the full model under all cases.
Source Emagge
Full Made
Bost Winning Tickets Marchin Suleetwerks (Sparnty:: TVUBEN)
Scurce Treege
Full Model (Sparety: 0%
Bes Winning Tickets Motching Sabuetworks O'paritt:: 39.94%)
Oppamit:::
(Sparatt:: 3884550
('panitt:: 79025)
Figure A10: Visualization of CycleGAN Winning Tickets found by IMP on summer2winter. Spar- sity of best winning tickets: 59.04%. Extreme sparsity of matching subnetworks: 79.02%. Left: visualization results of task summer2winter. Right: visualization results of task winter2summer.
A2.2 EXTRA METRICS FOR EVALUATING SNGAN
We also evaluate the quality of images generated by SNGAN using precision and recall Sajjadi et al. 2018). The results shown in Table A7 and Figure A9 The results show that the best winning tickets have higher F& and Fi,s compared to the full model, and the extreme winning tickets have on-par performance.
A13

---

Published as a conference paper at ICLR 2021
Seuree Iaaage
Full Madkl Opaniiy (%))
Whening: Fixters Matehing Nahretterrks
Neurea Imaag
Full Mecki Cpperityy (%)
Bot Winning Ticken ('guritty 50.84%)
Nateiing Nehretwerkk (Syper skyy 3455
(Spardigs 53 (5.8%%)
ilpur 67.34%%)
Figure All: Extra visualization of CycleGAN Winning Tickets found by IMP on horse2zebra. Sparsity of best winning tickets: 59.04%. Extreme sparsity of matching subnetworks: 67.24%.
Sencce Irage
Full Madel Opparity: 0%)
Bost Wisning Tinkets Matching Sebnetwerks
Soucee Inage
Full Modkl
Bet Wisning Tickets Matching Sebeetweris
(Sparalt:: 99.89%%
Opaaralyy 8%6)
(Sparslty: 50.84%)
(5par 70.0566
Figure A12: Extra visualization of CycleGAN Winning Tickets found by IMP on sunmer2winter. Sparsity of best winning tickets: 59.04%. Extreme sparsity of matching subnetworks: 79.02%.
Model
8.25
70
8.00
CIFAR-10
GS-32
horseZzebra
GS-32
Remaining Model Sime (36)
Remaining Weights (%)
Figure A13: Relationship between the best IS score of SNGAN subnetworks generated by channel pruning and the percent of remaining model size. GS-32: GAN Slimming without quantization (Wang et al., (0200b). Full Model: Full model trained on CIFAR-10.
Figure A14: Relationship between the best FID score of CyeleGAN subnetworks generated by channel pruning and the percent of remaining weights. GS-32: GAN Slimming without quan- tization (Wang et al. 2020b). Full Model: Full un-pruned CycleGAN trained on horse2zebra.
A2.3 CHANNEL PRUNING FOR SNGAN
We study the relationship between the Inception Score and the remaining model size, i.e. the ratio between the size of channel-prunce model and its original model. The results are plotted in Fig- ure A13 A similar conclusion can be drawn from the graph that matching networks exist, and at the same sparsity, the matching networks can be trained to outperform the current state-of-the-art GAN compression framework.
A2.4 CHANNEL PRUNING FOR CYCLEGAN
We also conducted experiments on CycleGAN using channel pruning. The task we choose is horse- to-zebra, i.e., we prune each of the two generators separately, which is aligned with SNGAN, which has only one generator. We prove that channel pruning is also capable of finding winning tickets in CycleGAN in Figure A14 Moreover, extreme sparsity, the sparse subnetwork that obtain
A14

---

Published as a conference paper at ICLR 2021
be trained to reach slightly better results than the current sfate-of-the-art GAN compression framework without quantization.
Algorithm 1:
Finding winning tickets by
Algorithm 2: Finding winning tickets by
Iterative Magnitude Pruning
Channel Pruning
Input: The desired sparsity Output: sparse GAN g(: mg 0g) and d( ma @ 0a)
Input: A threshold of importance score, number of steps for training N
Output: A sparse GAN g(:; mg @ 20g)) and d(=: m. : 0d.)
Set mg E 11 and
ma 1 E R1
Randomly initialize 7g for every
Set 0g == initial weights of the
normalization layers in generator G and Y for discriminator D. y (7g. Ya)
generator model, 0d initial weights of the discriminator model.
2 Set 6 = initial weights of the generator model, 9d initial weights of the discriminator model.
Iteration i
while the sparsity of mg < < do
Train the generator g(: mg @ 0g)
and the discriminator
4 while i < N do
d(: ma O 0dg) for N epochs to get parameters 9N and 02
5 Compute masks mn from Y and mg
from "gg
Compute Leps LGAN and Ldist- Update 0g and 0a by training the generator g(: m- @ 0g) and the discriminator d(-; ma O Oa) for one step.
if praning the discriminator then Prune 20% of the parameters in
0f and 0án, creating two mask my and m'y-
else
Update Y: Y prox pm (y Noppp
Prune 20% of the parameters in 0f ON. creating a mask m'y m remains 1 E IR 100aa llo
where
prox x (1) sgn(x) max(|a] 1.
i +
10
end
end
11
Update mg my and ma m'a
Calculate the final masks my from 7g and m, from a
12
end
A15


Reviews:
[{"review_id": "JmK_mblJkL6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors provide an empirical study on lottery ticket hypothesis on GANs. To do this, the authors use two GAN models and two datasets: SNGAN/CycleGAN and CIFAR-10/horse2zebra. Extensive experiments show that matching subnetworks can be found using unstructured magnitude pruning and channel pruning and they are transferrable to other tasks. The performance of subnetworks found is competitive and even surpasses state-of-the-art performance.\n\nThis paper is well-written and addresses an interesting problem in GANs compression. This paper first studies to verify if lottery ticket hypothesis works on GANs. Although the concept of LTH is not new, empirical verification on GANs is of value to this community. The experimental set-up including methods, datasets, and claims is solid and empirical results for the claims are convincing.\n\nI am not familiar with the original work of lottery ticket hypothesis, but it is easy to understand the concept of it without reading the related works except for the following:\n1. Is the iterative magnitude pruning (IMP) equivalent to the unstructured pruning described in p3?\n2. What is the difference between matching subnetworks and winning tickets?\nFor self-contained paper and readers unfamiliar with LTH, it would be better if these terms are clarified in the revision.\n\nThe authors mainly use FID scores and additionally compare Inception Scores. It would be better to see if experiments on recent metrics (e.g. precision/recall) show the same result.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CKZZjOHaZrz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the ideas from lottery ticket hypothesis are evaluated in the case of SNGAN and CycleGAN. \n\nThe evaluations aim to validate the following empirical results with respect to the lottery ticket hypothesis i.e. the iterative magnitude pruning and channel pruning ideas work, discriminator can also be pruned in addition to generator, it outperforms other compression methods such as GAN slimming and that instead of re-initializing the weights of the masked network to the initialization, it could be reset to that from an earlier epoch.\n\nWhile most of the ideas evaluated have already been proposed for general model compression, in this paper they are specifically evaluated for GANs. \n\nThe proposed work is useful in terms of the study and analysis of a particular approach from model compression to that of GANs, it would be useful to understand if the results are meaningful specially for the much larger GAN models such as Progressive GANs, BigGAN for high resolution images. This is because, compression is especially useful for the larger models and more so when one is generating high resolution images. Many of the early results on GANs trained on CIFAR 10/100 and SVHN do not hold for the larger GAN models. \n\nTo summarize the pros and cons of the paper are as follows:\nPros:\n1) Evaluates various aspects of the Lottery ticket hypothesis for specific GAN models\n2) Demonstrates results for both image to image and generation from latent cases and shows improvements over GAN slimming\n3) Provides useful analysis for the lottery ticket hypothesis by analysing specific claims that are relevant\n\nCons:\n1) Larger GAN models and larger datasets are not evaluated. The approach could be more meaningful for those models and datasets\n2) The approach evaluates a specific lottery ticket hypothesis and does not provide new insights into the actual hypothesis itself. That is, by applying it to GANs we could only verify most of the already valid claims that were earlier proposed\n3) It is not clear that the ideas proposed would generalize to other GAN models. There are a huge number of GAN models proposed in literature and validating the ideas on all would be infeasible. It would however be useful to provide some more results on a few more GAN models, particularly addressing the first point.\n\n---\nThe response from the authors satisfactorily addresses several points that I raised. However, I am not fully convinced that the LTH on GANs has provided significant new insights. However, based on the response I am inclined to raise my score to above acceptance threshold and tend towards acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ArRl3AYQXPn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe authors study the lottery ticket hypothesis  for generative adversarial networks. Specifically, they attempt to answer the following questions: the existence of winning tickets in GANs; the effect of discriminator pruning in finding such winning tickets; the effect of initialization during the rewinding steps; and finally if the subnetworks found transfer across datasets. They provide extensive empirical evidence using that ```winning' tickets exist in GANs. Further they show that iterative magnitude pruning and channel pruning successfully find such `winning' subnetworks. They analyse the effect of discriminator pruning and find that initialization during the rewind step matters more than the actual pruning of the discriminator. Finally, they show state-of-the-art results on GAN compression through channel pruning.\n\n**Strengths**\n1. The paper is well-written and has cited relevant related work. \n\n2. The work is well-motivated and novel. The authors answer some important questions about LTH based methods.\n\n3. The experiments are extensive and help prove the authors' claims. I especially appreciate the attention to detail in the experiments, with various comparisions and ablation studies. \n\n**Weaknesses and Clarifications**\n\nFrom the given qualitative results, there does seem to be a loss in the finer features (edges and textures) upon sparsification. However, the FID scores do not seem to reflect this. Could the authors provide more visual results to analyse this?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8YDpDeT2JIc", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "GANs Can Play Lottery Tickets Too\n\nSummary: \n\nThis paper investigates model pruning and the Lottery Ticket Hypothesis in the context of GAN training. The authors apply a set of pruning techniques to SN-GANs on CIFAR and CycleGANs and examine whether pruning masks can be applied to the models at their initialization state (or their state 20% into training), taking into consideration the GAN-specific choice of whether to independently prune G or to prune G and D jointly. The results suggest that the sparsity masks obtained from pruned networks can be applied to networks at initialization up to a given sparsity level while recovering similar or better performance on the target task.\n\n\nMy take:\n\n\nThis is an empirical paper which focuses its efforts on experimental design and elucidating some details of pruning in GAN training. While I think the experiments are reasonably well-designed for their intention, my primary concern is that I am not convinced that there’s any reason that lottery tickets should matter in this context at all, given that the process (a) requires one to train a model to completion in order to obtain the masks, and then (b) require access to the trained model for a distillation loss. \n\nFrom my point of view, while it is interesting that one can use these masks on the networks in their init state, given that the actual goal of pruning is to obtain a slimmer model for inference (or to train models more quickly if pruning during training, which this technique does not permit), it is quite clear that standard pruning (pruning the pre-trained model rather than doing any sort of rewinding) is highly preferable in almost every way. That this result is downplayed in favor of highlighting that the authors have found lottery tickets is also concerning, albeit keeping with the general trend in the lottery ticket line of work which is more concerned with elucidating phenomenology than obtaining practical pruning or sparsity techniques. I feel strongly that the significance of this work will be diminished by this. On the other hand, I think that given that the intent of the paper is not as much to obtain the best possible sparsification technique as it is to investigate lottery tickets, this is perhaps okay, and I feel that the experimental design on this front is fairly strong and methodologically sound.\n\nMy other concern is that this paper’s clarity is poor. On the whole, the paper is not well presented, with poor or outright incorrect notation in many places. I struggled to read much of this paper due to this, and I feel that the paper’s impact would be strongly diminished due to this.\n\nOverall, I would rate this paper about a 4.8. I will argue against acceptance, but I don’t think this is 100% a clear reject and depending on the opinions of the other reviewers I would not feel that accepting this paper was completely out of bounds.\n\n\nDetailed and minor notes:\n\n\n\n-Bibliography is inconsistently formatted: many of the references are missing information such as venue. If a paper is only on arXiv, the reference should indicate this. Note that one can use Scholar to automatically get the bibtex for a paper, and that one should do one’s best to get the most recent information on where a paper has landed. For example, the citation for the ResNet paper has no indication of the venue, when it was accepted by CVPR after being on arXiv first. \n\n\n-” SNGAN with ResNet (He et al., 2015) is one of the most popular noise-to-image GAN network and has state-of-the-art performance on several datasets like CIFAR10.”\n\nThis is not true, SNGAN is far from SOTA on CIFAR. In general I don’t think we should care much about SOTA or “a few points below SOTA,” especially on datasets like CIFAR, but if the authors are specifically going to use the term “state-of-the-art” it is not acceptable to omit related work from the past three years. Please consider consulting paperswithcode for at least a fairly broad overview of recent literature that reports results on these datasets.\n\n\n-“let g(x; θg) be the output of the generator network G “.\n\nThe authors re-use the notation “x” (typically used to refer to a sample from the dataset) to refer to the random noise latent input to G, instead of the standard practice “z.” This might be acceptable if the authors did not immediately then re-use “x” to refer to the input to the discriminator, and then again as the inputs to both G and D in the image-to-image task. Please correct this misuse of notation.\n\nThe authors incorrectly use the notation “. in R^d1” in multiple places to refer to multiple different things which certainly have different dimensionalities, and additionally mismatch the notation when referring to variables which *should* have the same dimensionality (such as referring to m in {0,1}^d, the mask for a parameter theta in R^d1. This is further exacerbated as the subscript “d” is used to refer to the discriminator parameters in e.g. theta_d. Please correct this misuse of notation.\n\n-Figure 9: I can’t see any difference between the source image and the output of the full model, but I can see checkerboard artifacts in the highly sparse models. I found this figure confusing and out of place.\n\n-In the background and related work, I feel that the authors omit (or do not make clear) the critical detail of the Lottery Ticket Hypothesis, which is that these subnetworks *exist at initialization* assuming one uses the same initialization weights (i.e. initializes uses the same random seed). It is also important to note that the Lottery Ticket Hypothesis in this form does not hold past the very small-scale tasks on which it was originally tested, which is what necessitates the “rewinding” technique (and, in this reviewer’s opinion, greatly weakens the hypothesis).\n\n-“We use a benchmark pruning approach named Standard Pruning”\n\nIs there a reference for standard pruning, or is this novel? If this is something the authors did not introduce, appropriate citations should be included.\n\n\n-”In summary, initialization benefits finding winning tickets”\nI was not able to understand this statement; please consider rewording it.\n\n-It is not surprising that resetting the generator but not resetting the Discriminator fails to train (IMPFG in Figure 4), as Gs and Ds are basically always intrinsically paired in this manner.\n\n=========================================================================================\n\nEdit post Rebuttal:\n\nThank you to the authors for their in-depth response and the effort they put into responding to this review, and my apologies for not engaging during the discussion phase. I'll respond to several points with the goal of furthering the discussion to try and avoid unfairly \"having the final say\" when the authors cannot respond.\n\n-\"The biggest potential impact of our work is that it provides empirical evidence that lottery tickets exist in GANs\"\nThis reviewer's opinion is that this is not at all surprising. While GANs do have unusual and unique training dynamics on a number of levels, and interact interestingly with many typical building blocks, many of the aspects of neural-network based GAN models (prunability, the relationship between signal propagation and trainability, model capacity, etc) are largely indistinguishable from those of more \"typical\" neural nets. Nonetheless I do agree that, if one holds the lottery ticket line of work to be important or relevant (which I personally do not, but I will withhold my bias and not \"legislate from the bench\" here), its extension into the realm of GANs does first require verification of its existence in this regime. I apologize to the authors if it seems like they're pushing against a brick wall here because this reviewer is not a disciple of the lottery ticket hypothesis--please note that to the best of my ability this review is meant to be calibrated around this bias and instead focus on the strength of the manuscript.\n\n-Transfer between tasks: I appreciate the authors response on this point; transfer of masks between tasks (rather than pretrained weights) does indeed represent a different modality of transfer learning, and one which may be well-suited to GANs which are known to be difficult to re-train or fine-tune in many situations (although there is a growing body of work on this topic outside of the pruning/lottery ticket context).\n\n-\"Lottery tickets without early weight rewinding techniques can still hold to small-scale tasks. \"\nYes, this is precisely the problem--lottery tickets without early weight rewinding *do not, in general, tend to hold on large scale tasks*. This again returns to my baseline issue (one of many) with the hypothesis in the first place, that it has to be modified to work on even models like VGG.\n\n-on checkerboard artifacts: This reviewer is very well calibrated to viewing GAN samples, and holds that the checkerboard artifacts are vastly more visible in the sparsified models. I would encourage the authors to, in future work,  consider why this might be (what about sparse networks leads to increased checkerboard artifacts?) rather than seek to claim that the unpruned models are also checkerboard-y.\n\n-Additional experiments: I commend the authors for repeating their experiments on a range of architectures, and agree that the improvements for the sparsified models are in support of their argumentation and conclusions.\n\n-Thank you to the authors for updating their notation and bibliography.\n\nOn the whole, I still feel that this paper is borderline. While the author's responses are fairly strong in context, re-reading the manuscript, I still do not feel that the paper (which is most of what matters here) is especially strong. I am upgrading my score to around a 5.5 (which I will simply round up to a 6 on OpenReview). I think this paper is true borderline--I won't argue for its rejection, but I don't feel especially strongly in favor of it and cannot champion it.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommended accept or weak accept. There are some concerns on the novelty of this approach since this work mainly validates that lottery tickets can be found on GANs, which seems like applying an existing idea to a new problem. Nevertheless, there are two reasons that such an effort is interesting: first, the implementation of this idea may be harder than it seems; second, it was not known a priori whether lottery tickets would exist for GANs due to the significantly different optimization problem (game instead of minimization). I think this work will be of interest to the community, and recommend acceptance. 
Decision: Accept (Poster)