Double dynamic sparse training for GANs
Authors: Yite Wang
Jing Wu
Naira Hovakimyan
Ruoyu Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: empirical deep learning
neural network pruning
dynamic sparse training

Abstract: The past decade has witnessed a drastic increase in modern deep neural networks (DNNs) size, especially for generative adversarial networks (GANs). Since GANs usually suffer from high computational complexity, researchers have shown an increased interest in applying pruning methods to reduce the training and inference costs of GANs. Among different pruning methods invented for supervised learning, dynamic sparse training (DST) has gained increasing attention recently as it enjoys excellent training efficiency with comparable performance to post-hoc pruning. Hence, applying DST on GANs, where we train a sparse GAN with a fixed parameter count throughout training, seems to be a good candidate for reducing GAN training costs. However, a few challenges, including the degrading training instability, emerge due to the adversarial nature of GANs. Hence, we introduce a quantity called balance ratio (BR) to quantify the balance of the generator and the discriminator. We conduct a series of experiments to show the importance of BR in understanding sparse GAN training. Building upon single dynamic sparse training (SDST), where only the generator is adjusted during training, we propose double dynamic sparse training (DDST) to control the BR during GAN training. Empirically, DDST automatically determines the density of the discriminator and greatly boosts the performance of sparse GANs on multiple datasets.

Under review as a conference paper at ICLR 2023
DOUBLE DYNAMIC SPARSE TRAINING FOR GANs
Anonymous authors
Paper under double-blind review
ABSTRACT
The past decade has witnessed a drastic increase in modern deep neural networks (DNNs) size, especially for generative adversarial networks (GANs). Since GANs usually suffer from high computational complexity. researchers have shown an in- creased interest in applying pruning methods to reduce the training and inference costs of GANs. Among different pruning methods invented for supervised learn- ing, dynamic sparse training (DST) has gained increasing attention recently as it enjoys excellent training efficiency with comparable performance to post-hoc pruning. Hence, applying DST on GANs, where we train a sparse GAN with a fixed parameter count throughout training, seems to be a good candidate for reduc- ing GAN training costs. However, a few challenges, including the degrading train- ing instability, emerge due to the adversarial nature of GANs. Hence, we introduce quantity called balance ratio (BR) to quantify the balance of the generator and the discriminator. We conduct a series of experiments to show the importance BR in understanding sparse GAN training. Building upon single dynamic sparse training (SDST), where only the generator is adjusted during training, we propose double dynamic sparse training (DDST) to control the BR during GAN training. Empirically, DDST automatically determines the density of the discriminator and greatly boosts the performance of sparse GANs on multiple datasets.
INTRODUCTION
In the past decade, the training and inference costs of modern deep neural networks (DNNs) are gradually becoming prohibitive (He et al.) 2016 Dosovitskiy et al., 2020) Liu et al.) 2021d). espe- cially for large language models (Brown et al. 2020). Among all these large models, generative adversarial networks (GANs) (Goodfellow et al. 2020 have been widely investigated for years and achieved remarkable results. However, similar to other giant models, GANs are notably computa- tionally intensive. For example, BigGAN (Brock et al. 2018) trained on 8 NVIDIA V100 GPUs with full precision will take 15 days. Consequently, train GANs in broader resource-constrained scenarios, this computational bottleneck of training needs to be resolved urgently.
Neural network pruning has recently emerged as powerful tool to reduce training and inference costs of DNNs for supervised learning. There are mainly three genres of pruming methods, namely sumiingat--iitiialzztooo
and post-hoc pruning methods. Post-hoe prun- ing (Janowsky) 1989 LeCun et al. 1989] Han et al., 2015) can date back to the 1980s, which was first introduced for reducing inference time and memory requirements; hence does not align with our purpose of efficient training. Later,
pruning-during-traiie
antiinitializatinnn (Lee et al.) [2018] Wang et al., 2020a Tanaka al. 2020) and gruning-during-raann
methods (Louizos et al. [201] Wen al. 2016) were introduced to prune the networks before training and throughout the training, respec- tively. Most early pruning-during -uring-traainnng training algorithms (Savarese al., (020) gradually decrease the density of the neural networks and hence do not bring much training efficiency compared to post- hoc pruning. However, recent advances in dynamic sparse training (DST) (Evei et al., 2020 Liu ct al. b Mocanu al. 2018) for the first time show that pruning-during-trin training methods can have comparable training FLOPs as pruning-at-ine at-initiaaiiztiion methods while having competing performance with respect to post-hoc pruning. Therefore, applying DST GANs seems be promising choice.
Although DST has attained remarkable achievements in supervised learning. the application of DST on GANs is less explored due to newly emerged challenges. The main difficulty stems from the fact that the training procedure of GANs is notoriously brittle. To ensure successful training,

---

Under review as a conference paper at ICLR 2023
usually need carefully chosen architectures and finely-tuned hyper- ppprr-paameters.. One possible cause the difficulty of balancing the generator and the discriminator throughout training (Bai et al. 2018 overly-strong discriminator will lead to overfitting, while weak Arora et al. 2017). Specifically,
discriminator will result in mode collapse. As a consequence, the requirement of balanced training brings even more challenges to sparse GAN training. On the one hand, find that performance degradation caused by the unbalance of GANs is even more severe when sparsity is introduced. On the other hand, for directly applying DST the generator (or both) like the pioneering work STU- GAN (Luu et al. 2022), it is unclear how to determine reasonable density of the discriminator. To this end, we propose metric called balance ratio (BR), which measures the degree of balance of the two components, to study sparse GAN training.
We find that BR useful in (1) understanding the interaction between the discriminator and the generator, (2) identifying the cause of training failure, and (3) helping stabilize sparse GAN training as an indicator. To our best knowledge, this is the first study investigate the balance of sparse GANs and may even provide new insights into dense GAN training.
Using BR as an indicator, we further propose double dynamic sparse training (DDST) to adjust the density and the connections of the discriminator automatically during training.
Our contributions are summarized below:
We introduce quantity named balance ratio to quantify the degree of balance in GAN training, which also helps understand the cause of some training failure cases.
We first consider single dynamic sparse training (SDST), which is generalization of STU- GAN (Liu et al.) 2022]: applying DST to only the generator with varying discriminator density ratios. We show that SDST does not necessarily outperform the static sparse train- ing baselines.
We provide two strategies to determine the discriminator density for SDST, and we find that using a relatively larger density usually generates stable and better performance.
Using the balance ratio as an indicator, we propose double dynamic sparse training (DDST), which makes the discriminator dynamic both in density level and parameter level. Empiri- cally, DDST outperforms baselines with reasonable computational cost on several datasets.
2 RELATED WORKS
2.1 NEURAL NETWORK PRUNING
Based on the smallest granularity of pruned units, neural network pruning can be categorized into structured (Liu al. [0177] 2018 Huang & Wang 2018 Luo al. (017) and unstructured pruning (Frankle & Carbin| 2018 Han et al. (015). In this work, we mainly focus on unstructured pruning where individual weight the finest resolution.
Post-hoc pruning. Post-hoc pruning prunes weights of a fully-trained neural network, and they usually have high computation cost due to the multiple rounds of train-prune-retran procedure (Han
et al. 2015 Renda et al. 2020). Some use specific criteria (Han et al. [2015] LeCun et al. 1989
Hassibi et al. al..|11..1.1...1.1 | Molchanov al., 2019 Dai al. 2018 Guo al. [0066] Dong et al.
2017
Yu et al. 2018) to remove weights, while others perform extra optimization iterations (eera &
Pesquet 2021). Post-hoc pruning was initially proposed to reduce the inference time, while later
work on lottery ticket works (Frankle & Carbin 2018 Renda et al. 2020) aimed to mine trainable
sub-networks.
methods. SNIP (Lee et al. 2018) is one of the pioneering works which aim to find trainable sub-networks without any training. Some following works (Wang et al. [020aa]
mouning-atiiittiaiiine
Tanaka et al. 2020 de Jorge al. 2020 Alizadeh et al. (022)) aim propose different metrics to prune networks at initialization. Among them, Synflow (Tanaka et al. (020). SPP (Lee et al. 2019),
and FORCE (de Jorge al. 2020) try address the problem of layer collapse during pruning.
Neural tangent transfer (Liu & Zenke (2020) learns a sub-network by aligning the empirical neural tangent kernel and network output the dense counterpart.
2

---

Under review as a conference paper at ICLR 2023
Pruning-during-rrainie methods. Another genre of pruning algorithms gradually prunes dense DNNs throughout training. To mitigate performance drop after pruning, early works add explicit lo (Louizos et al. 2017) or & (Wen et al., 2016) regularization terms to encourage sparse solution.
Later works learn the subnetworks structures through projected gradient descent (Zhou et al. or trainable masks (Kang & Han
2021
2020
Kusupati al. 2020 Liu al., 2020) [aavarese al.
2020
Srinivas et al. [0017] Xiao ct al.,
2019).
However, these werriinndduinn-rriiiinn
methods often do
not enjoy memory sparsity during training. As remedy, DST methods (Bellec et al. [2017] Dettmers
& Zettlemoyer 2019: Evci et al.
2020, Liu et al. 2021abbe Mocanu et 2018 Mostataa & Wang, 2019 Graesser et al. 2022) were introduced to train the neural networks under a given parameter
budget while mask change is allowed during training.
2.2 GENERATIVE ADVERSARTAL. NETWORKS
Generative ad advessarial networks (GANs). GANs (Goodfellow et al. 2020) have drawn consider- able attention and have been widely investigated for years. Various architectures have been proposed to enhance the capability of GANs. Deep convolutional GANs (Radford et al.) 2015) replace fully- connected layers in the generator and the discriminator. After that, follow-up works (Brock et al.) 2018 Gulrajani al. 2017: Karras et al. 2017 Zhang al. [019) employed more advanced meth- ods to improve the fidelity of generated samples. Due to the difficulty of finding Nash Equilibrium, training of GAN is highly unstable. Therefore, several novel loss functions (Mao et al. 2017: Ar-
jovsky et al. 2017 Salimans et 2016 Gulrajani et al. 2017 Sun et al. (020), normalization and regularization methods (Miyato et al. 2018 Wu et al. 2021] lerjek 2019) were proposed sta- bilize the adversarial training. Besides the efforts devoted to the training of GAN, image-to-image translation is also extensively explored. Specifically, this direction includes semantic image synthe- sis (Zhu et al. (011b)) style transfer (Karras et al., (202bb: Choi et al. 2018: Zhu et al. 2017a), super resolution (Ledig et al. 2017: Wang et al. (018) etc.
GAN compression and pruning. Like other deep neural networks, the training and inference pro- cess of GANs requires massive resource consumption and memory. One of the promising ways is based neural architecture search and distillation algorithm (Li et al., 2020) Fu et al.) 2020 Hou et al. 2021). Another part of the work applied prune- based methods for GANs' generator com- pression (Shu al., 2019 Jin al. 2021: Yu & Pool 2020). Yet, they only focus on the pruning of generators, thus potentially posing a negative influence on Nash Equilibrium between generators and discriminators. Later, works by (Wang et al. 20206) presented unified framework by combing the methods mentioned above. Follow-up work by Li et al. (2021) compresses both components of GANs by letting the student GANs also learn the losses. Another line of work (Kalibhat al. [02T]
Chen et al. 2021) tries to test the existence of lottery tickets in GAN. However, most mentioned methods are not prepared for training efficiency and require over- peee-paraettrizedd GAN models in ad- vance. Directly training sparse GANs has been less explored so far. To the best of our knowledge, STU-GAN Liu et al. 2022 is the only work that tries to apply DST to GANs.
3 BALANCE RATIO: QUANTIFYING THE BALANCE OF SPARSE GANS
3.1 PRELIMINARY AND SETUPS
Generative adversarial networks (GANs) have two fundamental components, generator G(-; 0a) and discriminator ((: 0p). Specifically, the generator maps a sampled noise z from a multivariate normal distribution p(z) into a fake image to cheat the discriminator, whereas the discriminator distinguishes the generator's 's output and the real images Φ, from the distribution g(x:). Formally, the optimization objective of the two-player game defined in JS-GAN (Goodfellow et al. 2020 is defined as follows:
L(OD. 0G) = Ey (g.))())) [log( ((x,; 0D))] + Exsp(()) (log(1 D(G(2;
(1)
To be more specific, different loss can be used, including Wasserstein loss (Gulrajani et al., 2017) and hinge loss (Miyato et al., 2018). In this work, we use hinge loss for all GANs.
3

---

Under review as a conference paper at ICLR 2023
GAN sparse training. In this work, interested in sparse training for GANs. Specifically, the objective of sparse GAN training can be formulated as:
min max L(Op @ mmD, Og ④ ma) |maallope S de.
(2)
OG On
s.t. mn [0,1) mc {0, 1) mmpllopp dp,
where is the Hadamard product; Pp, mp, pp, dp the sparse solution, mask, number of parameters, and target density for the discriminator, respectively. The corresponding variables for the generator are denoted with subscript G. For pruning-at-initiann -tiitialiiinniiiiattionn methods, masks m are determined before training whereas m are dynamically adjusted for dynamic sparse training (DST) methods.
3.2 BALANCE OF GAN DURING TRAINING
As discussed in Seeciin 1 it is essential to maintain the balance of generator and discriminator during GAN training. As pointed out by Bai et al. (018)) and Arora et al. (2017), discriminators that are too strong lead over-fitting whereas weak discriminators comes to sparse GAN training, the consequences caused by the unbalance can be further amplified. Specifically, different from dense GAN training, densities of generators and discriminators can be varied significantly, leading to a more unbalanced worst-case scenario.
unable detect mode collapse. When it
To support claim, conduct experiments with SNGAN (Myytoo et al.
2018) the CIFAR-10 dataset. Following Liu et al. (2022), we start with static sparse training where densities of gener- ators and discriminators and masks ma, mp are determined using Brdōs-Rényi-Ken 2020) and are fixed throughout the training. More experiment details can be found Appendix A
chosen from (10% 20%, 30%, 50%, 100%). Layer-wise sparsity ratio (ERK) graph topology (Evci et al.,
Experiment results. Results are reported in Figure 2 First three plots in Figure 2 show the re- sults when varying density of discriminator dp for weak generators (dg E {10%, 20%, 30%]). )- We observe FID first decreases then increases. Specifically, neither overly-weal discriminators nor overly-strong discriminators can provide satisfactory performance. Similarly, for stronger genera- tors (de E (50%, 100%), only the dense discriminator with dG = 100% is not too weak to have satisfactory FID result. Hence, to ensure balanced training of GAN, it crucial to find the suitable sparsity ratio for the discriminator.
3.3 BALANCE RATIO
The observation in Section 3.2 raises a funda- mental question: is there way to quantify the degree of balance between the generator and the discriminator? To answer the question, introduce balance ratio (BR), which is, to the best of our knowledge, the first quantity that measures the balance of sparse generators and discriminators.
D(x))
33--
(()))))))
Balance
-100
At each training iteration, we draw random noise from a multivariate normal distribution and real images æ, from the training set. We denote the discriminator after gradient descent up- date as D(:; Op). We denote generator before and after gradient descent training as G (:; and Grost (:; O.), respectively. Then the balance ratio is defined as:
Figure I: Illustration of balance ratio.
BR D(GroR(2))
D(x,)) D(G (z))
3
Figure I also provides an illustration of BR. Specifically, BR measures how much improvement the generator achieve in the scale measured by the discriminator for a specific random noise z. When BR small (e.g., BR 30%). it means that the updated generator is too weak to trick the discriminator as the generated images are still considered fake. Similarly, for the case where BR large (c.g., BR> 80%), the discriminator considered too weak hence it will not provide useful information to the generator.
4

---

Under review as a conference paper at ICLR 2023
Deralty of peserater 10%
Daraly of gererator 20%
Daraiy of persratio 22.5
Damaly of gararatio dly5
Demity panerator 100100% 100%
5A Desily of dicovinstrr
24.
Desity of dhoteenate
Desity dechnenate
Deady doonninaaa
Dunsty diahnineee
Static
DA-strong
DA-mid
Figure 2: FID (4) of static sparsely trained SNGAN with and without DA on CIFAR-10 with different sparsity ratio combinations. The shaded areas denote the standard deviation.
Desuity penemter
Damiity peneator 20%20%
Deniity guneraor 4o-30%
Demity panerator 50m50%
Density panerator do- 100%
am who sit Traning
wit mit
09d Extaton
Foraton
Bassicn
dg-10% 40-200
30%
4g-50%
dl-100%
Figure 3: Balance ratio of static sparsely trained SNGAN on CIFAR-10 with different sparsity ratio combinations.
We now visualize the BR evolution throughout the training for the experiments in Section 3.2 The results are shown in Figure 3
The effectiveness of BR. We first check if BR can reflect the density increase (hence representation power increase) of the discriminator. In Figure we can see that for larger discriminator density dp. the BR is much lower throughout the training. Furthermore, the best density indicated by Figure 2 has overall BR in the range [0.3, 0.7].
Overly weak diseriminators lead to training failure. For the cases where the discriminators too weak compared to the generators, e.g., all cases where dp = 10%, we are able to observe the strongly oscillatory behavior of BR. More precisely, BR starts oscillate after it reaches value that is higher than 1.0. During the experiments, we also empirically observe that the FID gradually increases after such a turning point. As also shown in Figure 2 FID of overly-strong discrimina- tors (e.g., dp = 100%) are lower than overly-weak discriminators (e.g., dp = 10%). The such phenomenon seems to imply that performance degradation caused by overly-strong discriminators is better than failure caused by overly-weak discriminators.
3.4 DYNAMIC DENSITY ADJUSTMENT OF THE DISCRIMINATORS
We have shown in /eccion 3.3 that BR able to capture the degree of balance of the generators and discriminators. Hence, natural leverage BR dynamically adjust the density of discrimina- tors during sparse GAN training. Specifically, we initialize the initial density of the discriminator junt -G. After specific training iteration interval ATp. will adjust the density of the discrim- inator based on the averaged BR over last few iterations with pre-defined density increment Ad. With pre-defined BR bounds [B_. B+ we decrease dp by Ad when BR smaller than B_, and vise versa. Notice that the DA algorithm is in spirit very similar to MyyleGAN--ADA (Karras) al. 2020a) which adjusts augmentation probability with ADA. Out of simplicity, we increase density by growing the connections with the largest gradient magnitude (Evci et al. 2020). Global magnitude pruning used to drop connections so as to decrease the density. The algorithm is shown in Appendix ClAlorithm
We test our proposed methods dynamic density adjust (DA) with two target BR intervals, namely DA-strong ([__, B+] |0.3, 0.4]) DA-mild ([B_, B+] [0.45, 0.55)). DA-strong tends to find a relatively stronger discriminator, which results in lower BR throughout the training, whereas DA-mi d tends make the discriminator and the generator relatively balanced, i.e., BR xs 0.5.
5

---

Under review as a conference paper at ICLR 2023
Dauily genzulor Nim100
Deady of gerpuder dgw20ts
Denily goscatoo dpw22%
Darily of gonaetor Spw50h
Deaily of gossetor
Derity contrnrnee
Derody ducie Nat
Unmely of Store Buter
Dmmby of dioverate do
Derddy
Static
5D5T gh)
SDST(SET)
Figure 4: FID (4) comparison of SDST against static sparse training for SNGAN on CIFAR-10 with different sparsity ratio combinations. The shaded areas denote the standard deviation.
Experiment results. Results are shown Figure with dashed lines. For stronger generators (dg E (30%, 50%, 100%}), both DA-st and DA-mi 1d are able to identify reasonable dis- criminator densities. While for weak generators (dg E {10%, 20%}), DA-mild shows a more stable performance. The experiments show the significant benefits brought by BR. Furthermore, they again support our claims that neither overly-strong nor weak discriminators can lead to bal- anced and successful GAN training.
4 IS ONLY ADJUSTING THE GENERATOR ENOUGH FOR SPARSE GANS?
In this section, method on GAN where only the generator dynamically adjusts masks during the training. We do not consider naively applying DST both generators and discriminators, et al, 2022). it is empirically shown that adjusting both components simultaneously generates worse performance with more severe training instability. We name such method single dynamic sparse training (SDST) as only one component of the GAN, i.e., the generator, is dynamic. Hence, STU- GAN is a special case for SDST, which grows connections based gradients
going to test DST on GANs. We first test SDST, direct application of DST
in STU-GAN (Liu
We follow the same setting considered in Section 3.2 where the densities of the generators de and discriminators dD are chosen from (10%, 20%, 30%, 50%, 100%). Detailed DST procedure and corresponding pprem-rareeterss can be found in Appendix B
Experiment results. We show the experiment results in Figure 4 The first observation is that the performance of RigL and SET does not vary much in general. The second observation that SDST better than static sparse training when the discriminator is strong enough. More specifically, for de E (10%, 20%). SDST method is worse than static sparse training when the density of the discriminator is weak (d, 10%). On the contrary, when the discriminator strong enough, dp E {20%, 30%, 50% 100%). reason is that the in-time over- paramcterrizionoonnn induced by DST increases the representation power of the generator. Such boost is beneficial only when the discriminator has matching better representation power.
great performance boost brought by SDST. The
Despite the superior performance of STU-GAN (or SDST in general) at higher density ratios, there exist some limitations for SDST, which are summarized as follows:
When using SDST, dp is manually chosen before training. However, it is unclear what is good choice. In real-world scenarios, it not practical to search for the optimal dp for each de-
The issue of GAN unbalance is unresolved during training since the density of the discriminator fixed. As shown in Figure 4. the best performance is not always obtained with the maximum dn = 100%. If cost for a worse performance.
using an overly-strong discriminator, we wasting extra computational
Hence, STU-GAN (or SDST in general), which directly applies DST the generator, may only be useful when the corresponding discriminator is strong enough. In this sense, to deal with more complicated scenarios, obtaining balanced training in automatic way is essential in GAN dynamic sparse training.
Notice that STU-GAN almost identical
SDST (Ri gl.) with EMA tailored for DST.

---

Under review as a conference paper at ICLR 2023
Table 1: FID (L) of different sparse training methods on CIFAR-10 and STL-10 datasets with no constraint on the density of the discriminator. Best results are in bold; second-best results are underlined.
CIFAR-10
STL-10
Generator density
10%
20 %
30
50 %
10%
20
30
%
(Dense Baseline)
10.74
29.71
Batio-Balance
26.73
18.04
14,38
12.22
50.08
44.19
43.96
37.21
Matic-Strong
26.60 60
19.47
14.60
11.28
52.03
44.04
42.53
38.33
DST-balance (SET)
32.02
18.54
14.74
13.23
49.91
33.71
32.92
31.75
DST-balance (RigL)
24.56
15.53
13.62
12:51
66.90
50.34
44.57
32.63
SDST B-Blancee (SET)
27.80
18.13
14.15
12.32
63.57
49.09 09.055
43.74
31.29
SDST-Strong (SET)
12.29
78.34
54.31
41.77
32.32
SDST-Balance Balance (RigL.)
30.38
17.89
14.95
12.09
46.17
38. 12
31.88
31.30
SDST DosTn- (Rigl.)
16.95
14.26
12.36
11.47
48.04
34.24
32.67
30.40
R-DDST (SET)
11.71
10.97
56.15
45
R-DDST (RigL)
13.77
12.33
11.46
11.18
42.72
33.12
32.44
30.88
Table 2: FID (L) and normalized training FLOPs of different sparse training methods with BigGAN on CIFAR-10 dataset. Best results are in bold; second-best results are underlined.
FID (1)
Normalized training FL.PPs
Generator density
10%
20 %
30
50 %
10%
20
30 %
(Dense Baseline)
8.43
6.80 x10 (100%)
Stutic-Balance
17.46
13.13
10.90
8.54
9.78%
19,04%
28.68%
49.12%
Static-Strong Strong
22.96
13.54
11.54
9.02
83.90%
84.93%
86.32%
90.15%
DOST-Balance (RigI.)
11.98
9.58
8.96
7.92
9.91%
19.41%
28.90%
38%
SDST-Strong (RigL)
10.79
9.30
8.82
8.30
84.04%
85.22%
86.54%
89. 56%
R. R-DDST (Rigl.)
9.58
8.77
8.11
8.17
9.77%
24.85%
40.00%
77.13%
DOUBLE DYNAMIC SPARSE TRAINING FOR GANS
STU-GAN (or SDST in general) considered in the last section cannot generate stable and satisfy- ing performance. This implies that we should utilize the discriminator in a better way rather than just directly applying DST to the discriminator. Consequently, DA (Section 3.4), which adjusts the diseriminator density to stabilize GAN training, is a favorable candidate to address the issue. We name the proposed method double dynamic sparse training (DDST), which adjusts the density of the discriminator during training with BR as the indicator while the generator performs DST. We propose two DDST methods, namely R-DDST and S SDDDST based on whether we give constraints on the maximum density of the discriminator. We present them in Section 5.1 and [Section 5.2 We use the word double for the following two reasons: 1 both the generators and the discriminators dynamic (both R-DDST and S S-DDST); @ the discriminator enjoys two levels of dynamic flexibility, namely density level and parameter level (S-DDST). Such method has much more flexibility and generates more stable performance compared to SDST.
5.1 RELAXED DOUBLE DYNAMIC SPARSE TRAINING
We first investigate the direct combination of SDST with DA. Specifically, the generator is adjusted using SDST mentioned in Section 4 while the density of the generator is dynamically adjusted with DA as mentioned in in[eection 3.4 We call such combination relaxed double dynamic sparse training (R-DDST) as it does not necessarily introduce sparsity to the discriminator, and the density of the discriminator can as high as 100% (hence dense discriminator). For a fair comparison, baseline methods can use the discriminator with arbitrary sparsity ratio, i.c., dp e [dmin dmax] [0%, 100%].
Comparison to STU-GAN (SDST). Compared to STU-GAN (or SDST in general) which pre- defines the discriminator density before training, the difference is that for R-DDST, the density of the discriminator is adjusted during the training process automatically through DA. Given the initial discriminator density d'd de. R-DDST automatically increases the discriminator density if a stronger discriminator is needed, and vice versa.
7

---

Under review as a conference paper at ICLR 2023
Datasets, architectures, and target sparsity ratios. We first conduct experiments SNGAN with ResNet architectures on CIFAR-10 10 (Krizhevsky et al. 2009) and STL-10 (Coates et al. 2011)
datasets. Target density ratios of the generators de are chosen from {10%, 20%, 30%, 50% Please
see Appendix A for more experiment details.
Baseline methods and R-DDST. We static (Section 3.2) and SDST (Section 4) base- lines. Since these baselines use pre-defined discriminator density ratios, we propose two strategies define the discriminator density ratios based on the results from [Section 3.2 and [Section 4 O balance strategy, where we set the density of the discriminator dp the same as the density of the generator de; strong strategy, where we set the density of the discriminator as large as possible, i... dp= = dmax. In this section, we use dmax = 100% for the strong strategy. For SDST methods, we test both grow methods, i.e., + SDST (SET) (Mocanu et al. 2018) which grows connections randomly and SDST (RigT) (Evci et al. 2020 which grows connections via gradient.
Similar to SDST, we again consider R-DDST (SET) and R-DDST (RigL) which differ based on how R-DDST grows connections. One thing to notice is that we use the same growth criterion for the generator and the discriminator out of simplicity. More training details can be found in Ap- pendix B FID results on the training set are shown in Table T More results of SNGAN on CIFAR-T0 test set can be found in Appendix Training costs comparison can be found in Appendix
The strong strategy and the balance strategy. For almost all density ratios of SNGAN (CIFAR-10) experiments, using the strong strategy is always comparable to or better than the balance strategy. The difference between the two is almost negligible when applied to static methods. However, for SDST methods, using stronger discriminators always leads to a large performance gain.
For SNGAN on the STL-10 dataset, the advantage of the strong strategy over the balance strategy is no longer obvious. Precisely, for 3 out of 8 cases, using the strong strategy is better than using the balance strategy. The explanation is that the size difference between generators and discriminators is larger for STL-10. Hence, the degree of unbalance is more severe and leads to more detrimental effects.
R-DDST identifies reasonable discriminator density. For the CIFAR-I0 dataset, we find that R-DDST consistently performs better than the corresponding baselines with the same grow methods. This illustrates that R-DDST is flexible and able to find suitable discriminator density compared to the two baseline strategies, i.c., the strong and the balance strategy.
For the STL-10 dataset, R-DDST (RigL) performs consistently better than R-DDST (SET) and baselines, whereas * R-DDST (SET) not competitive any more. We postulate that under such setting where the dataset scales up and the training is more difficult, gradient growth not only identifies important connections of the generator but also provides efficient representation power growth of the discriminator to balance the growth of the generator. Please also see Apppeddix D for the time evolution of BR and the discriminator density during training for R-DDST methods.
Larger GAN model experiments. We have also conducted experiments with BigGAN (Brock ct al. 2018) on CIFAR-10 datasets. Based on the SNGAN results, we compare all RigL variants with static ic baselines. FID and normalized training FLOOPs with respect to dense training shown in Table 2 The results show that + R-CDST shows stable performance and outperforms other baselines most of the time. Moreover, compared to the second best method SDST-Strong, R-DDST not only shows lower FID but also requires much less training cost.
Main takeaway. In this section, we compared R-DDST with sparse training baselines. We find that Rigl. and strong strategy are preferred compared to + SET and balance strategy. SDST (Ri gT.) with strong strategy generally generates better performance compared other sparse training base- lines. Finally, R- RDDDST (RigL) beats SDST (RigL) with much less computational cost and always ranked top two among all methods.
5.2 STRICT DOUBLE DYNAMIC SPARSE TRAINING
R-DDST introduced in the previous section does not necessarily introduce sparsity for the discrim- inator, which provides less memory/traning resources saving for larger generator density ratios. Hence, we further present strict double dynamic sparse training (S-DDST) in this section which enforces the discriminator to be sparse, i.e., dp & dmax < 100%. In this section, assume that
8

---

Under review as a conference paper at ICLR 2023
Table FID ()) of different sparse training methods on CIFAR-IO and STL-10 datasets. The density of the discriminator is constrained to be lower than 50%. Best results are in bold; second-best results are underlined.
CIFAR-10
STL-10
Generator density
10%
20
30 %
%
10%
20
30 %:
%
(Dense Bascline)
10.74
29.71
Static-balance
26.73
18.04
14.38
12.22
50.08
44.19
43.96
37.21
Static-strong
13.47
12.22
44.95
42.12
DST halanaa (SET)
32.02
18.54
14.74
13.22
49.91
33.71
32.92
31.75
DST-bulance (RigL)
13.62
50.34
44.57
32.63
SDST-ballacc (SET)
27.80
18.13
14.15
12:32
63.57
49.05
43.74
31.29
SDST-strong (SET)
13.31
13.17
12.32
33.56
32.19
31.29
SDDTT-balance (RigL)
30.38
17.89
14.95
12.09
46.17
38.12
32.48
31.30
SDST-stuong (RigL)
13.20
12.99
12.09
33.45
32.09
S-DDST (SET)
14.22
13.30
12.39
11.97
51.72
35,
42.36
31.68
S-DDST (Rigl.)
14.13
12.87
12.15
12.17
44.28
32.84
32,00
30.28
we can use the discriminator with sparsity ratio dp E [dmin dmax] = [0%, 50%]. Compared with R-DDST, the learning process will be more challenging with the introduced constraints on the max- imal discriminator density. S-DDST consists of two phases and works as follows:
Density exploration of the discriminator. During the first phase, S-DDST performs just like R-DDST, with the exception that we apply the constraint dp S dmax < 100%. Concretely, S-DDST aims to find a suitable discriminator density db with DA algorithm in the first half of training.
Paramter exploration of the discriminator. During the second phase, both the generator and discriminator are adjusted using DST with fixed discriminator density d' found in the first phase.
Baseline methods and S-DDST. We use the same baselines and adopt the same general setup in Section 5.T We divide the training iterations evenly for two phases. For comprehensive com- parison, we continue to report FID results from two growth methods, i.e., S-DDST (SET) and and other results can found Appendix E.
S-DDST (RigT.)
Table
S-DDST shows stable and superior performance. For the CIFAR-10 dataset, we notice that S DDDDST stably surpasses its corresponding baselines regardless of grow methods and initial density of discriminators and generators. Even with further constraint on the discriminator, DA is still able to improve GANs training and can explore more reasonable density than the strong and the balance strategy. For STL-10 dataset, S-DDST (RigL) again shows the most promising performance. Please also see Figure 7 in Appendix D for discriminator density and BR evolution during training.
Main takeaway. In this section, we report the results from S-DDST (RigT) with its baselines. Generally, Ri Ri still demonstrates encouraging results compared with SET in most experiments when extra sparsity is introduced. While the strong strategy shows favorable performance in the CIFAR-10 dataset, the gain is not salient when the size of the backbone increase and the training dataset scales up to STL-10. Most importantly, S S-DDST (RigL) is able to have comparable performance in some cases when compared to R-DDST (RigL) and outperforms SDST (RigL) after we restrict the maximal density of discriminators.
CONCLUSION
In this paper, we study DST for GANs. We find that simply applying DST methods the generator is not sufficient to improve the performance of sparse GANs. Hence, we propose to use BR to measure the degree of unbalance between the generator and the discriminator. We find that the application of DST only on the generator is beneficial when the discriminator is relatively stronger. Furthermore, we propose two methods, namely RDDDST and S S-DDST, to dynamically adjust the discriminator in both parameter and density levels. Both of these methods demonstrate encouraging results. Our study may help researchers have a better understanding of the balance of GAN training and encourage more researchers to investigate sparse training for generative models.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we will include a link to an anonymous repository after the discussion forums are open. All the experiment details can be found in Section Section 5.1, Section 5.2
Appendix A and Appendix B
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Milad Alizadeh, Shyam A Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initializa- tion using meta-grat meta-gradients. arXiv preprint arXiv:2202.08/32, 2022.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pp. 224- 232. PMLR, 2017.
Yu Bai, Tengyu Ma, and Andrej Risteski. Approximability of discriminators implies diversity in gans. arXiv preprint 1806 10586, 2018.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv: /711.5 136, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. arXiv preprint ar arXiv:2106.00134, 2021.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMIR Workshop and Conference Proceedings, 2011.
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the varia- tional information bottleneck. In International Conference on Machine Learning, pp. 1135-1144. PMLR, 2018
Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Tor, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint (wxXi:200660908/1 2020.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 803300365, 2018.

---

Under review as a conference paper at ICLR 2023
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan-distiller: Searching to compress generative adversarial networks. arXiv preprint arXiv:2006.000 20f1::2000008/18,, 2020.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):1139-44 2020.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792. PMLR. 2022.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299, IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Liang Hou, Zehuan Yuan, Lei Huang. Huawei Shen, Xueqi Cheng, and Changhu Wang. Slimmable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7746-7753, 2021.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-30 -20, 2018.
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, (911))6600,, 1989.
Qing Jin, Jian Ren, Oliver Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13600-13611, 2021.
Neha Mukund Kalibhat, Yogesh Balaji, and Soheil Feizi. Winning lottery tickets in deep generative models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8038- 8046, 2021.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pp. 5122-5131. PMLR, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances Neural Information Processing Systems, 13:12104-121 14, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

Under review as a conference paper at ICLR 2023
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, pp.
PMLR, 2020.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv: ( 2018.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv: 100.:::990600600.,, 2019.
Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun- Yan Zhu, and Song Han. Gan compression: Ef- ficient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5284-5294, 2020.
Shaojie Li, Jie Wu, Xuefeng Xiao, Fei Chao, Xudong Mao, and Rongrong Ji. Revisiting discrimi- nator in gan compression: A generator-Giseriinnal cooperative compression scheme, Advances in Neural Information Processing Systems, 34:28560-2857,, 2021.
Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So. Dynamic sparse train- ing: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005.0683 2020.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9922 2021a.
Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy. Selfish sparse training. In International Conference on Machine Learning, pp. 6893-6904. PMLR, 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense paraetettrtiiiinnnnn in-time wer-parramttrizzaioonn in sparse training, In International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021c
Shiwei Liu, Yuesong Tian, Tianlong Chen, and Shen. Don't be so dense: Sparse-to-sparse gan training without sacrificing performance. arXiv preprint arXhv:2203.027700 2022.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent trans- fer. In International Conference on Machine Learning, pp. 6336-6347. PMLR, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 10012-10022, 2021d.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint urXiv:1810.05270, 2018.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint 17 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-50666 2017.
13

---

Under review as a conference paper at ICLR 2023
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. Proceedings of the IEEE international confer- on computer vision, pp. 2794-2802, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 1r02v:1802.0533 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018.
Pavlo Molchanov, Arum Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11264-11272 2019.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference Machine Learning, pp- 4646-4655. PMLR, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.0644 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003. 02389, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 1380-1 1390, 2020.
Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co-evolutionary compression for unpaired image translation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3235-3244, 2019.
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 338-145, 2017.
Ruoyu Sun, Tiantian Fang, and Alexander Schwing. Towards better global loss landscape of gans. Advances in Neural Information Processing Systems, 33: 10186-10198, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems, 33:6377-6389, 2020.
Dávid Terjék. Adversarial lipschitz. arXiv preprint arXiv: 2019.
Sagar Verma and Christhrittephe Pesquet. Sparsifying networks via subdifferential inclusion. In International Conference Machine Learning, pp. 10542-10552, PMLR, 2021.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020a.
Haotao Wang, Shupeng Gui, Haichuan Yang. Ji Liu, and Zhangyang Wang. Gan slimming: All-in- one gan compression by a unified optimization framework. In European Conference on Computer Vision, pp. 54-73. Springer, 2020b.
Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
14

---

Under review as a conference paper at ICLR 2023
Yi-Lun Wu, Hong-Han Shuai, Zhi-Rui Tam, and Hong- Yu Chiu. Gradient normalization for gener- ative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 6373 6373-6382, 2021.
Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019,
Yasin Yaz, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, Vijay Chan- drasekhar, et al. The unusual effectiveness of averaging gan training. International Confer- ence on Learning Representations, 2018.
Chong Yu and Jeff Pool. Self-supervised generative adversarial compression. Advances in Neural Information Processing Systems, 33:8235-8222 2020.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9194- 9203, 2018
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference machine learning, pp. 7354-7363. PMLR, 2019.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020.
Xiao Zhou, Weizhong Zhang. Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 3599-3608, 2021.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imagee translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017a
Jun-Ya Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems, 30, 2017b.
15

---

Under review as a conference paper at ICLR 2023
A
EXPERIMENTAL SETUP
Our code is mainly based on the original code of ITOP (Liu et al. c02 and GAN ticket CCen
al. 2021
A.1
ARCHITECTURE DETAILS
We use ResNet-32 He et al. 2016) for CIFAR-10 dataset and ResNet-48 for STL-10 dataset. See Table 4 and Table 5 for detailed architectures. Spectral normalization is applied for all fully- connected layers and convolutional layers the discriminators.
For BigGAN architecture, we use the implementation used in Zhao et al. 2020
A.2 DATASETS
We use the training set of CIFAR-10 and unlabeled partition of 110 for GAN training. Train- ing images are resized to 32 x 32 and 48 48 for CIFAR-10 and STL.-10 datasets, respectively. Augmentation methods for both datasets random horizontal flip and per-channel normalization.
A.3 TRAINING HYPERPARAMETERS
SNGAN the CIFAR-10 and STL-10 datasets. We use learning rate of 2 X 10 for both generators and discriminators. The discriminator is updated five times for every generator update. We adopt Adam optimizer with $1 0 and 82 0.9. The batch size of the discriminator and the generator is set to be 64 and 128, respectively. Hinge loss is used following Brock et al. (2018); Chen et al. (202T). We use exponential exponential moving average (EMA) Yaz al. 2018 with 3
0.999. The generator is trained for total of 100k iterations.
of 2 x 10-4 for both generators and discriminators. The discriminator is updated four times for every generator update. We adopt Adam optimizer with 31 0 and B2 0.999. The batch size of both the discriminator and the generator is set to be 50. Hinge loss is used following Brock et al. (2018): [uu et al. (2021) We use EMA with 0.9999. The generator is trained for total 200k iterations.
BigGAN on the CIFAR-10 dataset. We
learning
A.4 EVALUATION METRIC
SNGAN on the CIFAR-10 and the STL-10 datasets. We compute Fréchet inception distance (FID) and Inception score (IS) for 50k generated images every 2000 iterations. Best FID and IS are reported. For the CIFAR-10 dataset, we report both FID for the training set and test set, whereas, for the STL-10 dataset, we report the FID of the unlabeled partition.
BigGAN on the CIFAR-10 dataset. We compute Fréchet inception distance (FID) and Inception score (IS) for 10k generated images every 5000 iterations. Best FID and IS are reported.
B DYNAMIC SPARSE TRAINING DETAILS
B. GENERAL DST IYPERPARAMETERS
Following Evci et al. 22020), we specify the hyper-parameters of DST through sparsity distribution, update schedule, drop criterion, and grow criterion. We explain the details of DST below.
: Liu et al. 2021c), only param- eters of fully connected layers and convolutional layers will be pruned. At initialization, we use the commonly adopted Erdōs-Rényi- Selis-Renii-Keeee Kernel (ERK) strategy (Evci et al. 2020 Dettmers & Zettlemoyer
Sparsity distribution at initialization. Following Evci et al. (2200):
2019 Liu et al. 2021c) to allocates higher sparsity to larger layers. Specifically, the sparsity of convolutional layers / is scaled with n! n'-14n' m thl where n' denotes the number of channels
/https://gi
han- wttp:///aaaiiiiiieeaeeaaeeeeeeee
Di Diffuugmett big.akggneeet--iiannn-nnne-eeee-
16

---

Under review as a conference paper at ICLR 2023
of layer/ 1 while w/ and h' are the width and the height of the corresponding kernel in that layer. For fully connected layers, Erdös-Rényí Rényi (ER) strategy is used, where the sparsity is scaled with
1- m
Drop and grow, After AT training iterations, we update the mask my by dropping//runing S (7, T Pode number of connections with the lowest magnitude, where pc, de are the num- ber of parameters and target density for the generator, faceay (7, is the update schedule, which will be explained in the next part. Right after the connection drop, we regrow the same amount of connections.
For the growing criterion, test both random growth SDST (SET) (Mocanu et al. 2018 Liu
et al. 2021c) and gradient-based growth * SDST (RigL) (Evci et al. 2020). Concretely y, gradient- based methods find newly-activater connections with highest gradient magnitude "0 while random based methods explore connections in a random fashion. All the newly-activated connec- tions set to O. One thing that should be noticed is that while previous works consider layer-wise connections drop and growth, we grow and drop connections globally as it grants more flexibility to the SDST method.
Update schedule. The update schedule can be specified by the number of training iterations be- tween sparse connectivity updates AT, the initial fraction of connections adjusted ", and decaying schedule faecay ( T) for-y. y.
EMA for sparse GAN. EMA (Yaz et al. 2018) is well-known for its ability alleviate the non- convergence of GAN. We also implement EMA for sparse GAN training. Specifically, we zero out the moving average of dropped weights whenever there mask change.
B.2 DST HYPERPARAMETERS FOR SDST
SNGAN on the CIFAR-10 and the STL-10 datasets. The connection update frequency of the generator AT is set to 500 and 1000 for the CIFAR-10 dataset and STL-10 dataset, respectively. The initial set to 0.5 and we use cosine annealing function faecay following Rigl. and ITOP.
BigGAN on the CIFAR-10 dataset. The connection update frequency of the generator AT is set to be 1000. The initial y is set to 0.5 and we use a cosine annealing function fdecay following Rigl. and ITOP.
B.3 DYNAMIC ADJUST AND DST HYPERPARAMETER FOR DDST
R-DDST. For R-DDST, only the generator is adjusted using DST while the discriminator is ad- justed using dynamic adjust (DA). The DA bounds are chosen to [0.475, 0.525), [0.45, 0.55). and [0.45, 0.55] for SNGAN (CIFAR-10). SNGAN (STL--10) and BigGAN (CIFAR-10), respectively. Ad is set to be 0.05, 0.025, 0.05 for SNGAN (CIFAR-1O), SNGAN (STL-10) and BigGAN (CIFAR- 10), respectively. The density of the discriminator is adjusted every 1000, 2000, and 5000 iterations for the three settings, respectively. Time- averggee BR over 1000 iterations is used as the indicator. We use the same setting used in Section B.2 for the generator.
S-DDST. For S-DDST, the discriminator is adjusted using DA in the first half of training, i.c., the first 50,000 iterations. In the second half of the training, the discriminator is adjusted using DST. The generator is only adjusted with DST. For the DA bounds, they are set as [0.45, 0.55] and [0.475, 0.525] for CIFAR-10 and STL-10 dataset, respectively. The density of the discriminator is adjusted every 2000 iterations for each dataset. The density of the generator is adjusted every 1000 iterations.
We compute BR for every iteration to visualize the BR evolution, whereas one should note that such computational cost can be greatly decreased if BR is computed every AT iterations.
C ALGORITHMS
In this section, we present the detailed algorithms for both DA and S-DDST. We do not present the algorithm of R-DDST as it is combination of DA and SDST.
17

---

Under review as a conference paper at ICLR 2023
Table 5: ResNet architecture for STL-10.
Table 4: ResNet architecture for CIFAR-10.
(a) Generator
(b) Discriminator
(a) Generator
(h) Discriminator
3 E T2 ~ (0),
image x é |-1,
2 12 (00,))
image ar с [-1, 1j
dense, 6 x 6 x 512
ResBlock down
dense, x 256
ResBlock 128
ResBlock up 256
ResBlock down 128
ResBlock 256
ResBlock down 128
ResBlock 128
ResBlock down 256
ResBlock up 256
ResBlock down 128
ResBlock up 64
ResBlock down 512
ResBlock up 256
ResBlock down
BN ReI.U, cony, Tanh
ResBlock down 1024
BN, ReLU, 3 conv, Tanh
ReLU 0.1
ReLU 0.1
Global sum pooling
Global avg pooling
dense
dense
C.I
DYNAMIC ADJUST ALGORITHM
We first present DA Algorithm
Algorithm Dynamic density adjust (DA) for the discriminator.
Require:
Generator G, discriminator D, DA upper bound B, and lower bound B., DA interval ATp, density increment Ad, grow method A. drop method B, iteration t.
then 2: Compute time-averaged BR with [Equation 3
1: if mod ATp
3: if BR is greater or equal to B4 then
4: Increase the density of discriminator from dp to dn + Ad using given grow method A.
5: else if BR is less equal B_ then
6: Decrease density discriminator from dn to dp Ad using given drop method B.
7: end if
8:
end if
C.2 STRICT DOUBLE DYNAMIC SPARSE TRAINING ALGORITHM
Details of S SDDDST algorithm presented in Algorithm 2
Algorithm 2 Strict double dynamic sparse training (S-DDST) for GANs.
Require: Generator G. discriminator D, total number of iterations T, number of training steps for discrimina- for in each iteration N, maximal density of discriminator dmwn.
1: for i in [1, .2" do
2: for in [1,-..,,, .N] do
3:
Compute the loss discriminator (p(0p) (p((p)).bcckuarr
4:
5:
end for
6:
Compute the loss of generator La(Oc)
7:
backkar)))
8:
if t is less than 0.5 + T and current density of discriminator dp is less than dmex then
9:
Apply DA in Algorithm to L)
10:
else
11:
Apply DST D end
13:
Apply DST to G
14:
end for
D DDST BALANCE RATIO EVOLUTION
In this section, we show that DDST methods are able to maintain a BR throughout training. We show the time evolution of BR and discriminator density for CIFAR-10 and STL-10 datasets.
18

---

Under review as a conference paper at ICLR 2023
Density of the generator 10=10%
Density of the generator de 20%
1.00
Densiay of the discriminaton
1.00
0.6
0.75
raño
0.75
0.25
0.25
Density
0.00
0.2
0.00
-0.25
20000
10000
60000
80000
100000
20000
40000
60000
80000
100000
Training
Training iteration
Density the generator d;= do=30%
Density of the generator de -5=50%
1.25
-10
Deniity of the disch discr minator
1.25
110
1.00
1.00
Balance rablo
0.8
0.50
0.50
0.6
0.25
0.25
0.4
0.6
Donsity
0.25
0.25
0.5
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
Training iteration
Training iteration
Figure 5: Balance ratio and discriminator density evolution during training for R-DDST (RigL) on CIFAR-10. Dashed lines represent BR values of 0.45 and 0.55. Density of the generator dg=20%
Density of the generator de= -0%
1.25
1.25
0.10
Densty of discriminator
0.35
1.00
1.00
Balance ratio
0.06
Balance ratio
0.30
0.50
0.06
0.50
0.25
the
0.25
0.25
0.04
0.20
0.15
00255
-0.25
20000
10000
60000
80000
100000
20000
20000
00000
80000
100000
Training
Training
Density of the generator do-30%
Density of the generator de 50%
1.25
1.26
1.0
of discriminatox
1.00
Balance rabl
Balance rablo
0.75
0.8
0.50
0.50
0.25
0.25
0.8
0.00
-0.25
-0.25
20000
40000
60000
80000
100000
20000
40000
60000
50000
10000
Training
Training
Figure 6: Balance ratio and discriminator density evolution during training for R-DDST (RigL) on STL-10. Dashed lines represent BR values of 0.45 and 0.55.
D. R-DDST
Results are shown in Figure and Figure 6, It clearly illustrates the ability of R R-DDST (RigL) to keep the BR controlled during GAN training.
D.2 S-DDST
Results of S S-DDST are shown Figure and Figure clearly illustrates the ability of S-DDST (RigT.) to keep the BR controlled during GAN training.
19

---

Under review as a conference paper at ICLR 2023
Density of the generator 10%
Density of the generator do=20%
1.26
Density discriinaoo
1.00
1.00
0.75
Balance rato
0.76
Balence
0.25
0.25
Density
0.00
0.00
0.255
Training iteration
Training iteration
Density of the generator de 30%
Density of the generator dg 50%
1.25
1.00
Demsity of discimiratoo
1.00
0.52
0.75
Bolance raño
0.75
0.57
Balance
0.25
0.35
0.26
0.49
0.00
0.00
0.48
0.30
60000
80000
Training iteration
Training iteration
Figure 7: Balance ratio and discriminator density evolution during training for S DDST (RigL) on CIFAR-10. Dashed lines represent BR values of 0.45 and 0.55. Density of the generator d0=10%
Density of the generator 2o=20%
1.25
1.25
1.00
0.75
0.3
Balance rablo
0.75
dicc
Balance
0.50
0.50
0.2
0.25
0.25
0.00
0.00
0.25
0.25
0.1
60000
80000
100000
10000
Training iteration
Training iteration
Density of the generator dc=30%
Density of the generator dg 50%
1.25
discrminstor
1.00
1.00
0.68
0.75
Balance abo
0.75
0.46
Balance
0.50
0.44
0.25
0.25
0.00
0.00
00.26
0.40
100000
Training iteaation
Training iteration
Figure 8: Balance ratio and discriminator density evolution during training for S-DDST (RigL) on STL-10. Dashed lines represent BR values of 0.45 and 0.55.
E MORE EXPERIMENT RESULTS
In this section. present IS scores results for Table I and Table 3 The corresponding results shown in Table and Table 7 respectively. We also include FID results of CIFAR-10 test set in
Table 8
20

---

Under review as a conference paper at ICLR 2023
Table 6: IS (higher is better) of different sparse training methods on CIFAR-10 and STL-10 datasets. There is no constraint on the density the discriminator, i.e., daux 100%.
Dataset
CIFAR-10
STL-10
Generator density
10%
20 %
30 %
50
10%
20 %
30 %
50 %
(Dense Buscline)
8.48
Static-Balance
7.18
7.76
8.01
8.31
7.84
8.07
8.35
8.60
Static-Strooms Strong
7.49
8.00
8.31
8.54
7.74
8.29
8,38
8.83
DDST-Balance (SET)
6.94
7.79
8.05
8,20
8.40
8.54
9.20
9.12
SDST- Strong (SET)
8.27
8.46
8.51
8,43
8.10
8.67
8.89
9.31
SDST-Balan Balance (RigL)
6.81
7.77
8.08
8,30
8.85
8.74
9.19
9.14
SDST-Strong (RigI.)
8.20
8.38
8.55
8.48
8.25
9.30
9.01
9.37
R-DDST (SFT)
8.55
8.50
8,40
8.56
8.33
8.62
9.04
9.34
R-DDST (RigL)
8.32
8.61
8.49
8.55
8.79
9.25
9.30
9.27
Table 7: IS (higher is better) of different sparse training methods on CIFAR-10 10 and STL-110 datasets.
The density of the discriminator constrained to be lower than dmax 50%.
Dataset
CIFAR-10
STL-10
Generator density
10%
20 %
10%
20
30 %
50 %
(Dense Baseline)
8.48
9.16
Matic-Balance
7.18
7.76
8.01
8.31
7.84
8.07
8.35
8.60
Static-Strong
7.86
8.21
8.35
8.28
7.81
8.05
8.26
8.37
SDST-Balance Balance (SET)
6.94
7.79
8.05
8.20
8.40
8.54
9.20
9.12
SDST-Strong (SET)
8.22
8.36
8.56
8.35
8.40
9.29
9.21
9.22
SDST-Balanee (RigL)
6.81
7.77
8.08
8.30
8.85
8.74
9.19
9.14
SDST-Strong (RigL)
8.24
8.51
8.20
8.18
7.70
9.32
9.19
9.33
S-DDST (SET)
8.08
8.25
8.45
8.23
8.07
8.91
9.11
9.50
S-DDST (RigL)
8.16
8.47
8.29
8.32
8.45
9.24
9.16
9.03
Table FID of test set (4) of different sparse training methods on CIFAR-10 dataset. Best results are in bold; second-best results are underlined.
Maximal discriminator density dasa
100 %
%
Generator density
10%
20 %
30 %
50 %
10%
20 %
30 %
50 %
(Dense Baseline)
13.32
Static-Balance Static-Strong
29.53
20.83
17.09
14.21
29.53
20.83
17.09
14.21
29.15
22.17
17.37
14.04
1..98
19.35
16.52
14.84
SDST Balance (SET)
30.34
21.00
16.84
15.53
30.34
21.00
16.84
15.53
SDST SDST-Strong (SET)
19.95
17.05
15.16
14.10
18.83
15.96
15.61
14.53
SDST-Balance (RigL)
33.25
20.74
17.78
14.75
33.25
20.74
17.78
14.75
SDST-Strong (RigL)
19.67
15.79
13.89
14.36
18.60
16.01
15.84
14.67
R- DDST (SET)
16.34
15.29
14.30
13.85
R-DDST (Rigl.)
16.65
14.87
14.49
14.05
S-DDST (SHT)
17.75
15.74
15.07
14.91
S-DDST (Rigl.)
17.07
15.50
15.02
14.67
21

---

Under review as a conference paper at ICLR 2023
Table 9: Training FLOPs (x10"7) of different sparse training methods on CIFAR-10 dataset.
Dataset
CIFAR-10
Generator density
10%
20 %
30 %
50 %
(Dense Baseline)
(1.74, 1.00x)
Static-Strong
(0.63, 0.36x)
(0.70, 0,40x)
(0.80, 0.46x)
(1.07, 0.61 x)
dmx 100%
SDST-Strong
(0.63, 0.36x)
(0.70, 0,40x)
(0.80, 0.46x)
(1.07. 0,61
R-DDST
(0.63, 0.36x)
(0.70, 0.40x)
(0.80, 0.46x)
(1.07, 0.
Strong
(0.36, 0.21
(0.43, 0.25x)
(0.53, 0.30x)
(0.79, 0.46x)
dmxx 50%
SDST-Strong
(0.36, 0.21 x)
(0.43, 0.25x)
(0.53, 0,30x)
(0.79, 0.46x)
S-DDST
(0.36, 0.21
(0.43, 0.25x)
(0.53, 0.30x)
(0.79, 0.46x)
Table 10: Training FLOPs (x10") of different sparse training methods on STL-10 dataset.
Dataset
STL-10
Generator density
10%
20 %
30 %
50 %
(Dense Baseline)
(1.85, 1.00x)
Static-Strong
(1.30, 0.75x)
(1.34, 0.77
(1.36, 0.78x)
(1.41, 0.81 x)
dmxx 100%
SDST-Strong
(1.30. 0.75x)
(1.34, 0.77x)
(1.36, 0.78x)
(1.41, 0,81
R-DDST
(1.30, 0.75x)
(1.34, 0.77x)
(1.36, 0.78x)
(1.41, 0.
Static- Strong
(1.07, 0.62x)
(L.11, 0.63x)
(1.13, 0.65x)
(1.18, 0.68
dmax 50%
SDST-Strong
(1.07, 0.62x)
(1.11, 0.63x)
(1.13, 0.65x)
(1.18, 0.68x)
S-DDST
(1.07, 0.62x)
(1.11, 0.63x)
(1.13, 0.65x)
(1.18, 0.68x)
F A ROUGH ESTIMATION OF COMPUTATIONAL COSTS ON SNGAN
In this section, we provide very rough estimation the computational cost of different sparse training methods in terms of training FLOPS. Please see Appendix G for a more accurate compari- son. We approximate the number of backward FLOPs with two times the number of forward FLOPs. We compare the following methods under two settings where dmax E {100%, 50%]:
Dense training.
stati o-Strong.
SDST-Strong.
R-DDST.
S-DDST.
We choose static-Stront and SDST-Strong they perform relatively better than their counterparts with the balance strategy. To simplify our calculation, we compute the FLOPs of R-DDST and S-DDST assuming the discriminator density dp dmax= We also assume that DST may not cause the change of FLOPs. The results are shown in Table and Table 10
It can be that the extra computational cost introduced by D2 which computes BR, and RigL, which computes gradient magnitude for connection growth, is negligible compared to the total training cost as they only happen every several hundred iterations.
G A DETAILED COMPARISON OF TRAINING COSTS
In this section, we compute the computational cost of RigL vairants and stat ic baselines more accurately, More specifically, we take into account the density redistribution over different layers in this section. Also, neglect the computational overhead introduced by computing BR.
In our experiment, compute BR for every iteration to visualize its evolution. However, BR only needs be calculated for every several hundred iterations compute the time-averaged BR.
22

---

Under review as a conference paper at ICLR 2023
Training FLOPs (x10"?) and normalized training FLOPs with respect to dense training of different sparse training methods with SNGAN on CIFAR-10 dataset.
Table 11:
Dataset
CIFAR-10
Generator density
10%
%
(Dense Baseline)
(2.67, 100%)
Static-Balance
(0.24, 9.00%)
(0.46, 17.08%)
(0.70, 26.29%
(1.26, 47.34%)
Static-Strong
(1,56, 58.29%)
(1.63, 60.94%)
(1.72, 64.57%)
(1.99, 74.64%)
SDST-Balance (Rigl)
(0.24, 9. 14%)
(0.46, 17.19%)
(0.68, 25.62%)
(1.16, 43.45%)
SDST-Strong (RigL)
(1.57, 58.80%)
(1.64, 61.42%)
(1.71, 64.04%)
(1.90, 71.16%)
R-DDST (RigL)
(0.49, 18.20%)
(1.14, 42.73%)
(1.63, 60.94%)
(1.85, 69.33%)
Table 12: Training FLOPs (x10!)) and normalized training FLOPs with respect to dense training of different sparse training methods with BigGAN on CIFAR-10 dataset.
Dataset
CIFAR-10
Generator density
10%
20 %
30 %
50 %
(Dense Baseline)
(6.80, 100%)
Static-Balance
(0.67, 9.78%)
(1.30, 19.04%)
(1.95, 28.69%)
(3.34, 49.09%
Static-Strong
(5.71, 83.90%
(5.78, 83.90%
(5.87, 36.34%)
(6.14, 90.26%)
SDST-Balance (Rigl.)
(0.67, 9.91%) %)
(1.32, 19.41%)
(1.96, 28.82%)
(3.29, 48.38%)
SDST-Strong (RigL)
(5.72, 84.04%)
(5.80, 85.22%)
(5.89, 86.54%)
(6.09, 89.56%)
R-DDST (RigL)
(0.67, 9.77%)
(1.69. 24.85%)
(2.72, 40.00%)
(5.25, 77.13%)
G.1 SNGAN ON THE CIFAR-10 DATASET
We first show the results of SNGAN (CIFAR-10) 10) in Table 11 Combined with the results shown in Table it shows that generally R RDDT is able achieve promising performance with reasonable computational costs. More precisely, R-DDST outperforms SDST-Strong (RigL) with much fewer training FLOPs. The reason that SDST-Strong (RigL) uses unnecessarily strong (dense) discriminators.
G.2 BIGGAN ON THE CIFAR- 10 DATASET
In this subsection, we show the results of BigGAN (CIFAR-10). We have included the simplified version in the Table Here give detailed results in Table 12 The results are similar to SNGAN on the CIFAR dataset.
H ONE-SHOT PRUNING AFTER TRAINING WITHOUT FINE-TUNING
In this section, we perform one-shot pruning after training for GANs without any fine-tuning. The results of SNGANs the CIFAR-10 and STL-10 datasets shown in Table 13
23

---

Under review as a conference paper at ICLR 2023
Table 13: FID (L) of different sparse training methods on CIFAR-10 and STL-10 datasets. The density of the discriminator is constrained to be lower than 50%. Best results are in bold; second- best results are underlined.
CIFAR-10
STL-10
Generator density
10%
20
30 %
50 %
10%
20 %
30 %
50 %
(Dense Baseline)
10.74
29.71
PF without fine-tuning
305.81
247.99
89.29
30.03
339.95
195.69
156.29
66.66
Static-balance Static-strong
26.73
18.04
14.38
12.22
50.08
44.19
43.96
37.21
22.35
16.57
13.47
12.22
50.28
44.95
42.12
37.21
DST-balance (SET)
32.02
18.54
14.74
13.23
49.91
33.71
32,92
31.75
DS DST-balance (RigL.
24.56
15.53
13.62
12.51
66.90
50.34
44.57
32.63
SDST-balance (SET)
27.80
18.13
14.15
12.32
63.57
49.05
43.74
31.29
SDST-strong (SET)
16.00
13.31
13.17
12.32
48.40
33.56
32.19
31.29
SDST-balance (RigL)
30.38
17.89
14.95
12.09
46.17
38.12
32.48
31.30
StSo-strong (RigL)
15.66
13,20
12.99
12.09
63.65
33.45
32.09
31.30
S-DDST (SET)
14.22
13.30
12.39
11.97
51.72
35.74
42.36
31.68
S-DDST (RigL)
14.13
12.87
12.15
12.17
44.28
32.84
32.00
30.28
24


Reviews:
[{"review_id": "pH_X0obmCk", "reviewer": "Reviewer_edZK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary.\n\nThis paper is dedicated to developing efficient generative adversarial networks (GANs). The authors apply dynamic sparsity training to GAN models. They point out that the balance of the generator and discriminator is the key to reaching good performance. The authors propose a quantity BR to measure the balance and use dynamic sparsity training for both the generator and discriminator. STL-10 and CIFAR-10 are used for their experiments.", "summary_of_the_review": "Limited novelty and highly insufficient experiments.", "strength_and_weaknesses": "Pros.\n\n1. The writing is good and easy to follow.\n2. The design of the balance ratio makes sense.\n\n\n\nCons.\n\n1. Lack of insights and incremental novelty. The balance problem has already been pointed out in previous GAN pruning papers. The authors directly apply dynamic sparse training to the GAN model. Furthermore, directly training sparse GANs has also been explored by (Liu et al., 2022).\n2. The balance ratio also has design limitations since it is sparsity/density invariant. It can not directly reflect the capacity balance or topology balance between the generator and discriminator.\n3. It is very hard to read the results in Fig 3.\n4. The paper claims efficient GAN. However, only small GANs like SNGAN are evaluated on small datasets like CIFAR-10 and STL-10. Large datasets with advanced GANs are highly needed, like ImageNet/FFHQ with BigGAN and StyleGAN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "bSLB5memxW", "reviewer": "Reviewer_cSFY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed double dynamic sparse training to train sparse GANs from scratch dotted DDST. It directly follows the work of STU-GAN (Liu2022) (a sparse GAN training approach only dynamically change the topology of generators) and proposes balance ratio to further adjust the sparsity of discriminators during training. They first confirm the finding of STU-GAN that the balance between discriminator and generator is important for sparse GAN training and then observe stronger discriminators usually lead to better GAN performance. The proposed balance ratio is a good indicator to check the overfitting problem of discriminators and further adjust the sparsity. The results on the small dataset CIFAR-10 and STL-10 demonstrate the effectiveness of DDST.", "summary_of_the_review": "Overall, I like the idea of dynamic adjusting both discriminators and generators. However, the experiment is not very solid and the method itself is too complicated compared with the marginal performance gains. ", "strength_and_weaknesses": "I will first go through the strength of the paper:\n## Strength\n(1) The paper is well-motivated. STU-GAN only dynamically changes the sparsity of the generator, leaving the discriminator fixed. The proposed double dynamic sparsity can balance the sparsity of GAN's two components during training, which plays a similar role as the famous Stylegan2-ADA in dense GAN training (https://arxiv.org/pdf/2006.06676.pdf). \n\n(2) The capacity of balance ratio to reflect the overfitting problem of sparse GANs is interesting. \n\n(3) The performance improvement over SGAN demonstrate the effectiveness of DDST.\n\n## Weakness\n(1) While the proposed method is sound, the evaluation is insufficient. DDST is solely evaluated with SNGAN on small datasets such as cifar10 and STL-10. To draw solid conclusions on the practical usage, the evaluation of DDST on large-scale settings is required, such as the de-facto standards for conditional image generation BigGAN on ImageNet, or StyleGAN2 on FFQH. \n\n(2) The method is relatively complicated (many hyperparameters and dynamically adjusted sparsity levels), while the performance improvement over the baseline SDST-strong (RigL) is marginal. I have a concern that the method is too complicated and not practical. \n\n(3) While DDST shows better performance than STU-GAN, it uses relatively larger discriminators (around 50% sparsity), which intuitively increases the training FLOPs. Given the marginal performance improvement, the contribution of DDST is diluted. Moreover, why the training FLOPs of DDST and SDST are exactly the same? I expect DDST to have higher training FLOPs than SDST. Coming up with some smart ways to adjust the capacity of discriminators without changing sparsity would be more convincing, like playing with random pruning or adding some perturbations when overfitting happens. \n\n(4) The balance ratio is very similar to the overfitting heuristics used in StyleGAN2-ADA (https://arxiv.org/pdf/2006.06676.pdf), but without clearly citing it. Is balance ratio a better heuristic than ADA? \n\n(5) I am confused by the second meaning of the word double: the discriminator enjoys two levels of dynamic flexibility, namely density level and parameter level.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2KEvSA7G6MF", "reviewer": "Reviewer_Fvhb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new dynamic sparse training method for GANs. The key idea is to balance the performance of the generator and discriminator during training by developing a quantity called balance ratio. Based on this quantity, the authors adjust the densities of the generator and discriminator during training to keep balance. This quantity can be integrated with existing sparse training algorithms flexibly. Some experiments are conducted to evaluate the performance of the proposed method. ", "summary_of_the_review": "The performance of the proposed method is marginal in some tasks.\n\nThis paper looks too heuristic since no theoretical analysis are provided for the proposed balance ratio.  \n\nThis paper is also not well organized. \n", "strength_and_weaknesses": "Strength:\n\n1. The proposed quantity can be integrated with existing sparse training algorithms flexibly to stabilize the training process. \n\n2. The experimental results show that the proposed method can stabilize the training process. \n\nWeaknesses:\n\n1. The performance in Table 2 shows that the improvements achieved by the proposed method over SDST are marginal in some tasks. The authors are recommended to look into this phenomenon.\n\n2. Some theoretical analysis is needed for the proposed balance ratio. I understand it is difficult to prove the convergence of training.  But some analysis is essential to show the insights and properties of the proposed balance ratio, this paper looks too heuristic. \n\n3. This paper is not well organized. For example, the authors present SDST for more than 1 page, which can potentially make the readers confused. \n\n4. Some notations are abused. For example, the $\\alpha$ in balance ratio and $f_{decay}$ should be different. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0JTp38AyGO", "reviewer": "Reviewer_J5kM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors extend the dynamic sparse training (DST) technique to train unconditional GANs(SNGAN). In addition, the authors propose a new metric called balance ratio to explore the role of sparsity in generator and discriminator. Extensive experiments are conducted to show the influence of different sparsity of generators and discriminators in SNGAN. ", "summary_of_the_review": "See the comments above.", "strength_and_weaknesses": "The authors propose a novel metric to balance the dynamic sparse training technique in generative neural networks. The problem is challenging and the experiments are extensive. However, before the reviewer recommends the acceptance of this manuscript, several concerns should be solved:\n(i)\tApplying the DST technique for training GAN is not new. Liu et al 2022 also propose STU-GAN to train GANs. We recommend the authors give a more detailed discussion with STU-GAN in the introduction and related work section. The current discussion with STU-GAN needs further clarification. \n(ii)\tThe results in the experiments merely restrict to SNGAN, which is too specific to conclude the phenomenon. We recommend the authors provide more exploration on other typical unconditional GANs, such as BIGAN (CIFAR10 or ImageNet with low resolution) and StyleGAN2 (CIFAR10 or FFHQ). \n(iii)\tNo baselines are provided in the experiments. We recommend the authors compare their results with several unconditional GAN baselines. The current experiments cannot tell the readers which sparse training technique is better for obtaining a lightweight GAN. \n(iv)\tThe unstructured sparse GAN training can not achieve real training acceleration or inference acceleration.  We recommend the authors extend the weight dynamic sparsity to channel dynamic sparsity. In addition, more baselines are required to demonstrate the effectiveness of the proposed approach. \n(v)\tCan the author explain why the performance is largely degraded compared with dense training? In general, applying dynamic sparse training techniques on training ResNet, there is almost no performance degradation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject