Balanced Training for Sparse GANs
Authors: Yite Wang
Jing Wu
Naira Hovakimyan
Ruoyu Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: Dynamics sparse training; pruning; neural network pruning; empirical deep learning
TL;DR: We propose ADAPT, which aims to achieve a good trade-off between performance and computational cost for GAN dynamic sparse training.

Abstract: Over the past few years, there has been growing interest in developing larger and deeper neural networks, including deep generative models like generative adversarial networks (GANs). However, GANs typically come with high computational complexity,  leading researchers to explore methods for reducing the training and inference costs.  One such approach gaining popularity in supervised learning is dynamic sparse training (DST), which maintains good performance while enjoying excellent training efficiency.  Despite its potential benefits, applying DST to GANs presents challenges due to the adversarial nature of the training process. In this paper, we propose a novel metric called the balance ratio (BR) to study the balance between the sparse generator and discriminator. We also introduce a new method called balanced dynamic sparse training (ADAPT), which seeks to control the BR during GAN training to achieve a good trade-off between performance and computational cost. Our proposed method shows promising results on multiple datasets, demonstrating its effectiveness.

Balanced Training for Sparse GANs
Yite Wang't, Jing Wu't, Naira Hovakimyan', Ruoyu 2.+ University of Illinois Urbana- Siranaa-hamaainn. USA School of Data Science, The Chinese University of Hong Kong, Shenzhen, China Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data [yitew2, ingwu6, (etoovaaiinmillioooooii edu, sunruoyu@cuhk. edw.
Abstract
Over the past few years, there has been growing interest in developing larger and deeper neural networks, including deep generative models like generative adversar- ial networks (GANs). However, GANs typically come with high computational complexity, leading researchers to explore methods for reducing the training and inference costs. One such approach gaining popularity in supervised learning is dynamic sparse training (DST), which maintains good performance while enjoy- ing excellent training efficiency. Despite its potential benefits, applying DST GANs presents challenges due to the adversarial nature of the training process. In this paper, propose novel metric called the balance ratio (BR) study the balance between the sparse generator and discriminator. We also introduce a new method called balanced dynamic sparse training (ADAPT), which seeks control the BR during GAN training to achieve good trade-off between perfor- mance and computational cost. Our proposed method shows promising results on multiple datasets, demonstrating its effectiveness. Our code is available at https: //github. cww http//tttaanng///eoo /AAAP ADAPT
Introduction
Generative adversarial networks (GANs) (20 70 44 are type of generative model that has gained significant attention in recent years due to their impressive performance in mage-genneaatin tasks. However, the mainstream models in GANs are known be computationally intensive, making them challenging to train in resource-constraiigg effectively reduce the computational cost of training GANs while maintaining their performance, making GANs more practical and applicable in real-world scenarios.
settings. Therefore, crucial to develop methods that
Neural network pruning has recently emerged as powerful tool reduce the training and inference costs of DNNs for supervised learning. There are mainly three genres of pruning methods, namely pruming-oe at- att-initializaton,, pruning- during-uuunn-taaiii training, and post-hor hoc pruning methods. Post-hoc pruning can date back to the 1980s, which was first introduced for reducing inference time and memory requirements for efficient deployment; hence does not align with our purpose of efficient training. Later, mruning-t-iiitiallzzan
32 42 25
methods [84] were introduced to circumvent the need fully train the dense networks. However, early pruning-during- training algorithms [58] do not bring much training efficiency compared to post-hoc pruning, while pruning-at-ininn at-initialization methods usually suffer from significant performance drop [18). Recently,
[4 78 75 and gruning-during-traine
advances in dynamic sparse training (DST) 622 [6] 51 52 (54 for the first time show that pruning- during-training methods can have comparable training FLOPs as moning-at-initialiin
methods
"Corresponding author.
'Euual contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

while having competing performance to post-hoe pruning. Therefore, applying DST on GANs seems be promising choice.
Although DST has attained remarkable achievements in supervised learning, the application of DST on GANs is not successful due newly emerging challenges. One challenge is keeping the generator and the discriminator balanced. In particular, using overly strong discriminators can lead to overfitting, while weaker discriminators may fail effectively prevent mode collapse 4 Hence, balancing the sparse generator and the (possibly) sparse discriminator throughout training is even more difficult. To mitigate the unbalance issue, recent work STU-GAN [53] proposes to apply DST directly the generator. However, we find empirically that such an algorithm is likely to fail when the generator already more powerful than the discriminator. Consequently, it remains unclear how to conduct balanced dynamic sparse training for GANs.
To this end, we propose metric called balance ratio (BR), which measures the degree of balance of the two components, to study sparse GAN training. We find that BR is useful in (1) understanding the interaction between the discriminator and the generator, (2) identifying the cause of certain training failure/collapse 77 [8. and (3) helping stabilize sparse GAN training as an indicator. To our best knowledge, this is the first study to quantify the unbalance of sparse GANs and may even provide
new insights into dense GAN training.
Furthermore, using BR as indicator, propose bAlanced DynAmic sParse Training (ADAPT) to adjust the density and the connections the discriminator automatically during training.
Our main contributions are summarized below:
We introduce novel quantity named balance ratio to study the degree of balance in sparse GAN training.
We find empirically that the balance ratio is problematic certain practical training scenarios and that existing methods are inadequate for resolving this issue.
We propose ADAPT, which makes real-time monitoring of the balance ratio. By dynam- ically adjusting the discriminator, ADAPT enables effective control of the balance ratio throughout training. Empirically, ADAP T achieves a good trade-off between performance and computational cost on several datasets.
2
Related works
2.1 Neural network pruning
In deep learning, efficiency is achieved through several methods. This paper primarily focuses on model training and inference efficiency, which is different from techniques for data efficiency
[3 85 [6). These include neural architecture search (NAS) [80 49 to discover optimal network structures, quantization [3 677 for computational efficacy, knowledge distillation (28) to leverage the knowledge of larger models for smaller counterparts, and neural network pruning remove unnecessary connections. Among these, neural network pruning is the focal point of our research. where individual weight where entire neurons or
More specifically, we narrow our focus on unstructured pruning 25
the finest resolution. This contracts with structured pruning [50 59 channels are pruned.
57
30
Post-hoc pruning. Post-hoe pruning method prunes weights of a fully-trained neural network. It usually requires high computational costs due the multiple rounds of the train-prune-retrainn
procedure 25 (8). Some use specific criteria 42 26
to remove weights, while others perform extra optimization iterations 77 Post-hoc pruning was initially proposed to
25
15
12 91 63
reduce the inference time, while lottery ticket works 17
68 aimed to mine trainable sub-networks.
Pruning-at- att-nitiiaizzaiinnn methods. SNIP [46] is one of the pioneering works that aim to find trainable sub- networkss without any training. Some follow-up works [78] 75 13
65 I aim to propose
different metrics to prune networks at initialization. Among them, Synflow 775,, SPP 45). and
FORCE [3] try to address the problem of layer collapse during pruning. NTT (55], PHEW 65 and
NTK-SAP
82 draw inspiration from neural tangent kernel theory.
methods. Another genre of pruning algorithms prunes or adjusts DNNs throughout training. Early works add explicit &o [58) or é [84] regularization terms to encourage

---

sparse solution, hence mitigating performance drop incurred by post-hoc pruning. Later works learn the subnetworks structures through projected gradient descent [94] or trainable masks 773 88 35 41
50 71. However, these pruning-during dduiing-training methods often do not introduce memory sparsity
during training. As a remedy, DST methods [S] 62 64 16 51 52
54 21 were introduced to train the neural networks under given parameter budget while allowing mask change during training.
2.2 Generative adversarial networks
Generative adversarial networks (GANs). GANs (0 have drawn considerable attention and have been widely investigated for years. Deep convolutional GANs [66] replace fully-connected layers employed more advanced methods to improve the fidelity of generated samples. After that, several novel loss functions [69] 660 2 [2 (44, normalization and regularization methods
in the generator and the discriminator. Follow-up works 22
36 9
87] were proposed stabilize the adversarial training. Besides the efforts devoted to training GANs, image-to-iee to-imge translation is
[6]]
also extensively explored 996 95
81
GAN balance. Addressing the balance between the generator and discriminator in GAN training has been the focus of various works. However, directly applying existing methods to sparse GAN training poses challenges. For instance, 3 offer theoretical analyses on the issue of imbalance but may have limited practical benefits, e.g., they require training multiple generators and discriminators. Empirically, BEGAN [6] proposes to use proportional control theory to maintain a hyper-paramettr
BIG(())D(((())) but it is only applicable when the discriminator is an auto-encoder. Unbalanced
Bl D(*)
GAN 24 pretrains VAE to initialize the generator, which may only address the unbalance near initialization. GCC 48 considers the balance during GAN compression, while its criterion requires trained (dense) GAN, which not given in the DST setting. Finally, STU-GAN [53] proposes to use DST to address the unbalance issues but may fail under certain conditions. as demonstrated in our experiments.
GAN compression and pruning. One the promising ways is based on neural architecture search and distillation algorithm [7
19 29. Another part of the work applied mauning-based methods for
generator compression 72 90 33 Later, works by (79 presented unified framework by combing the methods mentioned above. Nevertheless, they only focus on the pruning of generators, thus potentially posing a negative influence on Nash Equilibrium between generators and discriminators. GCC [8 compresses both components of GANs by letting the student GANs also learn the losses. Another line of work [34 9 S tries to test the existence of lottery tickets in GANs. To the best of our knowledge. STU-GAN
5 is the only work that tries to directly train sparse GANs from scratch.
Preliminary and setups
Generative adversarial networks (GANs) have two fundamental components, a generator G(:; 0G) and discriminator D(:; 0p). Specifically, the generator maps a sampled noise from multivariate normal distribution p(z) into fake image to cheat the discriminator. In contrast, the discriminator distinguishes the generator's output and the real images æ, from the distribution Pdam(x,). (z,).
Formally, the optimization objective of the two-player game defined GANs can be generally defined as follows:
Lo(Op, 0G) = E + (f((( (c(0g) = =Ez<p [g) (D(G(x;0o))).
(1)
(2)
To be more specific, different losses can be used, including the loss in the original JS-GAN 20 where fi (x) - log(æ), f2(2)= = gn (x) - log(1 1 - æ:;; Wasserstein loss [22 where f1
-f((x) = 91 (x) = - or; and hinge loss [6]l where (xx = max(0, x)) (2 = max(0, +x).
and gı (z) -z. The two components optimized alternately achieve the Nash equilibrium.
3

---

16
70
75%
Dersty
Deaily
Static
SDST-Rig!
SDST-SFT
Figure 1: FID (L) comparison of SDST against STATIC sparse training for SNGAN on CIFAR-I0 with different sparsity ratio combinations. The shaded areas denote the standard deviation.
GAN sparse training, In this work, are interested in sparse training for GANs. In particular, the objective of sparse GAN training can be formulated as follows:
O = min (D( C mD, Og c mm)
0p
0 min (a @ @mG)
0G
s.t. mD E 40, 10.l), mg E 1]
1] Øcl,
|mploo/10pl S dp.
|mallo/ S da,
where @ is the Hadamard product; 0, mp, 1ppl dp are the sparse solution, mask, number of parameters, and target density for the discriminator, respectively. The corresponding variables for the generator are denoted with subscript G. For pruning-at-imitaalinn at-initialization methods, masks m are determined before training, whereas m are dynamically adjusted for dynamic sparse training (DST) methods.
usually start with sparse network parame- terized by 0 @ m with randomly initialized mask m. After a constant time interval AT, it updates mask by removing fraction of connections and activating total number of active parameters ||mm]|o is hence kept under certain threshold d(O). Please see Appendix B for more details.
Dynamic sparse training (DST). DST methods 62 16
ones with certain criterion. The
Motivating observations: The unbalance sparse GAN training
As discussed in sectionn it is essential to maintain the balance of generator and discriminator during GAN training. As strong discriminators may lead over-fitting, whereas weak discriminators may be unable to detect mode collapse. When it comes sparse GAN training, the consequences caused by the unbalance can be further amplified. For example, sparsifying a weak generator while keeping the discriminator unmodified may lead to an even more unbalanced worst-case scenario.
To support our claim, we conduct experiments with SNGAN 61 on the CIFAR-I0 dataset. We consider the following sparse training algorithms:
Static sparse training (STATIC). For STATIC, layer-wise sparsity ratio and masks ma, mD are fixed throughout the training.
@ Single dynamic sparse training (SDST). SDST is direct application of the DST method on GANs where only the generator dynamically adjusts masks during the training. We name such method SDST only one component of the GAN, i.c., the generator, is dynamic. Furthermore, we call the variant which grows connections based on gradients magnitude as SDST-RigL [6),, and randomly as SDST-80 SET [62]. Note that STU-GAN [53] is almost identical to SDST-RigL with EMA [89] tailored for DST. We do not consider naively applying DST on both generators and discriminators, in STU-GAN, it is empirically shown that simply adjusting both components generates worse performance with more severe training instability
We test the considered algorithms with de E (10%, 20%, 30%, 50%) and dp {10%, 20%, 30%, 50%, 100%]. More experiment details can be found at Appendix A and Appendixx B
We also perform small experiment in subsection E.2 to validate their findings.

---

Galane rate
1o-10%%
Traitin Eaaaiinn
Traning Stration
Franing tteation
Teat forabiz
Trasnin teration
Boteon 1910
40-20%
20g wal alc
20A 20t0 Mbx
20G
dia
240 shll
Traning
Seate ride
Saa pee Whin
2000
2000 and
3000 2000 Gna
20ar 2000 oino
20ap ablo 2000
12
Doloce o31
Colonce oil
lance ote
do-500
Galane
t
200 2000 smm 100
pin
1000
1000
20m anc 800
Static
SOST-Ripl
SDST-SET
Figure 2: BR comparison of SDST against STATIC sparse training for SNGAN on CIFAR- 10 with different sparsity ratio combinations.
4.1 Key observations
We report the results in Figure and summarize our critical findings follows:
Observation 1: Neither strong nor weak sparse discriminators can provide satisfactory results. The phenomenon is most noticeable when de 10%, where the FID initially decreases but then increases. The reasons may be as follows: (1) Overly weak discriminators may cause training collapse as they cannot provide useful information to the generator, resulting in sudden increase in FID at the early stage of sparse GAN training. (2) Overly strong discriminators may not yield good FID results because they learn too quickly, not allowing the generator to keep up. Hence, to ensure balanced training of GAN for sparse training methods, is crucial to find an appropriate sparsity ratio for the discriminator.
Observation 2: SDST is unable to give stable performance boost compared to the STATIC baseline. Another critical observation is that SDST is better than STATIC only when the discrimina- tor is strong enough. More specifically, for all selected discriminator density ratios, SDST method not better than STATIC when using small discriminator density (dD 10%). On the contrary, for the cases where dp > do, we generally see a significant performance boost brought by SDST.
Balance ratio: Towards quantifying the unbalance in sparse GAN training
5.1 Formulation of the balance ratio
To gain a deeper understanding of the phenomenon observed in the previous section, and to better monitor and control the degree of unbalance in sparse GANs, we introduce novel quantity called the balance ratio (BR). This quantity defined as follows.
At each training iteration, we draw random noise from multivariate normal distribution and real images ar, from the training set. We denote discriminator after gradient descent update (c: Op). We denote generator before and after gradient descent training (Pe 0g) and
5

---

Density generator 10%
Density gener atorr do-22f%
Density generator 4p-30%
Density generator do
Dessity generator 100100%
dsoiminaior
Static
Dynamic density adjust
Figure 4: FID (1) of STATIC sparsely trained SNGAN with and without DDA on CIFAR-10 with different sparsity ratio combinations. The result of DDA is independent of dp it is determined automatically. The shaded areas denote the standard deviation.
(: (;:)). respectively. Then the balance ratio is defined as: Ezmpl D (z) - (2)))((2)) a
BR
(3)
E Eup [(x,))
Rz Bz ((Gro(z))
8
Precisely, BR measures how much improvement the generator can achieve in the scale measured by the discriminator for a specific random noise
D(
z. When BR is too small (e.g., BR 30%), the updated generator is too weak to trick the discriminator, as the generated images are still considered fake. Similarly, for the case where BR is too large (e.g.. BR> 80%), the discrimi- nator is considered too weak hence it may not provide useful information to the generator. We also illustrate BR in Figure 3
4.5m
GG (G
Balance ratio
Fake images
Real images
X
Figure 3: Illustration of balance ratio.
5.2 Understanding observation 1: Analysing GAN balance with the balance ratio
We visualize the BR evolution throughout the training for the experiments in seseectionn 4 to show the effectiveness of BR in quantifying the balance of sparse GANs. We show the results in Figure 2
It illustrates that BR can distinguish the density difference (hence the representation power difference) of the discriminator. Specifically, we can see that for larger discriminator density dp. the BR much lower throughout the training, indicating strong discriminators. On the contrary, for the cases where the discriminators are too weak compared to the generators, e.g., all cases where dp = 10%, we can observe BR first increases and then oscillates wildly. We believe this oscillatory behavior related to the training collapse. Empirical results also show that the FID metric experiences sudden increase after this turning point.
5.3 Dynamic density adjust: A first attempt utilize the balance ratio
As demonstrated in the previous section, the balance ratio (BR) effectively captures the degree of balance between the generators and discriminators sparse GANs. Hence, is natural to leverage BR to dynamically adjust the density of discriminators during sparse GAN training so that a reasonable discriminator density can be found.
To demonstrate the value of BR, we propose simple yet powerful modification to the STATIC baseline. This method, which we call dynamic density adjust (DDA), is explained below. Specifically, we initialize the initial density of the discriminator dipi: dG- After specific training iteration interval, adjust the density of the discriminator based the BR over the last few iterations with pre-defined density increment Ad. With a pre-defined BR bounds [B_, B, we decrease d, by Ad when BR is smaller than B_, and vise versa. We show the algorithm in Appendix A]Algorihm
Comparison to ADA 37 In this paragraph, we compare ADA and DDA. (1) Notice that DDA algorithm is orthogonal to ADA in sense that StyleGAN2-ADA adjusts the data augmentation

---

probability while DDA adjusts the discriminator density. (2) Moreover, the criterion used in DDA, i.c.
BR, is very different from the criterion proposed in StyleGAN2-ADA, i.e. Ty [ - (=)) and r. = E [sign( D ((aaai))]] In particular, To requires a separate validation set, while F. only quantifies the overfitting of the discriminator to the training set. (3) Another note is that DDA is flexible framework, where its criterion, i.c. BR, can be potentially replaced by Te. Tt. and such.
D(trn) - (2
Experiment results. We test DDA with target BR interval [B_, B+] = [0.5, 0.65]. Precisely, DDA tends to find a suitable discriminator where the generator can just trick the discriminator throughout the training. We show the results in Figure 4 with red lines. The experiments show that DDA can identify reasonable discriminator densities to enable balanced training for sparse GANs.
5.4 Understanding observation 2: Analysing the failure of SDST with the balance ratio
By leveraging BR, we can also gain further insights into why some configurations do not benefit from SDST compared to STATIC.
Regarding SDST as way of increasing the generator capacity. Our findings suggest that SDST possibly enhance the generator's representation power, as demonstrated by the higher BR values compared to STATIC observed in Figure 2 We attribute this effect to the in-time over-parameteriizaton (ITOP) 54 induced by dynamic sparse training.
SDST does not address training collapse. The increase generator's representation power resulting from SDST is only beneficial when the discriminator has matching or superior representation power. Therefore, if the training has already collapsed for the static baseline methods (STATIC), meaning that the generator is already stronger than the discriminator, SDST may not be effective in stabilizing sparse GAN training. This evident from the results shown in FFiuure 2 first-row column 1, second- row column 1, third-row columns 1-2, and fourth-row columns 1-3.
Despite the superior performance of STU-GAN (or SDST in general) higher discriminator density ratios dp, there exist some limitations for SDST, which we summarize below:
SDST requires pre-defined discriminator density dp before training. However, it is unclear what a good choice. In real-wordd scenarios, it is not practical to manually search for the optimal dp for each de. A workaround may be using the maximum allowed density for the discriminator. However, as shown in Figure I the best performance is not always obtained with the maximum do 100%. Moreover, we are wasting extra computational cost for worse performance if an overly-strong discriminator.
@ SDST fails if there is an additional constraint on the density of the discriminator dp. As Figure I suggests, for weak discriminators, SDST unable to show consistent improvement compared to the STATIC baseline.
Hence, STU-GAN (or SDST in general), which directly applies DST to the generator, may only be useful when the corresponding discriminator is strong enough. In this sense, obtaining balanced training automatically essential in GAN DST deal with more complicated scenarios.
Balanced dynamic sparse training for GANs
In this section, we describe our methodology for balanced sparse GAN training.
STU-GAN (or SDST in general) considered in the last section cannot generate stable and satisfying performance. This implies that should utilize the discriminator in better way rather than do nothing (like SDST) or directly apply DST to the discriminator (see (susecttion E.2 for additional experiments). Consequently, DDA (subsection S.3). 5.3), which adjusts the discriminator density to stabilize GAN training, is a favorable candidate to address the issue. To this end, we propose bAlanced DynAmic sParse Training (ADAPT), which adjusts the density of the discriminator during training with DDA while the generator performs DST.
We further introduce two variants, namely ADAPT and ADAPT based on whether we force the discriminator to be sparse. We present them in subsection 6. and (ubssecion 6.2 These methods more flexible and generate more stable performance compared to SDST.
7

---

Table 1: FID (L) of different sparse training methods with no constraint on the density of the discriminator. Best results are in bold: second best results are underlined.
Dutoset
CIFAR-10 (SNGAN)
STL-10 (SNGAN)
CIFAR-HO 10 (BigGAN)
Timy TinylnneeNN (BigGAN)
Generator densuty
10%
20%
30%
50%
10%
20%
30%
50%
10%
2016
30%
50%
10%
20%
30%
50%
(Dese Hascline)
10.74
29.71
8.11
15.43
STATIC Bulance STATIC Stroog
26.75
19.04
15.05
12.24
48.18
44.67
41.73
37.68
16.96
12.81
10.33
8.47
28.78
21.67
18.86
17.51
26.79
19.65
14.38
52.48
43.85
42.06
37.47
23.48
14.26
11.19
3.64
31.44
22.51
18:22
18.00
SDST B-Balancc SET
26.23
17.79
13.21
11.79
56.41
46.38
39.93
30.37
12.41
9.87
9.13
8.01
25.
21.30
21.30
21.20
SDDST=Sinong-SET SFT
16.49
13.36
11.68
10.68
67. 37
49. 96
37.99
31.08
18.94
9.64
8.75
8.36
22.30
20.56
21:70
18:32
SDST Balance Righ.
27.06
16.36
14.00
12.28
43.08
33.90
31.83
30.30
12,45
9.42
8.86
8.03
21
19.33
18.57
17.45
SDSTT-Sinog-Re Rigl.
53.65
17.75
ADAPT, (Ours)
14.19
13.19
12.38
10.00
35,98
33.06
31.71
29.96
10.19
8.56
8.36
8.22
19.42
17.99
17.06
14.15
6.1 ADAPTreaax Balanced dynamic sparse training in the relaxed setting
In this section, we consider the relaxed setting where a dense discriminator can be used, i.c., dp S dmax 100%. This relaxed scenario gives the greatest flexibility to the discriminator. However, it does not necessarily enforce the sparsity of the discriminator (hence, no computational savings for the discriminator) because the density of the discriminator can be as high as 100%.
For the relaxed setting, we use the direct combination of SDST with DDA, Precisely, the generator adjusted using DST as mentioned in section 4 while the density of the discriminator is dynamically adjusted with DDA as mentioned in subsection 5.3 We call such a combination relaxed balanced dynamic sparse training (ADAPT Please see Appendix Algoorithm 2 for more details.
Comparison to STU-GAN (or SDST in general). Compared STU-GAN (or SDST in general), which pre-defines and fixes the discriminator density during training, the difference is that for Treass the density of the discriminator is adjusted during the training process automatically through real-time monitoring of the balance ratio. Given the initial discriminator density d = da. ADAPT relax increases the discriminator density if stronger discriminator is needed, and vice versa.
6.2 ADAPTstiat Balanced dynamic sparse training in the strict setting
Different from subsection 6.
now consider strict setting where there is additional sparsity constraint on the discriminator density in this section, i.e., dp I dmmx < 100%.
ADAPTrelax introduced in the previous section does not necessarily enforce sparsity for the discrimi- nator, which provides less memorytraining resources saving for larger generator density ratios. Note that the discriminator does not take advantage of DST to explore the structure of the dense network. Hence, we further present striet balanced dynamic sparse training (ADAPT in this section. This method allows the discriminator to perform DST in a controlled manner, which can lead to better exploration of the dense network structure while maintaining the balance between the generator and the discriminator. We explain how ADAP arict differs from ADAP Trllax below:
Capacity increase of the discriminator. The essential difference lies when the observed BR is higher than B+, which means we need stronger discriminator. In this case, if the discriminator density lower than the constraint, i.e., dp < dmax. ADAP SRAPTsttic strict will perform just like ADAP to increase the discriminator density. However, if the discriminator is already the maximum density, i.e., dn = dmmx. the discriminator will alternatively perform DST as a way of increasing the discriminator capacity (See subsection 5.4 for intuition).
Capacity decrease of the discriminator, Similar to ADAPT, yellxx when the observed BR is lower than B_, we will decrease the discriminator density.
Hence, ADAPT sarcc makes the discriminator adaptive both in the density level (through density adjustment) and the parameter level (through DST). The algorithm of ADAFT is shown in Appendix C[Algorithm
6.3 Experiment setting
Datasets, architectures, and target sparsity ratios. We conduct experiments on SNGAN with ResNet architectures on the CIFAR-10 44) and the STL-10 [T] datasets. We have also conducted experiments with BigGAN 7 the CIFAR-10 10 and TinylmageNet dataset (with DiffAug 193).
8

---

Table 2: FID (L) of different sparse training methods. The density of the discriminator is con- strained to be lower than 50 %. Best results are in bold; second-best results are underlined.
Datteet
CIEAR-10 (SNGAN)
SIL-IO(SNGAN)
CIEAR-10 (BIGAAN)
IaasImageNet (BigGAN)
Generater density
10%
20%
50%
50%
10%
20%
30%
50%
10%
20%
30%
50%
10%
20%
50%
50%
(Dusse Baseline)
10.74
29.71
8.11
15:43
STATIC Ballace STATIC Strong
26.75
19.04
15,05
12.58
48.18
44.67
41.73
37.68
16.95
12.81
10.33
8.47
28.78
21.67
18.86
17.51
21.73
16.69
13.48
12.58
50.36
44,00
40.73
37.68
18.91
13:43
10.84
5.47
33.01
23.93
17:90
17.51
+ SISTT-Robmee-ET
26.23
17.79
13.21
11.79
56.24
44.51
41 1.
30.80
12.41
9.57
9:13
8.01
25.39
21. 30
21.90
21.30
SDDST-Strung SET
15.68
12.75
11.98
11.79
57.91
50.05
38.1 1.3
30.80
11.85
9.39
8.61
8.01
22.68
20.24
22.00
21.20
DOST-Bullanc--R R.
27.06
16.36
14,00
12.28
13.08
33,90
31
10.
12.45
9.42
3.86
8.03
21 21
19.33
18.57
17.45
SSS S-Stong Rigl.
15.19
12.95
12.28
53.74
37.34
31.
30.50
9.12
8.35
21.90
20.45
18.29
ADAPTasa (Dar))
14:53
12.73
12.30
12 11
41.18
31,59
31.16
29.11
9.29
8.64
8.44
7.90
18,89
17.37
16.93
16.02
Table 3: Normalized training FLOPs (4) of different sparse training methods with no constraint on the density of the discriminator.
CIFAR.I0 (SNGAN)
STL-19 (SNGAN)
CIFAR-10 (RgGAN)
TinsEnngeNNe (BIGANN)
Cenerstte denvity
10%
705
10%
99%
107
2095
30%
41%
1055
775-
10FS
10%
70%
30%
50%
Dene Bastins)
(2,00 x 1091)
100% (3.94
x (9"1)
100% (6.51
10"1)
1007%
(1.85 x 100))
17.88%
16.29%
47.25%
27 1955
49.14%
90.325
73.09%
4.70%
19175%
7% 66'5
40.09%
21.35%
64.87%
00.915
79-599
Stringtin
$1.294;
64.55%
74.61%
96.12%
$6.949
$7.606
$8.54%
53.78%
$4.80%
26.21%
90.15%
45.02%
61.62%
12.799
85.794
SEST SRTT
18915
78: 19.15%
45-4155
27 5.5
47-405
60:175
75.38%
18 19/5
70.755
79 W/W
69 827
71.175
17/19/
48.495
61 5815
SDS Stroey SET
$9.256
02.54/2
56,59%
75.96%
86. 365
$7.45%
85.45%
50.28
$4.30%
85.00%
K7 57.52%
50.55%
45 65,656
53.01%
60.61%
71.88%
BalaBas
1971%
17.43%
25.99/
43.50%
63.34 34.344
0.92%
24.9766
43.90%
57.366
04.57%
ADAPT
57.62%
46.73%
77.82%
83.62%
10.39%
25,00%
40.65%
29.75%
Target density ratios of the generators de are chosen from (10%, 20%, Appendix for more experiment details.
20%,30% 30%, 50%}. Please see
Baseline methods and two practical strategies. We use STATIC and SDST (ection 4) as our baselines. Note that in real-world application scenarios, it is not practical to perform grid search for a good dp as in Section 4 Hence, we propose two practical strategies to define the constant discriminator density for these baseline methods: (1) balance strategy, where we set the density of the discriminator dp the same as the density of the generator de; (2) strong strategy, where we set the density of the discriminator as large as possible, i.c., dD = dmax. For SDST methods, we test both SDST-SET and SDST-Righ. For fair comparison, dmax is set to be 100% and 50% for the relaxed setting and the strict setting, respectively.
For ADAPT, we use the + RigL version, which grows connections of the generators and discriminators based on gradient magnitude. The gradient information enables two components react promptly according to the change of each other. Different from the value used in subsection 5.3] we control the balance ratio in the range (0.45, 0.55) unless otherwise mentioned to have a slightly stronger discriminator, potentially avoiding training collapse. More details can be found in Appendix B
6.4 Experiment results
We show the experiment results in Table I and Table 2 for the relaxed setting and the strict setting, respectively. We also present the training FLOPs normalized by the dense counterpart for the relaxed setting Table 3. We defer the results for the strict setting to Appendix F Table 12 We show FID for the CIFAR-10 test set, Inception scores, and comparison with post-hoc pruning baseline in Appendix
We also show ADAPT BR evolution in AAppendxx D We summarize our findings below.
The strong strategy and the balance strategy for baselines. Generally, using the strong strategy has some advantages over the balance strategy. Such an observation is most prominent in the CIFAR-10 dataset. For the cases where the balance strategy is better, e.g., SNGAN on the STL-11 dataset, our explanation is that the size difference between generators and discriminators more significant. Hence, the degree of unbalance is more severe and leads to more detrimental effects.
Comparison of RigL and SET for SDST. We found that RigL has an advantage over SET when dealing with more sparse generators. Our hypothesis is that gradient information can effectively guide the generator to identify the most crucial connections in such cases. However, this advantage is not as apparent for more dense generators.
Are achieves good trade-off between performance and computational cost. Experiments show that ADAP ADAPTrelx shows promising performance by being best for 13 out of 16 cases. The advantage of ADAPT is most prominent for the most difficult case, i.e., de 10%. Specifically, it
9

---

shows 2.3 and 7.1 FID improvements over the second-best best methods for the SNGAN on the CIFAR-10 and the STL-10, respectively. Moreover, compared to the competitive baseline methods that the strong strategy, i.e., Stng- computational cost reduction. For example, it outperforms SDST-SttrongRRA on BigGAN (CIFAR-10) with much-reduced training FLOPs (10.39% v.s. 83.97%)
and SDST-St rong--SRT,, ADAPT shows great
sttiettriet shows stable and superior performance. Similar ADAPTreelax we notice that aDri also delivers promising results compared to baselines, even with further constraint on the discriminator. More precisely, among all the cases, ADAPTsune ranks top 2 for all cases, with 13 cases being the best. Moreover, ADAP Ttrip striet again shows comparable or better performance compared SDST-Strong-RigL with reduced computational cost.
more interesting observation is that ADAPT sometimes outperforms ADAPT We speculate that this phenomenon occurs because changes in density may result in larger influence the GAN balance during training compared to DST. Hence, the striet version, whose discriminator density range smaller, may offer more consistent performance.
7 Conclusion
In this paper, we investigate the use of DST for GANs and find that solely applying DST to the generator does not necessarily enhance the performance of sparse GANs. To address this, we introduce BR to examine the degree of unbalance between the sparse generators and discriminators. We find that applying DST to the generator only benefits the training when the discriminator is comparatively stronger. Additionally, we propose ADAPT, which can dynamically adjust the discriminator at both the parameter and density levels. Our approach shows promising results, and we hope it can aid researchers in better comprehending the interplay between the two components of GAN training and motivate further exploration of sparse training for GANs. However, we must note that we have not yet evaluated our methods the latest GAN architectures due to computational constraints.
8
Acknowledgement
This work utilizes resources supported by the National Science Foundation's Major Research Instru- mentation program, grant No. 1725729 (9). well as the University of Illinois Jirbaa--Champaige This work is supported in part by Hetao Shenzhen-Hong Kong Science and Technology Innova- tion Cooperation Zone Project UDF01001499 from the Chinese University of Hong Kong, Shenzhen; Guangdong Key Lab on the Mathematical Foundation of Artificial Intelligence, Department of Science and Technology of Guangdong Province. Prof. NH is in part supported by Air Force Office of Scientific Research (AFOSR) grant FA9550-21-1-0411
University Development Fund
10

---

References
[1] M. Alizadeh, S. A. Tailor, L. M. Zintgraf, J. van Amersfoort, S. Farquhar, N. D. Lane, and Y. Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. arXiv preprint arXiv:2202.088 02,,0,88222000 2022.
[2] M. Arjovsky, S. Chintala, and L.. Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pages 214-223. PMLR, 2017.
[3] S. Arora, R. Ge, Y. Liang, T. Ma, and Y. Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pages 224-232. PMLR, 2017.
[4] Y. Bai, т. Ma, and A. Risteski. Approximability of discriminators implies diversity in gans. arXiv preprint arXiv: 10586, 2018.
[5] G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:
2017.
[6] D. Berthelot, Schumm, and I.. Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv: 10717, 2017.
[7] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 2018.
[8] T. Chen, Y. Cheng, Z. Gan, J. Liu, and Z. Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, $4:20941-209955 2021.
[9] X. Chen, Z. Zhang, Y. Sui, and T. Chen. Gans can play lottery tickets too. arXiv preprint 00X.v22106.00134, 2021.
[10] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain mmage-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018.
[11] A. Coates, Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[12] B. Dai, C. Zhu, B. Guo, and D. Wipf. Compressing neural networks using the variational information bottleneck. In International Conference Machine Learning, pages 1135-1144. PMLR, 2018
[13] de Jorge, A. Sanyal, H. S. Behl, P. H. Torr, G. Rogez, and P. K. Dokania. Progressive skele- tonization: Trimming more fat from network at initialization. arXiv preprint 1rXiv:2006.09081, 2020.
[14] T. Dettmers and L.. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1rXii:190700484/ 2019.
[15] X. Dong, S. Chen, and S. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
[16] U. Evci, T. Gale, Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[17] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint
2018.
[18] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Pruning neural networks at initialization: Why are we missing the mark? In ICLR, 2021.
[19] Y. Fu, Chen, H. Wang. Li, Lin, and 7. Wang- Autogan-distiller Searching to compress generative adversarial networks. arXiv preprint 200:::0000008998 2020.

---

[20] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 6444111199-44, 2020.
[21] L. Graesser, U. Evci, E. Elsen, and P. S. Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pages 7766-7792 7792. PMLR, 2022.
[22] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[23] Y. Guo, Yao, and Y. Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
[24] H. Ham, T. J. Jun, and D. Kim. Unbalanced gans: Pre-training the generator of generative adversarial network using variational autoencoder. arXiv preprint arXiv:2002.02/ 12, 2020.
[25] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[26] B. Hassibi, D. G. Stork, and G. Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pages 293-299, IEEE, 1993.
[27] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[28] G. Hinton, O. Vinyals, and Dean. Distilling the knowledge in neural network. arXiv preprint AXX3v:503.0255: 2015.
[29] L. Hou, Z. Yuan, L.. Huang, H. Shen, X. Cheng, and C. Wang. Slimmable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1746-7753, 2021.
[30] Z. Huang and N. Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304-320, 2018.
[31] I. Hubara, M. Courbariaux, D. Soudry, R. El- Yaniv, and Y. Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-688) 2017.
[32] S. A. Janowsky. Pruning versus clipping in neural networks. Physical Review A, (91))66600, 1989.
[33] Q. Jin, J. Ren, J. Woodford, J. Wang, G. Yuan, Y. Wang, and S. Tulyakov. Teachers do more than teach: Compressing image-to-image models. In Pruceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13600-13611, 2021.
[34] N. M. Kalibhat, Y. Balaji, and S. Feizi. Winning lottery tickets in deep generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2021.
[35] M. Kang and B. Han. Operation-ane aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 5122-510 -1311. PMLR, 2020.
[36] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 11t.:::1110109996 2017.
[37] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-12114, 2020.
[38] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020.
12

---

[39] V. Kindratenko, D. Mu, Y. Zhan, J. Maloney, S. H. Hashemi, B. Rabe, K. Xu, R. Campbell, J. Peng, and W. Gropp. Hal: Computer system for scalable deep learning. In Practice and experience in advanced research computing, pages 41-48. 2020.
[40] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[41] A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Leaminn, pages 5544-5555 5555. PMLR, 2020.
[42] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[43] C. Ledig, L. Theis, F. Huszár. J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, al. Photo-realistic single image super-resolution using a generative adversar- ial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[44] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11523-11532, 2022.
[45] N. Lee, T. Ajanthan, S. Gould, and P. H. Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv:1906.063077 2019.
[46] N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:/810.023400, 2018.
[47] M. Li, J. Lin, Y. Ding, Z. Liu, J.-Y. Zhu, and S. Han. Gan compression: Efficient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5284-5294, 2020.
[48] S. Li, J. Wu, X. Xiao, F. Chao, X. Mao, and R. Ji. Revisiting discriminator in gan compression: discrmiiitttt cooperative compression scheme. arXiv preprint arXiv:21 /0.14499 2021.
[49] H. Liu, K. Simonyan, and Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.00055 2018.
[50] Liu, Z. Xu, R. Shi, R. C. Cheung, and H. K. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005.066 066700 2020.
[51] S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9922, 2021.
[52] S. Liu, D. C. Mocanu, Y. Pei, and M. Pechenizkiy, Selfish sparse mn training. In International Conference on Machine Learning, pages 6893-6904. PMLR, 2021.
[53] S. Liu, Y. Tian, T. Chen, and I.. Shen. Don't be so dense: Sparse-to-sparse gan training without sacrificing performance. arXiv preprint arXiv:2203.02770, 2022.
[54] S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense over- parameterization? in-time over paramettettiiooooonnn sparse training. In International Conference on Machine Learning, pages 6989-7000. PMLR, 2021.
[55] T. Liu and F. Zenke. Finding trainable sparse networks through neural tangent transfer. In International Conference on Machine Learning, pages 6336-6347. PMIR, 2020.
[56] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEFE international conference on computer vision, pages 2736-2744 2017.
13

---

[57] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. arXiv preprint arXiv: 1810.0527, 2018.
[58] C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint orXiv: 2017.
[59] J. Luo, J. Wu, and W. Lin. Thinet: A filter level praning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 2017.
[60] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2802, 2017.
[61] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative 2018.
adversarial networks. arXiv preprint
[62] D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibeseu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91):1-11,, 2018.
[63] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1164-11272,, 2019.
[64] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646 4655. PMLR, 2019.
[65] S. M. Patil and C. Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pages 8432 8442. PMLR, 2021.
[66] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXiv preprint
2015.
[67] M. Rastegari, V. Ordonez, Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pages 525-542. Springer, 2016.
[68] A. Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruming. arXiv preprint arXiv:2003.02338) 2020.
[69] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[70] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl:
Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[71] Savarese, H. Silva, and M. Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 1380-11390, 2020.
[72] H. Shu, Y. Wang, X. Jia, K. Han, H. Chen, C. Xu, Q. Tian, and C. Xu. Co-evolutionary compression for unpaired image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3235-3244, 2019.
[73] S. Srinivas, A. Subramanya, and R. Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition workshops, pages 138-145, 2017.
[74] R. Sun, T. Fang, and A. Schwing. Towards a better global loss landscape of gans. Advances in Neural Information Processing Systems, 33:10186-10198 2020.
14

---

[75] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389, 2020.
[76] D. Terjék. Adversarial lipschitz. arXiv preprint arXiv: 2019.
[77] S. Verma and J.-C. Pesquet. Sparsitying networks via subdifferential inclusion. In International Conference on Machine Learning, pages 10542-10552. PMLR, 2021.
[78] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.073376 2020.
[79] H. Wang, S. Gui, H. Yang, J. Liu, and Z. Wang. Gan slimming: All-in-one gan compression by unified optimization framework. In European Conference on Computer Vision, pages 54-73. Springer, 2020.
[80] H. Wang, Y. Wang, R. Sun, and B. Li. Global convergence of maml and theory-inspired neural architecture search for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9797-9808, 2022.
[81] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[82] Y. Wang, D. Li, and Sun. Ntk-sap: Improving neural network pruning by aligning training dynamics, In The Eleventh International Conference on Learning Representations, 2022.
[83] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7278-7286 2018.
[84] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
[85] J. Wu, J. Hobbs, and N. Hovakimyan. Hallucination improves the performance of unsupervised visual representation learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 16132-16143, 2023.
[86] J. Wu, N. Hovakimyan, and J. Hobbs. Genco: An auxiliary generator from contrastive learning for enhanced few-shot learning in remote sensing. arXiv preprint #rXi::2307.1461,, 2023.
[87] Y.L. Wu, H.-H. Shuai, Z.-R. Tam, and H.-Y. Chiu. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6373-6382, 2021.
[88] X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
[89] Y. Yaz, C.-S. Foo, S. Winkler, K.-H. Yap, G. Piliouras, V. Chandrasekhar, et al. The un- usual effectiveness of averaging in gan training. In International Conference on Learning Representations, 2018
[90] C. Yu and J. Pool. Self-supervised generative adversarial compression. Advances in Neural Information Processing Systems, 33:8235 8246, 2020.
[91] R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. L. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
[92] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. International conference on machine learning, pages 7354-7363. PMLR, 2019.
[93] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020.
15

---

[94] X. Zhou, W. Zhang, H. Xu, and T. Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4599-3608 2021.
[95] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[96] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems, 30, 2017.
16

---

Overview of the Appendix
The Appendix is organized follows:
Appendix A
introduces the general experimental setup.
Appendix introduces the details of dynamic sparse training.
shows detailed algorithms, i.e., DDA, ADAPTrelxx and ADAPT striet Appendix shows the BR evolution during training for ADAPT.
Appendix C
Appendix shows additional results, including IS and FID of test sets of the main paper.
Appendix F
shows detailed FLOPs comparisons of sparse training methods.
A
Experimental setup
In this section, we explain the training details used in our experiments. Our code mainly based on the original code of ITOP (54) and GAN ticket [9].
A.1 Architecture details
We ResNet-32 (27 for the CIFAR-10 dataset and ResNet-48 for the STL-I0 dataset. See Table and Table for detailed architectures, We apply spectral normalization for all fully-connected layers and convolutional layers of the discriminators.
For BigGAN architecture, we use the implementation used in DiffAugment 93
A.2 Datasets
We use the training set of CIFAR-10, the unlabeled partition of STL-10, and the training set of TinylmageNet for GAN training. Training images are resized 32 x 32, 48 48. 64 64 for CIFAR- 10, STL-10, and TinylmageNet datasets, respectively. Augmentation methods for both datasets are random horizontal flip and per-channel normalization.
A.3 Training hyperparameters
SNGAN on the CIFAR-10 and STL--10 datasets. We use a learning rate of 2 x 10-4 for both generators and discriminators. The discriminator updated five times for every generator update. We adopt Adam optimizer with 31 = 0 and B3 = 0.9. The batch size of the discriminator and the generator set to 64 and 128, respectively. Hinge loss is used following 7 []. We use exponential moving average (EMA) 89 with B = 0.999. The generator is trained for a total of 100k iterations.
BigGAN on the CIFAR-10 dataset. We use learning rate of 2 x 10 for both generators and discriminators. The discriminator is updated four times for every generator update. We adopt Adam optimizer with 31 0 and B2 0.999. The batch size of both the discriminator and the generator set to 50. Hinge loss is used following 7 87). We use EMA with 3 = 0.9999. The generator is trained for total of 200k iterations.
BigGAN on the Inageet dataset. We use DiffAug 193 to augment the input. The learning rate of the discriminator and the generator are set to 4 x 10-4 and x 10-4, respectively. The discriminator is updated one time for every generator update. We adopt Adam optimizer with 3, = and B2 0.999. The batch size of both the discriminator and the generator is set to 256. Hinge loss used following 7 877. We use EMA with = 0.9999. The generator is trained for a total of 200k iterations.
A.4 Evaluation metric
SNGAN on the CIFAR-10 and the STL-10 datasets. We compute Fréchet inception distance (FID) and Inception score (IS) for 50k generated images every 5000 iterations. Best FID and IS are
htttp:://iithu: http:://///iihhu..ooottttttt--- cowmmitttttooooooooooooootiteoot han- hhw.aa/laaaaaaefffctaaaaaggg////
Diffuugeent- www.kkugnnnnnigggniiiiiiiie biggan-cifar
17

---

reported. For the CIFAR- 10 dataset, we report both FID for the training set and test set, whereas, for the STL-10 dataset, we report the FID of the unlabeled partition.
BigGAN on the CIEAR-10 and the manyemageNet dataset. We compute Fréchet inception distance (FII) and Inception score (IS) for 10k generated images every 5000 iterations. Best FID and IS reported.
B Dynamic sparse training details
B.1 How the generator performs DST
In this section, we explain how the generator performs DST below. Note that the generator performs the same for SDST and ADAPT. Sparsity distribution at initialization. Following RigL and ITOP 16 54], only parameters of fully connected and convolutional layers will be pruned. At initialization, we the commonly adopted SerrissRéeyi-Keee Kernel (ERK) strategy [16 14 54 to allocate higher sparsity to larger layers. Specifically, the sparsity of convolutional layers is scaled with 1 - n ='-'+n' + n n''n''
where n' denotes
the number of channels of layer / while w' and h' the widths and the height of the corresponding kernel in that layer. For fully connected layers, Erlös-Rény: Rényi (ER) strategy is used, where the sparsity scaled with d''+"' -
Update schedule. The update schedule controls how many connections are adjusted per DST operation. It can be specified by the number of training iterations between sparse connectivity updates ATg. the initial fraction of connections adjusted 7, and decaying schedule [accay T) for
Drop and grow. After AT training iterations, we update the mask mg by dropping/pruning fdecay( (7, T) 10cl dg number of connections with the lowest magnitude, where |0gl. de the number of parameters and target density for the generator, /decay(7, (7,T') I') is the update schedule. Right after the connection drop, we regrow the same amount of connections.
For the growing criterion, we test both random growth SET 62 and gradient-based growth RigL [16]. Concretely, gradient-based methods find newly-activatess connections with the highest 8CC
gradient magnitude (e while random-based methods explore connections in random fashion. All the newly-activated connections are set to 0. One thing that should noticed is that while previous works consider layer-wist wiiee connections drop and growth, grow and drop connections globally grants more flexibility to the DST method.
EMA for sparse GAN. EMA 89 is well-known for its ability to alleviate the non-convergence of GAN. We also implement EMA for sparse GAN training. Specifically, we zero out the moving average of dropped weights whenever there is a mask change.
B.2 DST hyperparameters for the generator
We use the same hyper-parameters for SDST and ADAPT. The initial y is set 0.5, and we use cosine annealing function fdecay following Rigl. and ITOP.
SNGAN the CIFAR-10 and the STL-10 datasets. The connection update frequency of the generator ATg is set to 500 and 1000 for the CIFAR-I0 dataset and STL-10 dataset, respectively.
BigGAN on the CIFAR-10 and the TinylmageNet dataset. The connection update frequency of the generator ATg is be 1000.
B.3 Density dynamic adjust (DDA) hyper-parameters
In this section, we provide hyper-parameter used in subsection 5.3 We set dp = 2000, ATD = 0.05, [B_, B+] = [0.5, 0.65). Time-averaged BR over 1000 iterations used the indicator.
B.4 DST hyperparameters for the discriminator in ADAPT
We use a constant BR interval [B_, B.]= = 10.45, 0.55) for SNGAN experiments and BigGAN on the CIFAR- 10 dataset. We set the BR interval [B_, B+] [0.3, 0.4] for BigGAN on the TinylmageNet
18

---

Table 5: ResNet architecture for STL-10.
Table 4: ResNet architecture for CIFAR-IO.
(a) Generator
(b) Discriminatur
(a) Gonerator
(b) Discriminator
E RRth ~ N100
image x é [-1.1ja8cthec (1
2 E 1228 (0, image a [-1, 2
dense, 6 x x 512
ResBlock
dense, 256
ResBlock down 128
ResBlock up 256
ResBlock down 128
ResBlock 256
ResBlock down 128
ResBlock 128
ResßBlock down 256
ResBlock up 256
ResBlock down 128
ResBlock up 64
ResBlock down 512
ResBlock up 256
ResBlock down 128
ReLU, conv, Tanh
ResBlock down 1024
BN, ReI.U, cony, Tanh
Rel.U 0.1
Rel.U
Global sum pooling
Glohal avg pooling
dense
since it uses DiffAug. Time-averaged BR over 1000 iterations is used as the indicator. Density increment Ad is set to be 0.05, 0.025, and 0.05 for SNGAN 100A SNGAN (STL-10), and BigGAN (CIFAR-10), respectively. We use the same setting used in subsection B.2 for the generator.
Hyper-parameetrs for ADAPT 2000, 5000, and 10000 iterations for SNGAN (CIFAR-10), SNGAN (STL-10), BigGAN (CIFAR-10), and BigGAN (TinylmageNet), respectively.
The density update frequency of the discriminator ATD is 1000,
Hyper-parameters for ADAPT The density/connections update frequency the discriminator ATD is 2000, 2000, 5000, and 10000 iterations for SNGAN (CIFAR-10), SNGAN (STL-10), BigGAN (CIFAR-10), and BigGAN (TinylmageNet), respectively.
Note that we compute BR for every iteration to visualize the BR evolution, whereas one should note that such computational cost can be greatly decreased BR computed every few iterations.
C Algorithms
In this section, we present the detailed algorithms for DDA, ADAPT relax, and ADAP 1DAPTariitt
C.1 Dynamic adjust algorithm
We first present DDA in Algorithm
Algorithm Dynamic density adjust (DDA) for the discriminator.
Require: Generator discriminator D, upper bound B., and lower bound B-, DA interval ATp, density increment Ad, current training itcration
1: if mod ATp = 0 then
2: Compute time-averaged BR with Equation 3
3: if BR B, then
4: Increase density discriminator from do dp + Ad.
5: else if BR < B then
6: Decrease the density of discriminator from dp to dp Ad.
7: end if
8: end if
C.2 Relaxed balanced dynamic sparse training algorithm
Details of ADAPT algorithm presented Algorithm
C.3 Strict balanced dynamic sparse training algorithm
Details of ADAPTaria algorithm is presented in Algorithm 3
19

---

Algorithm 2 Relaxed balanced dynamic sparse training (ADAP Trelax) for GANs.
Require:
Generator G. discriminator D, total number of training iterations T, number of training steps for discriminator in each iteration N, discriminator adjustment interval ATn, DST interval for the generator ATc, density increment Ad, target generator density dg. BR upper bound By, and lower bound B...
1: Set initial discriminator density do da
2: for t in [1,.... ,T) do
3: for v in [1,... ..] do
4:
Compute the loss of discriminator (p (D(OD).backwwn end for
5:
6:
7:
if mod ATb
then Compute the loss of generator Lc(Oc)
8:
9:
Compute avme-ave BR with Equation
B4 then Increase the density of discriminator from dp to min do + Ad).
12:
13:
else if BR < B_ then
14:
Decrease the density of discriminator from dp max(0%, dn Ad).
end
17:
if mod ATo == 0 then
18:
Apply DST to G
19:
end if
20:
end for
Algorithm 3 Strict balanced dynamic sparse training (ADAPTsnic) for GANs.
Require:
Generator G. discriminator D, total number of training iterations T, number of training steps for discriminator in each iteration N, given maximal density of discriminator dmn, discriminator adjustment interval ATp. DST interval for the generator ATe density increment Ad, target generator density de, BR upper bound B4 and lower bound B
1: Set initial discriminator density dp de
2: for t in [1, ... 2' do
3: for in [1......] . do
4: Compute the loss of discriminator (D(D (oo@D))backuuarr end for
5:
6:
7:
if mod АTD = 0 then
8:
Compute the loss of generator (o(O))
9:
backward()
10:
Compute time- averaged BR with Eqqation 3 if BR > B, and dp < danx then
11:
12:
Inerease the density of discriminator from dp min(d dp Ad). else BR B4 and dp dmux then
Apply DST to D else BR < B_ then
15:
16:
Decrease the density of discriminator from dp max(0%, dp Ad).
17:
end if
end
19: if mod ATG == then
20:
Apply DST to G
21:
end if
22:
end for
20

---

Density of the generator de 10%
Density of the generator 4o=20%
1.25
1.0
1.25
Density of the disciiinatee
discmmminatr
0.75
0.75
0.6
0.6
Balance
0.50
Balance
0.50
0.4
0.4
0.25
0.25
0.00
0.2
0.00
-0.25
Training iteration
Training iteration
Density the generator do= 30%
Density of the generator do=50%
Dersity the discriminator
Density the discriminatoo
1.00
0.8
1.00
0.8
Ballnce natoo
0.75
0.75
0.6
0.6
0.50
0.50
0.4
0.25
0.25
0.25
0.0
0.255
50000
100000
150000
200000
0
50000
100000
150000
200000
Training
Training
Figure Balance ratio and discriminator density evolution during training for ADAP rellxx BigGAN (CIFAR-10). Dashed lines represent BR values of 0.45 and 0.55.
Density of the generator dc=10%
Density of the generator 26%
1.25
1.0
1.25
Densiy of the Instriminator
0.8
0.75
0.75
Balance
0.50
0.50
0.4
0.00
0.2
0.00
0.2
-0.25
Training heration
Training iteration
Density of the generator de 30%
Density of the generator de 50%
Density of the dscriminator
1.00
0.8
1.00
Balance ratio
0.75
0.75
0.6
0.6
Balance
0.4
0.25
0.25
0.00
0.00
0.25
0.0
-0.25
50000
100000
150000
200000
50000
100000
150000
200000
Training
iteration
Figure 6: Balance ratio and discriminator density evolution during training for ADAP Tarret BigGAN (CIFAR-10). Dashed lines represent BR values of 0.45 and 0.55.
D ADAPT balance ratio evolution
In this section, we show that ADAPT methods are able to maintain BR throughout training. We show the time evolution of BR and diseriminator density for BigGAN the CIFAR-10 dataset.
clearly illustrates the ability of ADAPT to keep the BR controlled during GAN training.
Results of Treaa and ADAPT Taria are shown in Figure 3 and Figure
21

---

Table FID (L) of different sparse training methods along with post-hoe pruning baseline with no constraint on the density of the discriminator. Best results are in bold; second- boooddbestt results are underlined.
Dataset
CIFAR-10 (SNGAN)
STL-10 (SNGAN)
CIFAR-10 (BigGAN)
Generator dousity
10%
20%
30%
50%
10%
20%
30%
50%
10%
20%
30%
50%
(Dense Baseline)
10.74
29.71
8.11
Post-hoc pruning
20.89
14.07
12.99
11.90
57.28
37.12
31.98
29.70
15.44
10.84
9.65
8.77
STATIC-Balance
26.75
19.04
15.05
12.24
48. 18
44.67
41.73
37.68
16.98
12.81
10.33
8.47
ITATIC-Strong
26.79
19.65
14.38
11.91
52.48
43.85
42.06
37.47
23.48
14.26
11.19
8.64
SDST- SDST-Balance-SEL
26.23
17.79
13.21
11.79
56.41
46.58
39.93
30.37
12.41
9.87
9.13
8.01
SAST--Strong-SET
16.49
13.36
11.68
10.68
67.37 37
49.96
37.99
31.08
18.94
9.64
8.75
8.36
SDST- Balance- -RigI.
27.06
16.36
12.28
43.08
33.90
31.83
30,30
12.45
8.86
8.03
SDST-SSrron- Rigl.
17.02
13.86
12.51
11.35
53.65
33.25
31.41
30.18
10.58
9.11
8.69
8.33
ADAPT (Ours)
14.19
13.19
12.38
10.60
35.98
33.06
31.71
29.96
10.19
8.56
8.36
8.22
E
More experiment results
E.1 IS and FID for the CIFAR-10 dataset
In this section, present corresponding IS scores results for Table and Table The results are shown in Table and Table 9 respectively. We also include FID results of CIFAR-10 test set in Table 10
E.2 Naively applying DST both the generator and the discriminator
In this section, we follow STU-GAN to compare the baseline where applying DST on both generators and discriminators. We name it DST-bothGD.
We test on SNGAN (CIFAR-I0) with ATD = 1000, ATg = 500, and y = 0.5. Note that we use the balance strategy where dę = dp. The reason is that the strong strategy uses a dense discriminator, and it does not make sense to apply DST to a dense network.
We show the results in Table 7
It shows that it generates unstable results and consistenly performs worse than SDST-Strong, So we do not compare such baseline in the main body of the paper.
E.3 Post-hoc pruning baseline
In this section, we compare different sparse training methods with post-hoc magnitude pruning [68] baseline. Magnitude pruning involves first training dense generator, then pruning weights globally based on their magnitudes. The pruned generator is then fine-tuned with the dense discriminator. We perform additional fine-tuning for 50% of the original total iterations. Results are presented in Table 6)
Our experimental results clearly demonstrate the advantages of dynamic sparse training over post- hoe magnitude pruning. The latter typically requires around 150% normalized training FLOPs, while DST methods constantly achieve comparable or better performance with significantly reduced computational cost.
F A detailed comparison of training costs
In this section, we include the detailed computational cost of all sparse training methods. More specifically, we take into account the density redistribution over different layers in this section. Also, we make an assumption that the computational overhead introduced by computing BR can be neglected [l
Here provide training costs for the strict setting in Table 12
'This is true if compute BR less frequently.
22

---

Table 7: FID (1) of different sparse training methods CIFAR-10 datasets with no constraint on the density of the discriminator. Best results are in bold; second-best results are underlined.
Dataset
CIFAR-10
Generator density
10%
20 %
30 %
50 %
(Dense Baseline)
10.74
Static-Balance
26.75
19.04
15.05
12.24
Static-Strong
26.79
19.65
14.38
11.91
DSS-SO
20.57
14.90
12.58
11.86
DST-bothGD Riiil.
31.95
17.99
13.24
12.47
EDSTT-Bllacc--55
26.23
17.79
13.21
11.79
SOST--Stongg SET
16.49
13.36
11.68
10.68
D----allnce-Rigl
27.06
16.36
14.00
12.28
SDST-Strong-RR RigI.
17.02
13.86
12.51
11.35
ADAPT (Ours)
14.19
13.19
12.38
10.60
Table 8: IS (higher is better) of different sparse training methods. of the discriminator, i.e., dmax 100%.
There is constraint the density
Datuset
SNGAN(CIFAR-10
SNGAN(STL-110
BigGAN
BigGAN(Tiny
Generator donsity
10%
20 f
30 %
30 %
10%
20
30 %
50 f6
10%
20
30
%
50%
10%
20 %
30 %
50 %
(Dense Baseline)
8.48
9.16
8.99
14.65
Balic-Balance Static Strong
7.94
7.52
8.03
8.32
8.45
7.70
8.22
8.35
8.70
7.75
8.13
8.52
3.99
10.45
12.56
13.61
13.73
SDST Balance-SEe SET
7.28
7.89
8.22
8.38
8.43
8.92
9.26
9.31
$.62
8.57
8.82
8.98
11.75
12.60
12.30
12.21
SDSS T-ST-S Strong- SET
8.37
8.54
8.57
8.60
7.65
8,53
9.39
9.21
3.16
8.78
8.85
9.06
12.75
12.84
12.46
13:73
SDST B-S---Blccee-e-e Rigl.
7.19
7.94
8.18
8.34
8.98
9.07
9.12
9.28
8.64
8.71
8.93
12.67
13.32
13.18
13.61
Strong RigL.
9.16
13.35
13.60
ADAPTat (Ours)
8,42
8.14
8.51
8.60
9.08
9.29
9.06
9.26
8.74
9.07
8.98
9.00
13.09
13.57
13,68
15.77
Table 9: IS (higher is better) of different sparse training methods. The density of the discriminator is
constrained to be lower than dmax = 50%.
Dolaset
SNGAN(CIFAR-10)
SNGAN(STL-10)
BigGANiCIFAR-IO
TEeGAN(Tnyymaageeee
Generator density
10%
20 %
30 %
50
10%
20
30 %
50
10%
20
30 5.
50
10%
20 %
30 %
50
(Dense Baseline)
8.48
9.16
8.99
14.65
Static-Balanee
7.34
8.06
7.94
8.19
8.44
8.69
7.99
8.34
8.50
10.65
12.28
13.41
13.57
StaticSttrngg
7,85
9.99
11.61
13.77
13.57
SSDST- Balasce-ETT
7.28
7,89
8:22
8.38
9.43
8.92
9.26
9.31
8.62
8.67
8.82
3.98
11.75
12.60
12.30
12:21
SDST Soro--Srngg--a SET
8.33
9.53
8.40
8.38
8.50
8.77
9,46
9.26
8.55
8.77
8.84
8.98
12.00
12.87
12.16
12.21
SDST-Bulance RipL
7.19
7.94
8.18
8.34
8.98
9.07
9.12
9.28
8.64
8.71
8.91
8.93
12.67
13.32
13.18
13.61
SDST-Suung RigL
8.24
ADAPT (Ours)
8.27
8.36
8:48
8.47
8.98
9.17
9.20
9.19
8.90
8.89
8.92
9.10
13.85
13.61
14.05
14.40
Table 10: FID of test set (L) of different sparse training methods on SNGAN (CIFAR-10) dataset. Best results are in bold; second-best results are underlined.
Maximal discriminator density daax
100
%
Generator density
10%
20 %
30 %
50 %
10%
20 %
30 %
50 %
(Dense Baseline)
13.32
Mlaic-Balance
29.56
21.79
17.80
14.94
29.56
21.79
17.80
14.94
Static-Strong
29.50
22.45
17.12
14.58
24.62
19.43
16.32
14.94
POSST-allanc-SSR
28.84
20.31
15.95
14.35
28.84
20.31
15.95
14.35
SODST-StrongSSE
19.16
16.12
14.45
13.50
18.38
15.33
14.78
14.35
SDST-Balance Balance-Rigl
29.77
19.02
16.68
15.05
29.77
19.02
16.68
15:05
EDST-Stroong Strong-RRgL
19.72
16.50
15,20
14.09
17.92
15.51
15.52
15.05
ADAPT jelax (Ours)
16.82
15.85
15.14
13.37
ADAPT (Ours)
17.19
15.57
14.92
14.80
23

---

Table 11: FID of test set (1) of different sparse training methods on BigGAN (CIFAR-10) 10) dataset. Best results are in bold; second-best results are underlined.
Maximal discriminator density desax
100 %
50 %
Generator density
30
50 %
10%
20 %
50
(Dense Baseline)
10.36
Static-Balance Static-Strong
19.58
15.63
13.21
10.92
19.58
15.63
13.21
10.92
26.08
15.82
13.47
10.95
22.04
16.39
13.73
10.92
ADDST-Bllace-SSER
14.90
12.77
11.82
10.68
14.90
12.77
11.82
10.68
SIDST-Strong-SE
21.63
11.92
11.27
10.75
14.53
11.83
10.96
10.68
ADST-Balance-RRe
14.86
12.03
11.30
10.68
14.86
12.03
11.30
10.68
SDST-Strong RigL
13.35
11.58
11.00
10.88
12.59
12.03
10.89
10.68
ADAPT (Ours)
12.71
11.02
10.62
10.80
ADAPT strcc (Ours)
11.83
11.22
10.92
10.33
Table 12: Normalized training FLOPs (1) of different sparse training methods. The density of the discriminator is constrained to be lower than 50%
CIFAR-I0 (SNGAN)
STL-10 INGAAN
(RgGAN)
(BigGAN)
Ckreenate dennity
10%
20%
30%
53%
10%
2095
30%
50%
1096
27%
30%
50%
10%
20%
30%
50%
100R
10"))
100%
10"")
10"))
89P4
17:85%
76,295
45 2961
105
47.106
90-375
75.95
0.70%
199756
19.00
51.11%
61.8%%
40.915
76 20-199
Stale-Stnng
903805
11 N15
17.17%
17.99%
71 691
77.145
73.55%
$7.5975
41.6855
24 101
40 0V0
41 175
55.175
66.7993
70:305
SDS BaST--Balnw--EE SET
15
76 155
45.411
77 555
4075
60.175
75.38%
10 195
70.15 115
W/W
80 8275
31 1116
37 17/19/
46 18.355
61 591
51.51
71 975955
71/078
30
42
31.9555
11.93
055
175
117
47.505
41:095
175
41.255
51.71%
76.75%
ADAPT..
27 27.55%
11.
71.99%
M1/6%
10.35
21,68
33,836
77.456
24


Reviews:
[{"review_id": "SAJSGvl3Hz", "reviewer": "Reviewer_cAMN", "summary": "Motivated by the identified imbalance between the generator and discriminator during sparse GAN training, this work proposes a quantitative metric dubbed balance ratio as an indicator for the degree of balance in sparse GAN training. Leveraging this metric, this work further proposes the ADAPT framework to dynamically adjust the sparsity of the discriminator towards balanced GAN training. Experiments across various datasets validate the effectiveness of the proposed method in achieving a good trade-off between performance and computational cost.", "questions": "I have listed my questions in the weakness section. I am willing to adjust my scores if my concerns are properly addressed.\n\nMinor: Line 256 \"while the density of the generator is dynamically adjusted with DDA\" => Here \"generator\" should be the \"discriminator\"?", "limitations": "This work targets improving the training efficiency of GANs, thus not suffering from obvious negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is well-written with a clear logical flow. The well-organized structure from observations to understandings and solutions is appreciated.\n\n2. The proposed balance ratio can well indicate the balance between the generator and discriminator during GAN training, which can potentially serve as a useful metric for the community.", "weaknesses": "1. The imbalance between the generator and discriminator has been extensively studied and it is not clear why the proposed quantitative metric (i.e., the balance ratio) can outperform previous indicators or solutions under the scenario of sparse GAN training. The authors are expected to provide a literature review and discuss the key advantage of the proposed method that makes it particularly suitable for sparse GAN training.\n\n2. It is not clear what is the rationale for proposing the two variants ADAPT_relax and ADAPT_strict. What is the expectation of the performance ranking between the two according to the claim \"a more interesting observation is that ADAPT_strict sometimes outperforms ADAPT_relax\"?\n\n3.  For the experimental results, only limited/insufficient baselines are considered. The authors are expected to benchmark the proposed framework with other sparse GAN training methods, e.g., [1][2] cited below.\n\n[1] \"Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective\", T. Chen et al., NeurIPS'21.\n\n[2] \"Don't be so dense: sparse-to-sparse gan training without sacrificing performance\", T. Chen et al., IJCV'23.\n\n4. Since only unstructured sparsity is considered, the reported FLOPs reduction cannot be turned into real-device speedup. The authors are expected to perform experiments under structured sparsity to validate whether the proposed method is still effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our work is well-written and the proposed metric is useful. We hereby address the reviewer's concerns:\n\n> **Q1. It is not clear why the proposed metric can outperform previous indicators or solutions. The authors are expected to provide a literature review and discuss the key advantage of the proposed method that makes it particularly suitable for sparse GAN training.**\n\nWe express our gratitude to the reviewer for providing constructive suggestions. In fact, in our paper, we have included several related papers [1,2,5,6,7]. We hereby provide a more detailed literature review in the **global rebuttal** and we intend to include it in our forthcoming versions as well.\n\n> **Q2. It is not clear what is the rationale for proposing the two variants of ADAPT.**\n\nWe deeply appreciate the reviewer's insightful question. If we understand correctly, the reviewer raises the question regarding the \"rationale\" of the more difficult setting and its corresponding variant $ADAPT_{strict}$ (Please correct us if we are wrong). We want to clarify that we do so to **(1) provide a more comprehensive study by evaluating our proposed approach in a different setting, and (2) to further push the limit of DST to achieve even greater reductions in computational costs.**\n\nAs the reviewer mentions, normally, we would only consider the setting where we can choose  discriminators with arbitrary densities (as considered in STU-GAN [6]), i.e., the relaxed setting. Other than that, we further introduce the strict setting, which is much more difficult but ensures more computational savings, which is explained below.\n\nThe fundamental distinction between the two settings lies in the density constraint imposed on the discriminator. More specifically, in the relaxed setting, the discriminator can have densities in the range $d_D \\in (0\\\\%,100\\\\%]$, offering a wide range of density choices. In contrast, the strict setting with $d_D^{max}=50\\\\%$ limits the discriminator to choose densities in the range $d_D \\in (0\\\\%,50\\\\%]$.\n\nTo illustrate, let's consider a scenario where we require a stronger discriminator when the density $d_D$ is already at $50\\\\%$. In the relaxed setting, $ADAPT_{relax}$ has the freedom to increase the discriminator's density from $50\\\\%$ to $100\\\\%$ (resulting in a dense discriminator). However, such an option is unavailable for $ADAPT_{strict}$ since we are already utilizing the highest density permitted. Consequently, the relaxed setting ensures a minimum computational savings of approximately $50\\\\% * C_{G}$ where $C$ is the computational cost, whereas the strict setting guarantees at least around $50\\\\% * (C_{G}+C_{D})$ computational savings. In essence, the strict setting only permits the use of discriminators with low densities, but it ultimately leads to more substantial computational savings.\n\n> **Q3. What is the expectation of the results in the two settings.**\n\nGiven the constraints it imposes, we expect the performance of $ADAPT_{strict}$ to be inferior compared to $ADAPT_{relax}$ as more restrictions are introduced.\n\n> **Q4. For the experimental results, only limited/insufficient baselines are considered. The authors are expected to benchmark the proposed framework with other sparse GAN training methods, e.g.,[6,7] cited below.**\n\nWe express our gratitude to the reviewer for providing constructive feedback. In response, we would like to elaborate on our answers as follows:\n\n1. We want to kindly point out that, as mentioned in our section 4 (line 137), **STU-GAN [6] is indeed included as one of our baselines**, as it is almost similar to SDST-RigL in our work.\n\n2. The innovative work [7] primarily focuses on lottery ticket finding, which involves a costly train-prune-retrain process, aiming to improve data efficiency. However, our work centers on achieving efficient training. Therefore, we believe that [7] is not a necessary baseline for our specific research objectives.\n\n3. As indicated in [6], STU-GAN has been shown to outperform post-hoc pruning. We have also conducted validation of this finding in Appendix E.3, Table 6, further supporting our selection of STU-GAN as a strong baseline in our study.\n\nWe hope these elaborations provide better clarity on the choices we made for the baselines.\n\n> **Q5. The authors are expected to perform experiments under structured sparsity to validate whether the proposed method is still effective.**\n\nWhile we acknowledge the importance of structured pruning, it is crucial to take into account the current state of the GAN DST field, and the pruning field as a whole.\n\n1. It is important to highlight that a significant number of works (almost all pruning works mentioned in Section 3) in the field continue to focus on unstructured pruning, including follow-up works of LTH, DST, foresight pruning, and others. These works have contributed invaluable insights to the pruning research community.\n\n2. In fact, to the best of our knowledge, the only GAN DST work, i.e., STU-GAN, focuses on unstructured pruning. Our primary objective is to build upon and enhance the potential of STU-GAN by addressing its limitations and extending its capabilities. As a result, while we recognize the significance of structured pruning, it is not the foremost goal of our current work.\n\nHowever, we once again acknowledge that structured pruning plays a crucial role in enhancing model efficiency, and we intend to explore structured pruning extensively in our future research.\n\n> **Q6. \"while the density of the generator is dynamically adjusted with DDA\" => Here \"generator\" should be the \"discriminator\"**\n\nThank you for pointing out the typo. We will fix it in the next version."}, {"review_id": "xV83hiXsZy", "reviewer": "Reviewer_6W4E", "summary": "The paper addresses the challenge of reducing the computational complexity of training GANs by leveraging dynamic sparse training (DST) techniques. The authors propose a novel metric called the balance ratio (BR) to quantify the balance between the sparse generator and discriminator during GAN training. They also introduce a method called balanced dynamic sparse training (ADAPT) to control the BR and achieve a balance between performance and computational cost. \n\nThe paper begins by providing a thorough background on GANs, sparse training, and the challenges associated with applying DST to GANs. The motivation for the research is clearly explained, emphasizing the need for efficient training methods without sacrificing performance. The proposed metric, BR, is introduced and its significance in measuring the balance between the generator and discriminator is well-established.\n\nThe methodology section is comprehensive, detailing the steps involved in ADAPT. The authors describe the specific modifications to the GAN training process, incorporating sparse training techniques and controlling the BR. The mathematical formulations and algorithms are clearly presented, making it easier to understand the implementation details.\n\nThe paper is generally well-written and structured, with a logical flow of ideas. However, there are a few areas where the clarity could be improved. In some sections, the technical details and explanations are a bit dense, making it challenging for readers unfamiliar with the topic to follow along. Providing more intuitive explanations or examples could enhance the accessibility of the paper.", "questions": " ", "limitations": " ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-motivated.\n2. Using balance-ratio as a metric to understand generator and discriminator sparsity and affects is interesting and seems very useful.\n3. The performance improvement over STATIC demonstrate the effectiveness of DDST.", "weaknesses": "The technical details and explanations are a bit dense. Providing more intuitive explanations or examples could enhance the accessibility of the paper. Not required but this can be part of Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful feedback, which recognizes our work is well-motivated, interesting, and effective. Your positive evaluation is both encouraging and valuable.\n\nIn response to the reviewer's valuable suggestion, we will follow the reviewer's suggestion to provide additional details and establish stronger connections between different sections of the paper in the subsequent versions, thereby improving the overall clarity and coherence of our work.\n\nIf there are any further questions or concerns, please raise them and we are more than willing to answer them. "}, {"review_id": "bpIG91pqm1", "reviewer": "Reviewer_PnSi", "summary": "This paper presents a method for dynamic sparse training for GANs. In particular, the authors propose the balance ratio to study the balance status between the generator and discriminator. In addition, a balanced dynamic sparse training strategy is designed by applying BR to achieve a good trade-off between performance and computational cost. Experimental results proved the effectiveness of the proposed method.", "questions": "No.", "limitations": "No.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The motivation of balancing sparse GAN training under resource budget is well presented.\n\n(2) The explanations and illustrations of the balance ratio and balanced dynamic sparse training are well-formulated and mostly clear and intuitive.", "weaknesses": "(1) The motivation of applying DST to GAN is not that novel to the community and the authors listed STU-GAN as an example. In addition, the experimental improvements over STU-GAN is trivial according to Table 1-3.\n\n(2) The normalized training FLOPs in Table 3 fluctuate a lot and are not linearly consistent with the generator density. It contradicts the basic belief of balanced model size between the generator and discriminator for stable GAN training.\n\n(3) The DST strategy is in-time over-parameterization, what is the difference to a dynamic-ratio Dropout?\n\n(4) When the DST strategy is applied, the balance ratio cannot explain away other important factors for the GAN performance. For example, when spectral normalization is used in the discriminator, the GAN training can be very stable among a lot of practice even for unbalanced model size between the generator and discriminator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for recognizing our work is well-motivated, well-formulated and clear. We hereby address the reviewer's questions.\n\n> **Q1. The motivation of applying DST to GAN is not that novel.**\n\n**We want to point out that though STU-GAN [6] is the first to apply DST on GAN, it has its limitations and it requires a better understanding.** Our work is novel as it provides a way to analyze DST for GANs, identifies STU-GAN's limitations, and proposes improvements to address these shortcomings.\n\nMore specifically, in our Section 5, we study STU-GAN through BR and highlight several limitations of it:\n\n1. STU-GAN requires a pre-defined density for the discriminator, lacking a principled method for choosing this value, while the chosen density can significantly impact the final performance of STU-GAN (see Figure 1).\n\n2. STU-GAN may fail when the discriminator is initialized to be weak.\n\nIn essence, STU-GAN only mitigates the unbalance problem when the discriminator is stronger than the generator. The limitation originates from the fact that (1) STU-GAN does not have a principled way to measure the unbalance, (2) STU-GAN only adjusts the generator. \n\nOur work identifies these shortcomings through BR and proposes to address the limitations of STU-GAN.\n\n> **Q2. In addition, the experimental improvements over STU-GAN are trivial according to Table 1-3.**\n\nWe appreciate the reviewer's input, but we respectively do not agree with the reviewer's opinion as (1) the improvement is not trivial, and (2) our goal is not solely to become the SOTA method.\n\n1. **Our improvement is not trivial.** Our improvement is most prominent in very sparse cases. For example, consider the second-best method SDST-Strong-RigL which is well-performing and stable:  \n(1) Table 1, 10\\% gen density, SNGAN on CIFAR-10. Our method outperforms it by 2.83 FID with 63\\% FLOPs.  \n(2) Table 1, 10\\% gen density, SNGAN on STL-10. Our method outperforms it by 17.67 FID with 53\\% FLOPs.  \n(3) Table 1, 10\\% gen density, BigGAN on Tiny-ImageNet. Our method outperforms it by 1.72 FID with 59\\% FLOPs.  \nSimilar favorable outcomes are observed in Table 2. Therefore, our method improves the performance of STU-GAN while significantly reducing computational costs.\n\n2. **The primary goal of our paper extends beyond the SOTA method.** To summarize, (1) we propose a metric to study the balance in sparse GAN training, (2) we introduce different strategies for STU-GAN and study the behaviors of them, (3) we identify and propose solutions for certain limitations of STU-GAN. We believe these findings may pave the way for further research in this domain.\n\n> **Q3. The normalized training FLOPs in Table 3 fluctuate a lot and are not linearly consistent with the generator density. It contradicts the basic belief of balanced model size between the generator and discriminator for stable GAN training.**\n\nWe appreciate the reviewer for bringing up this question, and we would like to provide some insights into why the normalized training FLOPs (sum of the generator and discriminator) do not exhibit \"linear consistent (scaling)\"  with the generator density:\n\n1. **The discriminator density is not fixed.** In our experiments, we observed that the discriminator's density may first increase and then decrease until it stabilizes during the training process. This behavior can be observed in Figure 5 and Figure 6, which helps explain why the total training FLOPs do not linearly scale with the generator density.\n\n2. **The balancing density of the discriminator may not linearly scale with the generator density.** It is important to note that the generator's capacity does not necessarily linearly scale with its density. For example, having a generator with twice the number of parameters does **NOT** imply a proportional increase in representation power or capacity. The same holds true for the discriminator.  As a result, the non-linear relationship between generator density and discriminator density further contributes to the non-linear scaling of total training FLOPs.\n\n3. **Moreover, due to DST/sparse initialization, the allocation of parameters across different layers may vary for different G/D densities and even different epochs.**\n\nHence, we do not expect the total training FLOPs to exhibit linear scaling with the generator density.\n\n> **Q4. The DST strategy is in-time over-parameterization, what is the difference to a dynamic-ratio Dropout?**\n\nWe appreciate the reviewer's question. The key distinctions between ITOP (or DST in general) and dynamic-ratio Dropout lie in **resulting network sparsity**. \n\nDuring training, pruned weights in DST are set to zero and do not receive updates. In contrast, while dropout sets weights' gradients to zero, their magnitudes remain nonzero and may still receive updates when momentum is used. Consequently, DST results in a sparse network with many zero weights after training, while dynamic-ratio Dropout produces a (normal) dense network.\n\nWe hope this clarifies the distinction between ITOP and dynamic-ratio Dropout.\n\n> **Q5. When the DST is applied, BR cannot explain away other important factors for the GAN performance. For example, when spectral normalization (SN) is used in the discriminator, the GAN training can be very stable among a lot of practice even for unbalanced model size between the generator and discriminator.**\n\nWe thank the reviewer for the question. However, we want to emphasize that **(1) SN is indeed used in our models, and (2) SN alone is not enough to enable balanced training for sparse GANs.**\n\nWe want to kindly point out that in our experiments (Section 5), we indeed applied SN to the discriminator. The results shows:\n1. SN alone is not sufficient to stabilize the unbalanced model size between the two components. Despite using SN, the sparse GAN training may still exhibit instability due to the unbalanced components.\n2. The BR is able to quantify the unbalance of the two components when SN is applied."}, {"review_id": "KDjtY2OuyW", "reviewer": "Reviewer_sFon", "summary": "This work proposes a metric named balance ratio to represent the balance between the generator and discriminator in dynamic sparse training, and furthermore proposes balanced dynamic sparse training to balance the performance and computation cost.", "questions": "Could you please give a reasonable explanation for only involving small datasets? If there is a reason I did not notice, I will correspondingly update my score.", "limitations": "I did not find the discussion of limitations. I also did not notice the potential limitations of this work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "(++) There are not many previous works that tried to apply DST to GANs, so this research is very valuable, in my opinion. The insight of the problem of balance between the generator and discriminator is also accurate.\n(++) The motivation is clearly explained and discussed well.", "weaknesses": "(----) The experiments are the main problem. The baselines (SNGAN, BigGAN) seem to be out of date. The involved datasets are all small, but larger images (e.g., FFHQ) are not utilized. Recently, GAN frameworks can even generate images larger than 1024x1024. Generating small images is not so challenging today, as powerful frameworks, e.g., diffusion models, have achieved great success.  In my opinion, at least 256x256 images should be considered to show the value of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's recognition of the value and motivation behind our work. We would like to address the reviewer's concerns as follows:\n\n> **Q1. The experiments are the main problem. The baselines (SNGAN, BigGAN) seem to be out of date.**\n\nThank you for your valuable input. However, we respectfully disagree with the notion that BigGAN is outdated compared to SOTA, e.g. StyleGAN-XL. Although StyleGAN is currently the SOTA model for generating high-resolution images, it has some limitations that make it less versatile than BigGAN. For instance, research [8] has shown that the StyleGAN family struggles with generating images that have high inter-class variations, such as those in ImageNet. Therefore, we believe it is reasonable to conduct our experiments on BigGAN, which is still a highly effective and versatile model for image generation tasks. However, we appreciate your suggestion and will keep the latest SOTA models in mind for future work.\n\n> **Q2. In my opinion, at least 256x256 images should be considered to show the value of the proposed method.**\n\nWe want to explain that **(1) we mainly design our experiments following previous baselines, and (2) our contribution is not solely ADAPT's performance.**\n\n1. It is worth noting that the field of GAN dynamic sparse training (DST) is relatively new, and the pioneering work, STU-GAN, mainly draws its conclusions based on the CIFAR-10 dataset. In our experiments, we have observed excellent performance of our proposed methods on CIFAR-10, STL-10, and TinyImageNet datasets. Additionally, running experiments on larger datasets and higher resolutions can be computationally demanding, requiring multiple GPUs and several days to complete a single case. Given these resource constraints, conducting extensive experiments involving multiple density ratios, SDST variants, and two settings on larger datasets may currently be computationally heavy. \n\n2. The primary goal of our paper extends beyond the state-of-the-art (SOTA) method.  \nTo provide a concise summary, we accomplish the following:  \n(1) Propose a novel metric to study the balance in sparse GAN training.  \n(2) Introduce and analyze the behaviors of various strategies for STU-GAN, offering valuable insights into the effectiveness and limitations of it.  \n(3) Identify and propose solutions for certain limitations of STU-GAN, thereby contributing to the advancement of dynamic sparse training in the GAN domain.  \nWe firmly believe that these findings not only enhance the performance of STU-GAN but also have the potential to pave the way for further research and advancements in this field.\n\nNonetheless, we highly value the reviewer's suggestion, and we are actively working on testing our methods on larger datasets. As soon as these experiments are completed, we will incorporate the results into the next version of our work, providing a comprehensive evaluation of our proposed methods."}]

Author rebuttal:
We thank the reviewers for recognizing our work is well-written (cAMN, PnSi), useful (cAMN, 6W4E), effective (6W4E), well-motivated (6W4E, PnSi, sFon), and valuable (sFon). In response to the reviewers' requests, we have included the following additional content.

> **Literature review requested by reviewer cAMN**

Addressing the balance between the generator and discriminator in GAN training has been the focus of various works. However, directly applying existing methods to sparse GAN training poses challenges. For instance, [1,2] offer theoretical analyses on the issue of imbalance but may have limited practical benefits, e.g., they require training multiple generators and discriminators. Empirically, BEGAN [3] proposes to use proportional control theory to maintain a hyper-parameter $\frac{\mathbb{E}[|G(z)-D(G(z))|^\eta]}{\mathbb{E}[|x-D(x)|^\eta]}$, but it is only applicable when the discriminator is an auto-encoder. Unbalanced GAN [4] pretrains a VAE to initialize the generator, which may only address the unbalance near initialization. GCC [5] considers the balance during GAN compression, while its criterion requires a trained (dense) GAN, which is not given in the DST setting. Finally, STU-GAN [6] proposes to use DST to address the unbalance issues but may fail under certain conditions, as demonstrated in our experiments.   

In summary, the existing approaches cannot be directly applied to balanced GAN DST. The only metric that could potentially be helpful for sparse GAN training is the one presented by BEGAN [3], which has restrictions on the discriminator architecture. Unlike BEGAN, our metric isn't constrained to a specific discriminator architecture. Furthermore, it demonstrated simplicity in computation and effectiveness in a broad range of experiments shown in our paper.

[1] Arora, Sanjeev, et al. "Generalization and equilibrium in generative adversarial nets (gans)." ICML, 2017.

[2] Bai, Yu, Tengyu Ma, and Andrej Risteski. "Approximability of discriminators implies diversity in GANs." ICLR, 2018.

[3] Berthelot, David, Thomas Schumm, and Luke Metz. "Began: Boundary equilibrium generative adversarial networks." arXiv preprint arXiv:1703.10717 (2017).

[4] Ham, Hyungrok, Tae Joon Jun, and Daeyoung Kim. "Unbalanced gans: Pre-training the generator of generative adversarial network using variational autoencoder." ICML (2020).

[5] Li, Shaojie, et al. "Revisiting discriminator in GAN compression: A generator-discriminator cooperative compression scheme." NeurIPS (2021).

[6] Liu, Shiwei, et al. "Don’t be so dense: sparse-to-sparse gan training without sacrificing performance." IJCV (2023).

[7] Chen, Tianlong, et al. "Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective." NeurIPS (2021).

[8] Kang, Minguk, Joonghyuk Shin, and Jaesik Park. "Studiogan: A taxonomy and benchmark of gans for image synthesis." arXiv preprint arXiv:2206.09479 (2022).

Meta-review: 1x A, 2x BA, and 1x BR. This paper proposes a balance ratio metric between the generator and discriminator in sparse GAN training, and dynamically adjusts the sparsity of discriminator to improve the balance of GAN training. Most reviewers agree on accepting the paper due to its (1) clear presentation, (2) insightful motivations, (3) valuable topic, and (4) useful metric technique. The rebuttal has addressed the BR reviewer’s concern although his final rating is lacking.
Decision: Accept (poster)