SlowGuess/NeurIPS_2023_OCR huggingface dataset:
since it uses DiffAug. Time-averaged BR over 1000 iterations is used as the indicator. Density increment  $\Delta d$  is set to be 0.05, 0.025, and 0.05 for SNGAN (CIFAR-10), SNGAN (STL-10), and BigGAN (CIFAR-10), respectively. We use the same setting used in subsection B.2 for the generator.
Hyper-parameters for ADAPTrelax. The density update frequency of the discriminator  $\Delta T_{D}$  is 1000, 2000, 5000, and 10000 iterations for SNGAN (CIFAR-10), SNGAN (STL-10), BigGAN (CIFAR-10), and BigGAN (TinyImageNet), respectively.
Hyper-parameters for ADAPTstrict. The density/connections update frequency of the discriminator  $\Delta T_{D}$  is 2000, 2000, 5000, and 10000 iterations for SNGAN (CIFAR-10), SNGAN (STL-10), BigGAN (CIFAR-10), and BigGAN (TinyImageNet), respectively.
Note that we compute BR for every iteration to visualize the BR evolution, whereas one should note that such computational cost can be greatly decreased if BR is computed every few iterations.
In this section, we present the detailed algorithms for DDA, ADAPTrelax, and ADAPTstrict.
C.1 Dynamic adjust algorithm
We first present DDA in Algorithm 1.
Algorithm 1 Dynamic density adjust (DDA) for the discriminator.
Require: Generator  $G$  , discriminator  $D$  , BR upper bound  $B_{+}$  and lower bound  $B_{-}$  , DA interval  $\Delta T_{D}$  , density increment  $\Delta d$  , current training iteration t.   
1: if t mod  $\Delta T_{D} = = 0$  then   
2: Compute time-averaged BR with Equation 3   
3: if  $\mathrm{BR} > B_{+}$  then   
4: Increase the density of discriminator from  $d_{D}$  to  $d_{D} + \Delta d$    
5: else if  $\mathrm{BR} <   B_{-}$  then   
6: Decrease the density of discriminator from  $d_{D}$  to  $d_{D} - \Delta d$    
7: end if   
8: end if
C.2 Relaxed balanced dynamic sparse training algorithm
Details of ADAPTrelax algorithm is presented in Algorithm 2.
C.3 Strict balanced dynamic sparse training algorithm
Details of ADAPT<sub>strict</sub> algorithm is presented in Algorithm 3.
Algorithm 2 Relaxed balanced dynamic sparse training (ADAPTrelax) for GANs.
Require: Generator  $G$ , discriminator  $D$ , total number of training iterations  $T$ , number of training steps for discriminator in each iteration  $N$ , discriminator adjustment interval  $\Delta T_{D}$ , DST interval for the generator  $\Delta T_{G}$ , density increment  $\Delta d$ , target generator density  $d_{G}$ , BR upper bound  $B_{+}$  and lower bound  $B_{-}$ .  
1: Set initial discriminator density  $d_{D} = d_{G}$   
2: for  $t$  in  $[1, \dots, T]$  do  
3: for  $n$  in  $[1, \dots, N]$  do  
4: Compute the loss of discriminator  $\mathcal{L}_D(\pmb{\theta}_D)$   
5:  $\mathcal{L}_D(\pmb{\theta}_D)$ .backward()  
6: end for  
7: if  $t$  mod  $\Delta T_{D} == 0$  then  
8: Compute the loss of generator  $\mathcal{L}_G(\pmb{\theta}_G)$   
9:  $\mathcal{L}_G(\pmb{\theta}_G)$ .backward()  
10: Compute time-averaged BR with Equation 3  
11: if  $\mathrm{BR} > B_{+}$  then  
12: Increase the density of discriminator from  $d_{D}$  to  $\min(100\%, d_{D} + \Delta d)$ .  
13: else if  $\mathrm{BR} < B_{-}$  then  
14: Decrease the density of discriminator from  $d_{D}$  to  $\max(0\%, d_{D} - \Delta d)$ .  
15: end if  
16: end if  
17: if  $t$  mod  $\Delta T_{G} == 0$  then  
18: Apply DST to  $G$   
19: end if  
20: end for
Algorithm 3 Strict balanced dynamic sparse training (ADAPT_strict) for GANs.
Require: Generator  $G$  discriminator  $D$  total number of training iterations  $T$  number of training steps for discriminator in each iteration  $N$  given maximal density of discriminator  $d_{\mathrm{max}}$  discriminator adjustment interval  $\Delta T_{D}$  DST interval for the generator  $\Delta T_{G}$  density increment  $\Delta d$  target generator density  $d_{G}$  BR upper bound  $B_{+}$  and lower bound  $B_{-}$    
1: Set initial discriminator density  $d_D = d_G$    
2: for t in [1, ,T] do   
3: for n in [1, ,N] do   
4: Compute the loss of discriminator  $\mathcal{L}_D(\pmb {\theta}_D)$    
5:  $\mathcal{L}_D(\pmb {\theta}_D).backward()$    
6: end for   
7: if t mod  $\Delta T_{D} = = 0$  then   
8: Compute the loss of generator  $\mathcal{L}_G(\pmb {\theta}_G)$    
9:  $\mathcal{L}_G(\pmb {\theta}_G).backward()$    
10: Compute time-averaged BR with Equation 3   
11: if BR > B+ and  $d_D <   d_{\mathrm{max}}$  then   
12: Increase the density of discriminator from  $d_{D}$  to min  $(d_{\mathrm{max}},d_{D} + \Delta d)$    
13: else if BR > B+ and  $d_D = = d_{\mathrm{max}}$  then   
14: Apply DST to D   
15: else if BR < B- then   
16: Decrease the density of discriminator from  $d_{D}$  to max(0%,  $d_{D} - \Delta d$    
17: end if   
18: end if   
19: if t mod  $\Delta T_{G} = = 0$  then   
20: Apply DST to G   
21: end if   
22: end for
Figure 5: Balance ratio and discriminator density evolution during training for  $\mathrm{ADAPT}_{\mathrm{relax}}$  on BigGAN (CIFAR-10). Dashed lines represent BR values of 0.45 and 0.55.
Figure 6: Balance ratio and discriminator density evolution during training for ADAPTstrict on BigGAN (CIFAR-10). Dashed lines represent BR values of 0.45 and 0.55.
D ADAPT balance ratio evolution
In this section, we show that ADAPT methods are able to maintain a BR throughout training. We show the time evolution of BR and discriminator density for BigGAN on the CIFAR-10 dataset.
Results of ADAPTrelax and ADAPTstrict are shown in Figure 5 and Figure 6. It clearly illustrates the ability of ADAPT to keep the BR controlled during GAN training.
Table 6: FID (↓) of different sparse training methods along with post-hoc pruning baseline with no constraint on the density of the discriminator. Best results are in bold; second-best results are underlined.
Dataset	CIFAR-10 (SNGAN)	STL-10 (SNGAN)	CIFAR-10 (BigGAN)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	10.74	29.71	8.11
Post-hoc pruning	20.89	14.07	12.99	11.90	57.28	37.12	31.98	29.70	15.44	10.84	9.65	8.77
STATIC-Balance	26.75	19.04	15.05	12.24	48.18	44.67	41.73	37.68	16.98	12.81	10.33	8.47
STATIC-Strong	26.79	19.65	14.38	11.91	52.48	43.85	42.06	37.47	23.48	14.26	11.19	8.64
◆ SDST-Balance-SET	26.23	17.79	13.21	11.79	56.41	46.58	39.93	30.37	12.41	9.87	9.13	8.01
◆ SDST-Strong-SET	16.49	13.36	11.68	10.68	67.37	49.96	37.99	31.08	18.94	9.64	8.75	8.36
◆ SDST-Balance-RigL	27.06	16.36	14.00	12.28	43.08	33.90	31.83	30.30	12.45	9.42	8.86	8.03
◆ SDST-Strong-RigL	17.02	13.86	12.51	11.35	53.65	33.25	31.41	30.18	10.58	9.11	8.69	8.33
ADAPTrelax (Ours)	14.19	13.19	12.38	10.60	35.98	33.06	31.71	29.96	10.19	8.56	8.36	8.22
E More experiment results
E.1 IS and FID for the CIFAR-10 dataset
In this section, we present corresponding IS scores results for Table 1 and Table 2. The results are shown in Table 8 and Table 9, respectively. We also include FID results of CIFAR-10 test set in Table 10.
E.2 Naively applying DST to both the generator and the discriminator
In this section, we follow STU-GAN to compare the baseline where applying DST on both generators and discriminators. We name it DST-bothGD.
We test on SNGAN (CIFAR-10) with  $\Delta T_{D} = 1000$ ,  $\Delta T_{G} = 500$ , and  $\gamma = 0.5$ . Note that we use the balance strategy where  $d_{G} = d_{D}$ . The reason is that the strong strategy uses a dense discriminator, and it does not make sense to apply DST to a dense network.
We show the results in Table 7. It shows that it generates unstable results and consistently performs worse than SDST-Strong. So we do not compare such baseline in the main body of the paper.
E.3 Post-hoc pruning baseline
In this section, we compare different sparse training methods with post-hoc magnitude pruning [68] baseline. Magnitude pruning involves first training a dense generator, then pruning its weights globally based on their magnitudes. The pruned generator is then fine-tuned with the dense discriminator. We perform additional fine-tuning for  $50\%$  of the original total iterations. Results are presented in Table 6.
Our experimental results clearly demonstrate the advantages of dynamic sparse training over post-hoc magnitude pruning. The latter typically requires around  $150\%$  normalized training FLOPs, while DST methods constantly achieve comparable or better performance with significantly reduced computational cost.
F A detailed comparison of training costs
In this section, we include the detailed computational cost of all sparse training methods. More specifically, we take into account the density redistribution over different layers in this section. Also, we make an assumption that the computational overhead introduced by computing BR can be neglected. $^4$
Here we provide training costs for the strict setting in Table 12.
This is true if we compute BR less frequently.
Table 7: FID (↓) of different sparse training methods on CIFAR-10 datasets with no constraint on the density of the discriminator. Best results are in bold; second-best results are underlined.
Dataset	CIFAR-10
Generator density	10%	20 %	30 %	50 %
(Dense Baseline)	10.74
Static-Balance	26.75	19.04	15.05	12.24
Static-Strong	26.79	19.65	14.38	11.91
• DST-bothGD-SET	20.57	14.90	12.58	11.86
• DST-bothGD-RigL	31.95	17.99	13.24	12.47
• SDST-Balance-SET	26.23	17.79	13.21	11.79
• SDST-Strong-SET	16.49	13.36	11.68	10.68
• SDST-Balance-RigL	27.06	16.36	14.00	12.28
• SDST-Strong-RigL	17.02	13.86	12.51	11.35
ADAPTrelax (Ours)	14.19	13.19	12.38	10.60
Table 8: IS (higher is better) of different sparse training methods. There is no constraint on the density of the discriminator, i.e.,  $d_{\mathrm{max}} = 100\%$ .
Dataset	SNGAN(CIFAR-10)	SNGAN(STL-10)	BigGAN(CIFAR-10)	BigGAN(TinyImageNet)
Generator density	10%	20 %	30 %	50 %	10%	20 %	30 %	50 %	10%	20 %	30 %	50 %	10%	20 %	30 %	50 %
(Dense Baseline)	8.48	9.16	8.99	14.65
Static-Balance	7.24	7.83	8.06	8.38	7.94	8.19	8.44	8.69	7.99	8.24	8.68	8.90	10.65	12.28	13.41	13.57
Static-Strong	7.52	8.03	8.32	8.45	7.70	8.22	8.35	8.70	7.75	8.13	8.52	8.99	10.45	12.56	13.61	13.73
• SDST-Balance-SET	7.28	7.89	8.22	8.38	8.43	8.92	9.26	9.31	8.62	8.67	8.82	8.98	11.75	12.60	12.30	12.21
• SDST-Strong-SET	8.37	8.54	8.57	8.60	7.65	8.53	9.39	9.21	8.16	8.78	8.85	9.06	12.75	12.84	12.46	13.73
• SDST-Balance-RigL	7.19	7.94	8.18	8.34	8.98	9.07	9.12	9.28	8.64	8.71	8.91	8.93	12.67	13.32	13.18	13.61
• SDST-Strong-RigL	8.32	8.52	8.59	8.57	8.15	9.10	9.16	9.17	8.65	8.72	8.97	9.00	13.32	13.35	13.60	14.47
ADAPTrelax (Ours)	8.42	8.44	8.54	8.60	9.08	9.29	9.06	9.26	8.74	9.07	8.98	9.00	13.09	13.57	13.68	15.77
Table 9: IS (higher is better) of different sparse training methods. The density of the discriminator is constrained to be lower than  $d_{\mathrm{max}} = 50\%$ .
Dataset	SNGAN(CIFAR-10)	SNGAN(STL-10)	BigGAN(CIFAR-10)	BigGAN(TinyImageNet)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	8.48	9.16	8.99	14.65
Static-Balance	7.24	7.83	8.06	8.38	7.94	8.19	8.44	8.69	7.99	8.24	8.68	8.90	10.65	12.28	13.41	13.57
Static-Strong	7.85	8.14	8.31	8.38	7.89	8.22	8.38	8.69	7.75	8.03	8.52	8.90	9.99	11.61	13.77	13.57
• SDST-Balance-SET	7.28	7.89	8.22	8.38	8.43	8.92	9.26	9.31	8.62	8.67	8.82	8.98	11.75	12.60	12.30	12.21
• SDST-Strong-SET	8.33	8.53	8.40	8.38	8.50	8.77	9.46	9.26	8.55	8.77	8.84	8.98	12.00	12.87	12.16	12.21
• SDST-Balance-RigL	7.19	7.94	8.18	8.34	8.98	9.07	9.12	9.28	8.64	8.71	8.91	8.93	12.67	13.32	13.18	13.61
• SDST-Strong-RigL	8.24	8.48	8.37	8.34	8.28	9.05	9.11	9.28	8.61	8.83	8.84	8.93	12.04	12.66	13.57	13.61
ADAPTstrict (Ours)	8.27	8.36	8.48	8.47	8.98	9.17	9.20	9.19	8.90	8.89	8.92	9.10	13.85	13.61	14.05	14.40
Table 10: FID of test set  $(\downarrow)$  of different sparse training methods on SNGAN (CIFAR-10) dataset. Best results are in bold; second-best results are underlined.
Maximal discriminator density \(d_{\text{max}}\)	100 %	50 %
Generator density (Dense Baseline)	10%	20%	30%	50%	10%	20%	30%	50%
13.32
Static-Balance	29.56	21.79	17.80	14.94	29.56	21.79	17.80	14.94
Static-Strong	29.50	22.45	17.12	14.58	24.62	19.43	16.32	14.94
SDST-Balance-SET	28.84	20.31	15.95	14.35	28.84	20.31	15.95	14.35
SDST-Strong-SET	19.16	16.12	14.45	13.50	18.38	15.33	14.78	14.35
SDST-Balance-RigL	29.77	19.02	16.68	15.05	29.77	19.02	16.68	15.05
SDST-Strong-RigL	19.72	16.50	15.20	14.09	17.92	15.51	15.52	15.05
ADAPTrelax(Ours)	16.82	15.85	15.14	13.37	-	-	-	-
ADAPTstrict(Ours)	-	-	-	-	17.19	15.57	14.92	14.80
Table 11: FID of test set (↓) of different sparse training methods on BigGAN (CIFAR-10) dataset. Best results are in bold; second-best results are underlined.
Maximal discriminator density \(d_{\text{max}}\)	100 %	50 %
Generator density (Dense Baseline)	10%	20 %	30 %	50 %	10%	20 %	30 %	50 %
10.36
Static-Balance	19.58	15.63	13.21	10.92	19.58	15.63	13.21	10.92
Static-Strong	26.08	15.82	13.47	10.95	22.04	16.39	13.73	10.92
SDST-Balance-SET	14.90	12.77	11.82	10.68	14.90	12.77	11.82	10.68
SDST-Strong-SET	21.63	11.92	11.27	10.75	14.53	11.83	10.96	10.68
SDST-Balance-RigL	14.86	12.03	11.30	10.68	14.86	12.03	11.30	10.68
SDST-Strong-RigL	13.35	11.58	11.00	10.88	12.59	12.03	10.89	10.68
ADAPTrelax (Ours)	12.71	11.02	10.62	10.80	-	-	-	-
ADAPTstrict (Ours)	-	-	-	-	11.83	11.22	10.92	10.33
Table 12: Normalized training FLOPs (↓) of different sparse training methods. The density of the discriminator is constrained to be lower than  $50\%$ .
Dataset	CIFAR-10 (SNGAN)	STL-10 (SNGAN)	CIFAR-10 (BigGAN)	TinyImageNet (BigGAN)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	100% (2.67 × 1017)	100% (3.94 × 1017)	100% (6.81 × 1017)	100% (9.85 × 1017)
Static-Balance	8.97%	17.08%	26.25%	47.25%	27.30%	47.14%	59.22%	73.35%	9.79%	19.02%	28.66%	49.03%	23.25%	44.87%	60.91%	79.29%
Static-Strong	30.89%	33.58%	37.17%	47.25%	70.65%	71.48%	72.14%	73.35%	42.66%	43.69%	45.10%	49.03%	41.52%	55.03%	66.29%	79.29%
• SDST-Balance-SET	9.78%	18.91%	28.35%	48.44%	27.55%	47.60%	60.17%	75.38%	10.35%	20.12%	29.96%	49.82%	21.13%	37.06%	48.83%	65.58%
• SDST-Strong-SET	31.87%	35.51%	39.53%	48.44%	70.95%	71.97%	73.07%	75.38%	43.25%	44.80%	46.42%	49.82%	39.28%	47.31%	54.11%	65.58%
• SDST-Balance-RigL	10.71%	17.43%	25.66%	43.56%	29.51%	50.41%	63.34%	79.03%	9.92%	19.30%	28.90%	48.31%	24.97%	43.86%	57.26%	76.75%
• SDST-Strong-RigL	31.22%	33.93%	36.63%	43.56%	72.95%	75.05%	76.42%	79.03%	42.80%	44.08%	45.37%	48.31%	43.76%	53.71%	63.05%	76.75%
ADAPT_strict (Ours)	24.23%	27.55%	31.70%	37.83%	50.91%	70.18%	75.99%	80.68%	10.32%	23.69%	31.54%	33.83%	34.42%	51.68%	62.34%	77.46%