SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\bullet$  SDST requires a pre-defined discriminator density  $d_{D}$  before training. However, it is unclear what is a good choice. In real-world scenarios, it is not practical to manually search for the optimal  $d_{D}$  for each  $d_{G}$ . A workaround may be using the maximum allowed density for the discriminator. However, as shown in Figure 1, the best performance is not always obtained with the maximum  $d_{D} = 100\%$ . Moreover, we are wasting extra computational cost for a worse performance if we use an overly-strong discriminator.
$\Theta$  SDST fails if there is an additional constraint on the density of the discriminator  $d_{D}$ . As Figure 1 suggests, for weak discriminators, SDST is unable to show consistent improvement compared to the STATIC baseline.
Hence, STU-GAN (or SDST in general), which directly applies DST to the generator, may only be useful when the corresponding discriminator is strong enough. In this sense, obtaining balanced training automatically is essential in GAN DST to deal with more complicated scenarios.
6 Balanced dynamic sparse training for GANs
In this section, we describe our methodology for balanced sparse GAN training.
STU-GAN (or SDST in general) considered in the last section cannot generate stable and satisfying performance. This implies that we should utilize the discriminator in a better way rather than do nothing (like SDST) or directly apply DST to the discriminator (see subsection E.2 for additional experiments). Consequently, DDA (subsection 5.3), which adjusts the discriminator density to stabilize GAN training, is a favorable candidate to address the issue. To this end, we propose bAlancedDynAmic sParse Training (ADAPT), which adjusts the density of the discriminator during training with DDA while the generator performs DST.
We further introduce two variants, namely  $\mathrm{ADAPT}_{\mathrm{relax}}$  and  $\mathrm{ADAPT}_{\mathrm{strict}}$ , based on whether we force the discriminator to be sparse. We present them in subsection 6.1 and subsection 6.2. These methods are more flexible and generate more stable performance compared to SDST.
Table 1: FID (↓) of different sparse training methods with no constraint on the density of the discriminator. Best results are in bold; second-best results are underlined.
Dataset	CIFAR-10 (SNGAN)	STL-10 (SNGAN)	CIFAR-10 (BigGAN)	TinyImageNet (BigGAN)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	10.74	29.71	8.11	15.43
STATIC-Balance	26.75	19.04	15.05	12.24	48.18	44.67	41.73	37.68	16.98	12.81	10.33	8.47	28.78	21.67	18.86	17.51
STATIC-Strong	26.79	19.65	14.38	11.91	52.48	43.85	42.06	37.47	23.48	14.26	11.19	8.64	31.44	22.51	18.22	18.00
♦SDST-Balance-SET	26.23	17.79	13.21	11.79	56.41	46.58	39.93	30.37	12.41	9.87	9.13	8.01	25.39	21.30	21.80	21.20
♦SDST-Strong-SET	16.49	13.36	11.68	10.68	67.37	49.96	37.99	31.08	18.94	9.64	8.75	8.36	22.20	20.56	21.70	18.32
♦SDST-Balance-RigL	27.06	16.36	14.00	12.28	43.08	33.90	31.83	30.30	12.45	9.42	8.86	8.03	21.60	19.33	18.57	17.45
♦SDST-Strong-RigL	17.02	13.86	12.51	11.35	53.65	33.25	31.41	30.18	10.58	9.11	8.69	8.33	21.14	18.95	17.75	16.30
ADAPTrelax (Ours)	14.19	13.19	12.38	10.60	35.98	33.06	31.71	29.96	10.19	8.56	8.36	8.22	19.42	17.99	17.06	14.15
6.1 ADAPTrelax: Balanced dynamic sparse training in the relaxed setting
In this section, we consider the relaxed setting where a dense discriminator can be used, i.e.,  $d_{D} \leq d_{\max} = 100\%$ . This relaxed scenario gives the greatest flexibility to the discriminator. However, it does not necessarily enforce the sparsity of the discriminator (hence, no computational savings for the discriminator) because the density of the discriminator can be as high as  $100\%$ .
For the relaxed setting, we use the direct combination of SDST with DDA. Precisely, the generator is adjusted using DST as mentioned in section 4 while the density of the discriminator is dynamically adjusted with DDA as mentioned in subsection 5.3. We call such a combination relaxed balanced dynamic sparse training (ADAPTrelax). Please see Appendix C Algorithm 2 for more details.
Comparison to STU-GAN (or SDST in general). Compared to STU-GAN (or SDST in general), which pre-defines and fixes the discriminator density during training, the difference is that for  $\mathrm{ADAPT}_{\mathrm{relax}}$ , the density of the discriminator is adjusted during the training process automatically through real-time monitoring of the balance ratio. Given the initial discriminator density  $d_D^{\mathrm{int}} = d_G$ ,  $\mathrm{ADAPT}_{\mathrm{relax}}$  increases the discriminator density if a stronger discriminator is needed, and vice versa.
6.2 ADAPTstrict: Balanced dynamic sparse training in the strict setting
Different from subsection 6.1, we now consider a strict setting where there is an additional sparsity constraint on the discriminator density in this section, i.e.,  $d_{D} \leq d_{\max} < 100\%$ .
$\mathrm{ADAPT}_{\mathrm{relax}}$  introduced in the previous section does not necessarily enforce sparsity for the discriminator, which provides less memory/training resources saving for larger generator density ratios. Note that the discriminator does not take advantage of DST to explore the structure of the dense network. Hence, we further present strict balanced dynamic sparse training  $(\mathrm{ADAPT}_{\mathrm{strict}})$  in this section. This method allows the discriminator to perform DST in a controlled manner, which can lead to a better exploration of the dense network structure while maintaining the balance between the generator and the discriminator. We explain how  $\mathrm{ADAPT}_{\mathrm{strict}}$  differs from  $\mathrm{ADAPT}_{\mathrm{relax}}$  below:
$\bullet$  Capacity increase of the discriminator. The essential difference lies when the observed BR is higher than  $B_{+}$ , which means we need a stronger discriminator. In this case, if the discriminator density is lower than the constraint, i.e.,  $d_{D} < d_{\mathrm{max}}$ ,  $\mathrm{ADAPT}_{\mathrm{strict}}$  will perform just like  $\mathrm{ADAPT}_{\mathrm{relax}}$  to increase the discriminator density. However, if the discriminator is already the maximum density, i.e.,  $d_{D} = d_{\mathrm{max}}$ , the discriminator will alternatively perform DST as a way of increasing the discriminator capacity (See subsection 5.4 for intuition).
$\Theta$  Capacity decrease of the discriminator. Similar to  $\mathrm{ADAPT}_{\mathrm{relax}}$ , when the observed BR is lower than  $B_{-}$ , we will decrease the discriminator density.
Hence,  $\mathrm{ADAPT}_{\mathrm{strict}}$  makes the discriminator adaptive both in the density level (through density adjustment) and the parameter level (through DST). The algorithm of  $\mathrm{ADAPT}_{\mathrm{strict}}$  is shown in Appendix C Algorithm 3.
6.3 Experiment setting
Datasets, architectures, and target sparsity ratios. We conduct experiments on SNGAN with ResNet architectures on the CIFAR-10 [40] and the STL-10 [11] datasets. We have also conducted experiments with BigGAN [7] on the CIFAR-10 and TinyImageNet dataset (with DiffAug [93]).
Table 2: FID (↓) of different sparse training methods. The density of the discriminator is constrained to be lower than  $50\%$ . Best results are in bold; second-best results are underlined.
Dataset	CIFAR-10 (SNGAN)	STL-10 (SNGAN)	CIFAR-10 (BigGAN)	TinyImageNet (BigGAN)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	10.74	29.71	8.11	15.43
STATIC-Balance	26.75	19.04	15.05	12.58	48.18	44.67	41.73	37.68	16.98	12.81	10.33	8.47	28.78	21.67	18.86	17.51
STATIC-Strong	21.73	16.69	13.48	12.58	50.36	44.06	40.73	37.68	18.91	13.43	10.84	8.47	33.01	23.93	17.90	17.51
SDST-Balance-SET	26.23	17.79	13.21	11.79	56.24	44.51	41.23	30.80	12.41	9.87	9.13	8.01	25.39	21.30	21.80	21.20
SDST-Strong-SET	15.68	12.75	11.98	11.79	57.91	50.05	38.13	30.80	11.85	9.39	8.61	8.01	22.68	20.24	22.00	21.20
SDST-Balance-RigL	27.06	16.36	14.00	12.28	43.08	33.90	31.64	30.30	12.45	9.42	8.86	8.03	21.60	19.33	18.57	17.45
SDST-Strong-RigL	15.19	12.93	12.75	12.28	53.74	37.34	31.98	30.30	10.11	9.17	8.35	8.03	21.90	20.43	18.29	17.45
ADAPTstrict (Ours)	14.53	12.73	12.20	12.11	41.18	31.59	31.16	29.11	9.29	8.64	8.44	7.90	18.89	17.37	16.93	16.02
Table 3: Normalized training FLOPs (↓) of different sparse training methods with no constraint on the density of the discriminator.
Dataset	CIFAR-10 (SNGAN)	STL-10 (SNGAN)	CIFAR-10 (BigGAN)	TinyImageNet (BigGAN)
Generator density	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%	10%	20%	30%	50%
(Dense Baseline)	100% (2.67 × 1017)	100% (3.94 × 1017)	100% (6.81 × 1017)	100% (9.85 × 1017)
Static-Balance	8.97%	17.08%	26.25%	47.25%	27.30%	47.14%	59.22%	73.35%	9.79%	19.02%	28.66%	49.03%	23.25%	44.87%	60.91%	79.29%
Static-Strong	58.29%	60.94%	64.53%	74.61%	86.12%	86.94%	87.60%	88.84%	83.78%	84.80%	86.21%	90.15%	48.02%	61.62%	72.79%	85.79%
◆SDST-Balance-SET	9.78%	18.91%	28.35%	48.44%	27.55%	47.60%	60.17%	75.38%	10.35%	20.12%	29.96%	49.82%	21.13%	37.06%	48.83%	65.58%
◆SDST-Strong-SET	59.25%	62.94%	66.89%	75.96%	86.36%	87.43%	88.49%	90.82%	84.36%	85.90%	87.52%	90.95%	45.66%	53.91%	60.61%	71.88%
◆SDST-Balance-RigL	10.71%	17.43%	25.66%	43.56%	29.51%	50.41%	63.34%	79.03%	9.92%	19.30%	28.90%	48.31%	24.97%	43.86%	57.26%	76.75%
◆SDST-Strong-RigL	58.63%	61.35%	64.04%	71.01%	88.51%	90.24%	91.78%	94.57%	83.97%	85.24%	86.59%	89.54%	50.05%	61.02%	69.64%	83.35%
ADAPTrelax (Ours)	36.67%	57.62%	61.31%	70.11%	46.73%	77.92%	83.62%	90.49%	10.39%	25.90%	40.65%	80.76%	29.75%	51.98%	64.57%	80.81%
Target density ratios of the generators  $d_{G}$  are chosen from  $\{10\%, 20\%, 30\%, 50\% \}$ . Please see Appendix A for more experiment details.
Baseline methods and two practical strategies. We use STATIC and SDST (section 4) as our baselines. Note that in real-world application scenarios, it is not practical to perform a grid search for a good  $d_{D}$  as in section 4. Hence, we propose two practical strategies to define the constant discriminator density for these baseline methods: (1) balance strategy, where we set the density of the discriminator  $d_{D}$  the same as the density of the generator  $d_{G}$ ; (2) strong strategy, where we set the density of the discriminator as large as possible, i.e.,  $d_{D} = d_{\mathrm{max}}$ . For SDST methods, we test both  $\clubsuit$  SDST-SET and  $\clubsuit$  SDST-RigL. For a fair comparison,  $d_{\mathrm{max}}$  is set to be  $100\%$  and  $50\%$  for the relaxed setting and the strict setting, respectively.
For ADAPT, we use the  $\bullet$ RigL version, which grows connections of the generators and discriminators based on gradient magnitude. The gradient information enables two components to react promptly according to the change of each other. Different from the value used in subsection 5.3, we control the balance ratio in the range [0.45, 0.55] unless otherwise mentioned to have a slightly stronger discriminator, potentially avoiding training collapse. More details can be found in Appendix B.
6.4 Experiment results
We show the experiment results in Table 1 and Table 2 for the relaxed setting and the strict setting, respectively. We also present the training FLOPs normalized by the dense counterpart for the relaxed setting in Table 3. We defer the results for the strict setting to Appendix F Table 12. We show FID for the CIFAR-10 test set, Inception scores, and comparison with post-hoc pruning baseline in Appendix E. We also show ADAPT BR evolution in Appendix D. We summarize our findings below.
The strong strategy and the balance strategy for baselines. Generally, using the strong strategy has some advantages over the balance strategy. Such an observation is most prominent in the CIFAR-10 dataset. For the cases where the balance strategy is better, e.g., SNGAN on the STL-10 dataset, our explanation is that the size difference between generators and discriminators is more significant. Hence, the degree of unbalance is more severe and leads to more detrimental effects.
Comparison of  $\bullet$  RigL and  $\clubsuit$  SET for SDST. We found that  $\bullet$  RigL has an advantage over  $\clubsuit$  SET when dealing with more sparse generators. Our hypothesis is that gradient information can effectively guide the generator to identify the most crucial connections in such cases. However, this advantage is not as apparent for more dense generators.
$\mathbf{ADAPT}_{\mathrm{relax}}$  achieves a good trade-off between performance and computational cost. Experiments show that  $\mathbf{ADAPT}_{\mathrm{relax}}$  shows promising performance by being best for 13 out of 16 cases. The advantage of  $\mathbf{ADAPT}_{\mathrm{relax}}$  is most prominent for the most difficult case, i.e.,  $d_{G} = 10\%$ . Specifically, it
shows 2.3 and 7.1 FID improvements over the second-best methods for the SNGAN on the CIFAR-10 and the STL-10, respectively. Moreover, compared to the competitive baseline methods that use the strong strategy, i.e.,  $\clubsuit$  SDST-Strong-RigL and  $\clubsuit$  SDST-Strong-SET, ADAPT $_{\text{relax}}$  shows great computational cost reduction. For example, it outperforms  $\clubsuit$  SDST-Strong-RigL on BigGAN (CIFAR-10) with much-reduced training FLOPs (10.39% v.s. 83.97%).
ADAPT<sub>strict</sub> shows stable and superior performance. Similar to ADAPT<sub>relax</sub>, we notice that ADAPT<sub>strict</sub> also delivers promising results compared to baselines, even with a further constraint on the discriminator. More precisely, among all the cases, ADAPT<sub>strict</sub> ranks top 2 for all cases, with 13 cases being the best. Moreover, ADAPT<sub>strict</sub> again shows comparable or better performance compared to SDST-Strong-RigL with reduced computational cost.
A more interesting observation is that  $\mathrm{ADAPT}_{\mathrm{strict}}$  sometimes outperforms  $\mathrm{ADAPT}_{\mathrm{relax}}$ . We speculate that this phenomenon occurs because changes in density may result in a larger influence on the GAN balance during training compared to DST. Hence, the strict version, whose discriminator density range is smaller, may offer a more consistent performance.
In this paper, we investigate the use of DST for GANs and find that solely applying DST to the generator does not necessarily enhance the performance of sparse GANs. To address this, we introduce BR to examine the degree of unbalance between the sparse generators and discriminators. We find that applying DST to the generator only benefits the training when the discriminator is comparatively stronger. Additionally, we propose ADAPT, which can dynamically adjust the discriminator at both the parameter and density levels. Our approach shows promising results, and we hope it can aid researchers in better comprehending the interplay between the two components of GAN training and motivate further exploration of sparse training for GANs. However, we must note that we have not yet evaluated our methods on the latest GAN architectures due to computational constraints.
This work utilizes resources supported by the National Science Foundation's Major Research Instrumentation program, grant No.1725729 [39], as well as the University of Illinois at Urbana-Champaign. This work is supported in part by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project (No.HZQSWS-KCCYB-2022046); University Development Fund UDF01001491 from the Chinese University of Hong Kong, Shenzhen; Guangdong Key Lab on the Mathematical Foundation of Artificial Intelligence, Department of Science and Technology of Guangdong Province. Prof. NH is in part supported by Air Force Office of Scientific Research (AFOSR) grant FA9550-21-1-0411.
[94] X. Zhou, W. Zhang, H. Xu, and T. Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3599-3608, 2021.
[95] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[96] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems, 30, 2017.
Overview of the Appendix
The Appendix is organized as follows:
- Appendix A introduces the general experimental setup.
- Appendix B introduces the details of dynamic sparse training.
- Appendix C shows detailed algorithms, i.e., DDA, ADAPTrelax, and ADAPTstrict.
- Appendix D shows the BR evolution during training for ADAPT.
- Appendix E shows additional results, including IS and FID of test sets of the main paper.
- Appendix F shows detailed FLOPs comparisons of sparse training methods.
A Experimental setup
In this section, we explain the training details used in our experiments. Our code is mainly based on the original code of ITOP [54] and GAN ticket [9].
A.1 Architecture details
We use ResNet-32 [27] for the CIFAR-10 dataset and ResNet-48 for the STL-10 dataset. See Table 4 and Table 5 for detailed architectures. We apply spectral normalization for all fully-connected layers and convolutional layers of the discriminators.
For BigGAN architecture, we use the implementation used in DiffAugment [93].<sup>3</sup>
A.2 Datasets
We use the training set of CIFAR-10, the unlabeled partition of STL-10, and the training set of TinyImageNet for GAN training. Training images are resized to  $32 \times 32$ ,  $48 \times 48$ ,  $64 \times 64$  for CIFAR-10, STL-10, and TinyImageNet datasets, respectively. Augmentation methods for both datasets are random horizontal flip and per-channel normalization.
A.3 Training hyperparameters
SNGAN on the CIFAR-10 and STL-10 datasets. We use a learning rate of  $2 \times 10^{-4}$  for both generators and discriminators. The discriminator is updated five times for every generator update. We adopt Adam optimizer with  $\beta_{1} = 0$  and  $\beta_{2} = 0.9$ . The batch size of the discriminator and the generator is set to 64 and 128, respectively. Hinge loss is used following [7, 9]. We use exponential moving average (EMA) [89] with  $\beta = 0.999$ . The generator is trained for a total of 100k iterations.
BigGAN on the CIFAR-10 dataset. We use a learning rate of  $2 \times 10^{-4}$  for both generators and discriminators. The discriminator is updated four times for every generator update. We adopt Adam optimizer with  $\beta_{1} = 0$  and  $\beta_{2} = 0.999$ . The batch size of both the discriminator and the generator is set to 50. Hinge loss is used following [7, 87]. We use EMA with  $\beta = 0.9999$ . The generator is trained for a total of 200k iterations.
BigGAN on the TinyImageNet dataset. We use DiffAug [93] to augment the input. The learning rate of the discriminator and the generator are set to  $4 \times 10^{-4}$  and  $1 \times 10^{-4}$ , respectively. The discriminator is updated one time for every generator update. We adopt Adam optimizer with  $\beta_{1} = 0$  and  $\beta_{2} = 0.999$ . The batch size of both the discriminator and the generator is set to 256. Hinge loss is used following [7, 87]. We use EMA with  $\beta = 0.9999$ . The generator is trained for a total of 200k iterations.
A.4 Evaluation metric
SNGAN on the CIFAR-10 and the STL-10 datasets. We compute Fréchet inception distance (FID) and Inception score (IS) for 50k generated images every 5000 iterations. Best FID and IS are
<sup>3</sup>https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-cifar.
reported. For the CIFAR-10 dataset, we report both FID for the training set and test set, whereas, for the STL-10 dataset, we report the FID of the unlabeled partition.
BigGAN on the CIFAR-10 and the TinyImageNet dataset. We compute Fréchet inception distance (FID) and Inception score (IS) for 10k generated images every 5000 iterations. Best FID and IS are reported.
B Dynamic sparse training details
B.1 How the generator performs DST
In this section, we explain how the generator performs DST below. Note that the generator performs the same for SDST and ADAPT.
Sparsity distribution at initialization. Following RigL and ITOP [16, 54], only parameters of fully connected and convolutional layers will be pruned. At initialization, we use the commonly adopted Erdős-Rényi-Kernel (ERK) strategy [16, 14, 54] to allocate higher sparsity to larger layers. Specifically, the sparsity of convolutional layers  $l$  is scaled with  $1 - \frac{n^{l-1} + n^l + w^l + h^l}{n^{l-1}n^lw^lh^l}$ , where  $n^l$  denotes the number of channels of layer  $l$  while  $w^l$  and  $h^l$  are the widths and the height of the corresponding kernel in that layer. For fully connected layers, Erdős-Rényi (ER) strategy is used, where the sparsity is scaled with  $1 - \frac{n^{l-1} + n^l}{n^{l-1}n^l}$ .
Update schedule. The update schedule controls how many connections are adjusted per DST operation. It can be specified by the number of training iterations between sparse connectivity updates  $\Delta T_{G}$ , the initial fraction of connections adjusted  $\gamma$ , and decaying schedule  $f_{\mathrm{decay}}(\gamma, T)$  for  $\gamma$ .
Drop and grow. After  $\Delta T_{G}$  training iterations, we update the mask  $m_{G}$  by dropping/pruning  $f_{\mathrm{decay}}(\gamma ,T)|\pmb{\theta}_{G}|d_{G}$  number of connections with the lowest magnitude, where  $|\pmb{\theta}_G|$ ,  $d_G$  are the number of parameters and target density for the generator,  $f_{\mathrm{decay}}(\gamma ,T)$  is the update schedule. Right after the connection drop, we regrow the same amount of connections.
For the growing criterion, we test both random growth  $\spadesuit$  SET [62, 54] and gradient-based growth  $\bullet$  RigL [16]. Concretely, gradient-based methods find newly-activated connections  $\theta$  with the highest gradient magnitude  $|\frac{\partial\mathcal{L}}{\partial\theta} |$  while random-based methods explore connections in a random fashion. All the newly-activated connections are set to 0. One thing that should be noticed is that while previous works consider layer-wise connections drop and growth, we grow and drop connections globally as it grants more flexibility to the DST method.
EMA for sparse GAN. EMA [89] is well-known for its ability to alleviate the non-convergence of GAN. We also implement EMA for sparse GAN training. Specifically, we zero out the moving average of dropped weights whenever there is a mask change.
B.2 DST hyperparameters for the generator
We use the same hyper-parameters for SDST and ADAPT. The initial  $\gamma$  is set to 0.5, and we use a cosine annealing function  $f_{\mathrm{decay}}$  following RigL and ITOP.
SNGAN on the CIFAR-10 and the STL-10 datasets. The connection update frequency of the generator  $\Delta T_{G}$  is set to 500 and 1000 for the CIFAR-10 dataset and STL-10 dataset, respectively.
BigGAN on the CIFAR-10 and the TinyImageNet dataset. The connection update frequency of the generator  $\Delta T_{G}$  is set to be 1000.
B.3 Density dynamic adjust (DDA) hyper-parameters
In this section, we provide hyper-parameters used in subsection 5.3. We set  $d_{D} = 2000$ ,  $\Delta T_{D} = 0.05$ ,  $[B_{-}, B_{+}] = [0.5, 0.65]$ . Time-averaged BR over 1000 iterations is used as the indicator.
B.4 DST hyperparameters for the discriminator in ADAPT
We use a constant BR interval  $[B_{-}, B_{+}] = [0.45, 0.55]$  for SNGAN experiments and BigGAN on the CIFAR-10 dataset. We set the BR interval  $[B_{-}, B_{+}] = [0.3, 0.4]$  for BigGAN on the TinyImageNet
Table 4: ResNet architecture for CIFAR-10.
(a) Generator	(b) Discriminator
z ∈ R128 ∼ N(0, I)	image x ∈ [−1, 1]32×32×3
dense, 4 × 4 × 256	ResBlock down 128
ResBlock up 256	ResBlock down 128
ResBlock up 256	ResBlock down 128
ResBlock up 256	ResBlock down 128
BN, ReLU, 3 × 3 conv, Tanh	ReLU 0.1
Global sum pooling
dense → 1
Table 5: ResNet architecture for STL-10.
(a) Generator	(b) Discriminator
z ∈ R128 ∼ N(0, I)	image x ∈ [−1, 1]48×48×3
dense, 6 × 6 × 512	ResBlock down 64
ResBlock up 256	ResBlock down 128
ResBlock up 128	ResBlock down 256
ResBlock up 64	ResBlock down 512
BN, ReLU, 3 × 3 conv, Tanh	ResBlock down 1024
ReLU 0.1
Global avg pooling
dense → 1