SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Masked Feature Map
Figure 5: Illustration of the generated images, feature maps, and the corresponding masks. The feature maps are extracted from the discriminator of StyleGAN-V2 [36] trained on FFHQ [35], and the red region denotes the artifacts. By masking feature maps of the discrimination, it is possible to remove discriminative local regions/features (e.g., unrealistic regions) that are important for the sample's distribution, shirting the sample to a new distribution.
Masked Feature Map
similar to their pseudo samples, it is probable that the discriminator does not discriminate the distribution difference and is detected as retardation. Formally, given  $m$  samples at time  $t$ , we can define our heuristic Retardation metric,  $\mathcal{R}_t$ , as
$$
\mathcal {R} _ {t} = \frac {1}{m} \sum_ {i \in U _ {t}} \frac {\bar {\mathbf {F}} _ {i} ^ {(d) , t} \cdot \mathbf {F} _ {i} ^ {(d) , t}}{| \bar {\mathbf {F}} _ {i} ^ {(d) , t} | | \mathbf {F} _ {i} ^ {(d) , t} |} \tag {4}
$$
where  $U_{t}$  is the index of samples,  $\bar{F}_{i}^{(d),t}$  is the output of the  $d$ -th layer of the discriminator taking original sample as input,  $\bar{F}_{i}^{(d),t}$  is that of a pseudo sample where its image region or feature maps in the discriminator is masked. If  $\mathcal{R}_t$  is larger than a predefined threshold, the discriminator doesn't detect differences between original and pseudo samples and is detected as retardation. The pseudocode is given in the Appendix.
Dataset. We evaluate the performance of our method on six widely used datasets, including AFHQ-Cat [12], AFHQ-Dog [12], AFHQ-Wild [12], FFHQ [35], and LSUN-Church [90] with  $256 \times 256$  resolutions and CIFAR-10 [39] with  $32 \times 32$  resolutions. We elaborate on these datasets and implementation details in the Appendix.
Evaluation Metrics. Following prior works [50, 33, 30], we use two evaluation metrics, including Fréchet Inception Distance (FID) [25] and Inception Score (IS) [67], to evaluate the quality of generated results. The number of testing samples is set to  $50\mathrm{k}$  in our experiments.
Baselines. Following the previous studies [33, 30, 50], we integrate our method with StyleGAN-V2 [36], StyleGAN-V3 [34], and BigGAN [6]. We compare our method with state-of-the-art methods that improve discriminators via data augmentation, including ADA [33] and APA [30]. We also compare with GANs using regularization: LC-Reg [82], zCR [93], InsGen [88], Adaptive Dropout [33], AdaptiveMix [50], MEE [48] and DynamicD [87]. Besides StyleGAN, we compare with other generative models: DDPM [26], ImageBART [16], PGGAN [32] and LDM [65].
5.2 Comparison with State-of-the-art Methods
Main results. To show the superiority of the proposed method, we compare the performance of our approach with state-of-the-art methods on FFHQ. As shown in Table 1, our method achieves the best performance in terms of FID score on FFHQ. Recent methods [87, 50, 88] improve GANs by data augmentation, e.g., ADA [33] and APA [30]. By further combining with APA [30], FID score of our method reduces from 3.299 to 3.075, significantly surpassing the other approaches. In addition to the quantitative analysis, Fig. 6 and Fig. 7 show qualitative results of our method. StyleGAN-V2 introduces noticeable artifacts, while our method generates images with much better quality. This is because our discriminator effectively learns the time-varying distributions of generated samples,
Table 1: Our method compared to the generative models and the variants of StyleGAN-V2 [36] on FFHQ [35]. Comparisons are taken from [50, 87, 65].
Methods	FID↓
ImageBART [16] (NIPS'21)	9.57
U-Net GAN (+aug) [73] (CVPR'20)	10.9(7.6)
LDM [65] (CVPR'22)	4.98
StyleGAN-V2 [36] (CVPR'20)	3.862
StyleGAN-V2 (Re-Impl.)	3.810
DiffAugment [95] (NIPS'20)	4.840
Adaptive Dropout [33] (NIPS'20)	4.160
LC-Reg [82] (CVPR'21)	3.933
DynamicD-Decreasing [87] (NIPS'22)	3.740
ADA [33] (NIPS'20)	3.880
ADA (Re-Impl.) [33] (NIPS'20)	3.713
APA [30] (NIPS'21)	3.678
MEE (Re-Impl.) [48] (AAAI'23)	3.626
AdaptiveMix [50] (CVPR'23)	3.623
AdaptiveMix (+APA) [50] (CVPR'23)	3.609
DynamicD-Increasing [87] (NIPS'22)	3.530
zCR [93] (ICLR'20)	3.450
InsGen (+ADA) [88] (NIPS'21)	3.310
Ours: DMD	3.299
Ours: DMD (+APA)	3.075
Table 2: FID  $\downarrow$  of our method compared with other regularizations for GAN training on AFHQ-V2 [12]. Comparison results are taken from [87, 50].
Methods	Cat	Dog	Wild
StyleGAN-V2 [36] (CVPR'20)	7.924	26.310	3.957
LC-Reg [82] (CVPR'21)	6.699	-	-
ADA [33] (NIPS'20)	6.360	18.930	3.800
DynamicD (+ADA) [87] (NIPS'22)	5.410	16.000	3.340
AdaptiveMix [50] (CVPR'23)	4.477	-	-
MEE (Re-Impl.) [48] (AAAI'23)	4.453	-	-
Ours: DMD (+ADA)	5.223	12.764	3.261
StyleGAN-V2 (+APA)	4.645	13.913	2.919
Ours: DMD (+APA)	4.261	11.784	2.625
Table 3: Our method compared to other state-of-the-art generative models on LSUN-Church [90]. We re-implement StyleGAN-V2, and the other results are from LDM [65].
Methods	FID↓
DDPM [26] (NIPS'20)	7.89
ImageBART [16] (NIPS'21)	7.32
PGGAN [32] (ICLR'18)	6.42
StyleGAN-V2 (Re-Impl.) [36] (CVPR'20)	4.29
LDM [65] (CVPR'22)	4.02
DynamicD [87] (NIPS'22)	3.87
Ours: DMD	3.06
Figure 6: Qualitative comparison results of our method and StyleGAN-V2 on FFHQ dataset.
which improves the training of the generator. (More experimental results and details are provided in the Appendix.)
We also perform comparison experiments on dataset AFHQ-V2. As listed in Table 2, our method outperforms state-of-the-art approaches, boosting the performance on all three sub-datasets, showing the effectiveness of our method in improving the training of the generator. DynamicD [87] empirically increases/decreases the capacity of the discriminator for GAN training, which is most related to our work. However, our method outperforms DynamicD by a large margin. For example, FID of DynamicD is 16.000 on the dog dataset, and our method achieves better FID (11.784). We also evaluate our method on StyleGAN-V3 [34] and BigGAN [6] respectively in Table 4 and Table 5, which show consistent improvements by adding our proposed method into these two baselines.
Table 4: Our method over StyleGAN-V3 [34] on AFHQ-Cat [12].
FID ↓	IS ↑
StyleGAN-V3R	7.616	1.881
StyleGAN-V3R w/ Ours(DMD)	6.864	1.915
StyleGAN-V3T	5.850	1.916
StyleGAN-V3T w/ Ours(DMD)	4.921	1.969
Table 5: Our method over BigGAN [6] on CIFAR-10 [39].
FID ↓
Unconditional BigGAN	16.040
Unconditional BigGAN w/ MEE [48]	13.860
Unconditional BigGAN w/ Ours(DMD)	10.540
Performance on the large-scale training dataset. We evaluate the performance of our method trained on a large-scale dataset LSUN-Church [90] shown in Table 3. LSUN-Church contains 120k images and can be augmented to 240k with flipping. Diffusion models and auto-regressive models achieve impressive performance by training on large-scale datasets. We compare our method with representative diffusion models: LDM [65], DDPM [26], and ImageBART [16]. LDM outperforms StyleGAN-V2 slightly (4.02 vs. 4.29 FID). However, our method significantly reduces the FID of StyleGAN-V2 from 4.29 to 3.06 FID, achieving the best performance, compared with the other generative models.
5.3 Ablation Study and Analysis
Ablation studies. Since our method is plug-and-play, which is easy to be integrated into existing methods. We adopt StyleGAN-V2 as the baseline and evaluate the improvement of integration with our method. As shown in Table 6, our method improves StyleGAN-V2 on AFHQ-V2 and LSUN datasets. For example, our method enables better performance of StyleGAN-V2, where FID is improved by  $25.8\%$  on AFHQ-Cat. One of our contributions is to automatically switch the discriminator between mask and non-mask training, according to discriminator retardation detection. We evaluate the effectiveness of this automatic scheme. We test the performance of the baseline with different fixed intervals for the transition between the mask and non-mask training. As shown in Table 7, fixed interval improves the performance of StyleGAN-V2, indicating that dynamically adding/removing masks in the discriminator helps the training of GANs. Nevertheless, the proposed method can achieve the best performance, since discriminator retardation detection automatically detects when the discriminator slows down.
Table 6: The ablation study of our method on AFHQ-V2 [12] and LSUN-Church [90], where the baseline is the StyleGAN-V2 [36] and Ours is StyleGAN-V2+DMD.
AFHQ-Cat	AFHQ-Dog	AFHQ-Wild	LSUN-Church
FID ↓	IS ↑	FID ↓	IS ↑	FID ↓	IS ↑	FID ↓	IS ↑
Baseline	7.924	1.890	26.310	9.000	3.957	5.567	4.292	2.589
Ours	5.879(-25.8%)	1.988(+5.2%)	21.240(-19.3%)	9.698(+7.8%)	3.471(-12.3%)	5.647(+1.4%)	3.061(-28.7%)	2.792 (+7.8%)
Figure 7: The generated samples of the proposed method on (a) AFHQ-Cat, (b) AFHQ-Dog, (c) AFHQ-Wild, (d) FFHQ, and (e) LSUN-Church. All training data are in a resolution of  $256 \times 256$ .
Figure 8: Probe of hyper-parameters of the proposed method on FFHQ-70k. We discuss the depth, ratio, and probabilities for masking. Probabilities of Dyna. Mask control the times in the masking stage. In each iteration of mask training, masking the discriminator in the 5-th layer with 0.3 ratios can achieve the best result.
Hyper-parameters and masking strategies. We then search the hyper-parameters for the proposed method. As shown in Fig. 8, we separately investigate the efficacy of the depth for masking, the mask ratio, and the probabilities to trigger the mask. Accordingly, the best performance appears with the 5-th layer, 0.3 masking ratios, and masking discriminator every time (i.e., probability=1) in the mask training stage. Meanwhile, we investigate different masking strategies, including Vanilla Dropout, Input Masking, Dynamic Head, and Vanilla Dropout. We detail the corresponding implementations in the Appendix. As shown in Table 8, the proposed method can achieve the best performance among all the cases. Note that the empirical study shows that directly applying dropout will incur unstable training and hurt the capacity of the discriminator, leading to worse FID. This shows the advantage of our method, which automatically switches the discriminator at a time interval.
Table 7: FID  $\downarrow$  of our method compared to the StyleGAN-V2 with fixed interval mask on FFHQ-70k.
Inerval (kimg)	FID
4	3.823
8	3.362
16	3.365
24	3.391
DMD (Ours)	3.299
Table 8: FID ↓ of and the proposed method on AFHQ-V2 [12] and other masking strategies for StyleGAN-V2[36]. Note that Vanilla Dropout [76] is based on a 0.5 drop rate.
Cat	Dog	Wild
StyleGAN-V2 [36]	7.924	26.310	3.957
Attention-based Mask	7.294	23.782	3.800
Input Masking	6.809	22.054	3.515
Dynamic Head	7.388	23.050	3.736
Vanilla Dropout [76]	9.294	28.099	5.056
DMD (Ours)	5.879	21.240	3.471
In this paper, we propose a novel method named DMD for training GANs from a new perspective, i.e., online continuous learning. Our study shows that the distribution of generated samples is time-varying, while this problem has been underexplored. This makes the discriminator often largely rely on historically generated data, instead of learning new knowledge from the incoming generated samples, degrading generation performance. We propose to force the discriminator to fast learn and adapt to incoming generated samples. We propose a simple yet effective method that automatically detects whether the discriminator slows down learning, and adjusts the discriminator by dynamically imposing or removing masks of the discriminator per time interval. Experimental results show our method improves the learning of the discriminator on temporally-vary distribution, boosting the guidance of training the generator, achieving the best performance than state-of-the-art methods.
Limitations: Theoretical studies can make this work more comprehensive; however, we have not explored it in the paper, since it is beyond the scope of this study. Moreover, while the proposed method can effectively improve the training of the CNN-based GANs models, combining our method with Transformer-based ones is left to be investigated in the future.
Broader Impact: Our method can be used for various applications such as producing training data and creating photorealistic images. On the other hand, like other generative models, our method can be misused for the application of Deepfake [1], where fake content is synthesized to deceive and mislead people, leading to a negative social impact. Nevertheless, many researchers have considered this problem while exploring fake content detection and media forensics techniques. In addition, we believe there would be regulations on fake content generation, such as forcing synthesized content to be injected with identifications that indicate it to be fake.
We thank Hasan Abed Al Kader Hammoud for his valuable constructive suggestions and help. This work was supported by the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
[94] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. In ICLR, 2017.
[95] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. NeurIPS, 33:7559-7570, 2020.
[96] Kaiyang Zhou, Han Zhang, Wenqi Xie, Yunpeng Chen, Hongtao Liu, and Tong Zhang. Dynamic: High-fidelity image generation via dynamic capacity adjustment of discriminator. In CVPR, pages 15451-15460, 2021.
[97] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223-2232, 2017.
[98] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In CVPR, pages 5802-5810, 2019.
A Introduction
In this appendix, we first introduce the implementation details of our Dynamic Mask Discriminator (DMD), and then provide more technical details of the proposed DMD in Sec. B. In Sec. C.1 and C.2, we elaborate on the additional details of datasets and experiment settings. In Sec. C.3, we provide additional experimental results. We also report the error bars of our method in Sec. D.
In this section, we provide the implementation details of our proposed method. We have utilized the PyTorch [59] public platform to conduct our experiments. Our models were trained on a workstation equipped with 8 NVIDIA Tesla V100 GPUs with 32 GB memory, a CPU of 2.8GHz, and 512GB RAM.
B.1 More details of the proposed method.
Our method is designed to automatically detect the retardation of the discriminator and force it to fast learn the new knowledge given time-varying distributions of generated samples. To show the detailed strategy of the proposed DMD method, the pseudo-code is given in Algorithm 1. In our main paper, we have demonstrated the effectiveness of our proposed DMD method by integrating it with image generation techniques on StyleGAN-V2 [36].
Since the retardation of learning new knowledge can potentially be addressed by augmenting the input stream or the outcome of the discriminator, we also extend another two schemes to achieve dynamic discriminator adjustment. Besides the proposed dynamic feature masking scheme, we can dynamically mask the input and output of the discriminator when it is detected as retardation. The corresponding details are shown summarized in Algorithm. 2 and Algorithm. 3.
B.2 More details of model parameter difference
We detail model parameter difference which is used in the main paper Figure 3(a) for investigating the fixed discriminator of StyleGAN-V2. Model parameter difference metric is to measure the differences of the discriminator model weights between two adjacent training steps:
$$
\mathbf {d} _ {t _ {j}} = \left\| W _ {t _ {j}} ^ {d} - W _ {t _ {j - 1}} ^ {d} \right\| ^ {2} \tag {5}
$$
where  $W_{t_j}^d$  is the parameter weight of  $d$ -th layer of the discriminator during  $t_j$  training step, and  $\| \cdot \| ^2$  is 12 norm. A smaller  $\mathbf{d}_{t_j}$  indicates that the parameter weights of  $d$ -th layer at  $t_j$  training step are more similar to that at  $t_{j-1}$  training step, i.e., parameter weights are updated slowly. In other words, given new arrival generated samples with new distributions, a smaller  $\mathbf{d}_{t_j}$  indicates that the discriminator slows down the learning of new knowledge to some extent. In the main paper, we calculate  $\mathbf{d}_{t_j}$  in the full connection layer of the discriminator.
C Additional Details on Experiments
C.1 Datasets and Experimental Settings
AFHQ-V2 [12] consists of 3 independent sub-datasets, which include around 5,000 closeups of cat, dog, and wildlife faces, respectively (denoted as AFHQ-Cat, AFHQ-Dog, and AFHQ-Wild). We utilized a high-quality Lanczos filter [42] to resize all images to a resolution of  $256 \times 256$ . We then conducted experiments on three sub-datasets while setting StyleGAN-V2 [36] as the baseline model. We maintain consistency with ADA [33], by using identical network architectures [36], weight demodulation [36], style mixing regularization [35], path length regularization, lazy regularization [36], equalized learning rate for all trainable parameters [32], non-saturating logistic loss [19] with  $R_{1}$  regularization [54], and the Adam optimizer [37].
Algorithm 1 Dynamic Mask Discriminator
Require:   
Generator  $\mathcal{G}_{\theta^t}$  ; Current Discriminator  $D_{\phi}(\cdot)$  ; Non-Masked Discriminator  $\mathcal{D}_{\phi}(\cdot)$  ; Dynamically   
Masked Discriminator  $\bar{D}_{\phi}(\cdot)$  ; Training Step t;   
 $d$  -th Layer of Discriminator  $\mathbf{F}^{(d),t}$  ; Dynamic Mask  $\mathbf{m}_d^t$  .   
 $d$  -th Masking Layer of Discriminator  $\bar{\mathbf{F}}^{(d),t}$  ; Predefined Threshold  $\lambda$  ; Retardation Metric  $R_{t}$    
Set  $U_{t}$  Containing m Samples for Calculating Retardation Metric;   
The number of training steps  $n_t$  ; The number of images per training step  $n_s$  .   
Ensure: Initialize  $R_{t}\gets 0$  and  $t\gets 1$  ; Random  $\mathbf{m}_d^t$  .   
1: while  $\theta$  has not converged do.   
2: for  $t = 1$  to  $n_t$  do   
3: if  $R_{t} > \lambda$  then   
4:  $\mathbf{M}^t\gets \mathbf{m}_d^t;\mathbf{M}_T\gets \mathbf{M}^t;D_\phi (\cdot)\gets \bar{D}_\phi (\cdot)$    
5: else   
6:  $\mathbf{M}^t\gets \mathrm{vector}(1);\mathbf{M}_T\gets \mathbf{m}_d^{t + 1};D_\phi (\cdot)\gets \mathcal{D}_\phi (\cdot)$    
7: end if   
8: for  $s = 1$  to  $n_s$  do   
9:  $\bar{\mathbf{F}}^{(d),t}\gets \bar{\mathbf{F}}^{(d),t}\odot \mathbf{M}^t$    
10:  $L_{\phi (t)}\gets -\mathbb{E}_{I,t}[log(D_{\phi}(I))] - \mathbb{E}_{z\sim p_z,t}[log(1 - D_{\phi}(\mathcal{G}(z,\theta^t)))]$    
11:  $\theta_s\gets \mathrm{Adam}(\frac{\partial\phi(t)}{\partial\theta_{s - 1}})$    
12: end for   
13:  $\mathcal{R}_t = \frac{1}{m}\sum_{i\in U_t}\frac{\bar{\mathbf{F}}_i^{(d),t}\cdot\bar{\mathbf{F}}_i^{(d),t}}{|\bar{\mathbf{F}}_i^{(d),t}||\bar{\mathbf{F}}_i^{(d),t}|}$    
14: end for   
15: end while   
16: Return  $\theta$
FFHQ [35] comprises 70,000 images of human faces, which we used for training after downscaling them to a resolution of  $256 \times 256$ . In this case, we set StyleGAN-V2 [36] as the baseline and used the same settings as those for AFHQ-V2.
LSUN-Church [90] includes 126,000 images of outdoor church. We downscale them to  $256 \times 256$  as the training data. In this case, we also set StyleGAN-V2 [36] as the baseline and used the same settings as those for FFHQ.
In accordance with previous studies [33, 30, 50], we have integrated our proposed method with StyleGAN-V2 [36]. In order to assess the effectiveness of our method, we have compared it with state-of-the-art methods that improve discriminators through data augmentation, including ADA [33] and APA [30]. We have also compared our method with GANs that utilize regularization techniques, such as LC-Reg [82], zCR [93], InsGen [88], Adaptive Dropout [33], AdaptiveMix [50], MEE [48], and DynamicD [87].
C.3 Additional Experimental Results
Comparing our method with other masking strategies. Besides the masking strategies discussed in the paper, another masking strategy, namely Continually-Changed-mask-ratio Discriminator (CCD) is explored here, which replaces our dynamic discriminator adjustment module by gradually increasing the mask ratio from 0.1 to 0.9 or decreasing the mask ratio from 0.9 to 0.1 over time. We evaluate the CCD with StyleGAN-V2 on the AFHQ-Cat  $(256\times 256$  pixels) dataset. Compared with the StyleGAN-V2 (7.924 FID), CCD with StyleGAN-V2 achieves 8.441 FID. However, our method (5.879 FID) still outperforms CCD, since CCD increases instabilities in GAN training. More specifically, the process of GAN training is to play a min-max two-player game between the generator and discriminator, which is more unstable than typical classification problems. Compared with our method, CCD causes the discriminator's discrimination ability to be more
Algorithm 2 Dynamic Mask Discriminator Assert in Input (Input Masking)
Require: Generator  $\mathcal{G}_{\theta^t}$  ; Discriminator  $\mathcal{D}_{\phi}(\cdot)$  ; Training Step  $t$  ; Dynamic Mask  $\mathbf{m}^t$  Predefined Threshold  $\lambda$  ; Retardation Metric  $R_{t}$  Set  $U_{t}$  Containing m Samples for Calculating Retardation Metric;  $d$  -th Layer of Discriminator  $\mathbf{F}^{(d),t}$  ; After Masking Input  $\bar{\mathbf{F}}^{(d),t}$  The number of training steps  $n_t$  ; The number of images per training step  $n_s$    
Ensure: Initialize  $R_{t}\gets 0$  and  $t\gets 1$  ; Random  $\mathbf{m}^t$  .   
1: while  $\theta$  has not converged do   
2: for  $t = 1$  to  $n_t$  do   
3: if  $R_{t} > \lambda$  then   
4:  $\mathbf{M}^t\leftarrow \mathbf{m}^t;\mathbf{M}_T\leftarrow \mathbf{M}^t$    
5: else   
6:  $\mathbf{M}^t\gets$  vector(1);  $\mathbf{M}_T\gets \mathbf{m}^{t + 1}$    
7: end if   
8: for  $s = 1$  to  $n_s$  do   
9:  $\bar{I}\gets I\odot \mathbf{M}^t;\bar{\bar{I}}\gets \mathcal{G}(z,\theta^t)\odot \mathbf{M}^t$    
10:  $L_{\phi (t)}\gets -\mathbb{E}_{\bar{I},t}[log(D(\bar{I}))] - \mathbb{E}_{z\sim p_z,t}[log(1 - D(\bar{\bar{I}})]$    
11:  $\theta_{s}\gets \mathrm{Adam}(\frac{\partial\phi(t)}{\partial\theta_{s - 1}})$    
12: end for   
13:  $\mathcal{R}_t = \frac{1}{m}\sum_{i\in U_t}\frac{\bar{\mathbf{F}}_i^{(d),t}\cdot\mathbf{F}_i^{(d),t}}{| \bar{\mathbf{F}}_i^{(d),t}| | \mathbf{F}_i^{(d),t}|}$    
14: end for   
15: end while   
16: Return  $\theta$
frequently changed and uncertain, making it more difficult for the generators to fool the discriminator. The deteriorated performance of the generator would further negatively affects the training of the discriminator. Instead, when the improvement of the discriminator slows down, our method switches from non-mask training to mask training, otherwise maintains masking/non-masking at a time interval, which provides more stable training than CCD.
Experiments on AFHQ-Cat with  $512 \times 512$  pixels. We conduct experiments on the high-resolution AFHQ-Cat dataset, where the resolution of an image is  $512 \times 512$  pixels. By integrating our method with StyleGAN-V2, our method reduces the FID of StyleGAN-V2 by a margin of  $18.77\%$  (from 4.3160 to 3.5061), effectively improving the performance of StyleGAN-V2 on higher-resolution images.
Experiments on ImageNet-1000. We integrate the proposed DMD with an intermediate checkpoint of the BigGAN [6] model which has been trained for 100,000 iterations. We then further trained models from 100,000 to 107,500 iterations on ImageNet-1000 at  $128 \times 128$  resolution using two NVIDIA V100 GPUs. Compared with the original BigGAN that achieves the FID of 12.213 at the 107,500 iteration, our method (i.e., DMD) improves the training of BigGAN (FID of 11.212), outperforming the original BigGAN by  $8.196\%$ . This not only shows that our method benefits the training of BigGAN on large-scale datasets, but also demonstrates the flexibility and compatibility of our method in combination with GAN models such as pre-trained GAN models.