Dynamically Masked Discriminator for GANs
Authors: Wentian Zhang
Haozhe Liu
Bing Li
Jinheng Xie
Yawen Huang
Yuexiang Li
Yefeng Zheng
Bernard Ghanem
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative model
Generative Adversarial Network

Abstract: Training Generative Adversarial Networks (GANs) remains a challenging problem. The discriminator trains the generator by learning the distribution of real/generated data. However, the distribution of generated data changes throughout the training process, which is difficult for the discriminator to learn. In this paper, we propose a novel method for GANs from the viewpoint of online continual learning. We observe that the discriminator model, trained on historically generated data, often slows down its adaptation to the changes in the new arrival generated data, which accordingly decreases the quality of generated results. By treating the generated data in training as a stream, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data. Therefore, we can explicitly enforce the discriminator to learn new knowledge fast. Particularly, we propose a new discriminator, which automatically detects its retardation and then dynamically masks its features, such that the discriminator can adaptively learn the temporally-vary distribution of generated data. Experimental results show our method outperforms the state-of-the-art approaches.

Dynamically Masked Discriminator for GANs
Wentian Zhang 1:2t Yuexiang Li2400 Yefeng Zheng? Bernard Ghanem
Haozhe Liu
Bing Li
Jinheng Xie
Yawen Huang?
Al Initiative, King Abdullah University of Science and Technology
Jarvis Research Center, Tencent YouTu Lab 3 National University of Singapore
Life Sciences Institute, Guangxi Medical University
bing. 1 @kaust edu.sa, sa; leeyuexianger 6omcom
Abstract
Training Generative Adversarial Networks (GANs) remains challenging prob- lem. The discriminator trains the generator by learning the distribution real/generated data. However, the distribution of generated data changes through- out the training process, which is difficult for the discriminator to learn. In this paper, propose novel method for GANs from the viewpoint of online con- tinual learning. We observe that the discriminator model, trained on historically generated data, often slows down its adaptation to the changes in the new arrival generated data, which accordingly decreases the quality of generated results. By treating the generated data in training the discriminator slows down the learning of new knowledge in generated data. Therefore, we can explicitly enforce the discriminator learn new knowledge fast. Particularly, we propose a new discriminator, which automatically detects its retardation and then dynamically masks its features, such that the discriminator can adaptively learn the temporally-vary distribution of generated data. Experi- mental results show our method outperforms the state-of-the approaches.
stream, propose detect whether
Introduction
Generative Adversarial Networks (GANs) [19, 84, 20, 5, 72, 70, 71] have shown the remarkable performance various applications [89, 85, 81, 3, 44], which have attracted intensive interests. The main components of GANs are the generator and the discriminator, where the generator is trained to generate realistic samples, and the discriminator learns to distinguish real and generated samples. However, it is well-known that GANs are difficult to train [33, 30, 21J.
Many efforts have been devoted to alleviating training difficulties for GANs. Some studies balance the generator and discriminator from the side of network architecture, such as DCGAN [61], PG- GAN [32]. Style-Gann GANs [35, 36, 34], and BigGAN [6]. Besides, some studies [2, 94, 53, 86] address this challenge from the learning objective, e.g., WGAN [2], EBGAN [94], LSGAN[53] and realness- GAN [86]. These methods can further stabilize training, especially coupled with the Lipschitz reg- ularization [56, 21] and conditional information [55]. Recent works [87, 33, 30, 95, 93, 50, 48, 82] propose to improve GANs on the discriminator side. For example, different data augmentation strategies [33, 30, 95] are proposed to strengthen the discriminator for limited data regime. Reg- ularizations are applied to stabilize the training of discriminator [93, 50, 82] and combat the mode
Equal Contribution. Haozhe worked on this project before joining KAUST.
& Corresponding Author
Project Leader
Code is available at bttps::///// /ttp//gtttttt/ com www.//nt//////eec.ooeooaeeee.oe Went waw.aaaanaa//aaaaaa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training Steps
Training Steps
(a) StyleGAN-V2
(b) StyleGAN-V2 Ours
Figure Illustration of the advantage of our method, where the first and second rows show generated images and feature maps taken from discriminator layer, respectively. (a) StyleGAN-V. introduces similar artifacts into generated images, despite the increased training steps. Our method is simple yet effectively improves the training of GANs, boosting the quality of generated samples for StyleGAN-V2 V2 in (b). By fast learning incoming generated samples, the proposed discriminator captures artifacts in the local regions, better guiding the training of the generator. The averaging cosine similarity between the current and previous feature maps is 0.8353 for StyleGAN-Vi V2 and 0.4232 for ours, which indicates that our method enforces the attentive regions of the discriminator to be more different than the baseline, better adapting to time-varying distribution of generated data.
collapse [48]. These methods show that the discriminator is very crucial to the training of GANs. Nevertheless, stably training GANs remains challenging problem /33,31, 93].
In this paper, we rethink the difficulty of training GANs from the discriminator side. The discrim- inator guides the training of the generator via learning the distributions of real/generated samples. However, since it has been observed that the characteristics of generated samples vary during the whole training process [87], a question arises: how challenging is il for the discriminator to learn from such time-varying generated samples? Our study shows that the distribution of generated samples dynamically changes during training, posing significant challenges to the discriminator. In particular, the discriminator in existing work (e.g., [6, 36, 57, 50]) does not explicitly consider an online task that addresses time-varying distributions of generated samples but is presented for a typical classification task. We refer to such a discriminator as fixed. We find that the fixed discrim- inator often improperly relies on local features that are important to historically generated samples to distinguish incoming ones, although the distributions of these generated samples are different. As result, the discriminator insufficiently learns the distribution of incoming generated samples, leading to subpar guidance to the generator (see Fig. 1).
We propose novel method for training GANs, from the perspective of online continual learning. Online continual learning has attracted increasing attention [52, 14, 40]. Different from offline con- tinual learning focusing on the catastrophic forgetting problem [51, 38, 60], recent online continual learning methods [17, 7, 47, 22] pointed out that data with time-vary distribution requires learning methods to have the ability of fast learning and quick adaption. Motivated by these methods, we propose to detect whether the discriminator slows down the learning of new knowledge in gener- ated data by treating generated samples throughout training as stream of data. We then explicitly force the discriminator to learn new knowledge fast. To this end, we propose simple yet effective scheme to detect the retardation of the discriminator and force it to learn fast. Instead of designing sophisticated network architecture, we dynamically mask discriminator features when the discrim- inator is predicted to be retarded, such that the discriminator adaptively learns the temporally-war distribution of generated data.
Our contributions are summarized as follows.
From the perspective of online continual learning, we propose novel approach that effec- tively improves the training of GANs by explicitly considering the time-varying distribution of generated samples.
The proposed discriminator fast learns and adapts to the varying distribution of the generated samples via dynamically masking discriminator features, improving the training of GANs.

---

Extensive experimental results show that our method generates high-quality images, out- performing state-of-the-ar GAN approaches. Besides, the performance of our method surpasses advanced diffusion models.
2 Related Works
Generative adversarial networks. GANs have attracted increasing attention from researchers
[19, 84, 20, 5, 72]. By learning min-max objective, GANs generate samples from given data dis- tribution, which can serve for various applications, e.g., image editing [18, 28, 75, 3, 27, 97, 83, 43] and text-conditional image synthesis [69, 31, 78, 45, 64, 98]. As fundamental technology, GANs coupled with the diffusion model (DM) [65, 26, 58, 62, 66] and auto-regressive model (AR) [63, 91, 10, 9] indirectly turn on the light of AIGC [8]. Compared with the other emergent gen- erative models, GANs have the advantages of rapid inference [31], generating fixed objects [31, 65] and controllable latent distribution [35, 36, 34, 4. 23, 75]. Recently, as an answer ball against DM and AR, GigaGAN [31], StyleGAN-T[69) and GALIP [78] adapted GANs to large image-text datasets, demonstrating that GANs can be another viable solution for zero-shot text-to-image gen- eration. However, as "No Pain, No Gain", the payments of GANs are also high, i.e., GANs suffer from unstable training and mode collapse. As a directed result, GANs are difficult to be equipped with some standard modules, such as dropout [76] and masking operator [10, 24], which strengthen the model generalization via introducing randomeess into the training process, but further decrease the training stability. Existing studies [92, 74, 13, 96] illustrated that GANs learn from the stream, where the training distribution varies in the different time steps. Without any regularization, GANs may harmfully learn a bias historical data and neglect the current data. Our goal in this paper is to downgrade the potential payments of GANs and integrate the masking operator into GANs to overcome overfitting historical data.
Regularization on discriminators. As a model helping the generator to align with real distribu- tion, the discriminator plays pivotal role to train GANs. Various methods have been proposed to improve GANs by redesigning discriminators [2, 21, 50, 93, 87, 30, 48, 96, 32, 57]. WGAN [2] employs the discriminator to measure the synthetic and natural samples with Wasserstein distance. Since WGAN requires Lipschitz continuity to ensure model convergence, weight clipping [2] and gradient penalty [2]] were proposed to enhance the discriminators. Such technologies are then in- herited by the family of StyleGAN [35, 36, 34J. Thanks to the advanced architecture [6, 36, 32], several regularization technologies, including ADA [33], APA [30], LC-Reg [82]. DigfAuggent [95], Con-Reg [93], AdaptiveMix [50], DAG [80]. and MEE [48] were proposed to train GANs with low-data regime [50, 30, 33, 82, 80, 95] or combat mode collapse [48]. In addition to exploring training data, some studies redesigned the discriminator via proposing extra tasks [11, 79, 29, 88] and implementing large-scale pre-trained models [68, 41]. In this paper, the proposed method does not require additional training data, which can work as a plug-and-pla module for existing discrim- inators. Compared with the previous regularization focusing on data augmentation, the proposed method investigates the capacity of the discriminator from a new perspective, i.e., 'augmenting' the architecture to quickly adapt the current (new) knowledge. Note that the most related study this paper is DynamicD [87], which dynamically adapts the capacity of the discriminator to the training data. However, DynamicD suffers from divergent solutions on different data regimes. In contrast, our method can help to re-adjust the attentions of discriminators and obtain consistent improvement against varying regimes.
3 Pilot Study
We first conduct study to investigate the distribution shifts of generated samples over time during training. We then demonstrate the challenges posed by such a time-varying distribution to the fixed discriminator. We adopt representative GAN, i.e., StyleGAN-V2 [36], for the study, due to its advanced network architectures, training strategies, and impressive results.
Analysis on generated distribution over time. We train StyleGAN-V2 on widely- dataset FFHQ [35], to study the distribution changes of its generated samples. In particular, we collect 5k generated samples at each time interval during training and then analyze the distribution of gener- ated samples per time interval. The distribution is estimated with Fréchet Inception Distance (FID) [25]. More specifically, FID measures the distribution similarity between real and fake samples by
3

---

1209
2016
4836
6451
9878
11289
12499
20764
Training progress (kimg)
0.00105
Mann
Variance
0.16
0.00103
0014
0000001
10 Mim mm M
M
0.00099
0.100
5000
10000
15000
20000
25000
5000
10000
5500
20000
25000
Triining progres (kimg)
Training (kimg)
Distribation of Generated Images from StyloGAN-V2 (Mean & Variance)
Figure 2: Illustration of time-varying distributions of generated samples, where we trace the training process of StyleGAN-V2 [36] on FFHQ [35]. The mean and variance of generated 5k samples' features are computed per time interval, showing the generated distributions are dynamic and time- time-varyine varying during training, as the generator evolves. kimg refers to the number of images (measured in thousand) trained so far.
calculating the mean and covariance of real/fake samples. Similarly, we first extract the feature of a generated sample by an Inception model [77], and then represent the distribution of generated samples per time interval by the mean and variance of these samples' features.
We observe that the generated distributions undergo dynamical and complex changes over time (see Fig. 2), as the generator evolves during training. For example, the generated samples are not even independently and identically distributed (i.i.d) across the training progress in Fig. 2. Such time- varying distribution inevitably poses significant challenges to the discriminator, since this has not been explicitly considered in the discriminator typically.
AFHQ-Cat
150
130
110
Training progress (king)
400
800
1200
1600
Training progress (kimg)
1200
1400
1600
(n) Model parameter difference
(b) Feature maps at different training steps
Figure 3: We trace the training StyleGAN-V22 V2's discriminator. The curve in (a) shows the model parameter difference of the discriminator between & and ti-1 in the training progress, In (b), the attentive regions of the discriminator are almost fixed at training steps when the discriminator slows down its model parameter updating, where each feature map is represented by the feature space of the discriminator trained at a time step.
Adaption ability of fixed discriminator. Learning the distributions of generated samples vital to training the generator. We investigate whether the fixed discriminator, which does not explic- itly consider distribution changes, can effectively learn the time-vryying distributions of generated samples. We argue that the discriminator needs to adaptively adjust its model weights and rely on different local features for discriminating generated samples at different steps. Suppose the genera- tor synthesizes unrealistic hair for human faces at time interval [4 12). and synthesizes plausible hair but introduces artifacts in cheek regions at [Lg, 44] (see [1209, 4838) and [6451, 11289] kimgs in Fig. 2). It is expected that the discriminator pays less attention to hair regions and more attention to check regions [t3, ta], compared to [f, t2]. so as to fast distinguish fake samples. Accordingly, the parameters of the discriminator model are expected to be updated, such that the discriminator adapts to the distribution changes of generated samples for [ti, t2] to [ta, t4].
Here we investigate the discriminator of StyleGAN-V. V2 since it is representative fixed discrimina- tor. We find the parameter differences of the discriminator are usually small between two adjacent steps, (see Fig. 3 (a)), indicating the discriminator slows down learning. Moreover, we find the discriminator model trained different times pays attention to almost fixed local regions given an image, as indicated by the feature map extracted by the layers of the discriminator (see Fig. (b)). As a result, the discriminator relies on old knowledge learned from historical data, which insufficient to learn incoming generated samples under dynamic distribution shift.

---

22
Time
Discriminator
Non-Mast
Non-Mask Training
Non-Mask Training
Non-Mask Training
Training
Random Mask
Random
Mask Training
Mask Training
Mask
Training
Training
(4) Training Process
Training
Units for training
Input
Similarity
Partial Units for training
Input
Training
between Non-Mask Training Training
(c) Retordation Detection
Figure 4: The pipeline of the proposed method. Our method, named Dynamically Masked Dis- criminator (DMD), automatically adjusts the discriminator via dynamically marking the discrimi- nator. When DMD detects that the discriminator slows down learning, DMD dynamically assigns masks removes masks features of the discriminator per time interval, forcing the discriminator to learn new knowledge and preventing it from relying on old knowledge from historical samples.
4 Methodology
4.1 Problem Formulation
The discriminator learns the distribution of real and generated samples, which is vital to training the generator. Nevertheless, as studied in Sec. 3, the distribution of generated samples varies and drifts over time during training, posing challenges to existing discriminators (e.g.. [36, 50J). By treating the generated samples of the generator across training as a stream, we innovatively formulate learning the distribution of generated samples an online continual learning problem. Motivated by recent advances in online learning [17, 7, 47], our target is enable discriminator quickly adapt incoming generated data.
Since the generator is trained via min-max game with the discriminator, the distribution of gener- ated samples is significantly complex and unpredictable. Hence, it is non-trivial to accurately predict distribution changes. In other words, it is difficult for the discriminator to predict when it needs to be rapidly adjusted for the new distribution.
Overview. We address the problem by considering two questions: (1) when does the discriminator slow down the learning from incoming generated samples? (2) how to force the discriminator to fast learn new knowledge? By exploring the two aspects, we propose method named Dynamically Masked Discriminator (DMD) for GANs, as shown in Fig. 4.
4.2 Dynamically Masked Discriminator
Different from existing GANs methods [36, 50, 33, 48], we propose to automatically adjust the dis- criminator during training towards the time-varying distribution generated samples. Instead of designing complex network architectures, we propose two key modules which can be easily inte- grated into existing discriminators. The two modules are (1) discriminator retardation detection and (2) dynamic discriminator adjustment. The first module automatically determines when the discrim- inator slows down learning. Here, slow down referred to as the discriminator largely relies on old knowledge learned from historical data, given incoming generated samples with different distribu- tions. The second module adjusts the discriminator for fast learning via dynamically assigning or removing masks certain interval.
With the two proposed modules, we improve an existing discriminator into dynamic one at the time axis. That is, the discriminator in our method has two states, i.e., mask and non-mask training. D denotes the original discriminator without masks, and π) denotes the adjusted discriminator with masks. During training, we dynamically switch D and D for guiding the training of the generator
5

---

according to discriminator retardation detection. Let T (Dit) describe the probability to use D at time t. The probability of D is thus formed as 1 Λ (D)(t). Let {7*) denote a stream consisting of samples generated by the generator G, where I' = [22, 0*) is generated sample from vector at time during training, and denotes parameters of the generator at t. We formulate our training of
GANs as follows:
Lgz= == -E [m(DI))oo((((((()))
Lei := - Beppant -
+ (1- (00 (1)
π (og -1
(DI))
(2)
- -Egxps, [[1 (D(t)logg - - ()) + -10
where I is real sample and øf is the parameter of the discriminator at time t.
Dynamic discriminator adjustment. When the discriminator is detected to be retarded (i.e., slow down learning), we adjust the discriminator to force it to learn fast. In this paper, we mainly focus on the issue that the discriminator largely relies on old knowledge learned from historical data and can- not rapidly adapt to incoming generated samples. To address this issue, one possible solution is to design complex network architectures. Instead, we propose to dynamically switch the discriminator from mask/non-mask states to non-mas mon-mask/mask at time interval. We argue that such dynamic mask- ing at time intervals can break the original dependency of the discriminator on some local features that are important to distinguish historical samples, inspired by [15, 24, 46, 49, 76]. For example, by masking a feature map that is fed to a layer of the discriminator, we can control the layer of the discriminator and enforce it to pay attention to other regions of a generated image.
In this paper, we force the non-mask discriminator (i.e., original one) D with parameter d* to be converted into the mask state Das by masking its feature map or input. In particular, given the d-th layer of a discriminator, we propose to mask its input Fdd in a time interval (4g, (fte). Let m denote mask in the t-th training step with the same size F(d-1), We can dynamically mask the discriminator by masking the input features of its d-th layer:
p(d). L(F(d-1).. m/-1)) L(F(d-D) o m²-1); LE (1,,4+
where ((d). is the output of d-th layer of a discriminator by masking, @ denotes Hadamard product, T. is the convolutional operator. Different from the traditional dropout, the mask does not contin- uously change. Instead, this mask is fixed for a period of training steps, i.e., (f,,l Switching the discriminator from mask non-mask is simply removing all masks in the discriminator. Such dynamical switching encourages the discriminator to pay attention to various local regions/features over time, making our discriminator better adapt to the vimevnarying generated distribution and lead- ing to better guidance of training the generator, compared with StyleGAN-V2 (see Fig. 1).
Diseriminator retardation detection. This module is to detect whether the discriminator slows down the learning, i.e., the discriminator largely relics on old knowledge from historical data to distinguish future generated samples with new distributions. Given the current time step 4,, an ideal solution is to use anture-generated samples with different distributions at time (lg, Ly+e] to evaluate the discriminator. However, these future-generate samples are unavailable at the current time step. Moreover, is non-trivial to predict the distribution of snture-generated samples.
Instead, we propose to construct pseudo sample that possibly belongs to new distribution to detect the Retardation of the discriminator. Without loss generality, we assume the distribution changes of generated samples correspond to the changes in local regions/features of these samples. For example, some hair regions are unrealistic in a generated sample at time t,, and become realistic at ty. as the generator evolves in Fig. 2. By randomly masking a generated sample its intermediate features in the discriminator, it is possible to remove discriminative local regions/features that are important for the sample's 's distribution, shirting the sample to a new distribution. For example, if the right face region of the man is removed/masked in Fig. 5, the remaining regions become more realistic (i.e., belongs to new distribution).
Therefore, given generated sample j for a distribution, we construct pseudo sample that pos- sibly belongs to another distribution by randomly masking local image regions or intermediate fea- tures of the discriminator. For generated samples at time if the discriminator determines they are

---

Generated Image
Feature Map
Masked Feature Map
Generated Image
Feature Map
Masked
Feature Map
Figure 5: Illustration of the generated images, feature maps, and the corresponding masks. The feature maps extracted from the discriminator of StyleGAN-V2 [36] trained on FFHQ [35], and the red region denotes the artifacts. By masking feature maps of the discrimination, it is possible remove discriminative local regions/features (e.g.. unrealistic regions) that are important for the sample's s distribution, shirting the sample to a new distribution.
similar their pseudo samples, it probable that the discriminator does not discriminate the distri- bution difference and is detected retardation. Formally, given samples time t, define our heuristic Retardation metric, R2 as
Fidl F ild (d)
R.
(4)
BEU:
where U, is the index of samples, ( is the output of the d-th layer of the discriminator taking original sample as input, F((d)it is that pseudo sample where its image region or feature maps in the discriminator masked. If R2 is larger than predefined threshold, the discriminator doesn't 't detect differences between original and pseudo samples and detected as retardation. The pseudo- code is given in the Appendix.
5
Experiments
5.1 Experimental Setup
Dataset. We evaluate the performance of our method on six widely used datasets, including AFHQ- Cat [12], AFHQ-Dog [12], AFHQ-WA Wild [12], FFHQ [35], and LSUN-Church [90] with 256 x 256 resolutions and CIFAR-IO [39] with 32 x 32 resolutions. We elaborate these datasets and implementation details in the Appendix.
Evaluation Metrics. Following prior works [50, 33, 30]. we use two evaluation metrics, includ- ing Fréchet Inception Distance (FID) [25] and Inception Score (IS) [67]. to evaluate the quality of generated results. The number of testing samples set to 50k in experiments.
Baselines. Following the previous studies [33, 30, 50], we integrate our method with StyleGAN-V2 [36], StyleGAN-V3 [34], and BigGAN [6]. We compare our method with state-of-the-art methods that improve discriminators via data augmentation, including ADA [33] and APA [30]. We also
compare with GANs using regularization: LC-Reg [82]. zCR [93], InsGen [88], Adaptive Dropout
[33], AdaptiveMix [50], MEE [48] and DynamicD [87]. Besides StyleGAN, we compare with other
generative models: DDPM [26], ImageBART [16], PGGAN [32] and LDM [65].
5.2 Comparison with State-of-the-art Methods
Main results. To show the superiority of the proposed method, we compare the performance of our approach with of-te-of-the-eee art methods on FFHQ. As shown in Table 1, our method achieves the best performance in terms FID score on FFHQ Recent methods [87, 50, 88] improve GANs by data augmentation, e.g., ADA [33] and APA [30]. By further combining with APA [30]. FID score of our method reduces from 3.299 to 3.075, significantly surpassing other approaches. addition to the quantitative analysis, Fig. 6 and Fig. 7 show qualitative results of our method. StyleGAN-V2 introduces noticeable artifacts, while our method generates images with much better quality. This is because our discriminator effectively learns the time-varying distributions of generated samples,
7

---

Table I: Our method compared to the generative models and the variants of StyleGAN-V2 [36] on FFHQ [35]. Comparisons are taken from
Table 2: FID + of our method compared with other regularizations for GAN training on AFHQ-V2 [12]. Comparison results are taken from 50].
[50, 87, 65].
Methods
FID
Methods
Cat
Dog
Wild
StyleGAN- V2 [36] (CVPR'20)
7.924 26,310 3.957
ImageBART [16] (NIPS'21)
9.57
I.C. I.C-Reg [82] (CVPR'21)
6.699
U-Net GAN (+aug) [73] (CVPR'2)) 20)
10.9(7.6)
ADA [33] (NIPS 20)
6.360 18,930
3.800
LDM [65] (CVPR'22 22)
4.98
DynamicD (+ADA) [87] (NIPS'22)
5.410 16,000 3.340
StyleGAN-V2 [36] (CVPR'20)
3.862
AdaptiveMix [50] (CVPR 23)
4.477
StyleGAN-V2 V2 (Re-Imp.)
3.810
MEE (Re-Impl. [48] (AAAI'23)
4,453
Ours: DMD (+ADA)
5.223 12.764 3.261
DiffAugment [95] (NIPS'20)
4.840
StyleGAN V2 (+APA)
4.645 13.913 2.919
Adaptive Dropout [33] (NIPS'20)
4.160
Ours: DMD (+APA)
4.261 11.784 2.625
I.C-Reg [82] (CVPR'21)
3.933
DynamieD-Decreassing [87] (NIPS'22)
3,740
Table 3: Our method compared to other
ADA [33] (NIPS'20) 20)
3.880
otate-of-the-art generative models on LSUN- Church [90]. We re-implement StyleGAN- V2, and the other results are from LDM [65].
ADA (Re-Impl.) [33] (NIPS'20)
3.713
APA [30] (NIPS'21)
3.678
MEE (Re-Impl.) [48] (AAAI'23)
3.626
AdaptiveMix [50] (CVPR'233 23)
3.623
Methods
FID
AdaptiveMix (+APA) [50] (CVPR'23)
3.609
DDPM [26] (NIPS' 20)
7.89
DynamicD-Innreennn [87] (NIPS' 22)
3,530
ImageBART [16] (NIPS'21)
7.32
zCR [93] (ICLR'20)
3,450
PGGAN [32] (ICLR'18)
6.42
InsGen (+ADA) [88] (NIPS'21)
3,310
StyleGAN V2 (Re-Impl.) [36] (CVPR'20)
4.29
LDM [65] (CVPR 22)
4.02
Ours: DMD
3.299
DynamicD [87] (NIPS'22)
3.87
Ours: DMD (+APA)
3.075
Ours: DMD
3.06
StyleGhN- -V2
Ours
Figure 6: Qualitative comparison results of our method and StyleGAN-V2 on FFHQ dataset.
which improves the training of the generator. (More experimental results and details are provided in the Appendix.)
We also perform comparison experiments dataset AFHQ-V2. As listed in Table 2, our method outperforms state-of-the-arr approaches, boosting the performance on all three sub-datasets, show- ing the effectiveness of our method in improving the training of the generator. DynamieD [87] empirically increases/decreases the capacity of the discriminator for GAN training, which is most related to our work. However, our method outperforms DynamicD by a large margin. For example, FID of DynamicD is 16.000 on the dog dataset, and our method achieves better FID (11.784). We also evaluate our method on StyleGAN-Vi V3 [34] and BigGAN [6] respectively in Table and Table 5, which show consistent improvements by adding our proposed method into these two baselines.
Table 4: Our method over StyleGAN-V3 [34] on Table 5: Our method AFHQ-Cat [12].
BigGAN [6]
CIFAR-10 [39].
FID 4.
IS f
FID
StyleGAN-V3R StyleGAN-V3R w/ Ours(DMD)
7.616
1.881
Unconditional BigGAN
16.040
6.864
1.915
Unconditional BigGAN w/ MEE [48]
13.860
StyleGAN-V3T
5.850
1.916
Unconditional BigGAN w/ (Drs(DMD)
10.540
StyleGAN-V3T w/ Ours(DMD)
4.921
1.969
8

---

Performance on the large-scale training dataset. We evaluate the performance of our method trained on a large-scale dataset LSUN-Churee [90] shown in Table images and can be augmented to 240k with flipping. Diffusion models and auto-regressive models achieve impressive performance by training on large-scale datasets. We compare our method with representative diffusion models: LDM [65], DDPM [26], and ImageBART [16]. LDM outperforms StyleGAN-V2 slightly (4.02 vs. 4.29 FID). However, our method significantly reduces the FID of StyleGAN-V2 from 4.29 to 3.06 FID, achieving the best performance, compared with the other generative models.
CUUN-Church contains 120k
5.3 Ablation Study and Analysis
Ablation studies. Since our method plug-and-play, which is easy to be integrated into existing methods. We adopt StyleGAN-V2 as the baseline and evaluate the improvement of integration with our method. As shown in Table 6, our method improves StyleGAN-V2 on AFHQ-V2 and LSUN datasets. For example, FID is improved by 25.8% on AFHQ-Cat. One of our contributions is to automatically switch the discriminator between mask and non-mask training, according discriminator retardation detection. We evaluate the effectiveness of this automatic scheme. We test the performance the baseline with different fixed intervals for the transition between the mask and non-mask training. As shown in Table 7, fixed interval improves the performance of StyleGAN-Ve indicating that dynamically adding/removing masks in the discriminator helps the training of GANs. Nevertheless, the proposed method can achieve the best performance, since discriminator retardation detection automatically detects when the discriminator slows down.
method enables better performance of StyleGAN-V2, where
Table 6: The ablation study of our method on AFHQ-V2 V2 [12] and LSUN-Churde [90]. where the baseline is the StyleGAN- V2 [36] and Ours StyDeGAN-V++DMD.
AFHQ-Cat
AFHQ- Dog
AFHQ. Wild
SUUN-Church
FID
1S
FID
1S
FID
IS
FID
IS
7,924
1890
26.310
9.000
5.567
Ours
-5.8%)
1.988(45.2%) ())
11.2400-19.39
97.8%
3.41(-123%)
5.647(14%) 3.061 7% 2.792 (+7.8%)
(a) AFHQ-Cat
(b) AFHQ-Dog
(c) AFHO
(d) FFHQ
(e) LUUN-Church
Figure 7: The generated samples the proposed method on (a) AFHQ-Cat, (b) AFHQ-Dog, (c)
AFHQ-Wild, (d) FFHQ, and (e) Chhhhhh All training data are in resolution of 256 256.
Euseinee
Using Dyna Mask
16862
3.8
3.8
3.8
3.6
3,6
3.5
3.635
3.5
3-415
3.29
Layer Discriminator
Ration Dyns Mask
Probabiliries of Dym Mask (6)
(4)
Figure 8: Probe of hyper-parameters of the proposed method on FFHQ-70K. We discuss the depth, ratio, and probabilities for masking. Probabilities of Dyna. Mask control the times in the masking stage. In each iteration of mask training, masking the discriminator in the 5-th layer with 0.3 ratios can achieve the best result.

---

Hyper-parameters and masking strategies. We then search the hyper-pas parameters for the proposed method. As shown Fig. 8, we separately investigate the efficacy of the depth for masking, the mask ratio, and the probabilities to trigger the mask. Accordingly, the hest performance appears with the 5-th layer, 0.3 masking ratios, and masking discriminator every time (i.e., probability=1) in the mask training stage. Meanwhile, we investigate different masking strategies, including Vanilla Dropout, Input Masking, Dynamic Head, and Vanilla Dropout. We detail the corresponding imple- mentations in the Appendix. As shown in Table 8, the proposed method can achieve the best perfor- mance among all the cases. Note that the empirical study shows that directly applying dropout will incur unstable training and hurt the capacity of the discriminator, leading the advantage of our method, which automatically switches the discriminator at a time interval.
FID. This shows
Table 7: FID J of our method compared to the StyleGAN- V2 with fixed interval mask on FFHQ-70K.
Table 8: FID & of and the proposed method AFHQ-V2 V2 [12] and other masking strategies for SoyleGAN-V2330) Note that Vanilla
Dropout [76] is based 0.5 drop rate.
Cat
Dog
Wild
Inverval (kimg)
FID
StylcGAN-V2 [36]
7.924
26.310
3,957
4
3.823
Attention- Mask
7.294
23.782
3,800
3.362
Input Masking
6.809
22.054
3.515
16
3.365
Dynamic Head
7.388
23.050
3.736
24
3.391
Vanilla Dropout [76]
9.294
28.099
5.056
DMD (Ours)
3.299
DMD (Ours)
5.879
21.240
3.471
Conclusion
In this paper, we propose novel method named DMD for training GANs from new perspective, ie., online continuous learning. Our study shows that the distribution of generated samples is time- varying. while this problem has been underexplored. This makes the discriminator often largely rely on historically generated data, instead of learning new knowledge from the incoming generated samples, degrading generation performance. We propose to force the discriminator to fast learn and adapt to incoming generated samples. We propose a simple yet effective method that automatically detects whether the discriminator slows down learning, and adjusts the discriminator by dynamically imposing removing masks of the discriminator per time interval. Experimental results show our method improves the learning of the discriminator on temporally-vary distribution, boosting the guidance of training the generator, achieving the best performance than state-of of-the-art methods.
Limitations: Theoretical studies can make this work more comprehensive; however, we have not ex- plored it in the paper, since it is beyond the scope of this study. Moreover, while the proposed method can effectively improve the training of the CNNN-based GANs models, combining our method with Transformer- bhaddd ones is left to be investigated in the future.
Our method can be used for various applications such as producing training data and creating photorealistic images. On the other hand, like other generative models, our method can be misused for the application of Deepfake [1], where fake content synthesized deceive and mislead people, leading to negative social impact. Nevertheless, many researchers have considered this problem while exploring fake content detection and media forensics techniques. addition, we believe there would be regulations on fake content generation, such as forcing synthesized content be injected with identifications that indicate be fake.
Broader Impact:
Acknowledgments and Disclosure of Funding
We thank Hasan Abed Al Kader Hammoud for his valuable constructive suggestions and help. This work was supported by the SDAIA-KAUSE Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUS) AI).
10

---

References
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. ICML, pages 214-223, PMLR, 2017.
[3] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. arXiv preprint arXiv:2005.07727, 2020.
[4] Amit Bermano, Rinon Gal, Yuval Alaluf, Ron Mokady, Yotam Nitzan, Omer Tov, Oren Patashnik, and Daniel Cohen-Or. State-of-the-art in the architecture, methods and applications of stylegan. In Computer Graphics Forum, volume 41, pages 591-611. Wiley Online Library, 2022.
[5] Sam Bond-Tayle Rond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative mod- elling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregres- sive models. IEEE TPAMI, 2021.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICLR, 2019.
[7] Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distri- bution shifts: An empirical study with visual data. In ICCV, pages 8281-8290, 2021.
[8] Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aige): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.0422 2023.
[9] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: text-to- generation via masked generative transformers. arXiv preprint arXiv:2301.007044 2023.
[10] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked
generative image transformer. In CVPR, pages 11315-11325, 2022.
[11] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In CVPR, pages 12154-1216,, 2019.
[12] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wio Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, pages 8188-8197 2020.
[13] Yulai Cong, Miaoyun Zhao, Jianqiao Li. Sijia Wang, and Lawrence Carin, Gan memory with no forgetting. NeurIPS, 33: 33:16481-164999 2020.
[14] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gre- gory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. JEEE TPAMI, 44(7):3366-335, 2021.
[15] Terrance Deieies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1708. 04555,, 2017.
[16] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidi- rectional context with multinomial diffusion for autoregressive image synthesis. NeurIPS, 34:3518-3532, 2021.
[17] Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Ham- moud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new paradigm. arXiv preprint arXiv:2302.01047 2023.
[18] Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image properties. In ICCV, pages 5744-5753, 2019.

---

[19] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27:2672- 2680, 2014.
[20] Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Transactions on Knowledge and Data Engineering, 2021.
[21] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. NeurlPS, 30, 2017.
[22] Hasan Abed Al Kader Hammoud, Ameya Prabhu, Ser-Nam Lim, Philip HS Torr, Adel Bibi, and Bernard Ghanem. Rapid adaptation in online continual learning: Are we evaluating right? arXiv preprint 2rXtv:2305.09275. 2023.
[23] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. NeurIPS, 1199841-98500 2020.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NeurIPS, 30, 2017.
[26] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
33:6840-6851, 2020.
[27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pages 1125-1134, 2017.
[28] Ali Jahanian*, Lucy Chai*, and Phillip Isola. On the "steerability" of generative adversarial networks. In ICLR, 2020.
[29] Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint arXiv:2/03.09742,22 2021.
[30] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: adaptive pseudo aug- mentation for gan training with limited data. NeurlPS, 34:21655-21667, 2021.
[31] Minguk Kang, Jun- Yan Zhu, Richard Zhang. Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. arXiv preprint arXiv:2303.05511, 2023.
[32] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018
[33] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurIPS, 33:12104-12114 2020.
[34] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrIPS, 34:852-863, 2021.
[35] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[36] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 81 31108119, 19, 2020.
[37] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint wrXiv:1412.6900 2014.
[38] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Sarnskaaa-awiika,,..a et al. Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 11((1))3521-3355 2017.
12

---

[39] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmmun Diseases, 1(4). 2009.
[40] Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Benjamin Van Roy. Continual learning as computationally constrained reinforcement learning. arXiv preprint arXiv:2307.04345 2023.
[41] Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Ensembling off-the-shelf models for gan training. In CVPR, pages 10651-10662. 2022.
[42] Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 1950.
[43] Bing Li, Yuanlue Zhu, Yitong Wang, Chia- Wen Lin, Bernard Ghanem, and Linlin Shen. Ani- gan: Style-guided generative adversarial networks for unsupervised anime face generation. IEEE Transactions on Multimedia, 2021.
[44] Jianan Li, Jimei Yang, Jiamming Zhang, Chang Liu, Christina Wang. and Tingfa Xu. Attribute- conditioned layout gan for automatic graphic design. IEEE TVCG, 27(10):4039-400 2020.
[45] Jiadong Liang, Wenjie Pei, and Feng Lu. Cpgan: Content-parsing generative adversarial net- works for text-to-image synthesis. In ECCV, pages 491-508. Springer, 2020.
[46] Kongming Liang, Kai Han, Xiuli Li, Xiaoqing Cheng, Yiming Li, Yizhou Wang, and Yizhou Yu. commetry-enhaneeee attention network for acute ischemic infarct segmentation with non- contrast ct images. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VII 24, pages 432-441 Springer, 2021.
[47] Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on worl-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[48] Haozhe Liu, Bing Li, Haoqian Wu, Hanbang Liang, Yawen Huang, Yuexiang Li, Bernard Ghanem, and Yefeng Zheng. Combating mode collapse in gans via manifold entropy estima- tion. arXiv preprint arXiv:2208. 12055, 2022.
[49] Haozhe Liu, Haoqian Wu, Weicheng Xie, Feng Liu, and Linlin Shen. Group-wise inhibition based feature regularization for robust classification. In ICCV, pages 478-486, 2021.
[50] Haozhe Liu, Wentian Zhang, Bing Li, Haoqian Wu, Nanjun He, Yawen Huang, Yuexiang Li, Bernard Ghanem, and Yefeng Zheng. Adaptivemix: Improving gan training via feature space shrinkage. In CVPR, 2023.
[51] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. NeurlPS, 30, 2017.
[52] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputiing 469:28-55 51, 2022.
[53] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In CVPR, pages 2794-2802, 2017.
[54] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018
[55] Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXiv preprint arXiv: 1784, 2014.
[56] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normaliza- tion for generative adversarial networks. arXiv preprint arXiv:1802.05990 2018.
13

---

[57] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normaliza- tion for generative adversarial networks. In ICLR, 2018.
[58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ila Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021.
[59] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[60] Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K. Dokania, Philip H.S. Torr, Ser- Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1698-3707, June 2023.
[61] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint urXivv:51 15111.5111.,4,,.111 2015.
[62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[63] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-8831. PMLR, 2021.
[64] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, pages 1060-1069. PMLR, 2016.
[65] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[66] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photore- alistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479- 36494, 2022.
[67] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS, 2016.
[68] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. NeurlPS, 14:179 2021.
[69] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Un- locking the power of gans for fast large-scale text-to-image synthesis. arXiv preprint 0rXiv:2301.09515, 2023.
[70] Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self- neural networks for dynamic reinforcement learning and planning in non-stationary environ- ments. Institut für Informatik, Technische Universität München. Technical Report FKI-126, 90, 1990.
[71] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the International Conference on Simulation of Adaptive Behav- ior: From Animals to Animats, pages 222-227, 1991.
[72] Jürgen Schmidhuber. Generative adversarial networks special cases of artificial curios- ity (1990) and also closely related to predictability minimization (1991). Neural Networks, 127:58-66 66, 2020.
[73] Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. u-net based discriminator for generative adversarial networks. In CVPR, pages 8207-8216, 2020.
14

---

[74] Ari Seff. Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative ad adversarial nets. arXiv preprint arXiv:/755.08395, 2017.
[75] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, pages 9243-9252, 2020.
[76] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhut- dinov, Dropout: simple way to prevent neural networks from overfitting. The journal of machine learming research, 15(1):1929-1958 2014.
[77] Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, pages 1818,282,, 2016.
[78] Ming Tao, Bing-Kun Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis. arXiv preprint arXiv:2301.12959, 2023.
[79] Ngoe-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, and Ngai-Man Man Cheung. Self-super gan: Analysis and improvement with multi-class minimax game. NeurlPS, 32, 2019.
[80] Ngoe-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training, IEEE TIP, 30: 30:1882-1897, 2021.
[81] Hung-Yo Yu Tseng, Matthew Fisher, Jingwan Lu, Yijun Li, Vladimir Kim, and Ming- Ming-Hsuan Yang. Modeling artistic workflows for image generation and editing. In ECCV, pages 158-174. Springer, 2020.
[82] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang. and Weilong Yang. Regularizing gen- erative adversarial networks under limited data. In CVPR, pages 7921-7931, 2021.
[83] Sheng- Yu Wang, David Bau, and Jun- Yan Zhu. Rewriting geometric rules of gan. ACM Transactions on Graphics (TOG), 41(4):1-16, 2022.
[84] Zhengwei Wang, Qi She, and Tomas E Ward. Generative adversarial networks in computer vision: A survey and taxonomy. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
[85] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: network for controlling face generation using images, audio, and pose codes. In ECCV, pages 670-686, 2018.
[86] Yuanbo Xiangli, Yubin Deng, Bo Dai, Chen Change Loy, and Dahua Lin. Real or not real, that is the question. ICLR, 2020.
[87] Ceyuan Yang, Yujun Shen, Yinghao Xu, Deli Zhao, Bo Dai, and Bolei Zhou. Improving gans with dynamic discriminator. NeurIPS, 2022.
[88] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. NeurlPS, 34:9378-9390, 2021.
[89] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin. Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans. In CVPR, pages 10743-10752, 2019.
[90] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 03365, 2015.
[91] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Va- sudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint wwXiv:2206.10789, 2022.
[92] Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image generation. In ICCV, pages 2759-2768, 2019.
[93] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint 1xX:::9100.12021 2019.
15

---

[94] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy- generative adversarial network. In ICLR, 2017.
[95] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. NeurIPS, 33:7559-7570, 2020.
[96] Kaiyang Zhou, Han Zhang, Wenqi Xie, Yunpeng Chen, Hongtao Liu, and Tong Zhang. Dy- namicd: High-fidelity image generation via dynamic capacity adjustment of discriminator. In CVPR, pages 15451-15460, 2021.
[97] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, In ICCV, pages 2223-2232, 2017.
[98] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. CVPR, pages 5802-5810, 2019.
16

---

Appendix
A Introduction
In this appendix, we first introduce the implementation details of our Dynamic Mask Discriminator (DMD), and then provide more technical details of the proposed DMD in Sec. B. In Sec. C.1 and C.2. we elaborate on the additional details of datasets and experiment settings. In Sec. C.3, we provide additional experimental results. We also report the error bars of our method in Sec. D.
B Implementation Details
In this section, we provide the implementation details of proposed method. We have utilized the PyTorch [59] public platform to conduct our experiments. Our models were trained on a workstation equipped with 8 NVIDIA Tesla V100 GPUs with 32 GB memory, a CPU of 2.8GHz, and 512GB RAM.
B.1 More details of the proposed method.
Our method is designed to automatically detect the retardation of the discriminator and force it to fast learn the new knowledge given time-varying distributions of generated samples. To show the detailed strategy of the proposed DMD method, the pseudo-code is given in Algorithm. 1. In our main paper, we have demonstrated the effectiveness of our proposed DMD method by integrating with image generation techniques on StyleGAN-V2 [36].
Since the retardation of learning new knowledge can potentially be addressed by augmenting the input stream or the outcome of the discriminator, we also extend another two schemes to achieve dynamic discriminator adjustment. Besides the proposed dynamic feature masking scheme, dynamically mask the input and output of the discriminator when it is detected as retardation. The corresponding details are shown summarized Algorithm. and Algorithm.
B.2 More details model parameter difference
We detail model parameter difference which is used in the main paper Figure 3(a) for investigating the fixed discriminator of StyleGAN-V2. Model parameter difference metric is to measure the differences of the discriminator model weights between adjacent training steps:
de IWW W
(5)
where Wd. is the parameter weight of d-th layer of the discriminator during Lj training step, and ||- 21 is 12 norm. A smaller d,, indicates that the parameter weights of d-th layer at ij training step are more similar to that at f--1 training step, i.e., parameter weights are updated slowly. In other words, given new arrival generated samples with new distributions, smaller d,, indicates that the discriminator slows down the learning of new knowledge to some extent. In the main paper, we calculate d,, in the full connection layer of the discriminator.
C Additional Details on Experiments
C.1 Datasets and Experimental Settings
AFHQ-V2 [12] consists of independent sub-datasets, which include around 5,000 closeups of cat, dog, and wildlife faces, respectively (denoted AFHQ-Cat, AFHQ-Dog, and AFHQ-Wild), We utilized high-qualing Lanezos filter [42] resize all images to a resolution of 256 x 256. We then conducted experiments on three sub-datasets while setting StyleGAN-V2 [36] as the baseline model. We maintain consistency with ADA [33], by using identical network architectures [36], weight demodulation [36]. style mixing regularization [35], path length regularization, lazy regularization [36], equalized learning rate for all trainable parameters [32], non-saturating logistic loss [19] with Rj regularization [54], and the Adam optimizer [37].
17

---

Algorithm Dynamic Mask Discriminator
Require:
Generator Gø:; Current Discriminator Da(-): Non-Masked Discriminator Do(-); Dynamically
Masked Discriminator D)(:); Training Step t;
d-th Layer of Discriminator F(4),; Dynamic Mask m
d-th Masking Layer Diseriminator F( Predefined Threshold X; Retardation Metric Ri;
Set U. Containing m Samples for Calculating Retardation Metrie;
The number of training steps ny; The number of images per training step ns;
Ensure:
Initialize Rg < and & < 1: Random m
1: while has not converged do
2: for 1, 1 to n, do
3:
if R₂ > X then
4:
M m/z: Mr M'; D&(:) Do(:)
5:
else
6:
M vector(1); Mr m D+(+) D-())
7:
end if
8:
for
1 to ngg do
9:
Fdj, + F(d);t o M
(t()) f -E (log ( D+)) eloog( D2(0((,0))
11:
0g < Adam 00
86())
end for
R2 1 EiEU, p
13:
14: end for
end while
16: Return 0:
FFHQ [35] comprises 70,000 images of human faces, which we used for training after downscaling them to resolution of 256 256. In this case, set StyleGAN-V2 [36] the baseline and used the same settings as those for AFHQ-V2.
LSUN-Church [90] includes 126,000 images of outdoor church. We downscale them to 256 256 as the training data. In this case, we also set StyleGAN-V? [36] as the baseline and used the same settings those for FFHQ
C.2 Baselines
In accordance with previous studies [33, 30, 50], we have integrated our proposed method with StyleGAN-V2 [36]. In order assess the effectiveness our method, we have compared it with state-of-the-art methods that improve discriminators through data augmentation, including ADA [33] and APA [30]. We have also compared our method with GANs that utilize regularization techniques, such as LC-Reg [82], zCR [93], InsGen [88], Adaptive Dropout [33], AdaptiveMix [50], MEE [48], and DynamicD [87].
C.3 Additional Experimental Results
Comparing our method with other masking strategies. Besides the masking strategies dis- cussed in the paper, another masking strategy, namely sachinuull--hangeedeee -aseedd--aaa--iiooo Discrimi- nator (CCD) explored here, which replaces our dynamic discriminator adjustment module by gradually increasing the mask ratio from 0.1 to 0.9 or decreasing the mask ratio from 0.9 to 0.1 over time. We evaluate the CCD with StyleGAN-V2 on the AFHQ-Cat (256 256 pixels) dataset. Compared with the StyleGAN- V2 (7. 7.424 FID), CCD with StyleGAN-Vi achieves 8.441 FID. How- ever, our method (5.879 FID) still outperforms CCD, since CCD increases instabilities in GAN training. More specifically, the process of GAN training is to play min-max two-player game between the generator and discriminator, which more unstable than typical classification prob- lems. Compared with our method, CCD causes the discriminator's discrimination ability to be more
18

---

Algorithm 2 Dynamic Mask Discriminator Assert in Input (Input Masking)
Require:
Generator Ggr; Discriminator (-): Training Step t: Dynamic Mask m':
Predefined Threshold A; Retardation Metric Rg:
Set U, Containing m Samples for Calculating Retardation Metrie;
d-th Layer of Discriminator F(d),, After Masking Input F(d),;
The number of training steps n,; The number of images per training step Nai Ensure:
Initialize Rg < 0 and / <- 1; Random m':
1: while 0 has not converged do
2: for = 1 to n, do
3: if R2 > À then
4:
M + m'; MT t M';
5: else
6:
M' t vector(1); Mr t m
7:
end if
8:
for s = 1 to n. do
+ 1 M. i + g2, o M
9:
(10(t) s - F (l,.(log(((1))))))
11:
0. Adam 88
2010)
end for
13:
R. 1 Σ
LEU,
|F |
14: end for
end while
16: Return 0;
frequently changed and uncertain, making more difficult for the generators fool the discrimi- nator. The deteriorated performance of the generator would further negatively affects the training of the discriminator. Instead, when the improvement of the discriminator slows down, our method switches from non-mask training to mask training. otherwise maintains cactii/g/noommaskinn at time interval, which provides more stable training than CCD.
Experiments on AFHQ-Cat with 512x512 pixels. We conduct experiments on the high- high-resolution AFHQ-Cat dataset, where the resolution of an image is 512x512 512 pixels. By integrating our method with StyleGAN-V2, our method reduces the FID of StyleGAN-V. by a margin of 18.77% (from 4.3160 to 3,5061), effectively improving the performance of StyleGAN-V2 higher-resolution images.
Experiments ImageNet-1000. We integrate the proposed DMD with intermediate checkpoint of the BigGAN [6] model which has been trained for 100,000 iterations. We then further trained ImageNet- 1000 at 128x 128 resolution using two
models from 100.000 to 107,500 iterations NVIDIA V100 GPUs Compared with the original BigGAN that achieves the FID 12.213 at the 107,500 iteration, outperforming the original BigGAN by 8. 1.196. This not only shows that our method benefits the training of BigGAN on large-scale datasets, but also demonstrates the flexibility and compatibility of our method in combination with GAN models such as pre-trained GAN models.
method (i.e., DMD) improves the training of BigGAN (FID of 11.212),
Experiments historical knowledge retention of discriminator. We conduct experimental stud- ies inspired by the study in online continual learning for rapid adaptation [7]. Given the discriminator trained from the beginning time T', we evaluate its historical knowledge retention by evaluating its discrimination performance on historical data generated at time T t. and evaluate its rapid adaptation by evaluating its performance in current data at time T and future data at T + t.
Table shows that the discriminator of StyleGAN-V2 achieves high accuracy on historical data, however, performs much worse current and future data. This shows that the discriminator of StyleGAN-V2 retains historical knowledge (i.e., high accuracy historical data), yet, does not fast learn and adapt to the changes in the distribution of future data.
19

---

Algorithm 3 Dynamic Mask Discriminator Assert in Outcome Logits (Dynamic Head)
Require:
Generator Oge; Discriminator (-):); Training Step t;
Outcome Logit Number of Da (.) and Dz(.) k; Dynamic Mask m';
d-th Layer of Discriminator F(d),, After Masking Outcome Logits F( Predefined Threshold A; Retardation Metric Re;
Set U, Containing m Samples for Calculating Retardation Metrie;
The number of training steps nı; The number of images per training step ngi Ensure:
Initialize Rg <- 0 and & <- 1; Random m';
converged do 2: for t. = 1 to n, do
1: while 0 has
3:
if R2 > À then
4:
M t m'; MT <- M';
5:
else
6:
M - vector( 1); My - m
7:
end if
8:
for = 1 to n1 do
9:
Le(1) r [log( (D()) @ Mt)))] - MM)
10:
0. + Adam J8.-
Ba(l)
11:
end for
F. pl
12:
R, Eiet,
| | |(
13: end for
end while
15: Return 0:
Instead, by detecting retardation and masking the discriminator, method enforces the discrim- inator to reduce retained historical knowledge (see decreased accuracy on historical data), while effectively encouraging it to fast learn and adapt to new knowledge of future data (i.e., method achieves higher accuracy on current and future data than StyleGAN-V2).
Table 9: The accuracy of the discriminator on historical data and future data, where the discriminator trained 800 kimgs step, and (-) indicates the data time T T + t.
Method
Historical data (200) Historical (600) Current data (800)
Future data (1000)
StyleGAN-V2
0.9696
0.9488
0.8236
0.604
Ours (StyleGAN- + DMD)
0.9129
0.9229
0.8793
0.6543
In addition, we calculate the average gradient of the StyleGAN-V2 during the training phase in Table 10, showing the gradient does not vanish from 1200 kimgs.
Table 10: Average gradient of the StyleGAN-V2 during training.
kimgs
0
200
400
600
800
1000
1200
Gradient 3.5145c-08 10c-70 -9.4235c-07
5.6989c-07
-4.3328c-07
9.2649c-08
1.2138c-06
Model parameter difference of the proposed method. We also compute the model parameter difference of the proposed method in the training of the AFHQ-Cat dataset. Our method detects the discriminator to be retarded 1000 kimgs after 800 kimgs. Our method then applies the proposed DMD adjustment to train StyleGAN-V from 1000 to 1004 kimgs, which increases the model differ- ence of our method by 63.889%, compared with the model difference between 800 to 1000 kimgs. Instead, the model difference of StyleGAN-V2's discriminator is decreased by 2.778% without our method. These results show StyleGAN-V2's discriminator slows down learning and our method effectively encourages the discriminator to fast learn.
20

---

Computation cost. Our method incurs a negligible increase in memory cost, since our method only additionally detects discriminator retardation and introduces feature masks. For computational cost, our method does not affect the inference time, since our method is designed for the discriminator and only the generator is used during inference. For training, our method introduces moderate time cost; however, this is affordable (e.g., increasing GPU number to reduce training time), and the additional training time cost depends on the time interval estimating the retardation. Our method additionally introduces 6 hours 11 minutes for training AFHQ in our experiments, when our method integrated with StyleGAN-V2 and retardation is detected at every kimgs. When detecting retardation at every 20 kings, method only additionally takes 1.3 hours for training.
Additional Generated Distribution. Our main paper studies the generated distributions of StyleGAN-V2. Here, we additionally provide the distribution of the generated samples of APA [30] method on FFHQ[35] in Fig. 9. Fig. also indicates that the generated distributions undergo dynamical and complex changes over time as the generator evolves during training. As result, the generated samples are not independently and identically distributed (i.i.d) across the training progress, posing significant challenges in learning the generated distributions.
0.00101
0.02
0.001
0.015
www.m
0.00999
0.01
15000
20000
25000
5000
10000
15000
20000
Training progress (kimg)
Trinning progress (kimg)
Distribution of Generated Images from APA (Meun & Variance)
Figure 9: Illustration of time-varying distributions of generated samples in the training process of APA [30] on FFHQ [35]. The mean and variance generated samples' features show the generated distributions are dynamic and timevnarying during training, as the generator evolves.
D Error Bar
To evaluate the reproducibility of our method's results, we run our experiments three times using random seeds and the same dyper-parametert Table 11 and Table 12 list the mean and variance of experimental results to show the error bar.
As shown in Table 11 and Table 12, our method performs stably multiple datasets i.e., AFHQ-V2, FFHQ and SSUN-hhurch datasets, indicating the reproducibility of our method.
Table 11:
Quantitative results of our method on AFHQ-V2 dataset [12], error bars are reported in terms of mean and variance, and Ours is StylGGAN-V2+DMD.
AFHQ-Cat
AFHQ-Dog
AFHQ-Wii Wild
FID
1St
FID
IS
FID 1
IS
StyleGAN-V2 V2
7.924
1.890
26.310
9,000
3.957
5,567
Ours (Reported in the paper)
5,879
1,988
21.240
9,698
3.471
5.647
Ours (re-run-l))
5.896
1.944
20.016
9.807
3.473
5.803
Ours (re-run-2))
6.015
1.987
20.456
10.291
3.420
5.699
Ours(Mean Variance)
5.960.10.01
1.973
20.57110.506
9.922.10.258
3.405
5.716.10.05
Table 12: Quantitative results of method FFHQ [35] and LSUN--hurch [90], where error bars are reported in terms mean and variance, and Ours is StyleGAN-V2+DMD..
LSUN-Church (126K)
FFHQ(70K)
FID +
IS
FID
IS f
StyleGAN-V. V2
4.292
2.589
3.810
5.185
Ours (Reported in the paper)
3.061
2.792
3.299
5,204
Ours (re-run-1))
3.025
2.795
3.177
5.225
Ours (re-run-2))
2.993
2.787
3.285
5.200
Ours Variance)
3.026-+0.028
2.791400.003
3.254+00.055
5.210-F0.01
21


Reviews:
[{"review_id": "bkFJawnnMc", "reviewer": "Reviewer_zHGZ", "summary": "The paper proposes Dynamically Masked Discriminator (DMD) which automatically adjusts the discriminator by dynamically masking the features when learning slows down, forcing the discriminator to learn new knowledge in the generated data. It consists of two modules: (1) discriminator retardation detection and (2) dynamic discriminator adjustment. The first module detects when the discriminator starts to learn slower (i.e. rely on old knowledge rather than learn new distributions of generated data). The second module dynamically assigns or removes masks to the features of the discriminator. The experiment results show that their method helps increase the quality of generated samples across diverse model architectures and datasets. ", "questions": "* What is the benefit of strictly separating the non-mask and mask training stage rather than changing the mask ratio over time? \n* Has the authors experimented with higher-resolution images? This would increase the impact of this work. ", "limitations": "The authors do not mention the limitation of their work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-motivated and clearly presented. \n* Rather than designing a complex network architecture, their method is designed to be easily integrated into any existing discriminator or used in combination with data augmentation methods which increases the significance and broader impact of the work. \n* They perform extensive experiments on a wide range of datasets and baseline architectures against various existing methods that improve discriminator training. Experiment results show that their method significantly reduces the FID score when combined with state-of-the-art GAN architectures. Qualitative results demonstrate that the method helps generate higher-quality samples without artifacts. \n* The paper is clearly organized and provided with sufficient technical and implementation details. \n", "weaknesses": "* Masking seems to be an important design factor for the efficacy of this method. The authors should elaborate on why switching from non-mask training to mask training is more desirable than continuously changing the mask ratio over time. \n* There exists minor typos and grammar mistakes throughout the paper. \n* The paper lacks the limitation section, which I feel would improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful feedback and constructive comments. We are encouraged by the reviewer's positive comments, such as “well-motivated”, “clearly organized”, “easily integrated into any existing discriminator”, “significance and broader impact of the work”, \"extensive experiments\", \"improve discriminator training\", and \"significantly reduces the FID score\".\n\n**Q1: Comparison with changing the mask ratio over time.**\n\n**A**. Thanks for the valuable comments. Following your suggestion, we build a baseline, namely Continualy-Changed-mask-ratio Discriminator (CCD), which replaces our dynamic discriminator adjustment module by gradually increasing the mask ratio from 0.1 to 0.9 or decreasing the mask ratio from 0.9 to 0.1 over time. \n\nThe table below shows that our method outperforms CCD, since CCD increases instabilities in GAN training. More specifically, the process of GAN training is to play a min-max two-player game between the generator and discriminator, which is more unstable than typical classification problems. Compared with our method,  CCD causes the discriminator's discrimination ability to be more frequently changed and uncertain, making it more difficult for the generators to fool the discriminator. The deteriorated performance of the generator would further negatively affects the training of the discriminator. Instead, when the improvement of the discriminator slows down, our method switches from non-mask training to mask training, otherwise maintains masking/non-masking at a time interval, which provides more stable training than CCD.\n\n\n **Table**: Comparison with changing the mask ratio over time on AFHQ-Cat (256$\\times$256 pixels). \n\n|                 Method                |  FID $\\downarrow$ |\n|:-------------------------------------:|:-----:|\n|            StyleGAN-V2             | 7.924 |\n|         StyleGAN-V2 + CCD   | 8.441 |\n|     Ours (StyleGAN-V2+ DMD)    | **5.879** |\n\n\nIn addition, we observe that continually changing the mask ratio requires careful designs on the decay/growth strategy of the mask ratio, and needs to enforce it to be adaptive to retardation detection. We thank the reviewer's comments and think temporally changing the ratio can be a promising direction.  We will explore this further in future work.\n\n\n\n**Q2: Minor typos and grammar mistakes.**\n\n**A.** Thank you. We will correct typos and grammar mistakes.\n\n**Q3: Has the authors experimented with higher-resolution images?**\n\n**A.** Thank you for the constructive comments. We conduct experiments on the high-resolution AFHQ-Cat dataset, where the resolution of an image is 512 $\\times$ 512 pixels. By integrating our method with the SOTA method StyleGAN-V2,  our method reduces the FID $\\downarrow$ of StyleGAN-V2 by a margin of **18.77\\%** (from 4.3160  to 3.5061) on AFHQ-Cat (512 $\\times$ 512), effectively improving the performance of StyleGAN-V2 on higher-resolution images.\n\n**Q4: The limitation section.**\n\n**A.** Thanks for your suggestions. Due to the page limitation, we have to discuss the limitation of our method in  Sec. 5 of the appendix:\n\nTheoretical studies can make this work more comprehensive; however, we have not explored it in the paper, since it is beyond the scope of this study. Moreover, while the proposed method can effectively improve the training of the CNN-based GANs models, combining our method with Transformer-based ones is left to be investigated in the future."}, {"review_id": "iHAR8bgLu0", "reviewer": "Reviewer_ZBcx", "summary": "In this paper, the authors propose a novel perspective for generative adversarial training via continual learning, which achieves better results on the generative quality for local details and better quantitative metrics. To realize continual learning, the authors propose two models, 1) the discriminator slow-down detection mechanism and 2) the dynamical masked discriminator for different time-steps of training.", "questions": "Please refer to the above weakness section.", "limitations": "The limitation is not discussed, however I would concern if the method can be directly scaled up to the large scale dataset generation, such  as imagenet1000.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The novel perspective of modelling GAN with continual learning is interesting and promising, which tries to improve GAN training on discriminators with fast adapting features.\n2. The proposed method is simple and straightforward yet effective to be trained and probably would be general for any type of GAN network.\n3. The model is well-explained, and the paper is well-written.", "weaknesses": "1. Even though the idea of continual learning for GAN is novel and exciting, it seems that the connection between continual learning and the dynamically masked discriminator is not well-explained.\n2. It seems the improvements on the face dataset or animal face dataset shows marginal improvement; I would be more interested in the training on a more diverse dataset as the SOTA generative models are mostly capable of generating on large-scale datasets, such as imagenet1000. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable comments. We are encouraged that the reviewer agrees that our work is “interesting and promising”, “novel and exciting”, “simple and straightforward yet effective to be trained”, “probably would be general”, “well-explained”, and “well-written”. \n\n**Q1.The connection between continual learning and the dynamically masked discriminator.**\n\n**A.** Thank you for the comments. Our method is inspired by the recent work on online continual learning for rapid adaptation. The study pointed out that data with time-vary distribution requires learning methods to have the ability of fast learning and quick adaption (see Line 47-55 in the main paper). For GANs, the distribution of the generated data drifts over time, as the generator evolves during training. This also requires the discriminator to fast learn new knowledge of incoming data.\n\nHowever, online continual learning for rapid adaption has been underexplored  [16, 6, 39]. Moreover, different from most online continual learning methods for rapid adaption, training GANs models plays a min-max game between the generator and the discriminator, posing new challenges.  For example,  the incoming data of the discriminator is unknown and uncertain, while the generation results of the generator depend on the discriminator and the discriminator learns from data generated by the generator.  Yet, these challenges haven't been considered by existing online continual learning methods. \nTo address the challenges, we propose a new method for GAN models, which detects the discriminator retardation and masks the discriminator features. More specifically, masking the feature map of the discriminator is to break the original dependency of the discriminator on some local features that are important to distinguish historical data, and enforce the discriminator learn new knowledge of incoming data by rebuilding the dependency of remaining non-masked local features and incoming data.  \n\n\n\n**Q2: Scale up to the large-scale dataset generation, such as ImageNet-1000.**\n\n**A.** Thanks for the valuable comments. Following your suggestions, we train our method on ImageNet-1000. Yet, the settings of SOTA GAN methods typically require a larger number of iterations (e.g., 150,000 iterations for BigGAN [5]) on ImageNet-1000,  which takes more than two weeks to train from scratch with eight NVIDIA V100 GPUs [5]. Due to the time limitations and limited number of GPUs, we integrate our method with an intermediate checkpoint of the BigGAN model which has been trained for 100,000 iterations.  We then further trained models from 100,000 to 107,500 iterations on ImageNet-1000 at 128$\\times$128  resolution using two NVIDIA V100 GPUs.\n\nCompared with the original BigGAN that achieves the FID $\\downarrow$ of 12.213 at the 107,500$^{th}$ iteration, our method (i.e., DMD) improves the training of BigGAN, outperforming the original BigGAN by **8.196\\%** (see the table below). This not only shows that our method benefits the training of BigGAN on large-scale datasets, but also demonstrates the flexibility and compatibility of our method in combination with GAN models such as pre-trained GAN models.  \n\n**Table**:  Results of the proposed method on  ImageNet-1000.\n\n|                 Method                |  FID $\\downarrow$ |\n|:-------------------------------------:|:-----:|\n|           BigGAN           | 12.213|\n|     Ours (BigGAN+ DMD)    |**11.212** |\n\n\nIn addition,  Table 2 in the main paper evaluates our method on a large dataset LSUN-Church, which contains 126K images and is used to challenge generative methods in recent GAN and diffusion work (e.g., [54,74]). Our method outperforms StyleGAN-V2 by 28.7\\% on LSUN-Church, and surpasses a SOTA diffusion model LDM [54] by 23.9\\%. \n\n\n**Q3:  The limitation.**\n\n**A.** Thanks for your suggestions. Due to the page limitation, we have to discuss the limitation of our method in  Sec. 5 of the appendix:\n\nTheoretical studies can make this work more comprehensive; however, we have not explored it in the paper, since it is beyond the scope of this study. Moreover, while the proposed method can effectively improve the training of the CNN-based GANs models, combining our method with Transformer-based ones is left to be investigated in the future."}, {"review_id": "vTzVfakz3R", "reviewer": "Reviewer_J72A", "summary": "This paper proposes a new training method of generating adversarial networks from the perspective of online continuous learning. In order to address the challenges posed to the discriminator by the dynamic changes in the generated data distribution during training, the authors propose to detect whether the discriminator has slowed down its adaptation to newly generated data and dynamically mask its features to force it to quickly learn new knowledge. Experimental results on FFHQ, AFHQ, and LSUN-Church show that the proposed method is superior to the state-of-the-art methods in image generation.", "questions": "- Considering that the proposed method requires continual detection of discriminator retardation, will this result in significant additional training overheads compared to baselines?", "limitations": "The authors have not adequately addressed the **limitations of the proposed method** and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is well-organized and well-written, making it easy to follow. The analysis of the discriminator in the Pilot Study section is helpful for readers to understand the motivation of the work.\n- The paper provides a comprehensive review of related work, which helps readers to better understand its contributions.\n- The proposed method outperforms state-of-the-art methods, including diffusion models, in mainstream experiments according to the experimental results reported in the paper.", "weaknesses": "- The notation used in the method description is confusing and inconsistent. For example, $\\mathcal{D}$ and $\\bar{D}$, $\\theta^t$ and $\\phi(t)$. Additionally, $\\phi(t)$ is not included in discriminators as a parameter like $\\theta^t$.\n- The comparison of the visualization results is not significant. For instance, it is difficult to understand the improvements in the generated results of the proposed method compared to the baseline in Fig.1. Considering that this is a subjective evaluation, including objective evaluation metrics in this experiment would be helpful.\n- The proposed method is kind of heuristic in nature, hence the technical novelty and contribution are limited. More advanced methods for online continual learning may be a better choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers for the careful reviews and\nconstructive suggestions.  We are encouraged that the reviewer agrees that our paper is \"well-organized and well-written\", \"new training method \", \"provides a comprehensive review\", \"Pilot Study section is helpful for readers to understand the motivation\", and  \"outperforms state-of-the-art methods\". \n\n**Q1: The notation used in the method description.**\n\n**A.** Thank you for pointing this out. We will replace $\\bar{D}$ by $\\bar{\\mathcal{D}}$ to maintain notation consistency between the discriminator without masks and that with masks.  We will include $\\phi(t)$ in the discriminator like $\\theta^t$ in the final version.\n\n**Q2. Including objective evaluation metrics in Figure 1**\n\nThank you for your constructive suggestions. Firstly, we introduce FID $\\downarrow$ to evaluate generation quality. The FID of our method is 3.299, suppressing the baseline (FID 3.810). \n\nSecondly, we introduce the cosine similarity metric. We aim to enable the discriminator to pay attention to different local regions in samples at different steps, such that the discriminator adapts to the time-varying distributions of generated samples. We extract binary attention maps from the feature maps in the second row of Figure 1, and then calculate the cosine similarity of the attention maps between the current training step and the previous one. Lower value indicates better performance, i.e., attentive regions at the current step are more different from that at the previous one. The table below shows our method enforces the attentive regions of the discriminator to be more different at each training step than the baseline, better adapting to time-varying distribution of generated data. We will include objective evaluation results in the final version.  \n\n\n**Table 2**: Cosine similarity $\\downarrow$ of attention maps between current and previous steps in Figure 1.\n\n|          |   t1  | t2   |   t3  |\n|:--------:|:------:|:------:|:------:|\n| Baseline | 0.8672 | 0.8242 | 0.8145 |\n|   Ours   | **0.2871** | **0.4336** | **0.5488** |\n\n\n\n\n\n**Q3: More advanced methods for online continual learning.**\n\n**A.** We will explore more advanced methods in our future work, thank you for your suggestion. Yet, as recognized by Reviewer zHGZ, \"Rather than designing a complex network architecture, their method is designed to be easily integrated into any existing discriminator or used in combination with data augmentation methods which increases the significance and broader impact of the work\".  Reviewer dcqD also agreed that we propose \"a novel method for training Generative Adversarial Networks (GANs) based on online continual learning\" which achieves \"Promising image results\" on \"Multiple datasets\".\n\nWe design such a simple and straightforward method, due to the specific challenges introduced by GAN models.  Different from existing online continual learning methods for rapid adaption (e.g., for classification), GANs models posed new challenges in that the training GANs plays a min-max game between the generator and the discriminator. Sophisticated designs may increase the difficulties of optimizing the generator and the discriminator, leading to large instabilities in the training. \nDifferent from [74] which explores the influence of various capacities of the discriminator on training, \n we propose to detect the retardation of the discriminator and dynamically mask the discriminator to force it to learn fast, which is simple yet effective.\n\nNevertheless,  we thank the reviewer's suggestions.  We try to improve our discriminator retardation detection.  In particular, if the Retardation value $\\mathcal{R}_t$ of current time interval $t$ is higher than that of the previous one, the discriminator would be detected in retardation. With this new discriminator retardation detection, our method achieves  5.877 FID on AFHQ-Cat(256 $\\times$ 256), which is on par with ours (5.879 FID) in the main text. \n\n\n\n**Q4: Will the detection of discriminator retardation result in significant additional training overheads**\n\n**A.** Our method does not result in significant additional training overheads. In particular, the retardation of the discriminator is typically not an abrupt, but smooth transition process during training. This enables our method to detect retardation of the discriminator sparsely, instead of after each training image.  The additional time cost is 6 hours 11 minutes for the whole training on AFHQ in our experiments,  where our method is integrated with  StyleGAN-V2 and detects retardation at every 4 kimgs. When detecting at every  20k kimgs, our method additionally takes 1.3 hours.\n\n**Q5: Limitations of the proposed method and potential negative societal impact**\n\n**A.** Thank you for the comments.  Due to the page limitation, we discuss the limitation of the proposed method in Sec. 5 of the Appendix:\n\nTheoretical studies can make this work more comprehensive; however, we have not explored it in the paper, since it is beyond the scope of this paper. Moreover, while the proposed method can effectively improve the training of the CNN-based GANs models, combining our method with Transformer-based ones is left to be investigated in the future. \n\nPotential negative societal impact is also discussed in Sec. 5 of the Appendix:\n\nLike other generative models, our method can be misused for applications, such as Deepfake [1], where fake content is synthesized to deceive and mislead people, leading to a negative social impact. Nevertheless, many researchers have considered this problem, while exploring fake content detection and media forensics techniques.\n"}, {"review_id": "V4h3jQri7z", "reviewer": "Reviewer_zVaP", "summary": "The paper introduces a new method to enhance GAN training. Taking a continual learning perspective, the authors contend that the discriminator faces challenges in modeling the real/fake distribution due to the dynamic nature of the generated samples' distribution over time. Consequently, this slows down the learning process of the discriminator. The paper suggests that this slowdown occurs because the discriminator is trained using historical data. To address this issue, the authors propose a method to detect the sluggishness of the discriminator and dynamically mask the feature map to accelerate its learning.\n\nThe authors perform experiments on several benchmark datasets and demonstrate that the proposed method surpasses several existing GAN methods, as measured by FID and IS scores. Additionally, they provide an ablation study to examine the optimal choice of the layer to be masked and the corresponding masking ratio for achieving the best performance.\n\nIn general, the paper is well-written and easy to understand. However, it should be noted that the proposed improvement is incremental, as the concepts of continual learning perspective and dynamic discriminator training have previously been introduced in [74]. The main contribution of this paper lies in the use of masking to adjust discriminator training. It is important to address some claims made in the paper that currently lack empirical evidence, and it would be beneficial to provide further ablation study on retard detection.", "questions": "See questions and suggestions above.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method introduces a technique to identify the slowdown and presents a novel way to mask feature maps to accelerate the discriminator learning.\n\nThe results are encouraging.\n\nThe paper is easy to follow.\n", "weaknesses": "I find the claim that the retardation of the discriminator is attributed to learning on historical data somewhat unclear. The slowdown of the discriminator could also be a result of overfitting, causing the gradient to vanish and making learning difficult. It is necessary to provide further justification for the claim that historical data is the cause of the retardation. I suggest the authors include an experimental study to support their argument.\n\nI would suggest the author consider including a comparison of the mean and variance between the proposed method and StyleGAN-2 in Figure 2. This would provide valuable insights into how the generator distribution changes with the proposed method. Additionally, it would be helpful to have more details about the figure. For instance, it would be beneficial to know the number of samples used to calculate the mean and variance. Could the author clarify whether the plot represents early epochs or late epochs? If the figure displays early epochs, it would be intriguing to observe how the changes evolve in the later epochs.\n\nSimilarly, I recommend that the author consider showcasing the model parameter difference in Figure 3 to demonstrate that the learning of the proposed method does not slow down. This would provide valuable evidence to support the claim. Additionally, it would be helpful if the author could clarify where the retard detection takes place during the training process when the proposed method is applied in this example.\n\nThe detection of retardation is a crucial aspect of the proposed method and warrants an ablation study to assess its impact. It would be valuable if the authors could provide an ablation study that demonstrates how frequently retardation is detected during the training process of the proposed method. Additionally, it would be interesting to explore the consequences of removing retardation detection and using random masking consistently throughout the training.\n\nOverall, the paper presents several intriguing aspects. However, it is crucial to provide justifications and supporting evidence for the claims made. Without proper justification, there is a risk of potentially misleading the research community. I would consider the paper to be on the borderline, and I am looking forward to reading the authors' rebuttal addressing my questions and the feedback from other reviewers to make a final decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive comments and\nconstructive suggestions. The reviewer agrees that \"a new method\", \"novel way to mask feature maps\", \"well-written and easy to understand\",   \"the results are encouraging\", and \"surpasses several existing GAN methods\".\n\n\n**Q1: Experimental study to show that the retardation of the discriminator is attributed to learning on historical data.**\n\n**A.** Thank you for your suggestions. We conduct experimental studies inspired by the study in online continual learning for rapid adaptation [6].  Given the discriminator trained from the beginning to time $T$, we evaluate its historical knowledge retention by evaluating its discrimination performance on historical data generated at time $T$-$t$, and evaluate its rapid adaptation by evaluating its performance in current data at time $T$  and future data at  $T+t$. \n\nThe table below shows that the discriminator of StyleGAN-V2 achieves high accuracy on historical data, however, performs much worse on current and future data. This shows that the discriminator of StyleGAN-V2  retains historical knowledge (i.e., high accuracy in historical data),  yet, does not fast learn and adapt to the changes in the distribution of future data.\n\n**Table**. The accuracy of the discriminator on historical data and future data, where the discriminator is trained at 800 kimgs step, and $(\\cdot)$ indicates the data at time $T$-$t$ or $T+t$. \n\n\n|   Method |  Historical data (200)  | Historical data (600)    | Current data (800)    | Future data (1000)|\n|:-------:|:----------:|:----------:|:----------:|:----------:|\n|StyleGAN-V2|0.9696| 0.9488|0.8236|0.604  |\n|Ours (StyleGAN-V2 + DMD)|0.9129 |0.9229|0.8793|0.6543|\n\nInstead, by detecting retardation and masking the discriminator, our method enforces the discriminator to reduce retained historical knowledge (see decreased accuracy on historical data), while effectively encouraging it to fast learn and adapt to new knowledge of future data (i.e., our method achieves higher accuracy on current and future data than StyleGAN-V2).\n\nIn addition, we calculate the average gradient of the StyleGAN-V2 during the training phase, showing the gradient does not vanish from 0 to 1200 kimgs.\n\n**Table**: Average gradient of the StyleGAN-V2  during training\n\n| kimgs  |  Gradient |\n|:-----:|:------:|\n| 0  | 3.5145e-08   |\n| 200  | 4.6110e-07   |\n| 400  | -9.4235e-07   |\n| 600  | 5.6989e-07   |\n| 800  | -4.3328e-07   |\n| 1000  | 9.2649e-08   |\n| 1200  | 1.2138e-06   |\n\n\n\n\n\n \n **Q2: The number of samples used to calculate the mean and variance in Figure 2.**\n\n**A.** We use 50k samples to calculate each mean and variance values. \n\n\n**Q3: Whether the plot in Figure 2 represents early epochs or late epochs?**\n\n **A.** The plot in Figure 2 represents the whole training process from early epochs to late ones on FFHQ.\n \n\n\n**Q4: Model parameter difference of the proposed method and where the retard detection takes place.**\n\n \n**A.** Thanks for your valuable suggestions. Our method detects the discriminator to be retarded at 1000 kimgs after 800 kimgs.  Our method then applies the proposed dynamic discriminator adjustment to train StyleGAN-V2 from 1000 to 1004 kimgs, which increases the model difference of our method by 63.889\\%, compared with the model difference between 800 to 1000 kimgs. Instead, the model difference of StyleGAN-V2's discriminator is decreased by 2.778\\% without our method. These results show StyleGAN-V2's discriminator slows down learning and our method effectively encourages the discriminator to fast learn. \nWe will add our model's parameter difference in Fig. 3, as suggested.\n\n\n**Q5: How frequently retardation is detected during the training process of the proposed method.**\n\n**A.** The number of detected retardation is 256 times in total on AFHQ-Cat，when StyleGAN-V2 + DMD (ours) is trained for 1400 kimgs steps.\n\n\n**Q6: Removing retardation detection and using random masking consistently throughout the training.**\n\n**A.**  Thank you for the suggestions. We build a baseline, namely Random-Mask Discriminator (RMD), which removes retardation detection from our method and uses random masking consistently throughout the training. We also build the second baseline,  namely Time-Varying-mask Discriminator (TVD), which removes retardation detection and randomly changes the mask per mini-batch. \n\nThe table below shows that our method outperforms RMD and TVD by a large margin, thanks to the proposed retardation detection and dynamic discriminator adjustment modules.\n\n**Table**: Comparison with the different masking strategies on AFHQ-Cat (256 $\\times$ 256 pixels).\n\n|  Method |  FID $\\downarrow$ |\n|:---------------:|:-----:|\n|  StyleGAN-V2   | 7.924 |\n  StyleGANV2 + TVD       | 9.294 \n| StyleGAN-V2 + RMD      | 8.759 |\n|   Ours (StyleGANV2+ DMD)    | **5.879** |\n"}, {"review_id": "0W1gH4GkHJ", "reviewer": "Reviewer_dcqD", "summary": "This paper presents a novel method for training Generative Adversarial Networks (GANs) based on online continual learning, addressing the persistent challenge of GAN training instability. \nThe method considers the time-varying distribution of generated samples and prompts the discriminator to learn new information swiftly. It accomplishes this by detecting when the discriminator is lagging in learning from newly generated data and applying dynamic masking to discriminator features, thus forcing faster adaptation to the changing data distribution. \nImproves GAN training, but also outperforms existing SOTA and advanced diffusion models.\n", "questions": "1. Does the proposed method introduce a significant amount of computation cost?\n2. I can’t understand the feature maps used in the figure, asking for more explanation. \n3 .How well does the proposed method generalize to other GAN methods? \n", "limitations": "Refer to previous sections. \n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Excellent pilot study to understand the generated distribution. \n2. Method well explained. Nice flow chart\n3. Promising image results supported by FID. \n4. Multiple datasets\n", "weaknesses": "1. The video in the supplementary is confusing. \n2. Lack of intuition explanation or theory proof to the proposed “Dynamically masked discriminator”\n3. Fig3, 5 are both feature maps for visualization. I don’t understand the feature map, asking for more explanations.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's positive feedback and insightful comments. We are delighted to learn that the reviewer agrees that \" novel method\", \"excellent pilot study\", \"well explained\", \"nice flow chart\", \"promising image results\", and \"multiple datasets\".\n\n**Q1. Asking for more explanations on the feature maps in Figs. 3,5**\n\n**A.**  Feature maps in Fig. 3 are extracted from the layer of StyleGAN-V2’s discriminator trained at different training steps.  These feature maps visualize the attentive regions of the discriminator, showing that StyleGAN-V2’s discriminator pays attention to almost fixed local regions over time, given the input image. That is, StyleGAN-V2’s discriminator almost depends on fixed local regions for discrimination.  However, as the generator evolves during training,  the discriminative local regions distinguishing between real and generated samples change over time.\nHence, it is expected that the discriminator pays attention to different local regions in the samples at different steps, to learn the time-varying generated distributions(see Fig. 2).\n\nIn Fig. 5, feature maps also show the attentive regions of the discriminator. For example, the feature map in the fifth column assigns high values to the right eye regions. By masking the feature map in the sixth column,  our method breaks the original dependency of the discriminator on some local features that are important to distinguish historical samples and encourage the discriminator to learn new knowledge from incoming data.\n\n\n**Q2: Intuition explanation or theory proof to the proposed “Dynamically masked discriminator**\n\n**A.** Thank you for the comments.  We agree with the reviewer that theory proof can make our work more comprehensive. We will explore it in future work, as it is beyond the scope of this paper. \nIn this paper, we mainly show the challenges posed by the time-varying distributions, reveal that\ntypical discriminators slow down their adaptation to the changes in the incoming data,\nand propose a new method to address the challenges.\n\nFor the intuition explanation of our method,  as discussed in the responses to the above questions,  we aim to enable the discriminator to pay attention to different local regions for generated samples at different steps, such that the discriminator learns the time-varying distributions of generated samples.\nTo this end, when the discriminator is detected to be retarded, our method dynamically masks the discriminator features to reduce its dependence on some local features that are important to distinguish historical samples (see Fig. 5). As a result,  the discriminator is enforced to re-build the dependency of incoming data and remaining non-masked local regions, and hence  learn new knowledge from the incoming data.\n\n\n\n\n**Q3: Does the proposed method introduce a significant amount of computation cost**\n\n**A.** Thanks for the comments.  Our method incurs a negligible increase in memory cost, since our method only additionally detects discriminator retardation and introduces feature masks.\n\nFor computational cost, our method does not affect the inference time, since our method is designed for the discriminator and only the generator is used during inference. For training, our method introduces moderate time cost; however, this is affordable (e.g., increasing GPU number to reduce training time), and the additional training time cost depends on the time interval of estimating the retardation. Our method additionally introduces 6 hours 11 minutes for training on AFHQ in our experiments, when our method is integrated with StyleGAN-V2 and retardation is detected at every 4 kimgs. When detecting retardation at every 20 kimgs, our method only additionally takes 1.3 hours for training.\n\n\n\n\n**Q4:How well does the proposed method generalize to other GAN methods?**\n\n**A.** Thanks for the valuable comments. Following your suggestions, we evaluate the generalizability of our method to other GAN methods by incorporating our method with StyleGAN-V3 and unconditional BigGAN, respectively. Compared with StyleGAN-V3 using the original discriminator (FID  5.850),  our method improves the performance of StyleGAN-V3 by 15.88\\% in the FID $\\downarrow$ metric on AFHQ-Cat.  Similarly, our method effectively improves the training of unconditional BigGAN, reducing the FID  of unconditional BigGAN by 34.29\\% on CIFAR10."}]

Author rebuttal:

Meta-review: The paper studies the use of online continual learning techniques when training generative adversarial nets (GAN), which addresses the issue that the discriminator faces a changing distribution during GAN training. The paper received a bi-modal assessment. Two reviewers were excited, two more concerned and one unsure. Major concerns were related to additional explanations (why switching from masking to non-masking and the use of masking in general), technical novelty, confusing notation, among others. The rebuttal was able to address most of the concerns. One reviewer remained concerned, missing a more theoretical explanation while the other reviewer didn't respond. AC thinks the paper presents an interesting aspect for GAN training that's useful for the community to discuss. AC also thinks that some choices (e.g., masking) could be better motivated and studied (at least empirically). Overall benefits outweigh concerns.
Decision: Accept (poster)