SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wentian Zhang $^{1,2\dagger}$  Haozhe Liu $^{1,2\dagger}$  Bing Li $^{1\boxtimes \star}$  Jinheng Xie $^{3}$  Yawen Huang $^{2}$  Yuexiang Li $^{2,4\boxtimes}$  Yefeng Zheng $^{2}$  Bernard Ghanem $^{1}$
$^{1}$  AI Initiative, King Abdullah University of Science and Technology  
 $^{2}$  Jarvis Research Center, Tencent YouTu Lab  $^{3}$  National University of Singapore  
 $^{4}$  Life Sciences Institute, Guangxi Medical University  
bing.li@kaust.edu.sa; leeyuexiang@163.com
Training Generative Adversarial Networks (GANs) remains a challenging problem. The discriminator trains the generator by learning the distribution of real/generated data. However, the distribution of generated data changes throughout the training process, which is difficult for the discriminator to learn. In this paper, we propose a novel method for GANs from the viewpoint of online continual learning. We observe that the discriminator model, trained on historically generated data, often slows down its adaptation to the changes in the new arrival generated data, which accordingly decreases the quality of generated results. By treating the generated data in training as a stream, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data. Therefore, we can explicitly enforce the discriminator to learn new knowledge fast. Particularly, we propose a new discriminator, which automatically detects its retardation and then dynamically masks its features, such that the discriminator can adaptively learn the temporally-vary distribution of generated data. Experimental results show our method outperforms the state-of-the-art approaches.
Generative Adversarial Networks (GANs) [19, 84, 20, 5, 72, 70, 71] have shown the remarkable performance in various applications [89, 85, 81, 3, 44], which have attracted intensive interests. The main components of GANs are the generator and the discriminator, where the generator is trained to generate realistic samples, and the discriminator learns to distinguish real and generated samples. However, it is well-known that GANs are difficult to train [33, 30, 21].
Many efforts have been devoted to alleviating training difficulties for GANs. Some studies balance the generator and discriminator from the side of network architecture, such as DCGAN [61], PGGAN [32], Style-GANs [35, 36, 34], and BigGAN [6]. Besides, some studies [2, 94, 53, 86] address this challenge from the learning objective, e.g., WGAN [2], EBGAN [94], LSGAN[53] and realnessGAN [86]. These methods can further stabilize training, especially coupled with the Lipschitz regularization [56, 21] and conditional information [55]. Recent works [87, 33, 30, 95, 93, 50, 48, 82] propose to improve GANs on the discriminator side. For example, different data augmentation strategies [33, 30, 95] are proposed to strengthen the discriminator for a limited data regime. Regularizations are applied to stabilize the training of discriminator [93, 50, 82] and combat the mode
† Equal Contribution. Haozhe worked on this project before joining KAUST.
$\star$  Project Leader
Code is available at https://github.com/WentianZhang-ML/DMD.
(a) StyleGAN-V2
Training Steps
Figure 1: Illustration of the advantage of our method, where the first and second rows show generated images and feature maps taken from a discriminator layer, respectively. (a) StyleGAN-V2 introduces similar artifacts into generated images, despite the increased training steps. Our method is simple yet effectively improves the training of GANs, boosting the quality of generated samples for StyleGAN-V2 in (b). By fast learning incoming generated samples, the proposed discriminator captures artifacts in the local regions, better guiding the training of the generator. The averaging cosine similarity between the current and previous feature maps is 0.8353 for StyleGAN-V2 and 0.4232 for ours, which indicates that our method enforces the attentive regions of the discriminator to be more different than the baseline, better adapting to time-varying distribution of generated data.
(b) StyleGAN-V2 + Ours
Training Steps
collapse [48]. These methods show that the discriminator is very crucial to the training of GANs. Nevertheless, stably training GANs remains a challenging problem [33, 31, 93].
In this paper, we rethink the difficulty of training GANs from the discriminator side. The discriminator guides the training of the generator via learning the distributions of real/generated samples. However, since it has been observed that the characteristics of generated samples vary during the whole training process [87], a question arises: how challenging is it for the discriminator to learn from such time-varying generated samples? Our study shows that the distribution of generated samples dynamically changes during training, posing significant challenges to the discriminator. In particular, the discriminator in existing work (e.g., [6, 36, 57, 50]) does not explicitly consider an online task that addresses time-varying distributions of generated samples but is presented for a typical classification task. We refer to such a discriminator as fixed. We find that the fixed discriminator often improperly relies on local features that are important to historically generated samples to distinguish incoming ones, although the distributions of these generated samples are different. As a result, the discriminator insufficiently learns the distribution of incoming generated samples, leading to subpar guidance to the generator (see Fig. 1).
We propose a novel method for training GANs, from the perspective of online continual learning. Online continual learning has attracted increasing attention [52, 14, 40]. Different from offline continual learning focusing on the catastrophic forgetting problem [51, 38, 60], recent online continual learning methods [17, 7, 47, 22] pointed out that data with time-vary distribution requires learning methods to have the ability of fast learning and quick adaption. Motivated by these methods, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data by treating generated samples throughout training as a stream of data. We then explicitly force the discriminator to learn new knowledge fast. To this end, we propose a simple yet effective scheme to detect the retardation of the discriminator and force it to learn fast. Instead of designing a sophisticated network architecture, we dynamically mask discriminator features when the discriminator is predicted to be retarded, such that the discriminator adaptively learns the temporally-vary distribution of generated data.
Our contributions are summarized as follows.
- From the perspective of online continual learning, we propose a novel approach that effectively improves the training of GANs by explicitly considering the time-varying distribution of generated samples.
- The proposed discriminator fast learns and adapts to the time-varying distribution of the generated samples via dynamically masking discriminator features, improving the training of GANs.
- Extensive experimental results show that our method generates high-quality images, outperforming state-of-the-art GAN approaches. Besides, the performance of our method surpasses advanced diffusion models.
Generative adversarial networks. GANs have attracted increasing attention from researchers [19, 84, 20, 5, 72]. By learning a min-max objective, GANs generate samples from a given data distribution, which can serve for various applications, e.g., image editing [18, 28, 75, 3, 27, 97, 83, 43] and text-conditional image synthesis [69, 31, 78, 45, 64, 98]. As a fundamental technology, GANs coupled with the diffusion model (DM) [65, 26, 58, 62, 66] and auto-regressive model (AR) [63, 91, 10, 9] indirectly turn on the light of AIGC [8]. Compared with the other emergent generative models, GANs have the advantages of rapid inference [31], generating fixed objects [31, 65] and controllable latent distribution [35, 36, 34, 4, 23, 75]. Recently, as an answer ball against DM and AR, GigaGAN [31], StyleGAN-T[69] and GALIP [78] adapted GANs to large image-text datasets, demonstrating that GANs can be another viable solution for zero-shot text-to-image generation. However, as "No Pain, No Gain", the payments of GANs are also high, i.e., GANs suffer from unstable training and mode collapse. As a directed result, GANs are difficult to be equipped with some standard modules, such as dropout [76] and masking operator [10, 24], which strengthen the model generalization via introducing randomness into the training process, but further decrease the training stability. Existing studies [92, 74, 13, 96] illustrated that GANs learn from the stream, where the training distribution varies in the different time steps. Without any regularization, GANs may harmfully learn a bias on historical data and neglect the current data. Our goal in this paper is to downgrade the potential payments of GANs and integrate the masking operator into GANs to overcome overfitting on historical data.
Regularization on discriminators. As a model helping the generator to align with real distribution, the discriminator plays a pivotal role to train GANs. Various methods have been proposed to improve GANs by redesigning discriminators [2, 21, 50, 93, 87, 30, 48, 96, 32, 57]. WGAN [2] employs the discriminator to measure the synthetic and natural samples with Wasserstein distance. Since WGAN requires Lipschitz continuity to ensure model convergence, weight clipping [2] and gradient penalty [21] were proposed to enhance the discriminators. Such technologies are then inherited by the family of StyleGAN [35, 36, 34]. Thanks to the advanced architecture [6, 36, 32], several regularization technologies, including ADA [33], APA [30], LC-Reg [82], DiffAugment [95], Con-Reg [93], AdaptiveMix [50], DAG [80], and MEE [48] were proposed to train GANs with low-data regime [50, 30, 33, 82, 80, 95] or combat mode collapse [48]. In addition to exploring training data, some studies redesigned the discriminator via proposing extra tasks [11, 79, 29, 88] and implementing large-scale pre-trained models [68, 41]. In this paper, the proposed method does not require additional training data, which can work as a plug-and-play module for existing discriminators. Compared with the previous regularization focusing on data augmentation, the proposed method investigates the capacity of the discriminator from a new perspective, i.e., 'augmenting' the architecture to quickly adapt the current (new) knowledge. Note that the most related study to this paper is DynamicD [87], which dynamically adapts the capacity of the discriminator to the training data. However, DynamicD suffers from divergent solutions on different data regimes. In contrast, our method can help to re-adjust the attentions of discriminators and obtain consistent improvement against varying regimes.
3 Pilot Study
We first conduct a study to investigate the distribution shifts of generated samples over time during training. We then demonstrate the challenges posed by such a time-varying distribution to the fixed discriminator. We adopt a representative GAN, i.e., StyleGAN-V2 [36], for the study, due to its advanced network architectures, training strategies, and impressive results.
Analysis on generated distribution over time. We train a StyleGAN-V2 on a widely-used dataset FFHQ [35], to study the distribution changes of its generated samples. In particular, we collect 5k generated samples at each time interval during training and then analyze the distribution of generated samples per time interval. The distribution is estimated with Fréchet Inception Distance (FID) [25]. More specifically, FID measures the distribution similarity between real and fake samples by
Figure 2: Illustration of time-varying distributions of generated samples, where we trace the training process of StyleGAN-V2 [36] on FFHQ [35]. The mean and variance of generated 5k samples' features are computed per time interval, showing the generated distributions are dynamic and time-varying during training, as the generator evolves. king refers to the number of images (measured in thousand) trained so far.
calculating the mean and covariance of real/fake samples. Similarly, we first extract the feature of a generated sample by an Inception model [77], and then represent the distribution of generated samples per time interval by the mean and variance of these samples' features.
We observe that the generated distributions undergo dynamical and complex changes over time (see Fig. 2), as the generator evolves during training. For example, the generated samples are not even independently and identically distributed (i.i.d) across the training progress in Fig. 2. Such time-varying distribution inevitably poses significant challenges to the discriminator, since this has not been explicitly considered in the discriminator typically.
Figure 3: We trace the training of StyleGAN-V2's discriminator. The curve in (a) shows the model parameter difference of the discriminator between  $t_i$  and  $t_{i-1}$  in the training progress. In (b), the attentive regions of the discriminator are almost fixed at training steps when the discriminator slows down its model parameter updating, where each feature map is represented by the feature space of the discriminator trained at a time step.
Adaptation ability of fixed discriminator. Learning the distributions of generated samples is vital to training the generator. We investigate whether the fixed discriminator, which does not explicitly consider distribution changes, can effectively learn the time-varying distributions of generated samples. We argue that the discriminator needs to adaptively adjust its model weights and rely on different local features for discriminating generated samples at different steps. Suppose the generator synthesizes unrealistic hair for human faces at a time interval  $[t_1, t_2]$ , and synthesizes plausible hair but introduces artifacts in cheek regions at  $[t_3, t_4]$  (see [1209, 4838] and [6451, 11289] kings in Fig. 2). It is expected that the discriminator pays less attention to hair regions and more attention to cheek regions in  $[t_3, t_4]$ , compared to  $[t_1, t_2]$ , so as to fast distinguish fake samples. Accordingly, the parameters of the discriminator model are expected to be updated, such that the discriminator adapts to the distribution changes of generated samples for  $[t_1, t_2]$  to  $[t_3, t_4]$ .
Here we investigate the discriminator of StyleGAN-V2 since it is a representative fixed discriminator. We find the parameter differences of the discriminator are usually small between two adjacent steps, (see Fig. 3 (a)), indicating the discriminator slows down learning. Moreover, we find the discriminator model trained at different times pays attention to almost fixed local regions given an image, as indicated by the feature map extracted by the layers of the discriminator (see Fig. 3 (b)). As a result, the discriminator relies on old knowledge learned from historical data, which is insufficient to learn incoming generated samples under a dynamic distribution shift.
Figure 4: The pipeline of the proposed method. Our method, named Dynamically Masked Discriminator (DMD), automatically adjusts the discriminator via dynamically marking the discriminator. When DMD detects that the discriminator slows down learning, DMD dynamically assigns masks or removes masks to features of the discriminator per time interval, forcing the discriminator to learn new knowledge and preventing it from relying on old knowledge from historical samples.
The discriminator learns the distribution of real and generated samples, which is vital to training the generator. Nevertheless, as studied in Sec. 3, the distribution of generated samples varies and drifts over time during training, posing challenges to existing discriminators (e.g., [36, 50]). By treating the generated samples of the generator across training as a stream, we innovatively formulate learning the distribution of generated samples as an online continual learning problem. Motivated by recent advances in online learning [17, 7, 47], our target is to enable a discriminator to quickly adapt to incoming generated data.
Since the generator is trained via a min-max game with the discriminator, the distribution of generated samples is significantly complex and unpredictable. Hence, it is non-trivial to accurately predict distribution changes. In other words, it is difficult for the discriminator to predict when it needs to be rapidly adjusted for the new distribution.
Overview. We address the problem by considering two questions: (1) when does the discriminator slow down the learning from incoming generated samples? (2) how to force the discriminator to fast learn new knowledge? By exploring the two aspects, we propose a method named Dynamically Masked Discriminator (DMD) for GANs, as shown in Fig. 4.
4.2 Dynamically Masked Discriminator
Different from existing GANs methods [36, 50, 33, 48], we propose to automatically adjust the discriminator during training towards the time-varying distribution of generated samples. Instead of designing complex network architectures, we propose two key modules which can be easily integrated into existing discriminators. The two modules are (1) discriminator retardation detection and (2) dynamic discriminator adjustment. The first module automatically determines when the discriminator slows down learning. Here, slow down is referred to as the discriminator largely relies on old knowledge learned from historical data, given incoming generated samples with different distributions. The second module adjusts the discriminator for fast learning via dynamically assigning or removing masks at a certain interval.
With the two proposed modules, we improve an existing discriminator into a dynamic one at the time axis. That is, the discriminator in our method has two states, i.e., mask and non-mask training.  $\mathcal{D}$  denotes the original discriminator without masks, and  $\bar{D}$  denotes the adjusted discriminator with masks. During training, we dynamically switch  $\mathcal{D}$  and  $\bar{D}$  for guiding the training of the generator
according to discriminator retardation detection. Let  $\pi(\mathcal{D}|t)$  describe the probability to use  $\mathcal{D}$  at time  $t$ . The probability of  $\bar{D}$  is thus formed as  $1 - \pi(\mathcal{D}|t)$ . Let  $\{\tilde{I}^t\}$  denote a stream consisting of samples generated by the generator  $\mathcal{G}$ , where  $\tilde{I}^t = \mathcal{G}(z, \theta^t)$  is a generated sample from a vector  $z$  at time  $t$  during training, and  $\theta^t$  denotes parameters of the generator at  $t$ . We formulate our training of GANs as follows:
$$
\mathcal {L} _ {\theta^ {t}} := - \mathbb {E} _ {z \sim p _ {z}, t} \left[ \pi (\mathcal {D} | t) \log (\mathcal {D} (\mathcal {G} (z, \theta^ {t}), \phi^ {t})) + (1 - \pi (\mathcal {D} | t)) \log (\mathcal {D} _ {M} (\mathcal {G} (z, \theta^ {t}), \phi^ {t}) \right] \tag {1}
$$
$$
\begin{array}{l} \mathcal {L} _ {\phi^ {t}} := - \mathbb {E} _ {I \sim p _ {I}, t} \left[ (1 - \pi (\mathcal {D} | t)) \log (\mathcal {D} _ {M} (I, \phi^ {t})) + \pi (\mathcal {D} | t) \log (\mathcal {D} (I, \phi^ {t})) \right] \\ - \mathbb {E} _ {z \sim p _ {z}, t} \left[ (1 - \pi (\mathcal {D} | t) \log \left(1 - \mathcal {D} _ {M} \left(\mathcal {G} \left(z, \theta^ {t}\right), \phi^ {t}\right)\right) + \pi (\mathcal {D} | t) \log \left(1 - \mathcal {D} \left(\mathcal {G} \left(z, \theta^ {t}\right), \phi^ {t}\right)\right) \right] \\ \end{array}
$$
where  $I$  is a real sample and  $\phi^t$  is the parameter of the discriminator at time  $t$ .
Dynamic discriminator adjustment. When the discriminator is detected to be retarded (i.e., slow down learning), we adjust the discriminator to force it to learn fast. In this paper, we mainly focus on the issue that the discriminator largely relies on old knowledge learned from historical data and cannot rapidly adapt to incoming generated samples. To address this issue, one possible solution is to design complex network architectures. Instead, we propose to dynamically switch the discriminator from mask/non-mask states to non-mask/mask at a time interval. We argue that such dynamic masking at time intervals can break the original dependency of the discriminator on some local features that are important to distinguish historical samples, inspired by [15, 24, 46, 49, 76]. For example, by masking a feature map that is fed to a layer of the discriminator, we can control the layer of the discriminator and enforce it to pay attention to other regions of a generated image.
In this paper, we force the non-mask discriminator (i.e., original one)  $\mathcal{D}$  with parameter  $\phi^{t_i}$  to be converted into the mask state  $\mathcal{D}_M$  by masking its feature map or input. In particular, given the  $d$ -th layer of a discriminator, we propose to mask its input  $\mathbf{F}^{(d - 1),t}$  in a time interval  $(t_j,t_{j + \epsilon}]$ . Let  $\mathbf{m}_{d - 1}^t$  denote a mask in the  $t$ -th training step with the same size as  $\mathbf{F}^{(d - 1),t}$ . We can dynamically mask the discriminator by masking the input features of its  $d$ -th layer:
$$
\bar {\mathbf {F}} ^ {(d), t} = L \left(\mathbf {F} ^ {(d - 1), t}, \mathbf {m} _ {d - 1} ^ {t}\right) = L \left(\mathbf {F} ^ {(d - 1), t} \odot \mathbf {m} _ {d - 1} ^ {t}\right), \quad t \in \left(t _ {j}, t _ {j + \epsilon} \right] \tag {3}
$$
where  $\bar{\mathbf{F}}^{(d),t}$  is the output of  $d$ -th layer of a discriminator by masking,  $\odot$  denotes Hadamard product,  $L$  is the convolutional operator. Different from the traditional dropout, the mask does not continuously change. Instead, this mask is fixed for a period of training steps, i.e.,  $(t_i,t_{i + \epsilon}]$ . Switching the discriminator from mask to non-mask is simply removing all masks in the discriminator. Such dynamical switching encourages the discriminator to pay attention to various local regions/features over time, making our discriminator better adapt to the time-varying generated distribution and leading to better guidance of training the generator, compared with StyleGAN-V2 (see Fig. 1).
Discriminator retardation detection. This module is to detect whether the discriminator slows down the learning, i.e., the discriminator largely relies on old knowledge from historical data to distinguish future generated samples with new distributions. Given the current time step  $t_i$ , an ideal solution is to use future-generated samples with different distributions at time  $(t_j, t_{j + \epsilon}]$  to evaluate the discriminator. However, these future-generated samples are unavailable at the current time step. Moreover, it is non-trivial to predict the distribution of future-generated samples.
Instead, we propose to construct a pseudo sample that possibly belongs to new distribution to detect the Retardation of the discriminator. Without loss of generality, we assume the distribution changes of generated samples correspond to the changes in local regions/features of these samples. For example, some hair regions are unrealistic in a generated sample at time  $t_i$ , and become realistic at  $t_j$ , as the generator evolves in Fig. 2. By randomly masking a generated sample or its intermediate features in the discriminator, it is possible to remove discriminative local regions/features that are important for the sample's distribution, shirting the sample to a new distribution. For example, if the right face region of the man is removed/masked in Fig. 5, the remaining regions become more realistic (i.e., belongs to a new distribution).
Therefore, given a generated sample  $\tilde{I}^{t_i}$  for a distribution, we construct a pseudo sample that possibly belongs to another distribution by randomly masking local image regions or intermediate features of the discriminator. For generated samples at time  $t$ , if the discriminator determines they are
Generated Image