SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experiments on historical knowledge retention of discriminator. We conduct experimental studies inspired by the study in online continual learning for rapid adaptation [7]. Given the discriminator trained from the beginning to time  $T$ , we evaluate its historical knowledge retention by evaluating its discrimination performance on historical data generated at time  $T - t$ , and evaluate its rapid adaptation by evaluating its performance in current data at time  $T$  and future data at  $T + t$ .
Table 9 shows that the discriminator of StyleGAN-V2 achieves high accuracy on historical data, however, performs much worse on current and future data. This shows that the discriminator of StyleGAN-V2 retains historical knowledge (i.e., high accuracy in historical data), yet, does not fast learn and adapt to the changes in the distribution of future data.
Algorithm 3 Dynamic Mask Discriminator Assert in Outcome Logits (Dynamic Head)
Require:   
Generator  $\mathcal{G}_{\theta^t}$  ; Discriminator  $\mathcal{D}_{\phi}(\cdot)$  ; Training Step  $t$    
Outcome Logit Number of  $\mathcal{D}_{\phi}(\cdot)$  and  $\bar{\mathcal{D}}_{\phi}(\cdot)k$  ; Dynamic Mask  $\mathbf{m}^t$  .   
 $d$  -th Layer of Discriminator  $\mathbf{F}^{(d),t}$  ; After Masking Outcome Logits  $\bar{\mathbf{F}}^{(d),t}$  .   
Predefined Threshold  $\lambda$  ; Retardation Metric  $R_{t}$  .   
Set  $U_{t}$  Containing m Samples for Calculating Retardation Metric;   
The number of training steps  $n_t$  ; The number of images per training step  $n_s$  .   
Ensure: Initialize  $R_{t}\gets 0$  and  $t\gets 1$  ; Random  $\mathbf{m}^t$  .   
1: while  $\theta$  has not converged do.   
2: for  $t = 1$  to  $n_t$  do   
3: if  $R_{t} > \lambda$  then   
4:  $\mathbf{M}^t\gets \mathbf{m}^t;\mathbf{M}_T\gets \mathbf{M}^t;$    
5: else   
6:  $\mathbf{M}^t\gets$  vector(1);  $\mathbf{M}_T\gets \mathbf{m}^{t + 1}$  .   
7: end if   
8: for  $s = 1$  to  $n_s$  do   
9:  $L_{\phi (t)}\gets -\mathbb{E}_{I,t}[log(\sum (D(I)\odot \mathbf{M}^t)))] - \mathbb{E}_{z\sim p_z,t}[log(1 - \sum (D(\mathcal{G}(z,\theta^t))\odot \mathbf{M}^t))]$    
10:  $\theta_{s}\gets$  Adam  $(\frac{\partial\phi(t)}{\partial\theta_{s - 1}})$    
11: end for   
12:  $\mathcal{R}_t = \frac{1}{m}\sum_{i\in U_t}\frac{\overline{\mathbf{F}}_i^{(d),t}\cdot\mathbf{F}_i^{(d),t}}{|\overline{\mathbf{F}}_i^{(d),t}||\mathbf{F}_i^{(d),t}|}$    
13: end for   
14: end while   
15: Return  $\theta$
Instead, by detecting retardation and masking the discriminator, our method enforces the discriminator to reduce retained historical knowledge (see decreased accuracy on historical data), while effectively encouraging it to fast learn and adapt to new knowledge of future data (i.e., our method achieves higher accuracy on current and future data than StyleGAN-V2).
Table 9: The accuracy of the discriminator on historical data and future data, where the discriminator is trained at 800 kings steps, and  $(\cdot)$  indicates the data at time  $T - t$  or  $T + t$ .
Method	Historical data (200)	Historical data (600)	Current data (800)	Future data (1000)
StyleGAN-V2	0.9696	0.9488	0.8236	0.604
Ours (StyleGAN-V2 + DMD)	0.9129	0.9229	0.8793	0.6543
In addition, we calculate the average gradient of the StyleGAN-V2 during the training phase in Table 10, showing the gradient does not vanish from 0 to 1200 kings.
Table 10: Average gradient of the StyleGAN-V2 during training.
kings	0	200	400	600	800	1000	1200
Gradient	3.5145e-08	4.6110e-07	-9.4235e-07	5.6989e-07	-4.3328e-07	9.2649e-08	1.2138e-06
Model parameter difference of the proposed method. We also compute the model parameter difference of the proposed method in the training of the AFHQ-Cat dataset. Our method detects the discriminator to be retarded at 1000 kings after 800 kings. Our method then applies the proposed DMD adjustment to train StyleGAN-V2 from 1000 to 1004 kings, which increases the model difference of our method by  $63.889\%$ , compared with the model difference between 800 to 1000 kings. Instead, the model difference of StyleGAN-V2's discriminator is decreased by  $2.778\%$  without our method. These results show StyleGAN-V2's discriminator slows down learning and our method effectively encourages the discriminator to fast learn.
**Computation cost.** Our method incurs a negligible increase in memory cost, since our method only additionally detects discriminator retardation and introduces feature masks. For computational cost, our method does not affect the inference time, since our method is designed for the discriminator and only the generator is used during inference. For training, our method introduces moderate time cost; however, this is affordable (e.g., increasing GPU number to reduce training time), and the additional training time cost depends on the time interval of estimating the retardation. Our method additionally introduces 6 hours 11 minutes for training on AFHQ in our experiments, when our method is integrated with StyleGAN-V2 and retardation is detected at every 4 kings. When detecting retardation at every 20 kings, our method only additionally takes 1.3 hours for training.
Additional Generated Distribution. Our main paper studies the generated distributions of StyleGAN-V2. Here, we additionally provide the distribution of the generated samples of APA [30] method on FFHQ[35] in Fig. 9. Fig. 9 also indicates that the generated distributions undergo dynamical and complex changes over time as the generator evolves during training. As a result, the generated samples are not independently and identically distributed (i.i.d) across the training progress, posing significant challenges in learning the generated distributions.
Distribution of Generated Images from APA (Mean & Variance)
Figure 9: Illustration of time-varying distributions of generated samples in the training process of APA [30] on FFHQ [35]. The mean and variance generated samples' features show the generated distributions are dynamic and time-varying during training, as the generator evolves.
D Error Bar
To evaluate the reproducibility of our method's results, we run our experiments three times using random seeds and the same hyper-parameters. Table 11 and Table 12 list the mean and variance of experimental results to show the error bar.
As shown in Table 11 and Table 12, our method performs stably on multiple datasets i.e., AFHQ-V2, FFHQ and LSUN-Church datasets, indicating the reproducibility of our method.
Table 11: Quantitative results of our method on AFHQ-V2 dataset [12], error bars are reported in terms of mean and variance, and Ours is StyleGAN-V2+DMD.
AFHQ-Cat	AFHQ-Dog	AFHQ-Wild
FID ↓	IS ↑	FID ↓	IS ↑	FID ↓	IS ↑
StyleGAN-V2	7.924	1.890	26.310	9.000	3.957	5.567
Ours (Reported in the paper)	5.879	1.988	21.240	9.698	3.471	5.647
Ours (re-run-1))	5.896	1.944	20.016	9.807	3.473	5.803
Ours (re-run-2))	6.015	1.987	20.456	10.291	3.420	5.699
Ours(Mean±Variance)	5.930±0.061	1.973±0.021	20.571±0.506	9.932±0.258	3.455±0.025	5.716±0.065
Table 12: Quantitative results of our method on FFHQ [35] and LSUN-Church [90], where error bars are reported in terms of mean and variance, and Ours is StyleGAN-V2+DMD.
LSUN-Church (126K)	FFHQ(70K)
FID ↓	IS ↑	FID ↓	IS ↑
StyleGAN-V2	4.292	2.589	3.810	5.185
Ours (Reported in the paper)	3.061	2.792	3.299	5.204
Ours (re-run-1))	3.025	2.795	3.177	5.225
Ours (re-run-2))	2.993	2.787	3.285	5.200
Ours(Mean±Variance)	3.026±0.028	2.791±0.003	3.254±0.055	5.210±0.011