SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {G} \mathcal {L} _ {G} ^ {\mathrm {d a}} + \lambda_ {g} \cdot \left(\mathcal {L} _ {G} ^ {\text {c o l o r}} + \mathcal {L} _ {G} ^ {\text {t r a n s l a t i o n}} + \mathcal {L} _ {G} ^ {\text {c u t o u t}}\right), \tag {12}
$$
where  $\mathcal{L}_{\hat{D}_{\mathrm{color}}}^{\mathrm{color}}$ ,  $\mathcal{L}_{\hat{D}_{\mathrm{translation}}}^{\mathrm{translation}}$ ,  $\mathcal{L}_{\hat{D}_{\mathrm{cutout}}}^{\mathrm{cutout}}$ ,  $\mathcal{L}_G^{\mathrm{color}}$ ,  $\mathcal{L}_G^{\mathrm{translation}}$ , and  $\mathcal{L}_G^{\mathrm{cutout}}$  are defined as:
$$
\begin{array}{l} \mathcal {L} _ {\hat {D} _ {\mathrm {c o l o r}}} ^ {\mathrm {c o l o r}} = \mathbb {E} _ {\mathbf {x}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\mathrm {c o l o r}} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} _ {\mathrm {c o l o r}} ^ {+} \| _ {2} ^ {2} \right] \\ + \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\text {c o l o r}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\text {c o l o r}} ^ {-} \| _ {2} ^ {2} \right], \tag {13} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\hat {D} _ {\mathrm {t r a n s l a t i o n}}} ^ {\mathrm {t r a n s l a t i o n}} = \mathbb {E} _ {\mathbf {x}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\mathrm {t r a n s l a t i o n}} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} _ {\mathrm {t r a n s l a t i o n}} ^ {+} \| _ {2} ^ {2} \right] \\ + \mathbb {E} _ {\mathbf {z}, \omega} \left[ \| \hat {D} _ {\text {t r a n s l a t i o n}} (T (G (\mathbf {z}); \omega), G (\mathbf {z})) - \omega_ {\text {t r a n s l a t i o n}} ^ {-} \| _ {2} ^ {2} \right], \tag {14} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {\hat {D} _ {\mathrm {c u t o u t}}} ^ {\mathrm {c u t o u t}} = \mathbb {E} _ {\mathbf {x}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\mathrm {c u t o u t}} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} _ {\mathrm {c u t o u t}} ^ {+} \| _ {2} ^ {2} \right] \\ + \mathbb {E} _ {\mathbf {z}, \omega} \left[ \| \hat {D} _ {\text {c u t o u t}} (T (G (\mathbf {z}); \omega), G (\mathbf {z})) - \omega_ {\text {c u t o u t}} ^ {-} \| _ {2} ^ {2} \right], \tag {15} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {G} ^ {\mathrm {c o l o r}} = \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\mathrm {c o l o r}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\mathrm {c o l o r}} ^ {+} \| _ {2} ^ {2} \right] \\ - \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\text {c o l o r}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\text {c o l o r}} ^ {-} \| _ {2} ^ {2} \right], \tag {16} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {G} ^ {\mathrm {t r a n s l a t i o n}} = \mathbb {E} _ {\mathbf {z}, \omega} \left[ \| \hat {D} _ {\mathrm {t r a n s l a t i o n}} (T (G (\mathbf {z}); \omega), G (\mathbf {z})) - \omega_ {\mathrm {t r a n s l a t i o n}} ^ {+} \| _ {2} ^ {2} \right] \\ - \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\text {t r a n s l a t i o n}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\text {t r a n s l a t i o n}} ^ {-} \| _ {2} ^ {2} \right], \tag {17} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {G} ^ {\mathrm {c u t o u t}} = \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\mathrm {c u t o u t}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\mathrm {c u t o u t}} ^ {+} \| _ {2} ^ {2} \right] \\ - \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} _ {\text {c u t o u t}} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} _ {\text {c u t o u t}} ^ {-} \| _ {2} ^ {2} \right], \tag {18} \\ \end{array}
$$
where  $\hat{D}_{\mathrm{color}} = \varphi_{\mathrm{color}} \circ \phi$ ,  $\hat{D}_{\mathrm{translation}} = \varphi_{\mathrm{translation}} \circ \phi$ , and  $\hat{D}_{\mathrm{cutout}} = \varphi_{\mathrm{cutout}} \circ \phi$  share the backbone  $\phi$  but differ in the heads  $\varphi_{\mathrm{color}}$ ,  $\varphi_{\mathrm{translation}}$ , or  $\varphi_{\mathrm{cutout}}$ , respectively. For the simplicity of notations, we write Equations (5) and (6) in the main text as the objective function.
C Ablation Studies
Self-supervised tasks. We introduce two non-adversarial self-supervised tasks for comparison. The first version is that the discriminator only learns self-supervision on real data, defined as:
$$
\mathcal {L} _ {\hat {D}} ^ {\mathrm {s s}} = \mathbb {E} _ {\mathbf {x} \sim p _ {\text {d a t a}} (\mathbf {x}), \boldsymbol {\omega} \sim p (\boldsymbol {\omega})} \left[ \| \hat {D} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} \| _ {2} ^ {2} \right]. \tag {19}
$$
The second version is that the discriminator learns self-supervised tasks on both real and generated data simultaneously, given by:
$$
\mathcal {L} _ {\hat {D}} ^ {\mathrm {s s}} = \mathbb {E} _ {\mathbf {x}, \boldsymbol {\omega}} \left[ \| \hat {D} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} \| _ {2} ^ {2} \right] + \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} \| _ {2} ^ {2} \right]. \tag {20}
$$
For both versions, the generator is encouraged to produce augmentation-recognizable data, as follows:
$$
\mathcal {L} _ {G} ^ {\mathrm {s s}} = \mathbb {E} _ {\mathbf {z} \sim p (\mathbf {z}), \omega \sim p (\omega)} \left[ \| \hat {D} (T (G (\mathbf {z}); \omega), G (\mathbf {z})) - \omega \| _ {2} ^ {2} \right]. \tag {21}
$$
According to the self-supervised task, we denote different approaches as SS (Equations (19) and (21)), SS+ (Equations (20) and (21)), and ASS (Equations (5) and (6), short for adversarial self-supervised learning, i.e., the proposed AugSelf-GAN).
Table 7: IS and FID of AugSelf-BigGAN with different self-supervised signals on CIFAR-10 and CIFAR-100 with full and limited training data. The self supervised signal to be predicted is marked by the symbol  $\checkmark$ . Notice that all methods adopt color, translation, and cutout as data augmentation.
Self-Supervised Signals	100% training data	20% training data	10% training data
color	trans.	cutout	IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
CIFAR-10	X	X	X	9.29±.02	8.48±.13	8.84±.12	15.14±.47	8.80±.01	20.60±.13
X	X	✓	9.30±.05	7.48±.07	8.94±.07	11.73±.27	8.73±.12	16.66±.39
X	✓	X	9.39±.07	7.51±.12	8.95±.05	11.82±.21	8.80±.03	16.27±.35
X	✓	✓	9.38±.05	7.41±.11	8.87±.04	11.19±.08	8.63±.08	16.30±.57
✓	X	X	9.50±.07	7.57±.07	8.99±.06	11.38±.14	8.72±.09	16.50±.14
✓	X	✓	9.41±.07	7.51±.05	8.92±.12	11.20±.17	8.83±.07	15.55±.21
✓	✓	X	9.42±.06	7.43±.06	8.91±.09	11.19±.20	8.58±.02	15.17±.15
✓	✓	✓	9.43±.14	7.68±.06	8.98±.09	10.97±.09	8.76±.05	15.68±.26
CIFAR-100	X	X	X	11.02±.07	11.49±.21	9.45±.05	24.98±.48	8.50±.09	34.92±.63
X	X	✓	11.16±.13	10.03±.14	10.18±.09	19.45±.62	9.46±.12	25.99±.62
X	✓	X	11.35±.10	9.88±.08	10.01±.07	18.39±.07	9.29±.10	26.50±.36
X	✓	✓	11.26±.07	9.75±.05	10.35±.20	16.88±.46	9.92±.08	23.09±.72
✓	X	X	11.20±.08	10.01±.13	9.90±.11	18.16±.62	9.39±.04	21.48±.14
✓	X	✓	11.17±.12	10.12±.20	10.14±.09	17.11±.62	9.94±.12	24.52±.76
✓	✓	X	11.26±.16	9.65±.04	10.21±.12	17.32±.51	9.92±.06	22.94±.17
✓	✓	✓	11.19±.09	9.88±.07	10.25±.06	16.11±.25	9.78±.08	21.30±.15
Table 8: IS and FID of AugSelf-BigGAN with different architectures and fusions of  $\varphi$  on CIFAR-10 and CIFAR-100 with  $10\%$  training data. The best result is bold and the second best is underlined. Input can be concatenation  $[\phi (\mathbf{x}),\phi (\hat{\mathbf{x}})]\in \mathbb{R}^{2d}$ , subtraction  $\phi (\hat{\mathbf{x}}) - \phi (\mathbf{x})$ , and augmentation  $\phi (\hat{\mathbf{x}})$ .
Architecture	Input	CIFAR-10 10% data	CIFAR-100 10% data
IS (↑)	FID (↓)	IS (↑)	FID (↓)
two-layer MLP	concatenation	8.54±0.07	18.47±0.37	8.65±0.11	30.15±0.47
two-layer MLP	subtraction	8.62±0.07	16.94±0.49	8.88±0.06	28.63±0.36
two-layer MLP	augmentation	8.78±0.05	19.06±0.81	8.73±0.10	29.40±0.91
linear layer	concatenation	8.85±0.11	17.52±0.20	9.37±0.09	25.22±0.25
linear layer	subtraction	8.76±0.05	15.68±0.26	9.78±0.08	21.30±0.15
linear layer	augmentation	8.66±0.09	20.29±0.36	9.92±0.10	24.47±0.58
bilinear layer	-	8.57±0.04	25.27±0.16	9.03±0.03	26.72±0.17
Table 6 reports the comparison between methods with different self-supervised tasks. We also conducted generator-free self-supervised learning experiments by setting the hyper-parameter as  $\lambda_g = 0$  on each kind of self-supervised task. According to the FID score, ASS is significantly superior to SS and  $\mathrm{SS + }$ . Even without the self-supervised task of the generator, ASS ( $\lambda_g = 0$ ) outperforms SS and  $\mathrm{SS + }$  that include generator self-supervised tasks in limited (10% and 20%) data, and the introduction of generator self-supervised tasks further expands this advantage.
Self-supervised signals. We empirically analyze the role of different augmentation parameters as self-supervised signals for AugSelf-GAN. All methods employ three types of data augmentations (color, translation, and cutout), with the difference lying in the predicted self-supervised signals. According to the FID scores in Table 7, predicting any augmentation parameters can significantly improve the final generative performance compared to the baseline. Although there is no significant difference among them, we still choose to predict all augmentation parameters as the default setting for AugSelf-GAN. This experiment demonstrates that the self-supervised task itself plays a decisive role in training AugSelf-GAN, rather than the specific predicted augmentations. Therefore, we believe that our method is generalizable and can be extended to other advanced data augmentations.
Network architectures. We investigate the impact of different network architectures (two-layer multi-layer perceptron (MLP), linear layer, and bilinear layer) and input approaches (concatenation  $[\phi(\mathbf{x}),\phi(\hat{\mathbf{x}})]$ , subtraction  $\phi(\hat{\mathbf{x}})-\phi(\mathbf{x})$ , and augmented samples only  $\phi(\hat{\mathbf{x}})$ ) on AugSelf-GAN. As
Table 9: IS and FID of AugSelf-BigGAN with different loss functions on CIFAR-10 and CIFAR-100 with full and limited training data. The best result is bold and the second best is underlined.
Method	100% training data	20% training data	10% training data
IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
CIFAR-10	λd=0,λg=0	9.29±.02	8.48±.13	8.84±.12	15.14±.47	8.80±.01	20.60±.13
λd=1,λg=0	9.12±.05	8.90±.07	8.82±.03	13.65±.70	8.52±.04	18.14±.33
saturating	9.27±.06	8.13±.14	8.83±.04	12.76±.29	8.42±.05	18.43±.21
non-saturating	9.37±.07	7.60±.08	8.91±.09	11.44±.16	8.63±.10	15.86±.26
combination	9.43±.14	7.68±.06	8.98±.09	10.97±.09	8.76±.05	15.68±.26
CIFAR-100	λd=0,λg=0	11.02±.07	11.49±.21	9.45±.05	24.98±.48	8.50±.09	34.92±.63
λd=1,λg=0	10.82±.10	11.29±.12	9.96±.11	18.90±.41	9.45±.05	25.77±.95
saturating	10.90±.07	10.53±.21	9.71±.08	18.78±.43	9.30±.09	25.92±.12
non-saturating	11.22±.14	9.90±.09	10.07±.02	16.12±.26	9.81±.10	21.99±.87
combination	11.19±.09	9.88±.07	10.25±.06	16.11±.25	9.78±.08	21.30±.15
reported in Table 8, we found that more complicated architectures of the predict head  $\varphi$  such as two-layer MLP with input concatenation  $(\hat{D} (T(\mathbf{x},\omega),\mathbf{x}) = \mathrm{MLP}([\phi (T(\mathbf{x},\omega)),\phi (\mathbf{x})])$  or bilinear layer  $(\hat{D} (T(\mathbf{x};\omega),\mathbf{x}) = \phi (T(\mathbf{x};\omega))^{\top}W\phi (\mathbf{x}),W\in \mathbb{R}^{d\times d})$  works worse than the simple linear layer with input subtraction  $(\hat{D} (T(\mathbf{x};\omega),\mathbf{x}) = \varphi (\phi (T(\mathbf{x};\omega)) - \phi (\mathbf{x})))$  , which is the design of AugSelf-GAN. The reason might be that complicated architectures of the head  $\varphi$  actually discourage the backbone  $\phi$  from capturing rich and linear representations, resulting in poor generalization of discriminators.
Generator loss functions. We study the effects of AugSelf-GANs with different generator loss functions. As shown in Table 9, The hyper-parameters  $\lambda_{d} = 0,\lambda_{g} = 0$  represent the baseline BigGAN + DiffAugment. The generation performance is improved with only augmentation-aware self-supervision for the discriminator  $(\lambda_d = 1,\lambda_g = 0)$ . The saturating version of self-supervised generator loss shows no significant further improvement. The non-saturating version yields comparable performance with the combination one (AugSelf-GAN), and they both substantially surpass the others. The reason is that the non-saturating loss directly encourages the generator to generate augmentation-predictable real data, providing more informative guidance than the saturating one.
D Additional Results
Figure 7: FID plots comparison of AugSelf-GAN and baselines on CIFAR-10 and CIFAR-100.
Table 10: FID comparison on AFHQ. The number within the parentheses represents the percentage of improvement  $(\downarrow)$  in AugSelf-StyleGAN2 compared to the baseline StyleGAN2 + DiffAugment.
Method	Cat	Dog	Wild
StyleGAN2 [19]	5.13	19.4	3.48
+ DiffAugment [47]	3.49	8.75	2.69
AugSelf-StyleGAN2	3.23 (↓ 7.45%)	8.17 (↓ 6.63%)	2.48 (↓ 7.81%)
StyleGAN2 + DiffAugment
Ccat
AugSelf-StyleGAN2
Doos
PAA
Figure 8: Visual comparison between StyleGAN2+DiffAugment and AugSelf-StyleGAN2 on AFHQ.
Training stability. Figure 7 shows the FID plots during training of our method and baselines on CIFAR-10 and CIFAR-100. Overall, AugSelf-BigGAN demonstrates a more stable convergence compared to the baselines, while AugSelf-BigGAN+ goes even further.
AFHQ We also conduct experiments on the AFHQ dataset [6] to compare our proposed method with DiffAugment by employing StyleGAN2 as the backbone model. The hyperparameters are set as  $\lambda_{d} = 0.1$ ,  $\lambda_{d} = 0.5$ , and  $\lambda_{d} = 0.2$  on Cat, Dog, and Wild domains, respectively, and  $\lambda_{g} = 0.1$  on all three domains. Quantitative experimental results, as shown in Table 10, suggest that AugSelf-StyleGAN2 yields approximately a  $7\%$  improvements in FID across all three domains when compared to DiffAugment. Figure 8 presents a qualitative comparison between these two methods, demonstrating a significant improvement in image quality for our method.
FFHQ and LSUN-Cat Figures 9 and 10 present the randomly generated images produced by StyleGAN2 + DiffAugment and AugSelf-StyleGAN2+ on the FFHQ [17] and LSUN-Cat [45] datasets with varying amounts of training data, respectively. Our AugSelf-StyleGAN2+ demonstrates significant superiority over StyleGAN2 + DiffAugment in terms of visual quality of generated images.
StyleGAN2 + DiffAugment
1k training samples
AugSelf-StyleGAN2+
5k training samples
10k training samples
30k training samples
Figure 9: Comparison of random generated images between StyleGAN2 + DiffAugment and AugSelf-StyleGAN2+ on FFHQ with different amounts of training data.
30k training samples
Figure 10: Comparison of randomly generated images between StyleGAN2 + DiffAugment and AugSelf-StyleGAN2+ on LSUN-Cat with different amounts of training data.