SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $M_{\mathrm{AH}}(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) + M_{\mathrm{AH}}(p_G(\mathbf{x}, \hat{\mathbf{x}}) \| p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}})) = \Delta(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}}))$
-  $M_{\mathrm{AH}}(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) = 1 - W(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) \leq 1$
where  $\Delta$  is the Le Cam (LC) divergence [23], and  $W$  is the harmonic mean divergence [37].
Corollary 1 reveals an inequality  $M_{\mathrm{AH}}(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) \leq \Delta(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}}))$  because of non-negativity of AHM divergence  $M_{\mathrm{AH}}(p_G(\mathbf{x}, \hat{\mathbf{x}}) \| p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}})) \geq 0$ . Figure 4 plots the function  $f$  in AHM divergence and other common  $f$ -divergences in the GAN literature. The AHM divergence shows better robustness of the function  $f$  than others for extremely large inputs  $p(\mathbf{x}) / q(\mathbf{x}) = D^{*}(\mathbf{x}) / (1 - D^{*}(\mathbf{x}))$ , which is likely for the optimal discriminator  $D^{*}$  in data-limited scenarios.
Goodfellow et al. [10] analyzed that saturated GAN optimizes the Jensen Shannon (JS) divergence, but in fact it uses non-saturated loss. LeCam-GAN [40] showed a connection between the Le Cam (LC) divergence [23] and its objective function based on fixed regularization, but in practice it uses exponential moving average.
3The theoretical analysis of LS-GAN is actually inconsistent with its generator loss function.
Table 1: IS and FID comparisons of AugSelf-BigGAN with state-of-the-art methods on CIFAR-10 and CIFAR-100 with full and limited data. The best result is bold and the second best is underlined.
Method	100% training data	20% training data	10% training data
IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
CIFAR-10	BigGAN [2]	9.07	9.59	8.52	21.58	7.09	39.78
DiffAugment [47]	9.16	8.70	8.65	14.04	8.09	22.40
CR-GAN [46]	9.17	8.49	8.61	12.84	8.49	18.70
LeCam-GAN [40]	9.43	8.28	8.83	12.56	8.57	17.68
DigGAN [8]	9.28	8.49	8.89	13.01	8.32	17.87
Tickets [3]	-	8.19	-	12.83	-	16.74
MaskedGAN [14]	-	8.41±.03	-	12.51±.09	-	15.89±.12
GenCo [7]	-	7.98±.02	-	12.61±.05	-	18.10±.06
AugSelf-BigGAN	9.43±.14	7.68±.06	8.98±.09	10.97±.09	8.76±.05	15.68±.26
AugSelf-BigGAN+	9.27±.05	7.54±.04	9.08±.04	9.95±.17	8.79±.04	12.76±.14
CIFAR-100	BigGAN [2]	10.71	12.87	8.58	33.11	6.74	66.71
DiffAugment [47]	10.66	12.00	9.47	22.14	8.38	33.70
CR-GAN [46]	10.81	11.25	9.12	20.28	8.70	26.90
LeCam-GAN [40]	11.05	11.20	9.81	18.03	9.27	27.63
DigGAN [8]	11.45	11.63	9.54	19.79	8.98	24.59
Tickets [3]	-	10.73	-	17.43	-	23.80
MaskedGAN [14]	-	11.65±.03	-	18.33±.09	-	24.02±.12
GenCo [7]	-	10.92±.02	-	18.44±.04	-	25.22±.06
AugSelf-BigGAN	11.19±.09	9.88±.07	10.25±.06	16.11±.25	9.78±.08	21.30±.15
AugSelf-BigGAN+	11.12±.10	10.09±.05	10.14±.11	15.33±.20	9.93±.06	18.64±.09
Table 2: FID comparison of AugSelf-StyleGAN2 with competing methods on FFHQ and LSUN-Cat with limited training samples. The best result is highlighted in bold and the second best is underlined.
Method	FFHQ	LSUN-Cat
30K	10K	5K	1K	30K	10K	5K	1K
StyleGAN2 [19]	6.16	14.75	26.60	62.16	10.12	17.93	34.69	182.85
+ ADA [18]	5.46	8.13	10.96	21.29	10.50	13.13	16.95	43.25
+ DiffAugment [47]	5.05	7.86	10.45	25.66	9.68	12.07	16.11	42.26
AugSelf-StyleGAN2	4.95	6.98	9.69	23.38	9.22	11.98	14.86	36.76
AugSelf-StyleGAN2+	5.82	6.65	9.15	20.39	9.43	12.00	14.12	26.52
We implement AugSelf-GAN based on DiffAugment [47], keeping the backbones and settings unchanged for fair comparisons with prior work under two evaluation metrics, IS [34] and FID [11]. The mean and standard deviation (if reported) are obtained with five evaluation runs at the best FID checkpoint. Each of experiments in this work was conducted on an 32GB NVIDIA V100 GPU.
6.1 Comparison with State-of-the-Art Methods
CIFAR-10 and CIFAR-100. Table 1 reports the results on CIFAR-10 and CIFAR-100 [21]. These experiments are based on the BigGAN architecture [2]. Our method significantly outperforms the direct baseline DiffAugment [47] and yields the best generation performance in terms of FID and IS compared with SOTA methods. Notably, our method achieves further improvement when using stronger augmentation (see Table 5), i.e., AugSelf-BigGAN+ (translation↑ and cutout↑).
FFHQ and LSUN-Cat. Table 2 reports the FID results on FFHQ [17] and LSUN-Cat [45]. The hyper-parameter is  $\lambda_{g} = 0.2$ . AugSelf-GAN performs substantially better than baselines with the same network backbone. Also, stronger augmentation, i.e., AugSelf-StyleGAN2+ (translation↑ and cutout↑), further improves the performance when training data is very limited.
D
Obama
AnimalFace Cat
AnimalFace Dog
Figure 5: Qualitative comparison between DiffAug (DA) and AugSelf (AS) on low-shot generation.
Table 3: FID comparison of AugSelf-StyleGAN2 with state-of-the-art methods with and without pre-training on five low-shot datasets. The best result is in **bold** and the second best is **underlined**.
Method	Pre-training?	100-shot	AnimalFaces
Obama	Grumpy cat	Panda	Cat	Dog
Scale/shift [31]	Yes	50.72	34.20	21.38	54.83	83.04
MineGAN [42]	Yes	50.63	34.54	14.84	54.45	93.03
TransferGAN [41]	Yes	48.73	34.06	23.20	52.61	82.38
FreezeD [30]	Yes	41.87	31.22	17.95	47.70	70.46
StyleGAN2 [19]	No	80.20	48.90	34.27	71.71	130.19
+ AdvAug [3]	No	52.86	31.02	14.75	47.40	68.28
+ ADA [18]	No	45.69	26.62	12.90	40.77	56.83
+ APA [16]	No	42.97	28.10	19.21	42.60	81.16
+ DiffAugment [47]	No	46.87	27.08	12.06	42.44	58.85
AugSelf-StyleGAN2	No	26.00	19.81	8.36	30.53	48.19
Low-shot image generation. Table 3 shows the FID scores on five common low-shot image generation benchmarks [36] (Obama, Grumpy cat, Panda, AnimalFace cat, and AnimalFace dog). The baselines are divided into two categories according to whether they were pre-trained. Due to its training stability, we can train AugSelf-StyleGAN2 for 5k generator update steps to ensure convergence. The hyper-parameters are  $\lambda_d = \lambda_g = 0.1$  on Grumpy cat and AnimalFace cat, and the self-supervision is color on all datasets. Impressively, AugSelf-StyleGAN2 surpasses competing methods by a large margin on all low-shot datasets, approaching SOTA FID scores to our knowledge. Figure 5 visually compares the generated images of DiffAugment and AugSelf-GAN, revealing the latter as superior in generating images with more diversity and fewer artifacts.
6.2 Analysis of AugSelf-GAN
Fixed supervision. Table 4 reports the results of AugSelf-GAN on CIFAR-10 and CIFAR-100 compared to the setup using fixed self-supervision, i.e.,  $\omega^{+} = -\omega^{-} = 1$ , which corresponds to the assumption in Theorem 1. AugSelf-GAN outperforms the fixed self-supervision in terms of both IS and FID in all training data regimes. The reason is somewhat intuitive, as fixed self-supervision does not constitute self-supervised learning for the model, therefore it cannot enable the discriminator to learn semantic information related to data augmentation to optimize the generator. Interestingly, the fixed one beats the baseline, which may be attributed to the multi-input [27] and regression loss [29].
Table 4: FID comparison with fixed self-supervision. The best results are highlighted in bold.
Method	100% training data	20% training data	10% training data
IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
O	Fixed (c = 1)	9.25±.17	8.01±.05	8.70±.12	12.58±.16	8.53±.05	17.66±.55
AugSelf-GAN	9.43±.14	7.68±.06	8.98±.09	10.97±.09	8.76±.05	15.68±.26
O	Fixed (c = 1)	10.67±.06	12.02±.07	9.94±.06	17.70±.17	9.50±.13	22.84±.28
AugSelf-GAN	11.19±.09	9.88±.07	10.25±.06	16.11±.25	9.78±.08	21.30±.15
Table 5: Study on stronger augmentation. The best is in bold and the second best is underlined.
Method	100% training data	20% training data	10% training data
IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
C10	DiffAugment	9.29±.02	8.48±.13	8.84±.12	15.14±.47	8.80±.01	20.60±.13
+ trans.↑cut.↑	9.28±.06	8.42±.18	8.78±.06	14.28±.27	8.69±.07	20.93±.21
AugSelf-GAN	9.43±.14	7.68±.06	8.98±.09	10.97±.09	8.76±.05	15.68±.26
+ trans.↑cut.↑	9.27±.05	7.54±.04	9.08±.04	9.95±.17	8.79±.04	12.76±.14
C100	DiffAugment	11.02±.07	11.49±.21	9.45±.05	24.98±.48	8.50±.09	34.92±.63
+ trans.↑cut.↑	11.10±.08	11.28±.20	9.58±.05	24.10±.66	8.59±.04	35.32±.46
AugSelf-GAN	11.19±.09	9.88±.07	10.25±.06	16.11±.25	9.78±.08	21.30±.15
+ trans.↑cut.↑	11.12±.10	10.09±.05	10.14±.11	15.33±.20	9.93±.06	18.64±.09
Figure 6: FID curves with varying hyper-parameters  $\lambda = \lambda_d = \lambda_g \in [0,10]$  on CIFAR-10 and CIFAR-100. The hyper-parameter  $\lambda = 0$  corresponds to the baseline BigGAN + DiffAugment.
Stronger augmentation. Translation and cutout actually erase parts of image information, which help prevent the discriminator from overfitting, but could suffer from underfitting if excessive. Our self-supervised task enables the discriminator to be aware of different levels of translation and cutout, which helps alleviate underfitting and allows us to explore stronger translation and cutout. Table 5 compares AugSelf-GAN with DiffAugment in this setting. Overall, when data is limited, AugSelf-GAN can further benefit from stronger translation and cutout and achieve new SOTA FID results, while DiffAugment cannot. This implicitly indicates that our method enables the model to learn meaningful features to overcome underfitting, even under strong data augmentation.
Hyper-parameters. Figure 6 plots the FID results of AugSelf-GAN with different hyper-parameters  $\lambda = \lambda_{d} = \lambda_{g}$  ranging from [0, 10] on CIFAR-10 and CIFAR-100. Notice that  $\lambda = 0$  corresponds to the baseline BigGAN + DiffAugment. AugSelf-BigGAN performs the best when  $\lambda$  is near 1. It is worth noting that AugSelf-BigGAN outperforms the baseline even for  $\lambda = 10$  with  $10\%$  and  $20\%$  training data, demonstrating superior robustness with respect to the hyper-parameter  $\lambda$ .
This paper proposes a data-efficient GAN training method by utilizing augmentation parameters as self-supervision. Specifically, a novel self-supervised discriminator is proposed for predicting the augmentation parameters and data authenticity of augmented (real and generated) data simultaneously, given the original data. Meanwhile, the generator is encouraged to generate real rather than fake data of which augmentation parameters can be recognized by the self-supervised discriminator after augmentation. Theoretical analysis reveals a connection between the optimization objective of the generator and the arithmetic - harmonic mean divergence under certain assumptions. Experiments on data-limited benchmarks demonstrate superior qualitative and quantitative performance of the proposed method compared to previous methods.
Limitations. In our experiments, we observed less significant improvement of AugSelf-GAN under sufficient training data. Furthermore, its effectiveness depends on the specific data augmentation used. In some cases, inappropriate data augmentation may limit the performance gain.
Broader impacts. This work aims at improving GANs under limited training data. While this may result in negative societal impacts, such as lowering the threshold of generating fake content or exacerbating bias and discrimination due to data issues, we believe that these risks can be mitigated. By establishing ethical guidelines for users and exploring fake content detection techniques, one can prevent these undesirable outcomes. Furthermore, this work contributes to the overall development of GANs and even generative models, ultimately promoting their potential benefits for society.
We thank the anonymous reviewers for their valuable and constructive feedback. This work is funded by the National Natural Science Foundation of China under Grant Nos. 62272125, 62102402, U21B2046. Huawei Shen is also supported by Beijing Academy of Artificial Intelligence (BAAI).
[1] Gulcin Baykal, Furkan Ozcelik, and Gozde Unal. Exploring deshufflegans in self-supervised generative adversarial networks. Pattern Recognition, 2022. ISSN 0031-3203.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[3] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. In Advances in Neural Information Processing Systems, 2021.
[4] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[5] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. {GAN}s can play lottery tickets too. In International Conference on Learning Representations, 2021.
[6] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[7] Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genco: Generative co-training for generative adversarial networks with limited data. Proceedings of the AAAI Conference on Artificial Intelligence, 2022. doi: 10.1609/aaai.v36i1.19928.
[8] Tiantian Fang, Ruoyu Sun, and Alex Schwing. Diggan: Discriminator gradient gap regularization for gan training with limited data. In Advances in Neural Information Processing Systems, 2022.
[9] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
[12] Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised gans with label augmentation. In Advances in Neural Information Processing Systems, 2021.
Proof. The objective function of the self-supervised discriminator can be written as follows:
$$
\begin{array}{l} \mathcal {L} _ {\hat {D}} ^ {\mathrm {s s}} = \mathbb {E} _ {\mathbf {x}, \boldsymbol {\omega}} \left[ \| \hat {D} (T (\mathbf {x}; \boldsymbol {\omega}), \mathbf {x}) - \boldsymbol {\omega} ^ {+} \| _ {2} ^ {2} \right] + \mathbb {E} _ {\mathbf {z}, \boldsymbol {\omega}} \left[ \| \hat {D} (T (G (\mathbf {z}); \boldsymbol {\omega}), G (\mathbf {z})) - \boldsymbol {\omega} ^ {-} \| _ {2} ^ {2} \right] \\ = \iiint \left[ p _ {\mathrm {d a t a}} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) \| \hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {+} \| _ {2} ^ {2} + p _ {G} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) \| \hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {-} \| _ {2} ^ {2} \right] \mathrm {d} \mathbf {x} \mathrm {d} \boldsymbol {\omega} \mathrm {d} \hat {\mathbf {x}}. \\ \end{array}
$$
Minimizing the integral objective is equivalent to minimizing the objective on each data point:
$$
\mathcal {L} _ {\hat {D}} ^ {\mathrm {s s}} (\mathbf {x}, \hat {\mathbf {x}}) = \int \left[ p _ {\mathrm {d a t a}} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) \| \hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {+} \| _ {2} ^ {2} + p _ {G} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) \| \hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {-} \| _ {2} ^ {2} \right] \mathrm {d} \boldsymbol {\omega}.
$$
Getting its derivative with respect to the self-supervised discriminator and let it equals to 0, we have the optimal self-supervised discriminator as:
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\hat {D}} ^ {\mathrm {s s}}}{\partial \hat {D} (\hat {\mathbf {x}}, \mathbf {x})} = \int \left[ p _ {\mathrm {d a t a}} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) 2 (\hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {+}) + p _ {G} (\mathbf {x}, \boldsymbol {\omega}, \hat {\mathbf {x}}) 2 (\hat {D} (\hat {\mathbf {x}}, \mathbf {x}) - \boldsymbol {\omega} ^ {-}) \right] \mathrm {d} \boldsymbol {\omega} = 0 \\ \Rightarrow \hat {D} ^ {*} (\hat {\mathbf {x}}, \mathbf {x}) = \frac {\int p _ {\mathrm {d a t a}} (\mathbf {x} , \boldsymbol {\omega} , \hat {\mathbf {x}}) \boldsymbol {\omega} ^ {+} \mathrm {d} \boldsymbol {\omega} + \int p _ {G} (\mathbf {x} , \boldsymbol {\omega} , \hat {\mathbf {x}}) \boldsymbol {\omega} ^ {-} \mathrm {d} \boldsymbol {\omega}}{p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})}. \\ \end{array}
$$
Proof. The objective function of the self-supervised task for the generator can be written as follows:
where  $c = \| c\| _2^2$  is a constant scalar for the constant vector  $\pmb {c} = \pmb{\omega}^{+} = -\pmb{\omega}^{-}$  . And we have the optimal generator  $p_{G}(\mathbf{x},\hat{\mathbf{x}}) = p_{\mathrm{data}}(\mathbf{x},\hat{\mathbf{x}})\Rightarrow p_{G}(\mathbf{x}) = p_{\mathrm{data}}(\mathbf{x})$  to minimize the AHM divergence.
Table 6: IS and FID of AugSelf-BigGAN with different self-supervised tasks (SS: Equations (19) and (21); SS+: Equations (20) and (21); ASS: Equations (5) and (6)) on CIFAR-10 and CIFAR-100 with full and limited training data. The best is bold and the second best is underlined.
Method	100% training data	20% training data	10% training data
IS (↑)	FID (↓)	IS (↑)	FID (↓)	IS (↑)	FID (↓)
CIFAR-10	Baseline	9.29±0.02	8.48±0.13	8.84±0.12	15.14±0.47	8.80±0.01	20.60±0.13
SS (λg=0)	9.28±0.06	8.30±0.12	8.86±0.09	13.96±0.19	8.62±0.09	20.94±0.42
SS	9.29±0.07	8.24±0.10	8.98±0.04	13.42±0.36	8.69±0.05	19.40±0.28
SS+ (λg=0)	9.25±0.06	8.11±0.18	8.81±0.05	13.86±0.31	8.76±0.06	19.52±0.24
SS+	9.26±0.04	8.26±0.28	8.85±0.05	13.81±0.27	8.63±0.05	19.85±0.18
ASS (λg=0)	9.12±0.05	8.90±0.07	8.82±0.03	13.65±0.70	8.52±0.04	18.14±0.33
ASS	9.43±0.14	7.68±0.06	8.98±0.09	10.97±0.09	8.76±0.05	15.68±0.26
CIFAR-100	Baseline	11.02±0.07	11.49±0.21	9.45±0.05	24.98±0.48	8.50±0.09	34.92±0.63
SS (λg=0)	11.04±0.09	11.48±0.43	9.81±0.07	21.17±0.19	9.11±0.09	30.48±0.57
SS	10.96±0.11	11.04±0.15	9.99±0.09	20.72±0.33	9.23±0.07	31.54±1.12
SS+ (λg=0)	10.84±0.09	11.48±0.13	9.95±0.07	21.22±0.57	9.14±0.05	31.02±1.00
SS+	10.94±0.10	10.94±0.12	9.88±0.06	22.72±0.37	9.23±0.14	31.40±0.22
ASS (λg=0)	10.82±0.10	11.29±0.12	9.96±0.11	18.90±0.41	9.45±0.05	25.77±0.95
ASS	11.19±0.09	9.88±0.07	10.25±0.06	16.11±0.25	9.78±0.08	21.30±0.15
-  $M_{\mathrm{AH}}(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) + M_{\mathrm{AH}}(p_G(\mathbf{x}, \hat{\mathbf{x}}) \| p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}})) = \Delta(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}}))$
-  $M_{\mathrm{AH}}(p_{\mathrm{data}}(\mathbf{x},\hat{\mathbf{x}})\| p_G(\mathbf{x},\hat{\mathbf{x}})) = 1 - W(p_{\mathrm{data}}(\mathbf{x},\hat{\mathbf{x}})\| p_G(\mathbf{x},\hat{\mathbf{x}}))\leq 1$
Proof. We first prove the first corollary:
$$
\begin{array}{l} M _ {\mathrm {A H}} \left(p _ {\text {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {G} (\mathbf {x}, \hat {\mathbf {x}})\right) \\ \leq M _ {\mathrm {A H}} \left(p _ {\mathrm {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {G} (\mathbf {x}, \hat {\mathbf {x}})\right) + M _ {\mathrm {A H}} \left(p _ {G} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {\mathrm {d a t a}} (\mathbf {x}, \hat {\mathbf {x}})\right) \\ = \iint p _ {G} (\mathbf {x}, \hat {\mathbf {x}}) \frac {p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} + \iint p _ {\text {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \frac {p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {G} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} \\ = \iint p _ {G} (\mathbf {x}, \hat {\mathbf {x}}) \frac {p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} - \iint p _ {\mathrm {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \frac {p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} \\ = \iint \frac {\left(p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})\right) ^ {2}}{p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} \\ = \Delta (p _ {\text {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {G} (\mathbf {x}, \hat {\mathbf {x}})), \\ \end{array}
$$
where  $0 \leq \Delta(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) \leq 2$  is the Le Cam (LC) divergence [23].
The following proves the second corollary:
$$
\begin{array}{l} 0 \leq M _ {\mathrm {A H}} \left(p _ {\text {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {G} (\mathbf {x}, \hat {\mathbf {x}})\right) \\ = \iint p _ {G} (\mathbf {x}, \hat {\mathbf {x}}) \frac {p _ {G} (\mathbf {x} , \hat {\mathbf {x}}) - p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} \\ = \iint p _ {G} (\mathbf {x}, \hat {\mathbf {x}}) \left[ 1 - \frac {2 p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {\text {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \right] d \mathbf {x} d \hat {\mathbf {x}} \\ = 1 - \iint \frac {2 p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) p _ {G} (\mathbf {x} , \hat {\mathbf {x}})}{p _ {\mathrm {d a t a}} (\mathbf {x} , \hat {\mathbf {x}}) + p _ {G} (\mathbf {x} , \hat {\mathbf {x}})} \mathrm {d} \mathbf {x} \mathrm {d} \hat {\mathbf {x}} \\ = 1 - W \left(p _ {\text {d a t a}} (\mathbf {x}, \hat {\mathbf {x}}) \| p _ {G} (\mathbf {x}, \hat {\mathbf {x}})\right) \leq 1, \\ \end{array}
$$
where  $0 \leq W(p_{\mathrm{data}}(\mathbf{x}, \hat{\mathbf{x}}) \| p_G(\mathbf{x}, \hat{\mathbf{x}})) \leq 1$  is the well known harmonic mean divergence [37].
B Loss Functions
AugSelf-GAN adopts all kinds of augmentations of DiffAugment as the self-supervised signals by default, thus the self-supervised loss functions of the discriminator and the generator (Equations (5) and (6)) actually each comprise three sub self-supervised loss functions. Specifically, for AugSelf-GAN that uses color, translation, and cutout as self-supervision, the objective functions are:
$$
\min  _ {D, \hat {D} _ {\text {c o l o r}}, \hat {D} _ {\text {t r a n s l a t i o n}}, \hat {D} _ {\text {c u t o u t}}} \mathcal {L} _ {D} ^ {\mathrm {d a}} + \lambda_ {d} \cdot \left(\mathcal {L} _ {\hat {D} _ {\text {c o l o r}}} ^ {\text {c o l o r}} + \mathcal {L} _ {\hat {D} _ {\text {t r a n s l a t i o n}}} ^ {\text {t r a n s l a t i o n}} + \mathcal {L} _ {\hat {D} _ {\text {c u t o u t}}} ^ {\text {c u t o u t}}\right), \tag {11}
$$