Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models
Authors: Yong Zhong
Hongtao Liu
Xiaodong Liu
Fan Bao
Weiran Shen
Chongxuan Li
Venue: ICLR 2023 poster (2023)

Abstract: Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits. Inspired by the classical perspective of the bias-variance tradeoff, we propose regularized deep generative model (Reg-DGM), which leverages a nontransferable pre-trained model to reduce the variance of generative modeling with limited data. Formally, Reg-DGM optimizes a weighted sum of a certain divergence and the expectation of an energy function, where the divergence is between the data and the model distributions, and the energy function is defined by the pre-trained model w.r.t. the model distribution. We analyze a simple yet representative Gaussian-fitting case to demonstrate how the weighting hyperparameter trades off the bias and the variance. Theoretically, we characterize the existence and the uniqueness of the global minimum of Reg-DGM in a non-parametric setting and prove its convergence with neural networks trained by gradient-based methods. Empirically, with various pre-trained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs with limited data and achieves competitive results to the state-of-the-art methods. Our implementation is available at https://github.com/ML-GSAI/Reg-ADA-APA.

Published as a conference paper at ICLR 2023
DEEP GENERATIVE MODELING ON LIMITED DATA WITH REGULARIZATION BY NONTRANSFERABLE PRE-TRAINED MODELS
Yong Zhong Hongtao 2 Xiaodong Liu Fan Bao Weiran Shen 122 Chongxuan Li
1 Gaoling School of AI, Renmin University of China, Beijing, China Beijing Key Lab of Big Data Management and Analysis Methods, Beijing, China DDepartment of Computer Science Technology, Tsinghua University, Beijing, China fyongzhong, ht6, xiaodong. 1 1iuu@ruc.... edu.cn, cn, bf19@mails twingguu...... edu. cn, {shenweiran, chongxuanli eru.c.du..c..
ABSTRACT
Deep generative models (DGMs) are data-eager because learning complex model on limited data suffers from large variance and easily overfits. Inspired by the classical perspective the bias-variance tradeoff, we propose regularized deep generative model (Reg-DGM), which leverages nontransferable pre-trained model to reduce the variance of generative modeling with limited data. Formally, Reg-DGM optimizes weighted sum of certain divergence and the expectation an energy function, where the divergence between the data and the model dis- tributions, and the energy function is defined by the pre-trained model w.r.t. the model distribution. We analyze a simple yet representative Gaussian-fitting case demonstrate how the weighting hyperparameter trades off the bias and the vari- ance. Theoretically, we characterize the existence and the uniqueness of the global minimum of Reg-DGM in non- parametree setting and prove its convergence with neural networks trained by gradient- methods. Empirically, with various pre- trained feature extractors and a data-dependent energy function, Reg-DGM con- sistently improves the generation performance of strong DGMs with limited data and achieves competitive results the state-of-the-art methods. Our implementa- tion is available at https:: http::///gtthsse
WOOLL--SAIRRReS ADA-
INTRODUCTION
Deep generative models (DGMs) (Kingma & Welling 2013
Goodfellow et al. 2014; Sohl-
Dickstein et al.) 2015; Van den Oord al. Dinh et al. [016 Hinton & Salakhutdinov 2006)
employ neural networks to capture the underlying distribution of digh--imensional data and find
applications in various learning tasks (Kingma et al., 2014 Zhu et al. 2017 Razavi et 2019
Ramesh et al.) 2021: 2022: Ho et al. (022). Such models are often data-eager (Li et al. 2021
Wang et 2018) due to the presence of complex function classes. Recent work (Karras et al,, 2020a) found that the classical variants of generative adversarial networks (GANs) (Goodfellow ct al. 2014: Karras et al.) 2020b) produce poor samples with limited data, which shared by other DGMs in principle. Thus, improving the sample efficiency is common challenge for DGMs.
The root cause of the problem is that learning a model in a complex class on limited data suffers from large variance and easily overfits the training data (Mohri et al. 2018). To relieve the prob- lem, previous work either employed sophisticated data augmentation strategies (Zhao et al. 2020a
Karras et al. 2020a Jiang al. 2021), designed new losses for the discriminator in GANS (Cui et al. [0021] Yang et al. 2021). or transferred a pre-trained DGM (Wang et af.) 2018 Noguchi &
Harada 2019)] Mo al. 2020). Although not pointed out in the literature to our knowledge, prior work can be understood as reducing the variance of the estimate implicitly (Mohri et al. 2018). In
"Equal contribution.
1 Cooresponndnce Weiran Shen and Chongxuan Li.

---

Published as a conference paper at ICLR 2023
this perspective, we propose a complementary framework, named regularized deep generative model Beg, which employs a pre-trained model as regularization to achieve a better bias-variance tradeoff when training a DGM with limited data.
In Sec. 2 we formulate the objective function of regularization term weighted by hyperparameter. The divergence is between the data distribution and model distribution, and the regularization term can be understood as the negative expected log- likelihood of an energy-based model, whose energy function is defined by a pre-trained model, w.r.t. the model distribution. Intuitively, with an appropriate weighting hyperparameter, Reg-DGi balances between the data distribution and the pre-trained model achieve better bias-variance tradeoff with limited data, as validated by simple yet prototypical Gaussian-fittie example.
as the sum of a certain divergence and a
In Sec. 3 we characterize the optimization behavior of Reg-DGM in both non-parametric and parametric settings under mild regularity conditions. On one hand, we prove the existence and the uniqueness of the global minimum of the regularized optimization problem with the Kull- back-Leibler (KL) and Jensen-Shannon (JS) divergence in the non- non-parametric setting. On the other hand, we prove that, parameterized by a standard neural network architecture, with high probability to a global (or local) optimum trained by gradient-based methods.
converges
In Sec. we specify the components in Reg-DGM We employ strong variants of GANs (Karras et al.] 2020b al Jiang et al. [222]) as the base DGM for broader interests. We consider a nontransfer- able setting where the pre- pre-trained model does not necessarily have to be generative model. Indeed, employ several feature extractors, which trained for non-generative tasks such image clas- sification, representation learning, and face recognition. Notably, such models cannot be directly used in the fine-tuning approaches (Wang al. 2018 Mo et al. 2020). With comprehensive ab- lation study, we define our default energy function as the expected mean squared error between the features of the generated samples and the training data. Such an energy not only fits our theoretical analyses, but also results in consistent and significant improvements over baselines in all settings.
In Sec. 5 we present experiments on several benchmarks, including the FFHQ (Karras al. 2019), LSUN CAT (Yu et al. 2013), and CIFAR-10 (Krizhessky et al. 2009) datasets. We compare Reg- DGM with a large family of methods, including the base DGMs (Karras et al., 2020b a Jiang al., 2021]. the transfer-ban baaseer--aseddaee approaches (Wang et al., 2018: Mo et al. (020)) the augmentation-bans based methods (Zhao et al.) 2020a) and others (Cui et al. 2021 Yang et (02T). With a classifier pre- trained on ImageNet (Deng et al.] (2009), or an image encoder pre-trained on the CLIP dataset (Rad-] ford et al. 202I). or a face recognizer pre-trained on VGGFace2 (Cao et al.) 2018). Reg-DGM consistently improves strong DGMs under commonly used performance measures with limited data and achieves competitive results to the state-of-the-ar methods. Our results demonstrate that Reg- DGM can achieve good bias-variance tradeoff in practice, which supports motivation.
In Sec. 6
we present the related work. In Sec. 7 we conclude the paper and discuss limitations.
2
METHOD
The goal of generative modeling is learn a model distribution Py (implicitly explicitly) from a training set S = {x; y of size m on a sample space X. The elements in S are assumed to be drawn i.i.d. according to unknown data distribution ps Px: where Px is the set of all valid distributions over X. A general formulation for learning generative models is minimizing a certain statistical divergence D(-II-) between the two distributions follows:
min ((pallpg), P,H
where H C Px is the hypothesis class, for instance, a set of distributions defined by neural networks in deep generative model (DGM). Notably, the divergence in Eq. (L) is estimated by the Monte Carlo method over the training set S and its solution has a small bias if not zero. However, learning DGM with limited data is challenging because solving the problem in Eq. (1) with small sample size m essentially suffers from a large variance and probably overfits (Mohri et al. 2018).
Inspired by the classical perspective of the bias-variance tradeoff. we propose to leverage ternal model ff araitr on a related and large dataset (c.g.. ImageNet) as a data-dependent reg- ularization reduce the variance of training DGM on limited data (e.g., CIFAR- 10), which is
2

---

Published as a conference paper at ICLR 2023
0.000
6.001
0.001
0.012
PIIE
6/08
0.07
20106
1855-101
0.000
6/01
0001
00000 00
0.5
00.0
300
600
500
1000
120
0000
3.00
00.44
0.00
018
(a) Varying 8 A/(A + 1)
(b) Varying m
(c) Varying JARE - j
Figure I: MSE in the Gaussian- Fautsian-fitting example to validate Proposition 2.2 (a) The effect of given fixed and pArE μ², where the red circle indicates
m = (
and the black circle indicates 3 = min o 02+m -= ya, 1}. (b) The effect of the sample size m. (c) The effect of the bias of the pre-trained model. In both (b) and (c), the hyperparameter À is optimal.
max
20²
complementary to prior work (see more details in Sec. 66. In particular, given a pre-trained ʃ. we first introduce an energy function (LeCun al. 2006) Ef : X IR that satisfies mild regularity conditions (see Assumption A.I and then define a probabilistic energy-based model (EBM) Ps as (z()) exp(-E,(z))
We can treat py as special estimate of pa if f is pre-trained on a dataset closely related to Pa- In this perspective, Py has potentially large bias yet variance of zero. To trade off the bias and variance, we simply optimize a weighted sum of the statistical divergence as in Eq. (1) and the expected log-likelihood of the EBM as follows:
min (PPullp,)- - -E [ogg)(()) PgEN
<> min D(Pallng) + Xe (x)) P&EH
(2)
where > 0 is the weighting hyperparameter balancing the two terms. The first one encourages Pg to fit the data as in the original DGM and the second one encourages Pg to produce samples with a high likelihood evaluated by the EBM py- Naturally, a properly selected can hopefully achieve a better bias-variance tradeoff. We refer our approach regularized deep generative model (Reg-DGM) when H consists of distributions defined by neural networks.
In the following, we analyze a simple yet prototypical Gaussian-fitting example to demonstrate how a regularization term defined by a pre-trained model can relieve the bias-variance dilemma in generative modeling. Such an example is helpful to illustrate the motivation of Reg- Deg precisely and provide valuable insights on its practical performance. In the same spirit, representative prior work (Arjovsky al. [017 Mescheder et al. 2018) in the literature of DGMs has investigated similar examples with several parameters and closed-form solutions.
2.1 A PROTOTYPICAL GAUSSIAN- FIISSINN-FITTING EXAMPLE
Example 2.1 (Gaussian-fiitingg example). The data distribution is a (univariate) Gaussian pa(æ)
N(xlp",0 where o² is known and μ² is the parameter to be estimated. A training sample S {xr)m um is drawn i.i.d. according to Pa(x)) The hypothesis class for Pg is H [N(x)u x2) E
R). The regularization term in Eq. 22) is (z) == -log N (pARE, o²), Lee., pf(x) = NNolirre 0²).
Note that we let fiRRE + n² and their gap PAPRE 1e1 can be large in general.
For simplicity, we consider the classical maximum likelihood estimation (MLE) (i.e., using the KL divergence in Eq. ( as the performance measure), and its solution for Example 2.1 is given by the sample mean (Bishop & Nasrabadi, 2006): AAMLE
> Ii, AMLE ~ (n o²). The pre-trained model дiPEE Is another meanigfull baseline for our approach. Clearly, it has a bias of jUPRE nr and
variance. Formally, the solution of our approach based on MLE
is ÂREG +XXEMEE + TEXAPRE, ÂREG ~ N [ A TEXUPRE m(1+x]= We compare all estimators under the mean squared error (MSE which is a common measure in statistics and machine learning. Formally, the MSE of an estimator w.r.t. unknown parameter 0 is defined as:
The expected risk coincides with the MSE in the example. See Appendix A.2.2 for generalization analyses.
3

---

Published as a conference paper at ICLR 2023
2-01 Jatd
1-61 Jatd
16
00
0/gu
00
000
X
(a) Data distribution
(b) Energy function
(c) Varying À (KLD)
(d) Varying (JSD)
Illustration of the optimization analyses. (a) The density of the data distribution. (b) The energy function defined by the pre-trained model. (c) The global minimum of the problem Eq. 2 with the KL divergence (KLD) and different values of X. (d) The global minimum of the problem in Eq. ( with the IS divergence (JSD) and different values of X. See more details in Appendix B.1
Figure 2:
MSE 0 == Ei - d) which can be decomposed as the sum of the variance of the estimator and the squared bias of the estimator (Bishop Nasrabadi] (2066). As summarized Proposition [.2 our method achieves better Mlas-variance tradeoff than the baselines if À is in an appropriate range.
Proposition 2.2. Let B
A be the normalized weight of the regularization term. In the
X+I
Gaussian-fitting example 2.1 if max { m"==m(Bras-"") y²²
B min
20²
02 +m a
then the following inequalities holds:
MSEliaeal min min MSE((PRE))
Due to space limit, the proof is deferred to Appendix A.1.1 We plot the MSE curves for all estima- tors w.r.t. the value of A, and jPPE f1" in Fig. for clearer illustration. Fig. 1 (a) directly validates the results of Proposition 2.2 Fig. (b) and (c) show that with the optimal X, the regular- ization gains more w.r.t. MIE the sample size decreases and |uPRE μ² increases. See more details of the plot in Appendix B.I
Since the generalization analysis in deep learning is still largely open (Belkin 2021 Bartlett al., 2021], it is difficult to generalize our Proposition 2.2 to the cases with deep models due to lack of appropriate tools. However, the intuition behind the Gaussian example also holds in deep learning. Namely, training model on limited data suffers from a large variance (overfittin)) and using pre- trained model suffers from large bias (underfitting). Thus, Reg-DGM with a proper hyperparameter can balance between them and potentially achieve a better performance. Further, according to Fig. []] (c), Reg-DGM benefits the EBM defined by the pre-trained model is close the target distribution, which inspires data-dependent energy function presented in Sec. 4
3 CONVERGENCE ANALYSES
3.1 ANALYSES IN THE NON-PARE PARAMETRIC SETTING
We assume that our hypothesis class contains all valid distributions (i.c. H = Px), and the data distribution pa is accessible. Although the setting is impractical, such analyses characterize the existence and uniqueness of the global minimum in an ideal case and have been widely considered in deep generative models (Goodfellow et al. 2014 Arjovsky et al. 2017). Further, it is insightful to see how the regularization affects the solution of Reg-DGM in the ideal case.
Built upon the classical recipe of the calculus of variations and properties of the KL divergence in the topology of weak convergence, we establish our theory on the existence and uniqueness of the global minimum of Eq. (2) with the KL. and JS divergence | The results formally characterized in Theorem 3.1 and Theorem 3.2 respectively. We refer the readers to Appendix A.2. for the
2 Note that max {
(m -2 }} < min o2+m(Aue (= ja 1 always holds.
202
We consider two divergences that are employed in the two representative DGMs: variational auto-encoders
(VAE) (Kingma & Welling 2013) and generative adversarial networks (GANs) (Goodfellow al. 2014).
4

---

Published as a conference paper at ICLR 2023
proofs. To establish our theory, assume that (I) X is nonempty compact set; (2) E2) : X R continuous and bounded; (3) Jxx e-g(()) oc, which are common and mild.
Theorem 3.1. Under mild regularity conditions in Assumption A.7 for any X > 0, there exists unique global minimum of the problem in Eq. 2 with the KL divergence. Furthermore, the global minimum is in the form of p
pa(a) (x) where a* R.
Theorem 3.2. Under mild regularity conditions in Assumption A.1 for any X > O, there exists unique global minimum of the problem in Eq. ( with the JS divergence. Furthermore, the global minimum is in the form of Pz
Pala) -3 where a E R.
As shown in Theorem 3.1 and Theorem 3.2 the global minimum in the form of a reweighted data distribution and the weights are negatively correlated to the energy function defined by the pre-trained model. Qualitatively, the global minimum assigns high density for a sample x it has high density under the data distribution (i.e., Pa(z)) and low value of the energy function (i.e., (x)). Notably, the weights Theorem 3.1 and Theorem 3.2 are different because the different divergences. In particular, the effect of the pre-trained model is enlarged by the exponential term in Theorem 3.2 using JS divergence (JSD). Fig. 2 (c) and (d) show that with the same value of X, the weighting coefficients of JSD are distributed a larger range than KL divergence (KLD). Naturally, in both theorems, as À + O, the denominator of p9 (z) tends to a constant and Py (x) - pa(x), which recovers the solution of pure divergence minimization in Eq. ( Therefore, Reg-DGM is consistent if the weighting parameter À is function of m, and lim ((m) - 0.
(z(x)
3.2 ANALYSES IN THE PARAMETRIC SETTINGS
Although it provides theoretical insights of Reg-DGM, the non-parametric setting is far from prac- tice. In fact, in our experiments, the hypothesis class is parameterized by neural networks and the training data is finite. In such case, Reg-DGM can be formulated as non-convex optimization problem, which is solved by gradient-based methods. Therefore, we also analyze the convergence of Reg-DGM trained by (stochastic) gradient descent in the presence of neural networks upon the general convergence framework (Allen-Zhhe et al. 2019).
In particular, as summarized in Theorem 3.3 we show that Reg-DGM with a standard neural net- work architecture converges with high probability under mild smoothness assumptions. The as- sumptions, result and proof are formally presented in Appendix
Theorem 3.3 (Convergence of Reg-DGM (informal)). Under standard and verifiable smoothmess assumptions, with high probability, Reg-DGM with sufficiently wide ReLU CNN converges global optimum of Eq. (2) Irained by GD and converges to a local minimum trained by SGD.
IMPLEMENTATION
In this section, we discuss the base DGM, the pre-trained model and the energy function in practice.
4.1 BASE MODEL
Although Reg-DGM applies to variational auto-encoders (VAE) (Kingma & Welling 2013) and many other DGMs, we focus on GANs (Goodfellow et al. 2014), which are most representative and
popular in the scenarios with limited data (Karras al. 2020a Mo et al. 2020 Cui et al. 2021 Formally, GANs optimize an estimate of the JS divergence via a minimax formulation as follows:
min max (xppp(x) log D(x)] + (x) [ogg[1 - -(x))), D
where is a generator that defines Pg (x) and D is a discriminator that estimates the JS divergence by discriminating samples. Both G and D are parameterized by neural networks and Eq. is estimated by the Monte Carlo method over mini-batches sampled from the training set.
For broader interest, we adopt three strong GAN variants, StyleGAN2 (Karras et al., 2020b), adap- tive discriminator augmentation (ADA) (Karras et al., 2020a). and adaptive pseudo augmentation (APA) (Tiang al. 2021) the base DGMs. Please refer to Appendix for more details.
5

---

Published as a conference paper at ICLR 2023
4.2 PRE-TRAINED MODEL
As mentioned in Sec. 2 different from the transfer-based methods (Wang et al. 2018 Mo et al. 2020), Reg-DGM applies to a nontransferable setting where the pre-trained model does not nec- essarily have the same architecture or the same formulation as Pg or does not even have be generative model, enjoying the flexibility of choosing the pre-trained models.
In our implementation, the pre-trained model feature extractor f X - R., which trained for other tasks (c.g., classification or contrastive representation learning) instead of generation. We choose such models because they are easily available and achieve excellent performance in super- vised learning. In particular, we investigate three prototypical pre-trained models: a ResNet (He 2015) trained supervised manner on ImageNet, CLIP image encoder (Radford et al,, trained in self-supervised manner large-scale image-text dataset, and a FaceNet (Schroff) 2015) trained on a face recognition dataset. Please refer to Appendix B.2 for more details. ct al.
al.
202
Note that such models are nontransferable in the fine-tuning manner (Hinton et al., 2006). Never- theless, with such models and a data-dependent energy function presented later, Reg-DGM still competitive to the transfer-based approaches (Wang al. 2018 Mo et al. 2020) shown in Tab.
4.3 ENERGY FUNCTION
According the results in Fig. (c) and our intuition, should define E such that pf is close to as possible. In most of the cases, I is pre-trained on dataset with richer semantics than Pa- Therefore, it is necessary to involve training data (sampled from pa) in the energy function to reducing the distance between Ps and As presented above, we specify f as feature extractor and it natural match the features of samples from Pg and pa as the energy function.
Formally, the energy function is defined by the expected mean squared error between the features of a generated sample and a training sample as follows:
E (x) Engeop (
(5)
Notably, our implementation with the data-dependent energy function in Eq. (5) is a valid instance of the general Reg-DGM framework as formulated in Eq. (2). Furthermore, the convergence results in both the non- non-parametric and parametric settings (see Sec. 3 hold in this case. The expectation is estimated by the Monte Carlo method of a single sample for efficiency by default and increasing the number of samples will not affect the performance significantly (see results in Appendix C.4)
We emphasize that our main contribution not designing a specific energy function but the general framework of Begg-GGM.Bt Many alternative energy functions can be employed in Reg-DGM. Indeed, we perform a systematical ablation study of the energy functions in Sec. [.3 and find that Eq. (5) is the best among them considering the qualitative and quantitative results together. Moreover, evaluate the effectiveness of Reg-DGN DGM implemented by Eq. (5). with strong base DGMs (Karras) al. 2020b at Jiang al. 2021), different datasets, different backbones of ʃ and different pre- training datasets in the experiments. We observe a consistent and significant improvement over SOTA baselines across various settings. Based such a comprehensive empirical study, we believe that our implementation of Reg-DGM based on Eq. ( would be effective in new settings.
5 EXPERIMENTS
For a fair comparison to a large family of prior work, we evaluate Reg-DGMM on several widely adopted benchmarks with limited data (Karras et al.) 2020a), including the FFHQ (Karras et al.) 2019, 100-shot Obama (Zhao et al.) 2020a). LSUN CAT (Yu et al. 2015) and CIFAR-I0 Krizhevsky et al. (2009) datasets, and the data processing and metric calculating are the same as those of ADA (Karras et al. 2020a). We present the main results and analyses in the section and refer the readers to Appendix B.2 for details and Appendix for additional results. Throughout the section, we refer to our approach as the name of the base DGM with the prefix "Reg-". For instance, "Reg-ADA" denotes approach with ADA (Karras et al.! 2020a) the base DGM.

---

Published as a conference paper at ICLR 2023
Table Median FID & on FFHQ and LSUN CAT and mean FID & on CIFAR-I0. 1 and indicate the results taken from the references and Karras et al. (2020a) respectively. Otherwise, the results are reproduced by us upon the official implementation KKrras et al. 2020a Tiang et al. 2021
FFHQ
LSUN CAT
CIFAR-10
Method
1k
5k
1k
5k
50k
Transfer Wang et al.
2018
21.42
12.34
Freeze-Di Moo ct al. 2020)
19.77
12.69
DA
Zhao et al. 2020a)
25.66
10.45
42.26
16.11
8.49
InsGen Yang et al. 2021
19.58
GenCot (Cui et al.
2021
65.31
27.96
140.08
40.79
8.83 + 0.04
DA GenCol (Cui et al.
2021
6.57 0.01
ADA bCR (Zhao et
2020b
22.61
10.58
38.82
16.80
RLC (Tseng al. 22211
63.16
23.83
8.31 + 0.05
ADA + Ric (Teeng et al.
2021
21.7
2.47 + 0.01
APAT (Jiang et al. 2021
45.19
13.25
StyleGAN2 (Krrras et al. 2020b
103.66
52.71
186.55
115.16
7.16 + 0.12
Reg- Beg-StyleGAN22 (ours)
75.99
37.77
107.02
63.10
6.56 0.14
ADA (Karras et al.
2020a)
22.26
12.64
41.81
16.76
3.07 0.08
Reg-ADA (ours)
20.05
11.95
36.17
15.91
2.95 + 0.05
ADA APA (Jiang al. Reg-ADA-APA (ours)
2021)
19.71
8.84
24.09
11.79
2.64 + 0.08
17.88
8.02
21.88
11.27
2.58 + 0.04
(a) 100-shot Obama (FID 39.53)
(b) 5F (FID 11.69)
Figure Samples from the Reg-ADA, truncated (ib 0.7) as in prior work KKrrra et 2020a
5.1 BENCHMARK RESULTS WITH LIMITED DATA
We employ StyleGAN2 Karras et al. 2020b), ADA Karras et al., 2020a) and APA (Jiang et al. 2021 the base DGMs and ResNet-18 (He et al. 2015) classifier trained on ImageNet (Deng et al. 2009) as the pre-trained model by default. The associated energy function is defined in Eq. (5)
Quantitatively, we compare Reg-DGM with a large family of existing methods, including the base DGMs, the transfer-basee approaches, the augmentation- based methods and many others. Following the direct and strong competitor (Karras et al.) 2020a),, we report the median Fréchet inception distance (FID) (Heusel et al. 2017) FFHQ and L.SUN CAT, and the mean FID CIFAR-10 out of runs for fair comparison Tab. T For completeness, we also report the mean FID with the standard deviation FFHQ and LSUN CAT in Appendix C.I
As shown Tab. Stg-SttleGAN,, Reg-ADA and Reg- Reg-ADA-APA consistently outperform the corresponding base DGM in five settings, demonstrating that Reg- Reg-DGM can achieve a good bias- variance tradeoff in practice. Besides, the superior performance of Reg-ADA over ADA (and Reg- ADA-APA over ADA-APA) shows that contribution is orthogonal to the augmentation-baal approaches. Notably. the improvement of Reg- DGM over the base DGM larger when the sample size is smaller. This is expected because the relative gain of Reg-DGM over the base DGM

---

Published as a conference paper at ICLR 2023
Table Median FID & and the corresponding KIDx 10
using a pre-trained CLIP or FaceNet.
CLIP
FaceNet
Method
FFHQ-5k
LSUN CAT-5k
FFHQ-5k
FID
KID
FID
KID
FID
KID
StyleGAN2 Karras et al. Reg-StyleGAANZonn
2020b)
52.71
39.52
115.16
100.57
52.71
39.52
40.98
27.56
42.04
26.21
38.80
23.38
ADA
Karras et
2020a)
12.64
5.17
16.76
8.13
12.64
5.17
Reg- (ours)
11.09
3.91
14.15
6.72
11.37
4.01
ADA+APA Jiang et al. Reg-ADA (PAoors
2021
8.84
2.76
11.79
4.86
8.84
2.76
8.18
2.26
10.47
4.68
8.21
2.37
increases decreases given fixed pre-trained model and the optimal À (see Fig. (b)). Further, we evaluate the base DGMs and Reg-DGMs under the kernel inception distance (KID) (Binkowski al. 2018) metric in Appendix C.4 The conclusion remains the same. As suggested by Propo- sition 2.2 the value of the weighting hyperparameter À is crucial and there is an appropriate range of À such that Reg-DGM is better than the base DGM. We empirically validate the argument on FFHQ-5k with StyleGAN2 as the base model in Appendix C.5
Mo et al. 2020 in Tab. employ a GAN pre-trained on CelebA-HQ (Karras et al. 2017). which is also a face dataset of the same resolution as FFHQ. In comparison, our approach built upon a classifier pertained on ImageNet. As highlighted in (Wang et al. 2018). the density of the pre-training dataset is more important than the diversity. Nevertheless, Reg- Deg-DGM is competitive with these strong baselines while enjoying the flexibility of choosing the pre-trained model and dataset. Further, we directly adopt the same pre-trained model across all datasets including CIFAR-IO, where takes additional efforts to get suitable pre-trained generative model to fine-tune. Based the results, it is safe to emphasize the complementary role Reg-DGM to the approaches based on fine-tuning.
We mention that the methods based fine-tuning (Wang al. 2018:
Qualitatively, show the samples generated from Reg-DGM FFHQ-5k and 100-shot Obama in Fig. 3 It can be seen that with the regularization, our approach can produce faces of a normal shape with limited data. We present the results from the base DGMs and more samples in other settings in Appendix C.2 For a comprehensive comparison on the visual quality of samples, we perform a
human evaluation by the Amazon Mechanical Turk (AMT) in prior work (Choi al. 2020 Ac- cording with the FID results, Reg-StyleGAN2 trained on FFHQ-5k against the baseline StyleGAN2 chosen in 63.5% of the 3,000 image quality comparison tasks. See details in Appendix C.4
5.2 ABLATION OF PRE-TRAINED MODELS AND PRE-TRAINING DATASETS
For simplicity, the main results of Reg-DGM presented in Sec. 5.1 are based on a ResNet-18 model pre-trained ImageNet. Although its architecture is significantly different from the Inception- v3 (Szegedy et al. 2016) used in FID and KID calculation, is worth performing an ablation on the pre-trained models eliminate the potential bias caused by the pre-training dataset.
In particular, we test Reg- Deg- with the image encoder of the CLIP model (Radford et al. 2021 (an architecture very similar ResNet-50), which is pre-trained on large- noisy text-image pairs instead of ImageNet, and with the face recognizer FaceNet (Schroff et al. 2015), which is pre-trained on the large-scale face dataset VGGFace2 (Cao et al. 2018). See details in Appendix B.2 The median FID and corresponding KID results are shown in Tab. 2 Without heavily tuning the hyperparameter X, Reg- DGM shows consistent improvements the two baselines under both FID and KID metrics, and achieves comparable results to those presented in Table I The results with the CLIP model and the FaceNet model demonstrate that Reg-DGM works well with various backbones and pre-training datasets.
Sreptiion-RsNeetwve (Szegedy et al. 2017) of
8

---

Published as a conference paper at ICLR 2023
5.3 ABLATION OF ENERGY FUNCTIONS
As emphasized in Sec. 4 our main contribution is the general framework instead of specific energy function. Nevertheless, we still investigate two alternative regularization terms for completeness.
We first consider the famous entropy- minimmiziiiiiiiiion regularization (Grandvalet & Bengio 2004). which is data-independent. Intuitively, the regularization forces Pg that produces samples with clear semantics. We find that the entropy regularization achieves similar FID results of 50.87 on FFHQ-5k to the baseline 52.71, showing the importance of the data dependency in the energy function. We then investigate another data-dependent regularization term, i..., feature matching (Salimans al. 2016), which matches the averaged features between the model and data distributions. We find that feature matching can achieve FII 32.65 on FFHQ-5k greatly reducing the FID of StyleGAN2 while it cannot improve the visual quality of the samples. Please refer to Appendix C.4 for details of both energy functions. Therefore, Eq. (5) the best among them considering the qualitative and quantitative results together and we believe it can be transferred to new settings based our results in Sec. 5. and Sec. 5.2
6 RELATED WORK
Fine-tuning approaches. milestone of deep learning is that deep generative model fine-tuned
for classification outperforms the classical SVM on recognizing the hand-writing digits (Hinton
al.
2006). Since then, the idea of fine-tuming has a significant impact (Devlin et al. [2018] He
2020) including generative models with limited data (Wang et al. | Mo et 2020 Wang et al.
et al. 2020 Li al., 2020; Ojha et al. 202T). However, an inherent restriction of fine-tuning is that
the pre-trained model and target model should partially share common structure. Thus, it may
take additional efforts to find a suitable pre-trained model to fine-tune.
comparison, Reg-DGM provides an alternative way to make it possible exploit pre-traine pre-trained classifier help generative modeling. Notably, the latter is often thought of as much harder than the former.
Other generative adversaria networks with limited data. To relieve the overfitting problem of the discriminator, DA (Zhao et al.] 2020a), ADA (Karras et al., 2020a) and APA (Jiang et al.] 2021 design sophisticated data augmentation strategies. GenCo (Cui et al., 2021) designs training framework that introduces multiple complimentary discriminators. InsGien (Yang et al.
2021] improves the data efficiency of GANs via an instance discrimination loss (Wu et al. 2018).
We believe that Reg-DGM is orthogonal to these methods based on our results in Tab. [
Regularization in probabilistic models. Extensive regularization approaches have been developed in traditional Bayesian inference (Zhu et al.) 20144) and probabilistic modeling (Chang et al.) 2007 Liang et al. [200)] Ganchev al. (2010). Among them, posterior regularization (PR) (Ganchev al., 2010) encodes the human knowledge about the task as linear constraints of the latent representations in generative models for better inference performance. Such methods have been extended to deep generative models (Hu et al.) 2018 Du et al.) 2018 [huu et al. 2018:; Xu et al.) (019)] for similar reason. Technically, PR-based methods regularize the latent space via handcrafted or jointly trained constraints. In comparison, our approach regularizes the data space via pre-trained model. Besides, PR- PR-based methods are suitable for structured prediction tasks instead of generative modeling with limited data, which is the main focus of this paper.
7 CONCLUSIONS
In this paper, we propose regularized deep generative model (Reg-DGM), which leverages pre- trained model for regularization to reduce the variance of DGMs with limited data. Theoretically, we analyze the convergence properties of Reg-DGM. Empirically, with various pre-trained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs and achieves competitive results to the state-of-the-art methods. An interesting future work is to analyze the generalization behaviour of Reg-DGM in general and in- spire energy functions. Currently. the generalization analysis of deep learning is still largely open (Zhang et al. 2021 Bartlett et al., 2221]] Belkin 202I), and there lacks appropriate tools to formalize our intuition on the bias-variance tradeoff in general.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Guoqiang Wu for helpful discussions about the generalization analysis. This work was supported by NSF of China (Nos. 62076145, 62106273); Beijing Outstanding Young Scientist Major Innovation & Planning Interdisciplinary Plat-
Program (No. SJJW2YJH022011100220000 form for the "Double-Firs" Class" Initiative, Renmin University of China; the Fundamental Re- search Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13, 22XNKJ16). Part of the computing resources supporting this work, totaled 720 A100 GPU hours, were provided by High-Flyer AI. (Hangzhou High-Flyer AI Fundamental Research Co., Ltd.). C. Li was also sponsored by Beijing Nova Program.
ETHICS STATEMENT
This work presents a framework to train deep generative models on small data. By improving the data efficiency, it can potentially benefit real-world applications like medicine analysis and automatic drive. However, this work can have negative consequences in the form of "DeepFakes", as existing GANs. It worth noting that this work may exacerbate such issues by improving the data efficiency of GANs. How to detect "DeepFakes" is an active research area in machine learning, which aims to relieve the problem.
REPRODUCIBILITY STATEMENT
We submit the source code in the supplementary material and have released it. Datasets used in ex- periments are open and publicly available, and experimental details provided in the Appendix B In addition, the complete proof of the propositions and theorems is contained in the Appendix A
REFERENCES
Ganesh Ajjanagadde, Anuran Makur, Jason Klusowski, Sheng Xu, et al. Lecture notes on informa- tion theory. Lab. Inf. Decis. Syst., Massachusetts Inst. Technol., Cambridge, MA, USA, Tech. Rep, 2017.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242 052-252. PMLR, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-222222 PMLR, 2017.
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. Acta numerica, 30:87-201, 2021.
Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203 248, 2021.
Patrick Billingsley. Convergence of probability measures. John Wiley & Sons, 2013.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01 2018
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67-74. IEEE, 2018
Ming- Wei Chang, Lev Ratinov, and Dan Roth. Guiding semi cemi-supervision with constraint-driven learning. In Proceedings of the 45th annual meeting of the association of computational linguis- fics, pp. 280-287, 2007.
10

---

Published as a conference paper at ICLR 2023
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings the recognition, pp. 8188-8199 8197, 2020.
the IEEE/CVE conference on computer vision and pattern
Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genco: Generative co-training for generative adversarial networks with limited data. arXiv preprint arXiv:2110.01254 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1oXiv:/8/000 1810,08805 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv
preprint arXiv: 2016.
Chao Du, Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Learning implicit generative models by teaching explicit ones. arXiv preprint arXiv:1807.0380 2018.
Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11:2001-2049, 2010.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Yves Grandvalet and Yoshua Bengio. Sumi-supe learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint 1 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-577, 2006.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):157-1554, 2006.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:220..03458, 2022.
Zhiting Hu, Zichao Yang, Russ R Salakhutdinov, LIANHUI Qin, Xiaodan Liang, Haoye Dong, and Eric P Xing. Deep generative models with learnable knowledge constraints. Advances Neural Information Processing Systems, 31, 2018.
Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pseudo augmentation for gan training with limited data. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196. 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.

---

Published as a conference paper at ICLR 2023
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104412111111111111111111111 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 81 10-8119, 2020b.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,200 2013.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
Chongxuan Li, Kun Xu, Jun Zhu, Jiashuo Liu, and Bo Zhang. Triple generative adversarial net- works. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with clastic weight consolidation. arXiv preprint arXiv:2012.02788. 2020.
Ziqiang Li, Xintian Wu, Beihao Xia, Jing Zhang, Chaoyue Wang, and Bin Li. A comprehensive survey on data-efficient gans in image generation. arXiv preprint arXiv:2204.0888 2022.
Perey Liang, Michael I Jordan, and Dan Klein. Learning from measurements in exponential families. In Proceedings of the 26th annual international conference on machine learning, pp. 641-648, 2009.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490. PMLR, 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. arXiv preprint arXiv:2002. 10964, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning- MIT press. 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In lcml, 2010.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 2750-2758, 2019.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10743-10752 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine PMLR, 2021.
Learning, pp.
12

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, 2016.
pp.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recogmition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823,
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. Advances Neural Information Processing Systems, 31, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-forst AAAI con- ference on artificial intelligence, 2017.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing genera- tive adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7921-7931, 2021.
Aaron Van den Oord, Nal Kalehbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016
Onno Van Gaans. Probability measures metric spaces. Lecture notes, 2003.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gorzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234, 2018.
Yaxing Wang, Abel Gonzalez-Garee Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few im- ages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9332-9341, 2020.
Zhirong Wu. Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Taufik Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Multi-objects generation with amortized struc- tural regularization. Advances in Neural Information Processing Systems, 32, 2019.
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems, 34, 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 0tt.:/0066.33333 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt. Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
13

---

Published as a conference paper at ICLR 2023
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020a.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for gans. arXiv preprint arXiv:2002.04722 2020b.
Jun Zhu, Ning Chen, and Eric P Xing. Bayesian inference with posterior regularization and appli- cations to infinite latent svms. The Journal of Machine Learning Research, 15(997779-11877 2014.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
A PROOFS
A.1 THE GAUSSIAN-FITTING EXAMPLE
We first derive the solution of Reg-DGM in the main text. In the Gaussian fitting example, the regularized optimization problem can be written as
PREG
arg min log (e,, )+ N lolollloopppe logpy(z)]
(6)
(μ
xx)²
(1 - P
1
arg min
+ A
+
log (2TT² +
20²
202
2
m
(n x2))
(p (PPRE)
arg min
(8)
202
202
where the first equality holds by the definition and properties of Gaussian (Bishop & Nasrabadi 2006) and the second equality holds by omitting constant irrelevant to the optimization. It is easy solve the above quadratic programming problem analytically:
À
jREG
Ti
İAPRE
jOMLE +
1 X iPRE,
m(1 + A)
1
1 + X
where jAMLE N (nt, d² variable and thus is also Gaussian distributed as follows:
DIEG is obtained by an affine transformation of a Gaussian random
d²
PREG .
[ 1C + I+A
PPRE
(10)
m(1 + A)2
A.1.1 PROOF OF PROPOSITION 2.2
Proof. In the Gaussian- fittinn example, we have
ÂMLE ~ N [u", o²
(11)
and
PREG N (
d² 1 1 1 jAPRE m(1 + X)2
(12)
According to the bias-variance decomposition of the MSE, we have
MSE[ o²
(13)
14

---

Published as a conference paper at ICLR 2023
and
X² (APRE - -0)² + m(1 + X)
0²
MSFILREGG]
(14)
(1 + X)2
Let B = T+X (0,1) be the normalized weight of the regularization term. Then, we can rewrite MSE (ūRGG]
A
MSE[AREG] (APPE 1²)² + (1 - 3) o²
(15)
m
To satisfy [rRGR < MSE[(MME), we have
0² - 3< 02 +
20²
(iPRE 1=) (1-
3) m
(16)
m
mmjpPRE p0))2"
The pre-trained model ÄPRR is another meaningful baseline, which has bias of jUPRE p² and zero variance. Its MSE is given by
MSE| APRRE
= (APRE - 1 12
(17)
To satisfy [SE] < (SEE( we have
( - 2²)² + (1 -3)= 3)2
< (ÁPRE. - -1²)2 = 3 3> 0² -m(PPRE - 120)2
(18)
m
σ2 + mmpPPR 100 110)2"
which completes the proof.
We now computes the optimal MSE[EREG]- It is easy that the quadratic programming problem in Eq. achieves minimum at 3 mm+mmj - with the corresponding
x a² jiljava p The minimum value is of SEARE--""? 2² 3=4m(Avan 3²
PNILE MSF MSE PMnE MSE
1.1 COMPARISON UNDER THE EXPECTED RISK
We also evaluate all estimators in terms of expected risk, which a commonly used measure in statistical learning theory. In density estimation task for Pa. the expected risk of a hypothesis a which depends on the training sample S, is R(2)) := I- p(æ; pi]] We can show that the expectation of the expected risk w.r.t. S coincides with the corresponding MSE in the Gaussian-
fitting example and directly obtain the following Corollary fr.0. Proposition 2.2
Corollary A.0.1. the Gaussian canussian-fttiing example 2.1 if satisfies the same condition as Propo-
sition 2.2 then the following inequality holds:
Es Es|R(PREc) min (Es (((ML))) Es(R(peRs)]::
(19)
Proof. In the cunssian-fitting example, the expected risk for a hypothesis f is
R(p)=
Esupy B .m2)
[- log (pp
(20)
11
Es-pp
log (og(2mo²) + 2
(21)
20²
2
1 MSE[A]
1 log(2To²)
(22)
20
2'
which completes the proof together with Proposition 2.2
A.2 OPTIMIZATION ANALYSES
A.2.1 PROOF OF THEOREM 3.1 AND THEOREM 3.2
Our results rely the following regularity conditions.
15

---

Published as a conference paper at ICLR 2023
Assumption A.1.
1. X is nonempry compact set.
2. E X +R is continuous and bounded.
3. Ix d= < co.
The regularity conditions in Assumption A.1 are mild in the sense:
The sample space X' is often a subset of a n--dimensional Euclidean space. Then, X is compact if and only if X' is bounded and closed, which holds for extensive datasets in various types including images, videos, audios, and texts.
2. In this paper, E is defined by compositing a neural network and continuous real-valued function. Then Ey is continuous and bounded on X' if the neural network has bounded weights and uses continuous activation functions including ReLU (Nair & Hinton 2010), Tanh and Sigmoid.
3. Jx (x,()) <
holds following the choice of the sample space and energy function,
We first prove Theorem 3.1 as follows.
Proof. Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the KL divergence as follows:
x min (- ( Pa
(23)
subject to
Py pg(x)dx=11
Wr E p2(x)) 2 0.
For clarity, we denote the functional in problem Eq. 23 be optimized
J(p)) = / (())m((p())))dd +EE (z),
According to Assumption 3.1 that X' is a nonempty compact set, the feasible area characterized by the constraints (i.e., Px) is compact in the topology of weak convergence by the Prokhorov's Theorem (See Corollary 6.8 in (Van Gaans, 2003)). By Theorem 3.6 in (Ajjanagadde et al.] (017)) KL divergence is lower semi-continuous in the topology of weak convergence. According As- sumption 3.2 that & is continuous and bounded on X, the regularization term is continuous in the topology of weak convergence. Therefore. by the extreme value theorem, the global minimum of TP,) exists in the feasible area.
Note that the optimization problem Eq. (23 is convex. To obtain a necessary condition for the global minima, we get the Lagrangian with the equality constraint:
C(pv) = / (=(())(()))pa()) + (E)()) + al po(2)d - -1),
(24)
where R. Note that for simplicity, do not include the inequality constraint, which will be verified shortly. It is easy to check the constraint qualifications for problem Eg, (23). By the calculus
of variations, a necessary condition for a global minimum of problem Eq. (23 is
8C
pa(x)
xx) aa=
(25)
8pg
Pg(x)
which implies that
pa(x)
()(2)
(26)
a + (x)
16

---

Published as a conference paper at ICLR 2023
We define A = {a IR (a Vr l). According to Assumption 3.1 that Ey is bounded on X, we have * We define function @ I
Pa(a)
o(a) / a (x)
pa(x)
dr.
(27)
is easy see Φ(a) is monotonically decreasing and there is most o' A such that o(a*) = 1, which finishes the proof together with the existence of the global minimum.
now prove Theorem 3.2 as follows.
Ignoring some constant irrelevant to the optimization, we rewrite the optimization problem of our approach with the JS divergence as follows:
Proof. The proof here shares the same spirit of Theorem 3.1
min / (- (n(pp())) + pa(x))pa(x)) (n(p)()) (a( (x)p Ps xx
subject to / Po (o(x)d==1 11
Va E X. 2
(28)
For clarity, denote the functional in problem Eq. 23 be optimized
I(p)) = { (- (n)p(p(()) + pa(x))pa(a)) [n(p (x) + pa(x)))(pg (g) (x) + (n(pg(()))
(29)
(e)()) (30)
Similarly the proof of Theorem 3.1 the global minimum of J [Pg) exists in the feasible area due to Assumption A.. and the fact that the JS divergence is lower semi-continuous in the topology of weak convergence.
Notably, the optimization problem Eq. gence (Billingsley 2013). To obtain necessary condition for the global minima, we get the La- grangian with the equality constraint:
28 is convex due to the convexity of the JS diver-
C(pg) = / (- (n(pp()) pa(=))pd(z) (n(p(()) pa(x)))p(x)) (n(pg()))pp(0f+
(z)))) al Po(x)dx - 1), (31)
where a E R. Similarly to the proof of Theorem 3.1 a necessary condition for a global minimum of problem Eq. (3 is
8C
In(p, (x) + pa(x)) + In(p, + (E) + O,
(32)
8pg
which implics that
Pa(x) ca+2E,(*) -
(T((x)
(33)
Similarly to the proof in Theorem 3.1 there is at most one a* E R such that fx which finishes the proof together with the existence of the global minimum.
pa(x)
A.2.2 CONVERGENCE WITH NEURAL NETWORKS TRAINED BY (STOCHASTIC) GRADIENT DESCENT
We establish the convergence of with over-parameterized neural networks trained by (stochastic) gradient descent upon the general framework (AllenZhhu al. 2019).
17

---

Published as a conference paper at ICLR 2023
Theorem A.2. (General convergence guarantee (Allen Jillen-Zhueee et al. smooth loss function with probability at least é 1ilogm ReLU convolutional neural net- work with width and depth trained by gradient descent with appropriate learning rate satisfy the following
2019) For an arbitrary Lipschitz-
f
If C is non-convex, and o-gradient dominant, then GD finds c-error minimizer in (poly((n, log )) iterations long m (1polyyn, ))).
If C is non-convex, then SGD finds a point with || S in O(poly(m, 1, log !)) itera- tions long as m 2 (ppoly(n,4, }))
We assume the following standard smoothness conditions, which can be verified in practice with bounded data and weights.
Assumption A.3. (Smoothness conditions)
1. 3b 0 such that V0 O, Ya 2, 2
2. BT. 0 such that Vo E X, v0 O vo' E (00=))))))))) - T.1 -0
3. 3K 0 such that v0 O v0' e p(0;z) p00;; K|10 -01|.
4. supnex supyex [f(x)) fw)ll < B.
We consider the general density estimation setting where LMLE (0;;;) = (o;;). Note that the first conditions in Assumption A.3 directly imply that EMLE(1:2) ) f L-Lisschitz. Formally, given a set of samples S = 1 Reg DGM optimizes
1 R = n Σ (f) + (E)=))(E(()))
If E is independent from the training data x, then the overall loss function is also & L-Lisschitz and Theorem A.2 directly applies. Otherwise, we can show that the data- outa-dependent regularization used our experiments also Lipschitz-smoott By the linearity of expectation, we have
n
ŌREG == arg min 850 n
(MLLE(0; (0:x:)))) + Bumpo n
(1f(v) f(x))11
(34)
(0: + A Boynn (1 f(x))
arg min 858
(35)
We define CREG x.,) = Byypp lIf()) - f(x;)) which is Lipschitz-smooth:
X
(CREG(0;0) LREG (0'; xz)
Byrns (lf(2) f(x))1 Eyrrv (lf(w) - f())
/ (po(v) - (o - f(y)l1d
(sup (l)())) - s(w)11) / (o(()) po (v) dy
ABK ||0 8'||-
d
Therefore, Theorem A.2 applies to Reg-DGM with the data-dependent energy defined Sec. of the main text.
B EXPERIMENTAL DETAILS
Our implementation is built upon some publicly available code. Below, we include the links and please refer the licenses therein.
See additional assumptions regularity of the data Allen-Zhu 2019
18

---

Published as a conference paper at ICLR 2023
B.1 TOY DATA
In the toy example for optimization analyses, the data follows a uniform distribution over [0, 1]. The energy function is defined Ex(x) = 0.7x + 0.9. The optimal 9 is estimated by numerical integration.
In the experiments for the Gaussian fitting example, we 0² 1, m 150, and ÄPRE - =1" 0.1 by default.
B.2 GANS WITH LIMITED DATA
Datasets. In our experiments, we use FFHQ (Karras et al., 2019). which consists of 70, 000 human face images of resolution 256 256, LSUN CAT (Yu al. 2013). which consists of 200, 000 cat images of resolution 256 x 256, and CIFAR-10 (Krizhevsky et al. 2009). which consists of 50, 000 natural images of resolution 32 x 32. Specifically, we randomly split training subsets of size 1k and 5k from full FFHQ and LSUN CAT in the same way as ADA (Karras et al., 2020a). For reproducibility, we directly use the random seed provided by the official implementation In addition, we also experiment on the smaller dataset 100-shot Obama (Zhao et al. (0200a) with only 100 face images of 256 256 resolution. In all experiments, we do not use x-flips to amplify training data except for combining with APA (Jang et al. (021).
ADA 5
Metrics. To quantitatively evaluate the experimental results, we choose the Fréchet inception dis- tance (FID) (Heusel et al. (017) and the kernel inception distance (KID) (Binkowski et al., 2018) as our metrics. We compute the FID and KID between 50, 000 generated images and all real images instead of training subsets (Heusel et al. 2017). Following ADA (Karras et al. 2020a), we report and the medium FID and corresponding KID on FFHQ and LSUN CAT and the mean FID with standard
deviation on CIFAR-10 10 out of runs. We record the best FID during training in each run in ADA (Krras et al. 2020a).
Base DGM. In particular, the lighter-weight StyleGAN2 is the backbone for FFHQ and LSUN CAT, and the tuning StyleGAN2 is the backbone for CIFAR- 10 following ADA (Karras et al. 2020a). Compared to the official StyleGAN2, the lighter-weight StyleGAN2 has the same performance and less computing cost on the FFHQ and LSUN CAT and the tuning StyleGANZ is more suitable for CIFAR-10 (Karras et al.) 2020a). Adaptive discriminator augmentation (ADA) (Krrras et al. 2020a) is a representative way of data augmentation for GANs under limited data, and its combination with adaptive pseudo augmentation (APA) (Jiang al. 2021) the state-of-the the--rrrrt method for few-shot image generation (Li et al. 2022). Our implementation is based on the official code of ADA (5] and
APA (
Pre-trained model. We choose the ResNet- 18 trained on the ImageNet dataset as the pre-trained model by default due to its excellent performance on ImageNet (Deng et al.) (209) and little com- putational overhead. We normalize both the real and fake images based the mean and standard deviation of training data and then feed them into the classifier. We extract the features of the last fully connected layer in the pre-trained model, which is frozen during training of generative models. On CIFAR-10, we interpolate both the fake and real images to a resolution of 256 x 256 after normalization.
Other alternative pre-trained models are the image encoder Re of the CLIP model (Radford et al. 2021). which is pre-trained on large-scale noisy text-image pairs instead of ImageNet, and the face recognizer Inception-ResNet--v (Szegedy et al. 2017) of FaceNet (Schroff et al. 2015), which pre-trained on the large-scale face dataset VGGFace2 (Cao al. 2018). Notably, replace the last attention pooling layer in the image encoder of CLIP with the global average pooling layer, and we directly pass the image to the face recognizer without operating the face detector in FaceNet
htt.ppes/ppaorroooooroooo///aees interpolate.html
funcfunttii functional. onal.
19

---

Published as a conference paper at ICLR 2023
Table Hyperparameters in the experiments of GANs. Reg-DGM shares the same hyperparameters as the base DGM if not specified. All models converge with the corresponding training length.
Parameter
FFHQ-1k
FFHQ-5K
LSUN CAT-IK
LSUN CAT-5k
CIFAR-10 10
StyleGAN2
StyleGAN2
StyleGAN2
StyleGAN2
StyleGAN2
Buse DGM
Karras et al. 2020b
Karrus et al. 2020b
Karrus et al. 2020b
Karrus et al. 2020b
Karrus et al. 2020b
À
x 10
Number GPUs
4
Training length
5M
5M
5M
5M
25M
Minibatch size
64
64
64
64
64
ADA
ADA
ADA
ADA
ADA
Base DGM
Karras el al. 2020a
(Karra el al.
2020a
Karras el al.
2020a
(Krrra et al.
2020a
Karras el al.
2020a
0.01
0.001
0.01
0.0005
x 10
Number of GPUs
8
8 16M
8 60M
Training length
16M
16M
16M
Minibatch size
64
64
64
64
64
ADA+APA
ADA+APA
ADA+APA
ADA+APA
A
Base DGM
Jiang et al. 2021]
Jiang et al. U:UT
2021
Jiang et al. 0.005
2021
Jiang et al. 2021 0:001
Jiang et 2021
À Number of GPUs
0.01
8
8
8 100M 64
Training length
25M
25M
25M
25M 64
Minibatch
Table 4: The hyperparameter À with pre- pre-trained CLIP and FaceNet.
CLIP
FaceNet
Base DGM
FFHQ-5k
LSUN CAT-5k
FFHQ-5k
StyleGAN2 Karraa et al. 2020b)
50
50
4
ADA (Karras et al. 2020a
2
0.05
ADA+APA Jhang ct al. 2021)
0.5
1
0.001
to extract cropped faces. For pre-trained CLIP and FaceNe 10 feature extractors.
we also employ their last layers as
Hyperparameters. Some parameters are shown in Tab. The weight parameter À controls the strength of our regularization term. We choose the weighting hyperparameter À by perform- ing grid search over [50, 20, 10, 5, 4, 2, 1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005] for FFHQ and LSUN CAT, and [1, 0,1,0.05 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001, 0.000005] for CIFAR-10 according to FID following prior work (Karras et al.) 2020a). Other parameters remain the same settings as ADA (Karras et al. 2020a) and APA (Jiang et al. 2021). For pre-trained CLIP and FaceNet, the adopted parameter À shown in Tab. In addition, we simply set À as 0.1 for Reg-ADA trained on 100-shot Obama.
Computing amount. A single experiment can be completed on NVIDIA 2080Ti GPUs. It takes day 6 hours 19 minutes to run our method with ADA on FFHQ or LSUN CAT and 2 days 17 hours 10 minutes on CIFAR-10 at a time.
C ADDITIONAL RESULTS
C. STANDARD DEVIATION ON FFHQ AND LSUN CAT
As shown in Tab. 5 we also provide the mean FID and standard deviation on FFHQ and LSUN CAT. Reg-DGM can reduce the mean FID significantly (compared to the measurement variance) and achieve a similar if not smaller standard deviation.
httpsss//qithbbb.cmmmoeeaa// /CLIP
pytorrh
20

---

Published as a conference paper at ICLR 2023
Table 5: The mean FID and standard deviation on FFHQ and LSUN CAT which is a supplement to reported medium FID.
FFHQ
LSUN CAT
Method
1k
5k
1k
5k
StyleGAN2 KKrrras et al. Reg- 2og-StyleGAAN (ours)
2020b)
102.62 = 5.67
53.37 + 1.92
189.57 + 8.13
110.83 + 6.85
77.80 + 3.65
38.14 0.97
112.15 + 8.48
64.11
2.51
ADA (Karras et al. 2020a)
22.10 0.50
12.72 0.13
41.59 = 1.71
16.77 + 0.74
Reg-ADA (ours)
20.16 + 0.22
11.88 + 0.13
36.85 + 1.09
15.85 + 0.10
(a) StyleGAN2 (FID 51.41)
(b) Reg-StyleGAN2 (FID 37.17)
(c) ADA (FID 12.62)
(d) Reg-ADA (FID 11.69)
Figure Samples generated for FFHQ-5k, truncated (1 0.7).
C.2 MORE SAMPLES OF GANS
Fig. 4 and Fig. respectively show the samples randomly generated by models with best FID trained on FFHQ-Sk 5k and CIFAR-10, using slight truncation as in ADA (Karras et al.] (020a). Reg-DGM produces samples of better comparable quality to the corresponding base DGM.
21

---

Published as a conference paper at ICLR 2023
(a) StyleGAN2 (FID 7.02)
Reg-StyleGAN2 (FID 6.38)
(c) ADA (FID 2.96)
(d) Reg-ADA (FID 2.88)
Figure 5: Samples generated for CIFAR-10 10, truncated (1 0.7).
C.3
LEARNING CURVES
We show the learning curves of GANs FFHQ, LSUN CAT and CIFAR-10 in Fig. 6 Fig. 7 and Fig. respectiiell Reg-DGM consistently improves both baselines all settings and the improve- ments increase the number of the training data decreases. Moreover, the curves of Reg-DGM generally have a smaller fluctuation, which is consistent with our theory that the regularization re- duces the variance of the baselines. One exception Fig. [ (a), which shows that Reg-DGM is more unstable than the bascline, which is caused by a bad random initialization. We mention that the instability of Reg-ADA in Fig. (b) due to that of ADA.
C.4 ABLATION OF CLASSIFIER AND RESULTS UNDER MORE EVALUATION METRICS
In this section, to better understand the influence classifiers on our method, we try to use pre- trained classifiers with different regularization terms, layers, and backbones. We retain the same experimental setting as in Tab. 3
Regularization form. We first investigate the feature matching objective (Salimans al. 2016) as an alternative regularization term. Formally, it computes the square of the 2n-m between expected features of real and fake samples from layer of feature extractor f. which can be represented
22

---

Published as a conference paper at ICLR 2023
110 momeenmmmmmee (3 runs
SyLleGAN (FPHID-3R)
ADA (FFHOO-10)
lo StylcGAN2 IFFHQ 1KI
SyBeSAN2 (FFH1--50 (I--5))
(FFHO 54)
290
Neg Slyw IPttig 3k]
Reg (PFNO--76)
60
50
0
1000
2000
3000
4000
5C00
0
2000
4000
6000
0000
10000
12000
14000
16000
Training progness (kimg))
Thaining pingre (kimg)
(a) StyleGAN2 and Reg- Styl-ttyyGA22
(b) ADA and Reg-ADA
Figure 6: Learning Curves on FFHQ
.00
FD "edimmmmmmee (.70))
200
FDD modooooomiiii woodaootiiinn
ADA ILSUN CAT-UMI
Reg-4na (USINN CAT-14)
ALIA [LSAIN CAI-K)
SytleGAN2 ILSUN CAT-1K)
18
ADA (LSUN CAT-SK)
Rep-Rytee20M2 (USUN CAT-IR]
syskeuANz (LSUN LAI 5k)
StyfeGAM2 (LSUN CAT-SK1
%
5000
10
8000
14000 16000
Training progrets (kmmg)
Training (kimg)
(a) StyleGAN2 and Reg-StyleGAN2
(b) ADA and Reg-ADA
Figure 7: Learning Curves on LSUN CAT.
as follows:
||Ex'~aa [ Eu-p, [f.
(36)
Note that the feature matching objective cannot be rewritten as the expectation over Py and thus cannot be understood as an energy function. As before, we adopt the last fully-connected layer of ResNet-18, and the results of StyleGAN2 regularized by Eq. (36) are shown in Tab. 6 Feature matching can greatly reduce FID of StyleGAN2 while it cannot improve the visual quality of the
samples, as shown in Fig. 9
Then, evaluate the entropy monooyy-mmimiiztiionn regularization (Grandvalet & Bengio 2004) as follows:
(f/sstmaax((=))))
(37)
where f(x) outputs the logits for the prediction distribution. As shown in Tab. 7 the entropy regu- larization achieves similar FID results the baseline within small search space of X, showing the importance of the data dependency in the energy function.
Layers in f. To explore the different layers of pre-trained model, retrain GANs separately using the first convolution layer of ResNet-18 18 and the last layers of four modules in ResNet- 18. As shown in Tab. [8] our method with features of diverse single layers can all improve the baseline StyleGAN2, and the last layer of ResNet-18 is most beneficial for our regularization strategy.
Backbone of f. We employ ResNer-50 and ResNet-101 as feature extractors explore the effect of different backbones on Reg-DGM. Tab. Pub.99sa shows the results on FFHQ-5k. Even using the default without tuning. Reg-DGM with ResNet-30 and ResNet- 101 can achieve a similar FID to that of our default setting and outperform the baseline. We believe that Reg-DGM with ResNet-50 and ResNet- 101 can get better results if we finely search the hyperparameter
Monte Carlo estimate of f. We evaluate Reg-StyleGAN2 with 8, 16, 32, and 64 (the default batch size) samples to estimate the energy function in Eq. (5) of the main text, as shown in Tab, I0 We do not observe a significant improvement by increasing the number of samples. For instance, when = 1, the estimate with 64 samples achieves an FID of 38.10, which is similar to 37.77 of the single sample estimate. Intuitively, the features of faces are likely concentrated in small area of the feature space of f. which is discriminative to other classes of natural images like cars, making the variance negligible to the training process of the generative model.
23

---

Published as a conference paper at ICLR 2023
6.0
ADA
9.5
5.0
85
SANe
13000
20000
25000
7.0
10000
20003
300000
40003
50000
Traiiing (kmgg
Traiiing (kimg)
(a) StyleGAN2 and Reg StyleGAN2
(b) ADA and Reg-ADA
Figure 8: Learning Curves on CIFAR-10.
Figure 9: Samples generated for FFHQ-5k 5k using feature matching (FID 32.65), truncated (+) 0.7)-
KID metric. For a comprehensive comparison of Reg-DGM and baselines, we also evaluate the models under KID in Tab. II The conclusion remains the same FID. Namely, Reg-DGM consis- tently improves baselines in all settings.
Human evaluation. We compare the quality of generated images from the human perspective by Amazon Mechanical Turk (AMT) as in prior work (Choi et al. 2020). In particular, we randomly generate 000 pairs of images by StyleGAN2 and Reg-StyleGiAZ trained on FFHQ-5k using the truncation trick (1) = 0.7). For each image pair, three unique workers in AMT choose the image with the more realistic and high-quality face image. Finally, 3, 000 selection tasks are completed by 34 workers within 14 hours. As consistent with the comparison of image quality between Fig. 4 (a) and Fig. (b). Reg- StyleGAN2 chosen 63.5% of the 000 tasks, and other statistical information is shown in Tab. [2]
Results on more training sets with different size. To explore the influence of our method on training data with different sizes, we test the Reg-Stpe reg-StyleGAN2 with the pre-trained ResNet-18 on different subsets FFHQ. The results are shown Tab. I3 the hyperparameter is simply fixed as 1 for new data settings 100, 2k, 7k, 10k, and 15k. Without heavily tuning the hyperparameter X, Peg-StyleGAN2 shows consistent improvements over the baseline under the FID metric. There seems to be trend that method improves more smaller datasets, which agrees with our Gaussian case illustrated by the Fig. (b).
C.5 SENSITIVITY ANALYSIS OF THE WEIGHTING HYPERPARAMETER
We empirically analyze the sensitivity of the weighting hyperparameter À on FFHQ-5k with Style- GAN2 the base model in Tab. 14 It can be seen that À affects the performance significantly. Notably, although is nearly impossible to get the optimal À via grid search, there a range of (e.g. from 0.01 to 1 in Tab. 14] such that Reg-DGM outperforms the base DGM, which agrees with
24

---

Published as a conference paper at ICLR 2023
Table 6: The median FID & FFHQ and LSUN CAT with feature matching the regularization.
À is simply set as the same values in Tab. 3
FFHQ
LSUN CAT
Method
1k
5k
1k
5k
StyleGAN2 Karrra et al. Reg-StyleGANZ (ours) (ours)
2020b)
103.66
52.71
186.55
115.16
59.96
32.65
66.46
47.56
Table 7: Median FID on FFHQ-5k for the entropy-mininizzaion regularization.
À
(Baseline)
0.1
0.01
0.001
0.0001
0.00001
FID 4
52.71 145.55
92.11
56.41
53.37
50.87
55.76
theoretical analyses. Besides, Reg-DGM is not sensitive when the value of À is around the optimum. For instance, the gap between the results of À = 0,1 and À = 1 in Tab. [4]is much smaller than their gain compared the baseline. The performance of Reg-DGM deteriorates with large in Tab. 14 as expected. Proposition 2.2 shows that Reg-DGM is preferable if and only if its value in proper interval. Intuitively, very large value means that we almost ignore the training data, which should lead to inferior performance.
25

---

Published as a conference paper at ICLR 2023
Table 8: Results with different layers on FFHQ-5k. The "-1 layer" represents the last layer (i.e., default setting). Note that layers all indexed by the function named.modules in Pytorch. A is simply set as the same values from Tab. 3
Layer index
(Baseline)
49
65
-1 (by default)
FID 4
52.71
46.39
47.65
51.61
45.24
46.01
37.77
Table 9:
Results with different backbones on FFHQ-5k. X is simply set as the same values from
Tab.
Backbone (Baseline)
ResNet-18
ResNet-50
ResNet-101
FID
52.71
37.77
40.95
42.63
Table 10: FID on FFHQ-5k for different number of samples in MC.
MC
(by default)
8
16
37.77
40.97
40.78
37.54
38.10
18.10
À = 10
53.09
53.36
58.58
52.93
48.21
Table 11: KIDx 10 on FFHQ, LSUN-CAT, and CIFAR-10.
FFHQ
LSUN CAT
CIFAR-10
Method
1k
5k
5k
50k
StyleGAN2 (Karras et al. 2020b
98.06
39.52
161.95
100.57
3.66 + 0.07
Reg-StyleGAN2 (ours)
47.91
23.06
83.71
42.68
2.89 = 0.11
ADA (Karras et al. 2020a)
9.77
5.17
23.30
8.13
0.90 + 0.12
Reg-ADA (ours)
9.38
4.30
20.52
6.54
0.83 = 0.06
Table 12: Other statistical information about human evaluation. Number-i indicates the number of image pairs that at least workers think StyleGAN2 or Reg-StyleGAN2 generates more realistic face images.
Method
Number-1
Number-2
Number-3
StyleGAN2 (Karras et al. 2020b
610
339
146
Reg-StyleGAN2 (ours)
854
661
390
Table 13: The FID on different subsets of FFHQ.
FFHQ 5k
Method
100
1k
2k
7k
10k
15k
StyleGAN2
132.68
103.66
83.32
52.71
37.75
34.20
23.07
Reg-Stylle Peg-StyleGAN2
113.96
75.99
60.83
37.77
31.58
25.72
20.85
Table 14: Sensitivity analysis of À on FFHQ-5k with StyleGAN2 the base DGM.
Values of X
(base DGM) À
0.01
==
X
=
A = 10
À 100
FIDD 4
52.71
47.49
41.51
37.77
53.09
178.53
26


Reviews:
[{"review_id": "jqkjD-h6EVk", "reviewer": "Reviewer_5xKN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Potential double-blindness violation**\n\nThe source codes, scripts, and license files in the supplementary materials have the copyright notice of NVIDIA. It is not unnatural if only some of them have it since they may be third-party codes with the third-party being NVIDIA, but in this case, as far as I checked, all the files have it. I don't know if this would be regarded as a violation of double-blindness but report here just in case.\n\nBy the way, posting something independently from \"Official Review\" is not enabled at this stage, which I found a bit inconvenient for reporting this kind of thing.\n\n---\n\n**Summary of the paper**\n\nThe authors propose a regularization of learning generative models supposing the presence of an energy function that (hopefully) captures the true distribution's nature to some extent. Their theoretical discussion is twofold. One is about the fact that the regularization does not hurt the estimation as long as the hyperparameter is appropriately chosen and is shown for a prototypical Gaussian fitting example. Another is about the uniqueness of global optimum and probabilistic convergence of SGD to it. The authors show the benefits of the proposed regularization on different image datasets.", "summary_of_the_review": "This is a solid paper. While the presentation around the analyses could possibly be improved, otherwise it is nicely written, and the experimental results well support the claim of the paper.", "strength_and_weaknesses": "### Strengths\n\n- The paper is nicely written and easy to follow.\n- The experimental results are clearly reported and support the claim well.\n- The idea of using a non-generative model's features for defining an energy function is very interesting.\n\n### Weaknesses\n\nThe analyses in Section 3 are not wrong, but the current way of presentation looks a little awkward. What's described here is that the facts that hold without the regularizer also hold with the regularizer under some conditions. In this sense, discussing those conditions in the main text, instead of deferring them to the appendix, would be more natural.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eJyn0A4vgY", "reviewer": "Reviewer_rAFU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies generative modeling under a limited data regime and proposes a regularized generative model by utilizing a pre-trained model from an external dataset. The paper presented some theoretical justification and analysis of the proposed approach from the bias-variance trade-off perspective. Experiments on image generation are conducted in comparison to several baseline methods, using GANs as the backbone generative model, showing that the proposed model can achieve better performance with the help of the proposed regularization approach.", "summary_of_the_review": "This work studies the generative models under a limited data regime which is interesting, some parts including theoretical analysis and evaluation can be improved to make the contributions clearer in terms of soundness and novelty.", "strength_and_weaknesses": "- This paper is generally well-written and the overall structure is clear. The motivation is clearly stated.\n- The authors suggest looking at the problem and proposed model from the bias-variance trade-off perspective. However, the presented theoretical analysis does not seem to be very intuitive in explaining why and how this perspective would help. To put it in another way, bringing another pre-trained model from a larger-scale dataset into the training of a generative model under a limited data regime is not a very surprising idea in the first place, which makes the synthetic study and the theoretical analysis a bit obsolete.\n- Why does the author emphasize the role of an energy-based formulation of $f$ rather than a regularization term, which is pretty much a perceptual loss in previous works? What does the distribution induced by the EBM imply in the context of the proposed model and its training process?\n- The exact description of how the limited data is prepared is not very clear, as well as how the metrics are computed. For example, are the FID scores computed against the original whole training set or in the limited dataset?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PfQZ_WbRK7", "reviewer": "Reviewer_ZXuR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address the limited data in generative modeling, this paper proposes a regularized deep generative model (Reg-DGM) to reduce the variance and prevent over-fitting. There are two main contributions of this paper: 1) proposed a regularized deep generative model based on pre-trained model with an energy function. 2) Experiments on three main datasets and serval base model demonstrate that RegDGM improves the generation performance and outperform the SOTA methods.", "summary_of_the_review": "This paper proposes a simple regularized deep generative model by leveraging pretrained models to deal with limited training data. The experiments demonstrate the effectiveness of the method on some datasets compared with the unregularized versions. However, the motivation is not well demonstrated, and not enough evidence is shown to demonstrated the effectiveness of the proposed method.\n\nPost rebuttal:\nThanks for the clarifications. Most of my concerns are addressed, and I am happy to raise my score to 6. The reason of not raising to higher scores is that I think the proposed method is somewhat restricted. I think it improves the generative model only if the distribution of the pretrained models has a distribution similar enough to the training data, which is hard to guarantee in practice. In addition, the paper only shows successful cases, it would be great to see some failure cases to see how much worse a bad-pretrained model can bring to the generative model.", "strength_and_weaknesses": "Strength:\nThe paper contributes some new ideas. It provides an alternative way compared to fine-tuning-based methods to help generative modeling in limited data. Experiments with Reg-StyleGAN2, Reg-ADA, and Reg-ADA-APA show the proposed method outperforms the corresponding base models. \n\nWeaknesses:\nIt is not clear why incorporating pre-trained models as regularization can improve generative modeling. It is said that this can reduce variance. However, it will also introduce bias. Does any pretrained models always help?\n\nFurthermore, it is claimed in the paper that the paper focuses on the limited data setting. However, I don't see any data settings described in the experiments. In the appendix, it has some descriptions on how to split the training data, but it is too rough to know whether they use the original whole dataset? If so, I am not sure if it is on the setting of \"limited data\". In my opinion, limited data means few-shots. So it should at least have some studies on the performance w.r.t. the number of training data.\n\nIn addition, to show the effectiveness of the proposed method, existing methods to deal with limited data should be compared, e.g., the methods discussed in the Related Work. The current results do not have these comparisons, thus it is difficult to claim the proposed method is a better choice.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4rYlxCDpaJ_", "reviewer": "Reviewer_rg8D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a regularization strategy for training deep generative models such as GANs on limited data. The strategy involves adding an additional term to the objective function for the generator that penalizes the expected mean square error between the features of a generated sample and a real sample, where the features are extracted from a pre-trained model. The authors present some theoretical analysis to motivate their approach and provide comprehensive empirical results to demonstrate its effectiveness.", "summary_of_the_review": "The method of regularization (expected feature-wise MSE) is not a novel concept, but the authors motivate their use here with some theoretical analysis and show promising experimental results. I would suggest more clearly discussing the differences between this work and other regularization strategies such as feature matching and entropy minimization, as well as showing the quantitative comparison in the main paper.", "strength_and_weaknesses": "Strengths: \n\n1) The paper is very well-written and easy to understand.\n2) The authors present theoretical analysis to motivate/justify their approach.\n2) The authors provide comprehensive experiences to demonstrate the effectiveness of their approach. The proposed regularization is applied to three models (StyleGAN, ADA, APA) trained for generation on three datasets (FFHQ, LSUN CAT, CIFAR-10), and provides a modest boost to each of the models. The authors' best model outperforms all the baselines in the main table.\n4) The authors provide ablation studies and connect their proposed approach to other regularization strategies such as entropy minimization and feature matching.\n\nWeaknesses:\n1) There is a gap between the theoretical analysis and the actual method that is tested empirically. The bound of Section 2.1 is limited to the simple Gaussian example. The theoretical results do not provide intuition/justification for using a pretrained feature extractor. \n2) The comparison between the proposed approach and existing regularization strategies such as entropy minimization and feature matching should be emphasized, both in discussion and in results. At the moment, the results for the other two regularization strategies are in the Appendix and the presentation of the results does not easily allow the reader to compare them to the proposed approach. For example, it seems that feature matching achieves better FID than the proposed method, but this is not clear from the presentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster