CcGAN: Continuous Conditional Generative Adversarial Networks for Image Generation
Authors: Xin Ding
Yongwei Wang
Zuheng Xu
William J Welch
Z. Jane Wang
Venue: ICLR 2021 Poster (2021)
Keywords: Conditional generative adversarial networks
image generation
continuous and scalar conditions

Abstract: This work proposes the continuous conditional generative adversarial network (CcGAN), the first generative model for image generation conditional on continuous, scalar conditions (termed regression labels). Existing conditional GANs (cGANs) are mainly designed for categorical conditions (e.g., class labels); conditioning on a continuous label is mathematically distinct and raises two fundamental problems: (P1) Since there may be very few (even zero) real images for some regression labels, minimizing existing empirical versions of cGAN losses (a.k.a. empirical cGAN losses) often fails in practice; (P2) Since regression labels are scalar and infinitely many, conventional label input methods (e.g., combining a hidden map of the generator/discriminator with a one-hot encoded label) are not applicable. The proposed CcGAN solves the above problems, respectively, by (S1) reformulating existing empirical cGAN losses to be appropriate for the continuous scenario; and (S2) proposing a novel method to incorporate regression labels into the generator and the discriminator. The reformulation in (S1) leads to two novel empirical discriminator losses, termed the hard vicinal discriminator loss (HVDL) and the soft vicinal discriminator loss (SVDL) respectively, and a novel empirical generator loss. The error bounds of a discriminator trained with HVDL and SVDL are derived under mild assumptions in this work. A new benchmark dataset, RC-49, is also proposed for generative image modeling conditional on regression labels. Our experiments on the Circular 2-D Gaussians, RC-49, and UTKFace datasets show that CcGAN is able to generate diverse, high-quality samples from the image distribution conditional on a given regression label. Moreover, in these experiments, CcGAN substantially outperforms cGAN both visually and quantitatively.

Published as a conference paper at ICLR 2021
CCGAN: CONTINUOUS CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS FOR IMAGE GENERATION
Xin Ding", Yongwei Wang', Zuheng Xu, William J. Welch, Z. Jane Wang, The University of British Columbia (xin, dindiiin igssat,,, yongwe wêece, zuheng. xuêstat, will@stat, 2 janewecce) ca
ABSTRACT
This work proposes the continuous conditional generative adversarial network (CcGAN), the first generative model for image generation conditional on contin- uous, scalar conditions (termed regression labels). Existing conditional GANs (cGANs) are mainly designed for categorical conditions (e.g., class labels); condi- tioning on regression labels is mathematically distinct and raises two fundamental problems: (P1) Since there may be very few (even zero) real images for some regression labels. minimizing existing empirical versions of cGAN losses (a.k.a. empirical cGAN losses) often fails in practice; (P2) Since regression labels are scalar and infinitely many, conventional label input methods (e.g., combining a hidden map of the generator/discrimnnato with one-hot encoded label) are not applicable. The proposed CcGAN solves the above problems, respectively, by (SI) reformulating existing empirical cGAN losses to be appropriate for the continuous scenario; and (S2) proposing a novel method to incorporate regression labels into the generator and the discriminator. The reformulation in (S1) leads to novel empirical discriminator losses, termed the hard vicinal discrininator loss (HVDL) and the soft vicinal discriminator loss (SVDL) respectively, and novel empirical generator loss. The error bounds of discriminator trained with HVDL and SVDL are derived under mild assumptions this work. A new benchmark dataset, RC-49, is also proposed for generative image modeling conditional on regression labels. Our experiments on the Circular 2-D Gaussians, RC-49, and UTCe datasets show that CcGAN is able to generate diverse, high-quality samples from the image distribution conditional on given regression label. Moreover, in these experiments, CcGAN substantially outperforms cGAN both visually and quantitatively.
INTRODUCTION
Conditional generative adversarial networks (cGANs), first proposed in (Mirza & Osindero, 2014). aim to estimate the distribution of images conditioning on some auxiliary information, especially class labels. Subsequent studies (Odena et al., 2017: Miyato Koyama 2018 Brock et al., 2019 Zhang et al. 2019)) confirm the feasibility of generating diverse, high-quality (even photo-realistic,, and class-label consistent fake images from class-conditional GANs. Unfortunately, these cGANs do not work well for image generation with continuous, scalar conditions, termed regression labels, due to two problems:
(P1) cGANs are often trained to minimize the empirical versions of their losses (a.k.. the empirical cGAN losses) on some training data, a principle also known as the empirical risk minimization (ERM) (Vapnik 2000). The success of ERM relies large sample size for each distinct condition. Unfortunately, we usually have only a few real images for some regression labels. Moreover, since regression labels continuous, some values may not even appear in the training set. Consequently, a cGAN cannot accurately estimate the image distribution conditional on such missing labels.
(P2) In class-conditional image generation, class labels are often encoded by one-hot vectors or label embedding and then fed into the generator and discriminator by hidden concatenation (Mirza &
Eqqual contribution

---

Published as a conference paper at ICLR 2021
Osindero 2014).
auxiliary classifier (Odena et al. 2017) label projection (Miyato & Koyama 2018). A precondition for such label encoding is that the number of distinct labels (e.g., the number of classes) is finite and known. Unfortunately, in the continuous scenario, may have infinite distinct regression labels.
A naive approach solve (P1)-(P2) is "bin" the regression labels into series of disjoint intervals and still train a cGAN in the class-conditional manner (these interval are treated as independent classes) (Olmschenk 2019). However, this approach has four shortcomings: (1) our experiments in Section 4 show that this approach often makes cGANs collapse; (2) we can only estimate the image distribution conditional on membership in an interval and not on the target label; (3) large interval width leads to high label inconsistency; (4) inter-class correlation is not considered (images in successive intervals have similar distributions).
In machine learning, vicinal risk minimization (VRM) (Vapnik 2000; Chapelle et al. 2001)) is an alternative rule ERM. VRM assumes that a sample point shares the same label with other samples in its vicinity. Motivated by VRM, in generative modeling conditional on regression labels where we estimate a conditional distribution p( p(all)) (æ is an image and y is a regression label), is natural to assume that small perturbation to results in negligible change to p(æ) yy. This assumption is consistent with our perception of the world. For example, the image distribution of facial features for population of 15-year-old teenagers should be close to that of ye-year olds.
We therefore introduce the continuous conditional GAN (CeGAN) to tackle (P1) and (P2). To our best knowledge, this is the first generative model for image generation conditional regression labels. It is noted that |Rezagholizadeh et al. (018)) and RRezahhliiaddh & Haidar (2018) train GANs in an unsupervised manner and synthesize unlabeled fake images for a subsequent image regression task. [Olmschenk et al. (2019) proposes a semi-supervised GAN for dense crowd counting. CcGAN fundamentally different from these works since they do not estimate the conditional image distribution. Our contributions can be summarized as follows:
We propose in Section 2 the CeGAN to address (P1) and (P2), which consists of two novel empirical discriminator losses, termed the hard vicinal discriminator loss (HVDL) and the seft vicinal discriminator loss (SVDL), a novel empirical generator loss, and a novel label input method. We take the vanilla cGAN loss an example to show how to derive HVDL, SVDL, and the novel empirical generator loss by reformulating cxisting empirical cGAN losses.
We derive in Section the error bounds of a discriminator trained with HVDL and SVDL.
In Section 4 we propose a new benchmark dataset, RC-49, for the generative image modeling conditional on regression labels, since very few benchmark datasets are suitable for the studied continuous scenario. We conduct experiments on several datasets, and our experiments show that CcGAN not only generates diverse, high-quality, and label consistent images, but also substantially outperforms cGAN both visually and quantitatively.
2 FROM CGAN TO CcGAN
In this section, provide the solutions (S1)-(S2) to (P1)-(P2) in one-to-one manner by introducing the continuous conditional GAN (CcGAN). Please note that theoretically cGAN losses (e.g., the vanilla cGAN loss (Mirza & Osindero 2014), the Wasserstein loss (Arjovsky et al. 2017). and the hinge loss (Miyato et al. 2018)) are suitable for both class labels and regression labels; however, their empirical versions fail in the continuous scenario (i.e., (P1)). Our first solution (S1) focuses on reformulating these empirical cGAN losses to fit into the continuous scenario. Without loss of generality, we only take the vanilla cGAN loss as an example show such reformulation (the empirical versions of the Wasserstein loss and the hinge loss can be reformulated similarly).
The vanilla discriminator loss and generator loss (Mizza & Osindero 2014) are defined as:
L(D) - -Eyoep- (alp))))))) [log (D(x,, y)il| (xy (Ezesps (elv) (log (1 D(x,, v)][]
/
log( ((e, (,),, y)dxdy /
log(1 D(x, (x0 v)dxddy.
(I)
2

---

Published as a conference paper at ICLR 2021
[(G) (a ( [log (D(G(2,)) - -
(2)
where E X' is an image of size d x d, y E y is a label, Pr (v( and (2()) are respectively the true and fake label marginal distributions, p, (a( (æll)) yy) and Py( ((xly) (y) are respectively the true and fake
image distributions conditional on Pr(x, y) and PT( y) are respectively the true and fake joint distributions of z and y, and (()) the probability density function of (O I).
Since the distributions in the losses of Eqs. T and 2 are unknown, for class-condes caus--onnittioal image generation, Mirza Osindero (2014) follows ERM and minimizes the empirical losses:
No
C
No log(1 - D( c)),
(D) N
log D( 6))
(3)
N9
C
N
E (G) = N9 Σ
log ((
j=1
where C is the number of classes, N and N are respectively the number of real and fake images. No and Nº are respectively the number of real and fake images with label c, æi., and x are respectively the j-th real image and the j-th fake image with label and the Zej are independently and identically sampled from (z)). Eq. (3) implies we estimate (r y) and Py (a y) by their empirical probability density functions as follows:
C ΣΣ (1x-xx1,)80 c), () (x. v) = N c=1==1
1 No
Σ
8(x x22)0(0 - c). (5)
11
where 8(-) is a Dirac delta mass centered at However, p y) and (z y) in Eq. 5 are not good estimates in the continuous scenario because of (P1).
To overcome (P1), we propose a novel estimate for each of p. (ax y) and py y), termed the hard vicinal estimate (HVE). We also provide an intuitive alternative to HVE, named the soft vicinal estimate (SVE). The HVEs of p, (x. y) and Py (I., y) are:
N
(y 12
HIVE Pr (x, 1) = =C
Σ
Σ 1 110-0115 - x5))
exp
N
20
1=1
(6)
(y v
No
P5 SHVE (x, v) - C2
N Σ
Un- x4))
exp
20
where aç and x are respectively real image and fake image i, yf and yrare respectively the labels of aς and x! K and a are two positive hyper-parann paprr-parameters, C1 and C2 are two constants making these two estimates valid probability density functions, N is the number of the of satisfying ly- v/1 S K. Ng.x is the number the y satisfying ly - xe S K, and is an indicator function with support in the subscript. The terms in the first square brackets of p!!VE and pHIVE imply we estimate the marginal label distributions (y) and (y)) by kernel density estimates (KDEs) (Silverman, 1986). The terms y
designed based on the assumption that small perturbation results in negligible changes to p-(x (2) and Pg (x|y)) yy) If this assumption holds, we can use images with labels small vicinity of y to estimate Pr ly) and Py (xly)) The SVEs of p, (æ. y) and Pg (xz, y) are:
in the second square brackets
N
(g
19) 20
MN w (v) 8)((x - xx))
SVE (x,, (x. v) = C3
exp
N /=1
Σ w (16
No
(y )2
1==1 : (1. y) y)0(x x!))
SVE. (x, y) CCa
P9
N9
20
Σ w (y.
1=1
3

---

Published as a conference paper at ICLR 2021
where C3 and C4 two constants making these two estimates valid probability density functions,
w (vf v) e -(0 and w (v. = -
(8)
and the hyper-parameter > O. In Eq. 77. similar to the HVEs, estimate pr(3) and Pa (v) by KDEs, Instead of using samples in a hard vicinity, the SVEs use all respective samples to estimate (xlv) and Py (2 y) but each sample is assigned with weight based on the distance of its label from y. Two diagrams in Fig. [ visualize the process using hard/soft vicinal samples to estimate p(a (y)y, i.e., univariate Gaussian distribution conditional on its mean y.
The
Poetrs
1001
Y1
y1
y22
y-k
Hard Vicinity
y+x
Soft Vicinity
(a) Hard Vicinity
(b) Soft Vicinity
Figure I: HVE (Eq. (6 and SVE (Eq. estimate p(zlyy (a univariate Gaussian conditional on y) using two samples in hard and soft vicinities, respectively, of y. To estimate p(æly) (the red Gaussian curve) only from samples drawn from p(æly)) and p( (the blue Gaussian curves), estimation based on the samples (red dots) in a hard vicinity (defined by y + k) or a soft vicinity (defined by the weight decay curve) around y. The histograms in blue are samples in the hard or soft vicinity. The labels /ı, y, and y/2 the r-axis denote the means of conditional on /ı, y, and y2, respectively,
By plugging Eq. (6) and into Eq. T.. we derive the hard vicinal discriminator loss (HVDL) and the soft vicinal discriminator loss (SVDL) as follows:
N
Cs
tleftervilss log log(D(xf), V/ +ce)
HHDD (D)
E (
N
N torie
(9)
Co
N9
N
fluf +co - SK) log(1 - D( : 19 e01)
E Baa~N(0
N9
No +e0,0
1=1
C7
N
N
w (Ol' U3 + +c)
ESVDI =
E. Be~~000
log(D(20), + +6"))
N
ΣM w (Vf 1/ + +
Cs
N
N
wd 1 + E9)
E. (20
log(1 D(z 89 + +0))
Ng
No D w (y, v/ + cd)
(10)
where d" A y V/. の
vg. and Cs. Co. C7, and Cs
constants.
Generator training: The generator of CeGAN is trained by minimizing Eq.
N
(c) = No
Σ
E (, log(D vl + + vi + )))
(II)
(i) Given a label y as the condition, overcome (P1)? The solution (S1) includes:
How do HVDL, SVDL, and Eq.
images in hard/soft vicinity of y train the discriminator instead of just using images with label y. It enables us to estimate p, (xly) when there are not enough real images with label y.
(ii) From Eqs. 9 and (10), we can see that the KDEs in Eqs. 6 and 7 are adjusted by adding Gaussian noise to the labels. Moreover, in Eq. (T). add Gaussian noise seen labels (assume
4

---

Published as a conference paper at ICLR 2021
y 's are seen) to train the generator to generate images at unseen labels. This enables estimation of (r (/)) when y' is not in the training set.
How is (P2) solved? We propose novel label input method. For G. add the label y element- wisely to the output of its first linear layer. For D, an extra linear layer is trained together with D to embed y in latent space. We then incorporate the embedded label into D by the label projection (Miyato & Koyama 2018). Please refer to Supp. [S.3] for more details.
Remark 1. An algorithm is proposed in Supp. S.2 for training CcGAN in practice. Moreover, CeGAN does not require any specific network architecture, therefore it can also use the state-of-art architectures in practice such as SNGAN (Miyato al. 2018) and BigGAN (Brock et al. 2019).
ERROR BOUNDS
In this section, derive the error bounds of a discriminator trained with LHVDL and [ under the theoretical loss C. First, without loss of generality, we assume E [0, 1]. Then, we introduce some notations. Let D stand for the Hypothesis Space of D. Let p. and Pg 5KDE (y) stand for the KDEs
of (r)) and Py (g) respectively. Let (v)(1))))) & "" ""(y' s)p""y") ple (3)())) A w= ""(o" W" (y) & W=(g)
(=)y)
[ [w" (y', () )dy' and (y(y) A wd wy',, v)P, Jdy'. Denote by D the optimal discriminator Goodfellow et al. 2014) which minimizes L but may not be in D. Let & arg oigmin C(D Let
DUVDI A arg min min CIVDL similarly, define SSVDI Definition 1. (Hölder Class) Define the Hölder class of functions
[[) A (p 12 E V, A.. 0, s.1.1p(((1) p'()2) T/41 121)
(12)
Please see Supp. S for more details of these notations. Moreover, we will also work with the
following assumptions: (A1) All D's in are measurable and uniformly bounded by Let U a
max(sup |- log D) suppeD | log(1 D)}} and U < oo; (A2) For Vo E X and y,y' EV 3g > 0 and M > 0, s.t. (</l') Pr y) < g" (x)lv' - -yl with f (a dæ M"; (A3) For Vx E and y. y, 3 (x) and M > 0, s.t. [pg(all//) Py (2)l)) S (x)ly' vl
with J gd (x)dx = M (A4) (v( E E(L") and Py(v) E E(L?).
Theorem 1. Assume that (AI)-(A4) hold, then V8 E (0, 1). with probability at least 8.
L(HNVDL) - L(D")
OKDE log Nr Nro + L'O2
CKDE log No 2,6
22U
+ 2U
+ L9O2
+ + M9)
Noo
[V Ngg. )
(8) Byr-ppeee())
+ 2U log
8) )
+ E Egyn (1)
+ CL L(D'),
V
(13)
for some constants OKDE, C depending on 8.
Theorem 2. Assume that (AI)-(A4) hold, then V8 E (0, 1). with probability at least 8.
((SSVDL) - L(D")
CKDE log N Noo
CDD log No 2,8
<2U
+ 1
+ 2U
+ 1
Nag
log (15)
( (pDE)))))
(w((o)))
(14)
+ 2U
VNT E
+ VNO
E Bar (1)
(w(w))
+ (M'E (MEyy)ttE())
[Ey W - vl] + M M By-pp
( [Eyr
lv wll)
+ L(D) L(D"),
for some constant KDD CKDE depending 8.
5

---

Published as a conference paper at ICLR 2021
Remark 2. The error bounds in both theorems reflect the distance of DHVDI and DSVDL from D Enlightened by the two upper bounds. when implementing CcGAN, we should (1) avoid letting D output extreme values (close to 0 or 1) that U is kept at moderate level; (2) avoid using too small or too large k or v to keep the third and fourth terms moderate in Eqs. (13) and (14). Please see Supp. S.5.2.3 for a more detailed interpretation and Supp. [S.5.2 for the proofs.
4 EXPERIMENT
In this section, study the effecttivnness of CcGAN on three datasets where cGAN (Mirza & Osindero 2014) cannot generate realistic samples. For a fair comparison, cGAN and CcGAN use the same network architecture (a customized architecture for Circular 2-D Gaussians and the SNGAN (Miyato et al. 2018) architecture for RC-49 and UTKFace) except for the label input modules. For stability, image labels are normalized [0, 1] in the RC-49 and UTKFace datasets during training.
4.1 CIRCULAR 2-D GAUSSIANS
We first test on the synthetic data generated from 120 2-D Gaussians with different means.
Experimental setup: The means of the 120 Gaussians are evenly arranged on unit cirele centered at the origin O of a 2-D space. The Gaussians share common covariance matrix 82 121,,, where = 0.02. We generate samples (blue dots) from these Gaussians with their means (red dots) on a unit circle. The unit circle can be seen as a clock where we take the mean at 12 o' cclock (point A) as the baseline point. Given another mean on the circle (point B), the label y for samples generated from the Gaussian with mean B is defined as the clockwise angle (in radians) between line segments OA and OB. E.g., the label for samples from the Gaussian at A is O. Both cGAN and our proposed CeGAN are trained on this training set. When implementing cGAN, angles are treated as class labels (each Gaussian is treated as a class); while when implementing CeGAN, angles are treated as real numbers. The network architectures of cGAN and CcGAN are shown in Supp. S.6.1 Both cGAN and CcGAN are trained for 6,000 iterations. We use the rule of thumb formulae in Supp. S.4 select the syper-parameters of HVDL and SVDL, i.e., a se 0.074, K & 0.017 and = 3600 (see Supp. S.6.2 for details).
samples from each Gaussian for training. Fig. 2a shows 1,200 training
For testing, we choose 360 points evenly distributed on the unit circle as the means of 360 Gaussians. For each Gaussian, we generate 100 samples, yielding a test set with 36,000 samples. It should be noted that, among these 360 Gaussians, least 240 are not used at the training. In other words, there are at least 240 labels in the testing set which do not appear in the training set. For each test angle, we generate 100 fake samples from each trained GAN, yielding 36,000 fake samples from each GAN in total. The quality of these fake samples is evaluated. We repeat the whole experiment three times and report in Table Tthe average quality over three repetitions.
Evaluation metrics and quantitative results: In the label-conditional scenario, each fake sample with label y is compared with the mean (sin(y) (y). cos(y) (y)) of a Gaussian on the unit circle with label y. A fake sample is defined as "high-quality" if its Euclidean distance from to (sin()).cos({)) is smaller than 45 = 0.08. A mode (i.e., a Gaussian) is said to be recovered if at least one high-quality sample is assigned to it. We also measure the quality of fake samples with label y by computing the 2- 2-Wasserstein Distance (Wzz) (Peyré et al. [2019) between pr (xlyy) = - ((sin(y)) cos(y)) āT) and Py (xll) (ug M3)). where we assume Py [ (ly)) Gaussian and its mean and covariance are estimated by the sample mean and sample covariance of 100 fake samples with label In Table we report the average percentage of high-quality fake samples and the average percentage of recovered modes over 3 repetitions. We also report the average W2 over 360 testing angles. We can see CeGAN substantially outperforms eGAN.
Visual results:
We select 12 angles which do not appear in the training set. We then use cGAN and CcGAN to generate 100 samples for each unobserved angle. Fig. visually confirms the obervation from the numerical metrics: the fake samples from the two CcGAN methods are more realistic.
4.2 RC-49
Since most benchmark datasets in the GAN literature do not have continuous, scalar regression labels, we propose new benchmark -C-ns, a synthetic dataset created by rendering 49 3-D chair

---

Published as a conference paper at ICLR 2021
Table Average quality of 36,000 fake samples from cGAN and CcGAN over three repetitions with standard deviations after the "+" symbol. "L" ("9") indicates lower (higher) values are preferred.
Method
% High Quality
% Recovered Modes
2 2-Wasserstein Dist. 4
eGAN (120 elasses)
68.8 4.8
81.8 3.9
3.32 10 2 3.13 10-2
CoGAN (HVDL)
99.3 + 0.4
100.0 + 0.0
3.03 10 - + 5.05 10-5
CoGAN (SVDL)
99.6 1 0.1
100.0 1 0.0
2.56 10 1 8.95 10-6
(a) 1200 training samples and 120 means (red dots).
(b) cGAN
(c) CeGAN (HVDL)
(d) CeGAN (SVDL))
Figure 2: Visual results for the Circular 2-D Gaussians simulation. (a) shows 1,200 training samples
from 120 Gaussians, with 10 samples per Gaussian. In (b) to (d), each GAN generates 100 fake samples at each of 12 means not appearing in the training set, where green and blue dots stand for fake and real samples respectively.
models different yaw angles. Each of 49 chair models is rendered at 899 yaw angles ranging from 0.1 to 89.9 with step size 0.1. Therefore, RC-49 consists of 44,051 64 x 64 rendered RGB images and 899 distinct angles. Please see Supp. S.7 for more details of the data generation. Some example images are shown in Fig. 3
Not all images are used for the cGAN and CcGAN training. A yaw angle is selected for training if its last digit is odd. Moreover, at each selected angle, only 25 images are randomly chosen for training. Thus, the training set includes 1 1250 images and 450 distinct angles. The remaining images are held out for evaluation.
Experimental setup:
When training cGAN, we divide 10.1, 89.9] into 150 equal intervals where each interval is treated as class. When training CcGAN, we use the rule of thumb formulae in Supp- S.4 to select the three hyper-parameters of HVDL and SVDL, i.e., o Rs 0.047, K z 0.004 and v = 50625. Both cGAN and CcGAN are trained for 30,000 iterations with batch size 256. Afterwards, we evaluate the trained GANs on all 899 angles by generating 200 fake images for each angle. Please see Supp. S.7 for the network architectures and more details about the training/testing setup.
To evaluate (1) the visual quality, (2) the intra-label diversity, and (3) the label consistency (whether assigned labels of fake images consistent with their true labels) of fake images, we study an overall metric and three separate metrics here. (i) Intra-FID (Miyato & Koyama| 20188) is utilized as the overall metric. It computes the Fréchet inception distance (FID) (Heusel et al. 2017) separately at each of the 899 evaluation angles and reports the average FID score. (ii) Naturalness Image Quality Evaluator (NIQE) (Mittal et al. (0022) measures the visual quality only. (iii Diversity is the average entropy of predicted chair types of fake images over evaluation angles. (iv) Label Score is the average absolute error between assigned labels and predicted labels. for details of these metrics.
Quantitative and visual results:
Please see Supp. S.7.5
We report in Table 2 the performances of each GAN. The example fake images in Fig. 3 and line graphs in Fig. 5 support the quantitative results. cGAN often generates unrealistic, identical images for a target angle (i.e., low visual quality and low intra-label diversity). "Binning" [0.1, 89.9) into other number of classes (e.g., 90 classes and 210 classes) is also tried but does not improve cGAN's performance. In contrast, strikingly better visual quality and higher intra-label diversity of both CcGAN methods are visually evident. Please note that CcGAN is designed to sacrifice some (not too much) label consistency for better visual quality and higher diversity, and this explains why CeGAN does not outperform eGAN in terms of the label score in Table 2
7

---

Published as a conference paper at ICLR 2021
Table 2: Average quality of 179,800 fake RC-49 images from cGAN and CcGAN with standard deviations after the "+" symbol. "L" ("1") indicates lower (higher) values are preferred.
Method
Intra-
NIQE +
Diversity
Label Score
cGAN (150 classes)
1.720 = 0.384
2.731 = 0.162
0.779 + 0.199
4.815 + 5.152
CcGAN (HVDL)
0.612 + 0.145
1.869 + 0.181
2.353 + 0.121
5.617 1 4.452
CcGAN (SVDL)
0.515 + 0.181
1.853 = 0.159
2.610 + 0.113
4.982 + 4.439
4.3 UTKFACE
In this section, we compare CcGAN and cGAN on UTKFace Zhang et al. 2017). dataset consisting of RGB images of human faces which are labeled by age.
Experimental setup: this experiment, we only use images with age in [1. 60). Some images with bad visual quality and watermarks are also discarded. After the preprocessing, 14,760 images are left. The number of images for each age ranges from 50 to 1051. We resize all selected images to 64 x 64. Some example FTKFaae images are shown in the first image array in Fig
When implementing cGAN, each age is treated as a class. For CeGAN we use the rule of thumb formulae in Supp. SS.4 to select the three hyper- parameters of HVDL and SVDL,, i.e., a R 0.041, Re 0.017 and v 3600. Both cGAN and CcGAN are trained for 40,000 iterations with batch size 512. In testing, we generate 1,000 fake images from each trained GAN for each age. Please see Supp. S.8 for more details of data preprocessing, network architectures and training/testing setup.
Quantitative and visual results: Similar the RC-49 experiment, evaluate the quality of fake images by Intra-FID, NIQE, Diversity (entropy of predicted races), and Label Score. We report in Table 3 the average quality of 60,000 fake images. We also show in Fig. 4 some example fake images from cGAN and CeGAN and line graphs of FID/NIQE versus ages in Fig. 5 Analogous to the quantitative comparisons,
that CcGAN performs much better than cGAN.
Table 3: Average quality of 60,000 fake UTKFace images from cGAN and CeGAN with standard deviations after the "4" symbol. " ("1") indicates lower (higher) values are preferred.
Method
Intra-FII FID 4
NIQE 4
Diversity
Label Score 1
cGAN (60 classes)
4.516 0.965
2.315 0.306
0.254 0.353
11.087 + 8.119
CeGAN (HVDL)
0.572 0.167
1.739 0.145
1.338 0.178
9.782 + 7.166
CcGAN (SVDL)
0.547 + 0.181
1.753 + 0.196
1.326 0.198
10.739 + 8.340
4.5
13.5
22.5
15
21
40.5
26
32
58.5
39
45
78.5
51
85.5
Angle
Real
CGAN
OoGAN HVDL
CoGAN, SVDL
Age
Real
6GAN
CoGAN, HVDL CoGAN, SvoL
Figure 3: Three RC-49 example images for each of 10 angles: real images and example fake im- ages from cGAN and two proposed CcGANs, respectively. CeGANs produce chair images with higher visual quality and more diversity.
Figure Three UTKFace example images for each of 10 ages: real images and example fake images from cGAN and two proposed CcGANs, respectively. CcGANs produce face images with higher visual quality and more diversity.
8

---

Published as a conference paper at ICLR 2021
22-
213
10
(a) RC-49: FID vs Angle (b) RC-449: NIQE vs Angle (c) UTKFace: FID vs Age (d) UTKFaae: NIQE vs Age
Figure 5: Line graphs of FID/NIQE versus regression labels RC-49 and UTKFace. Figs. (
Show that two CeGANs consistently outperform cGAN across all regression labels. The graphs of CcGANs also appear smoother than those of cGAN because of HVDL and SVDL.
5 CONCLUSION
As the first generative model, we propose the CcGAN in this paper for image generation conditional on regression labels. In CcGAN, two novel empirical discriminator losses (HVDL and SVDL), novel empirical generator loss and a novel label input method are proposed to overcome the two problems of existing cGANs. The error bounds of a discriminator trained under HVDL and SVDL are studied in this work. A new benchmark dataset, RC-49, is also proposed for the continuous scenario. Finally we demonstrate the superiority of the proposed CcGAN cGAN on the Circular 2-D) Gaussians, RC-49, and UTKFaae datasets.
ACKNOWLEDGMENTS
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grants CRDPJ 476594-14 14, RGPIN-2019-05011 and RGPAS2011-500965.
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions paltern analysis and machine (acpligencee3877:14422333 2015.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 2017. PMLR.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Advances in neural information processing systems, pp. 416-422, 2001.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural informa- fion processing systems, pp. (672-2680, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.

---

Published as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of styleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1110--119, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 0rXiv:1411.117 2014.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):20-212, 2012.
Takeru Miyato and Masanori Koyama. eGANs with projection discriminator. In International Conference on Learning Representations, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 180.05970,. 2018.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. Proceedings of the 34th International Conference on Machine Learning-Vor Volume 70, pp. 2642 2651, 2017.
Greg Olmschenk. Semi-supervised Regression with Generative Adversarial Networks Using Minimal Labeled Data. PhD thesis, 2019.
Greg Olmschenk, Jin Chen, Hao Tang, and Zhigang Zhu. Dense crowd counting convolutional neural networks with minimal data using supi-supe dual-goal generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition: Learning with Imperfect Data Workshop, 2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6)355-660 2019.
Mchdi Rezagholiradeh and Md Akmal Haidar. Reg-GAN Semi-supervised learning based on gener- ative adversarial networks for regression. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2806 2810. IEEE, 2018.
Mehdi Rezagholizadeh, Md Akmal Haidar, and Dalei Wu. Semi-supervised regression with generative adversarial networks, November 22 2018. US Patent App. 15/789,518.
Bernard W Silverman. Density estimation for statistics and data analysis, volume 26. CRC press, 1986.
Vladimir Vapnik. The nature of statistical learning theory. Springer, 2nd edition, 2000.
Larry Wasserman. Density estimation @ONLINE. http://wwwz stat. cmu war...edu/aa -larry/
sml /tt.///iiiyestttattiii.too pdf
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 7354-7363, 2019.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regresioon by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5810-5818, 2017.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances neural information processing systems, 2020.
10

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL
S.1
GITHUB REPOSITORY
Please find the codes for this paper Github:
S.2 ALGORITHMS FOR CcGAN TRAINING
Algorithm I: An algorithm for CcGAN training with the proposed HVDL.
Data: N real image-label pairs { ........ (arrr, (oxt,9x=)) ;8x=). Ngy ordered distinct labels
T= {yii).... MNngi / in the dataset, preset a and K, number of iterations K, the discriminator
batch size m", and the generator batch size m
Result: Trained generator G.
for
to K do
Train D;
Draw m labels yd with replacement from T:
Create a set of target labels yd = {3 + ely, E Y (( L,... m (D training conditional on these labels);
Initialize 02 = 2 = 0;;
for = to m do
Randomly choose an image-label pair (æ, v) e Q2 satisfying ly S K where
y/ e ydd and let 525 = 22g U (a, yi + c). Randomly draw label y from Uw + R, yi e k) and generate fake image x' by
evaluating G(z, s)), where
(o0, f). Let 022 2 U (x' y. e).
end
Update D with samples in set O2 and 9/ via gradient- based optimizers based on Eq, 6 Train G;
10
12
Draw m² labels y with replacement from Y:
13
Create another set of target labels y {y. + clys Y e (V L...,,, m²) (G training conditional these labels)
Generate m² fake images conditional on ya and put these image-label pairs in 22/
15
Update G with samples in 2/ via gradient-based optimizers based on Eq. [1]
16
end
Remark S.3. It should be noted that, for computational efficiency, the normalizing constants
Ney+er.e Ng 1/y uc Em w" (v Vf + +"), and No w vi e9) in Eq. excluded from the training and only used for theoretical analysis.
and I00
S.3 MORE DETAILS OF THE PROPOSED LABEL INPUT METHOD IN SECTION 2
We propose a novel way to input labels to the conditional generative adversarial networks. For the generator, add regression label element-wise the feature map of the first linear layer. For the discriminator, labels are first projected to a latent space learned by an extra linear layer. Then, we incorporate the embedded labels into the discriminator by the label projection (Miyato & Koyama 2018). Figs. S.3.6 and S.3.7 visualizes our proposed label input method. Please refer to our codes for more details.
S.4 A RULE OF THUMB FOR HYPER-PAR PARAMETER SELECTION
In our experiments, normalize labels to real numbers in [0. 1] and the hyper-parameter selection is conducted based on the normalized labels. To be more specific, the hyper-parameter o is computed based rule-of-thumb formula for the bandwidth selection of KDE (Silverman, 1986). i.e.,
11

---

Published as a conference paper at ICLR 2021
Algorithm 2: An algorithm for CcGAN training with the proposed SVDL. Data: N" real image-label pairs Ω { vf)...., (xxxr,W6x=) Nay ordered distinct labels
T= [ B[Ng)) / in the dataset, preset and number of iterations K. the discriminator batch size vn and the generator batch size m
Result: Trained generator C.
for to K do
Train D:
Draw m labels yd with replacement from T;
I...., m (D training is Create set of target labels y = {y1 + clys E er EN,, l...,. conditional on these labels);
Initialize 225 = 2
for = to m 4 do
Randomly choose an image-lahel pair (æ, v) E Q2 satisfying e -(s-u--)) > 10 where ys + c E yd and let 22 22 U (x, di + c). This step is used to exclude real images with too small weights.
Compute w/ (y, + <"x
Randomly draw label y' from Uw +
10-3
3/4 - e
10 and generate
fake image x' by evaluating G(z, y') where ~N(0 Lot S = 12 v ve)
Compute (y y4 e -(="toct--01²²
11
end
Update D with samples in set 22 and 2/ via gradient-based optimizers based on Eq Train G;
12
13
Draw m² labels y with replacement from T;
15
Create another set of target labels y = [ys + clys E E training is conditional these labels)
mm) (G
Generate m fake images conditional on y and put these image-lahel pairs in 2/
16
17
Update G with samples in 22 via gradient-bas based optimizers based on Eq. [I]
18 end
noise
label
Image
label
z
y
Linear layer 2
Conv. layers
Linear layer
Inner product
Add element-wise
Linear layer
Conv. layers
Add
X
real/fake
Image
Figure S.3.6: The label input method for the generator in CcGAN.
Figure S.3.7: The label input method for the discriminator in CeGAN.
(463//3NT)) 1/5, where ōy is the sample standard deviation normalized labels in the training
set. Let Khase = max
M[a ..... VINE 2[Ng -1) where w/g is the I-th smallest
12

---

Published as a conference paper at ICLR 2021
normalized distinct real label and Ny is the number of normalized distinct labels in the training set. The x is set as a multiple of Kbase (i.e., K = myNbaae) where the multiplier my stands for 50% the minimum number of neighboring labels used for estimating p, (xly) given label y. For example, mR = 1 implies using neighboring labels (one on the left while the other one on the right). In our experiments, mhe is generally too few real samples, we may consider increasing m.. We also found v = 1/k² works well in our experiments.
or 2. In some extreme case when many distinct labels have
S.5
MORE DETAILS OF THEOREMS S.4 AND S.5
S.5.1 SOME NECESSARY DEFINITIONS AND NOTATIONS
The hypothesis space D is a set of functions that can be represented by D neural network with determined architecture).
In the HVDL case, denote by (x) J (ally )p) (pr(pp/(ddd)) (v')dy' the marginal distribution
of real images with labels in [y K. y x| and similarly to pl (x) of fake images.
if the number of real and fake images are infinite, the empirical density converges to pp:" (x) & I (x=ly')))) "v('".))pe((") dy' and pg (a)) J Pa(l)))") w w"(''" v))a(() dy' respectively, W*(y)
In the SVDL case, given and weight functions (E.q.
W=()) where (w) ww (1/,v)p, v)p,(y')dy and (* a [ww(( (y',,)), ()pg(//)dy
Let pT w"("" and pl (v/l)) A (" (v'))))(())) Wr(p))
Wo(g))
The Hölder Class defined in Definition T is set of functions with bounded second derivatives, which controls the variation of the function when parameter changes. (A4) implies the two probability density functions pr (y) and (v()) are assumed in the Hölder Class.
Given G, the optimal discriminator which minimizes is in the form of
(r y)
D (a, Pr v) + Pg (x,
(S.15)
However, D may not be covered by the hypothesis space D. The D the minimizer of C in the hypothesis space D. Thus, L(D) - L(D*) should be a non-negative constant. In CoGAN, we minimize (H)DL(D) or LHvDL(D) with respect to D E DD so we are more interested in the distance of DUVDL and DVVDL from D" i.e., L(DUVDL) L(D*) and C(DSvDL)- L(D*).
S.5.2 PROOFS OF THEOREMS 1 AND
S.5.2.1
TECHNICAL LEMMAS
Before we move the proofs Theorems [ndd]] we provide several technical lemmas used in the later proof.
Recall notations and assumptions Sections and I.5.I then we derive the following lemmas.
Lemma S.1. Suppose that (AI)-(A2) and (A4) hold, then 48 E (0, 1), with probability at least -
N
sup DED
Σ
of SN) |- log D(x), y)1 Eapp, [- log D(x; y)]
Nr U.K
jml
(6.16)
2
K
< U,
log
+
2N
8
2
for given y.
13

---

Published as a conference paper at ICLR 2021
Proof. Triangle inequality yields
N
sup
Cly (1v- lolog ( v)1 - E (mmp)(()))) log (a y>
DED
y.
N
sup DED
f1v- |- log Di= y)] B (a) |- log D y)1
Nr U.K
i=1
sup
T
(P (m)
log D(xr, v)] (elp) log ((x,
DED
We then bound the two terms of the RHS separately as follows:
1. Real images with labels in ly Ki k + x] can be seen as independent samples from (xx).
Then the first term can bounded by applying Hoeffding's inequality follows: 4A (0, 1), with at least probability 5,
N
U Ullo log D(x
log D( y)
Σ
sup DED
Nr
u
25 1SK}
U
(ax)
U
1=1
<
log
2Nr
(S.17)
2. For the second term, by the definition of pxx(( and defining Prr(y') = fll plsn)
we have
sup
E [x))""" (ax) [- log ((x, y)] Ex (m/y) |- log D(x, v)]
DED
(by the definition of total variation and the boundness of - D)
(S.18)
(x) (>+lv) dæ.
Then, focusing 1pp"*<x> (=(lll))
(ptl((x)) Pr (xlvv)] / p(a:l))pxa())) p(xlv)
1p(xl1/) p(zlu)) (v
(by (A2))
/ g" (x)ll/ - (PP(/)
s (g""(z))
Thus, Eq. (.118 is upper bounded follows,
sup E " (m) [- log D(r v)] E. (all)) |- log D(x, y)]
DED
rg" (x))dx
(S.19)
(by (A2))
R.M".
By combining Eq. S.17) and S.19 we can get Eq. (..66), which finishes the proof.
Similarly, apply identical proof strategy the fake images 2º and generator distribution Pg Pg(ar) (arly)) ly).
14

---

Published as a conference paper at ICLR 2021
Lemma S.2. Suppose that (A1). (A3) and (A4) hold, then V8 (0, 1), with probability least 1 8,
sup DED
Clu (1v-))1sm) 1- log
D( v))l Bunp, (mll)) |- log(1 (xx, y))l
(S.20)
2
RUM
S U
log
(5]
2N B.r
2
for given y.
Proof. This proof is omitted because it is almost identical to the one for Lemma S.T
The following two lemmas provide the bounds for SVDL. Lemma S.3. Suppose that (AI), (A2) and (A4) hold, then Vo (0, 1). with probability at least 1
8,
1
i= w" (vi y) -- log D( ,3]l Ex (mlw) |- log D(x,, y)]
Nr
sup DED
Ei w" (vf
N
(S.21)
U
4 8
UM
log
+
Pyreet Il ul],
Wr(y)
2N
2
for given
Proof. For brevity, denote by f(x, y) = log D(x, y) and llog D. Then,
No w² (Ve y) - log D(x) w)] E Earp, (alv) |- log D(a, v)]
sup DED
Enn w" (vF
xi= w (yf v))(.)
N'
sup fef
(mlo) [f(x.
ΣM w (VI+
(S.22)
N Li=1 w (yi 4) (1) (w)..)
S sup
E Eanpyi (m) [f(x, v)
fef
1= w (y)
Nr
-mrpz-" (z)) (f y)] (am (f v)]
sup
fEF
where the inequality is by triangular inequality. We then derive bounds for both two terms of the last line.
1. For the first term, we can further split it into two parts,
N Dm w" (v% u) [x)
E Bapp!!" y." (x) [fx v)]l
NF E w (v]. v)
N Ei w (vl .),,,,,.. f
Enn w (w7 2) 220
N
(S.23)
No Ee w (V.
1 No
(r(y)
N w" (V..v)) (x][))) Wr(y)
NF
E Bmppy x0 (mn) [f 3)]
Focusing on the first part of RHS of E.23))
By (A1),
1= w (V) v) f(( v)
N"
NO Σn w (vf (x(()))))
Di w (VF v)
W"(y)
NF IN' (v" v) wr
U
Wr(y)
15

---

Published as a conference paper at ICLR 2021
Note that Vy, vf, w (y',,, (y', = e- Σ 1 and hence given y, w" w" y) is random
variable bounded by 1. Apply Hoeffding's s incquality to the numerator of above, yielding that with probability at least 1 8',
N 1= w0 (0, u) f).
N Di= w (
U W
1
log
N E w (V.
Wr(y)
(v)
2N
(S.24)
Then, consider the second part of RHS of Eq. S.23
Recall that pz.w0 (x)
-
S (xely') wn w/p" (v') dy'. Thus,
((())
No w'(h y) (2) Di= N
E Doupp (x)) [(<x
Wr(v)
Σ
w (wl (f) u) B(a.o/))pp (w ()) (arf v)
(7
N
where (a, (a) "p' "p"(''"""'""""" (y') denotes the joint distribution of real image and its label. Again, since w² wy f y) uniformly bounded by U under (A1), we can apply Hoeffding's inequality. This implies that with probability at least 1 - 8', the above can be upper bounded by
U
log
(S.25)
(y)
V
2Nr
8'
Combining Eq. (S.24) and S.25 and by setting 8' = 5
2. we have with probability at least
NO
1 w (Mt f)
E xxpp." "f
(2) [f (r)
U
1
log
(3)
4
N E w' (Vf 8)
2N
Since this holds for Vf F, taking supremum over f. we have
N 1i w w'( v) f(]
1 log
sup fer
1 v)
E
w-pp "" (0) [f
WT V
2N
55)
(S.26)
2. For the second term on the RHS of Eq. S.22). By (A1) that |/| < U Exwp, (0lw) [(x.
sup fEF
aappiin or (x) [f y)]
< U Ul1pg."" (x) -pr(ar yy) TT
U Ipy.en (z) (zlv))
/
2
Note that by the definition of p & I (xly))) "0 (V) dy and ph, (V'l)) a W (x))
(v' (x))p((( (v/) (()))
we have
Ipp. (x) - (x/(v]l)] - // pr (x (v'lv) dy (x(wv)
$ f (x/(0))))) (=(l)))) Pi (v'l)) dy/.
By (A.2) and
10, 11, the above is upper bounded by g" (x (z)) -pt (w/l)) 18 Il. Thus,
y
sup
E
wapp " (a) [f( Raww,, (2olw) [f w)]
fEF
U
<
g" ( H1l vll dx
(S.27)
UM
Ey
Peymm (v 111l wll
2
16

---

Published as a conference paper at ICLR 2021
Therefore, combining both Eq. (S266) and (S.27). with probability least 1 5, w" (1/ y) -- log ( v)1
sup
(alp)( (alv) l-log (a v)
DED
Ei=1 (v..v) .))
U
4
UM
log
+
B By-ppt(rll 11/ -yl.
Wryy)
2N
2
This finishes the proof.
Lemma S.4. Suppose that (A7), (A3) and (A4) hold, then V8 E (0, 1), with probability at least 1
8,
No 1=
( y) |- log(1 D(x v))l No i==
sup DED
N
Hewpyy (mlv)) log(1 - ((x,,))))
Σ
w (y..9)
No
(S.28)
U
UM
(y(y)
V
2No
log
8
+
2
E p (w/l) I1 v|].
for given y.
Proof. This proof omitted because it almost identical to the one for Lemma S.21
As introduced in Section 2 KDE for the marginal label distribution with Gaussian kernel. The next theorem characterizes the difference between a pr(y) Py( and their KDE using n i.i.d. samples.
Theorem S.3. Let (y) and (y) stand for the KDE of pr (y( and Py (y) respectively, Under condition (A4), the KDEs are based on n i.i.d. samples from Pr/Pg and a bandwidth o, for all 5 E (0, 1), with probability at least 1 1
CKDE log n
KDF (y) pr(1)) S sup Pr
L'
(S.29)
na
2,8 log n
sup p9
KDE (v) Po(1)) S
+ 1L022
(S.30)
no
for some constants OEDE CKDE depending on 6.
Proof. By (Wasserman): P.12), for any p(t) E(L) (the Hölder Class, see Definition with probability at least 0,
CKDE log co²2 sup KDE (t) p(t) S
(S.31)
no
for some constants CKDE and c, where C' depends on 5 and c c= LJ ((s))a|dds. Since in this work, K is chosen as Gaussian kernel, T. K
((s) 2 L.
S.5.2.2 ERROR BOUNDS FOR HVDL AND SVDL
Based on the lemmas and theorems in Supp. S.5.2.1 we derive the error bounds of HVDL and SVDL, which will be used in the proofs of Theorems and
Theorem S.4. Assume that (AI)-(A4) hold, then Vō E (0, 1), with probability at least 5,
EHVDL C(D)
sup DED
OKDE log Nro
CKDE log No 2,8
RU RU + M9)
SU
+ L'O2 +U
L902
N9o
2
(S.32)
+ V log (6)
8
8
E By-ppparra))
[V Ny.n
+ y By-aaa(( (v)
(V No...
17

---

Published as a conference paper at ICLR 2021
for some constants CY, C2, depending on 6.
(HVDL(D) ( as follows
Proof. We first decompose suP
ChNDL (D) C(D)
sup DED
|- log D(,, ((r.)))l..=x))))d (pr(v) (y)ed)
sup
- log(1 D(x )))lpp (Po(v) - ") (u))dy
sup DED
N
Σ (1x-0/1sk
log D (x v)] Banp, (mlp) 1- log (x b)l pxDE (v)dy
sup
Nr v.re i=1
DED
N
sup DED
log(1 D( v))
E (aop), [- log(1 D( v)) (KDE
(y)dy
N9 Bis
These four terms in the RHS can be bounded separately as follows
1. The first term can be bounded by using Theorem S.3 and the boundness of D and y E [0, 1]. For the first term, 4g, E (0, 1), with least probability 51,
sup / / DED
- log D(æ, v) (pr(y) - 1r- D. (2))
(S.33)
CKDE log Nr Nro
<U
+ L'o2
for some constants CTDE depending on 61.
2. The second term can be bounded by using Theorem S.3 and the boundness of D and y E [0, 1]. For the first term, Vo2 E (0, 1), with at least probability 82.
/ [
1- log(1 D(x, y))l Py (xly)d (Po(v) -p3 -D (y))dy
sup DED
(S.34)
KDE log No 2,82 Nro
<U
+ L902
for some constants
2,82
depending on 6z.
3. The third term can be bounded by using Lemma S.1 and S.2 with at least probability 1 03.
For the third term, Vog E (0, 1).
N
sup DED
1 (1u (1v l- log D(x), v)] E (mppp. log D(x, v)]
BKDE (y)dy
NT U.N
1=1
N
f sup DED
log D(x w)) Eeep. log D(ac, w))
Dr
KDF (yDB
Nr
1=1
< <
RUM
VV
log
+
KDE (DD
2Nr
5g
2
18

---

Published as a conference paper at ICLR 2021
Note that g,r expressed as
1== ([)->])>> which is random variable of y/s. The above can be
N
Σ
(Iv u(n-) I- log D ((x,,,)) y)1 E. B pm-p.t |- log ((x, v\1 p)
sup DED
Nr U. ==1
RU
M
2
<
log
E Eyr (1)
(V
2
03
Min
(S.35)
4. Similarly, for the fourth term, Vo, (0, 1), with at least probability 61.
N
sup DED
No
flv (l |- log(1 - D( x))
E Eaw (2/y)
[- log(1 D y))l dæ 2KDE ( (S.36)
KUM
2
U
log
E pADE (1)
2
2
04
No.
With 81 62 63 8a
combining Eq. (S.33)
(S36)) leads to the upper bound in Theorem
S.4
Theorem S.5. Assume that (AI)-(A4) hold, then 45 E (0, 1), with probability at least 1 - 6,
(SVDL L(D)
sup DED
KDE 1,5 Noo
log Nr
CKDE log No
U
+ +
+U
I90²
N90
(S.37)
16
U
log
E Bynpsee
+
Pyrmpene (w)
w
N
Wr(y)
No
U
(M' Bpoe (v) By-ro lo' -vl] + M Ey Eyy (vrl)) lv v/]
+
2
for some constant CKDE C depending on 5.
Proof. Similar to the decomposition for Theorem S.4 supDeD [SVDL(D) (D)) into four terms which be bounded by using Theorem S.3 the boundness of D, Lemma S.3 and Lemma S.4 The detail is omitted because is almost identical the one of Theorem S.4
we can decompose
S.5.2.3 PROOF OF THEOREM 1
Based Theorem S.4
derive Theorem 1
19

---

Published as a conference paper at ICLR 2021
Proof. We first decompose C(DHvDI.)- L(D)) as follows
C(DHVDL)- C(D*)
-[(DHDLL L(DHVDL - - L(D) L(D) - L(D))
(by L(DHVDL)) - -((D) $ 50
LHVDL L(D) + - -(D")
22 sup DED
(by Theorem S.4
22U (V
CKDE
log Nr 1.0 Nro
CDD log No 2.5
+ 2U + L5O2
+ 1902
+ +UU(M* + M9)
Nao
[/
+ 2U
V
88 log
8
E (ymNRREEW (1)
| Nr.
+ Ey Lymppe
(X) (1)
N
+ -C(D)-C(L))
(S.38)
S.5.2.4
PROOF OF THEOREM 2
Based on Theorem S.5
we derive Theorem
Proof. The detail is omitted because it is almost identical to the one of Theorem 1 in Supp. S.5.2.3
S.5.2.5 INTERPRETATION OF THEOREMS 1 AND
Both theorems imply HVDL and SVDL perform well if the output D is not too close to 0 or (i.e., favor small U). The first two terms in both upper bounds control the quality of KDE, which implies KDE works better if we have larger N and No and a smaller a. The rest terms of the two bounds different. In the HVDL case, we favor smaller Mr, and M However,
should avoid setting i for too small value because we prefer larger N y,e and Ng. g.x- In the SVDL case, we prefer small M" and M but large W" and (y). Large W (y) and W (y) imply that the weight function decays slowly (i.e., small 1u; similar large Ny and 1. in (S.32) However, should avoid setting y too small because small leads to large By (1 1E yl and Ey'no' lv yl rule-of-
(i.e., y'ss which far away from y have large weights). In our experiments, thumb formulae to select x. and v. As a future work, a refined hyper- hyper- parameter selection method should be proposed.
S.6
MORE DETAILS OF THE SIMULATION IN SECTION 4.1
S. NETWORK ARCHITECTURES
Please refer to Table S.6. and Table S.6.2 for the network architectures we adopted for cGAN and CcGAN in our Simulation experiments.
S.6.2 TRAINING SETUPS
The cGAN and CcGAN are trained for 6000 iterations on the training set with the Adam (Kingma &
Ba 2015) optimizer (with 31 = 0.5 and 3g = 0.999), a constant learning rate x 10 and batch size 128. The rule of thumb formulae in Section S.4 are used to select the hyper- parameers for HVDL and SVDL, where we let m, = 2. Thus, the three snper-parameters in this experiments are set as follows: 0.074, 0.017. 3600.
20

---

Published as a conference paper at ICLR 2021
Table S. Network architectures for the generator and discriminator of cGAN in the simulation. "fc" denotes fully-connen colly-coneectd layer. "BN" stands for batch normalization. The label is treated as a class label and encoded by label-embedding (Akata et al. 2015) so its dimension equals to the number of distinct angles in the training set (i.e., R 2)).
(a) Generator
(b) Discriminator
2 E R² ~ NN, I): y E 120
A sample a E I??
concat(2, y) E
fc 100; ReLU
fc 100; BN; ReLU
fc 100; ReLU
Fe-> 100; BN; ReLU
f 100; ReLU
IC-> I00; BN; ReLU
fc 100; ReLU
fc-+ 100; BN; ReLU
coupat of previous layer, y) R
fc-> > I00; BN; ReLU
where E IR 120 is the label of æ.
f 100; BN; ReLU
fc > 100; ReLU
fc 2
fc 1; Sigmoid
Table S.6.2: Network architectures for the generator and discriminator of our proposed CeGAN in the simulation. The label y treated as a real scalar its dimension 1. We do not directly input y into the generator and discriminator. We first convert each y into the coordinate of the mean represented by this y, i.e., (sin(y), cos(y)). Then insert this coordinate into the networks.
(a) Generator
(b) Discriminator
E IR² ~ N(O, D: RR.
A sample x E IR² with label E R
concat(z, sin(y), COS cOs(y)) E R"
concat(r, sin cos(y)) E IR"
fc- > 100; BN: ReLU
fc-> 100; ReLU
fc 100; BN; ReLU
fc-> 100; ReLU
fc-> TOO; BN; ReLU
fc 100; ReL.U
fc-> > 100; BN; ReLU
fc 100; ReLU
fc + 100; BN; ReLU
fc 100; ReLU
fc 100; BN; ReLU
fc 1; Sigmoid
fc-+ 2
S.6.3 TESTING SETUPS
When evaluating the trained cGAN, if test label y' unseen in the training set, we first find its closest, seen label y. Then, we generate samples from the trained cGAN at instead of at y'. On the contrary, generating samples from CcGAN at unseen labels is well-defined.
S.6.4 EXTRA EXPERIMENTS
S.6.4.1 VARYING NUMBER OF GAUSSIANS FOR TRAINING DATA GENERATION
In this section, we study the influence of the number of Gaussians used for training data generation on the performance of cGAN and CcGAN. We vary the number of Gaussians from 120 10 with step size 10 but keep other settings in Section (.1oo 4.1 Junchanged and plot the line graphs of 2- --Wasersttein Distance (log scale) versus the number of Gaussians in F [..61 Reducing the number of Gaussians for training implies a larger gap between any two consecutive distinct angles in the training set. As the number of Gaussians decreases, the continuous scenario gradually degenerates to the categorical scenario, therefore the assumption that small perturbation to y results in negligible change to p(xly) is no longer satisfied. Consequently. the 2- Wasserstein distances of the proposed two CcGAN methods gradually increase and eventually surpass the 2- Wasserstein distance of cGAN when the number of Gaussians is small (c.g., less than 40). Note that reducing the number of Gaussians in the training data generation will not improve the performance of cGAN in the testing
21

---

Published as a conference paper at ICLR 2021
because many angles seen in the testing stage (we evaluate each method 360 angles) do not appear in the training set.
cGAN
2-Wasserstein Distance (in log scale)
(HVDL)
(SVDL)
-4
-5
-6-
-7
-8
120
110
100
90
80
70
60
50
40
30
20
10
Number of Gaussians for Training Data Generation
Figure S.6.1: Line graphs of 2- Wasserstein Distance (log scale) versus the number of Gaussians for training data generation. As the number of Gaussians decreases, the continuous scenario gradually degenerates to the categorical scenario, therefore the assumption that a small perturbation to results negligible change p(æ) ((z)) (y) no longer satisfied. Consequently, the 2-Wasserstein distances of two CcGAN methods gradually increase and eventually surpass the 2-Wasserstein distance of cGAN when the number of Gaussians is small (e.g., less than 40).
S.7 MORE DETAILS OF THE EXPERIMENT ON RC-49 IN SECTION 4.2
S.7.1 DESCRIPTION OF RC-49
To generate RC-49, firstly we randomly select 49 3-D chair object models from the "Chair" category provided by ShapeNet (Chang et al. 2015). Then we use Blender v2.79 to render these 3-D models. Specifically, during the rendering, we rotate each chair model along with the yaw axis for degree between 0.1* and 89.9 (angle resolution as 0.1%) where we use the scene image mode to compose our dataset. The rendered images are converted from the RGBA to RGB color model. In total, the RC-49 dataset consists of 44051 images of image size 64 the PNG format.
S.7.2 NETWORK ARCHITECTURES
The RC-49 dataset is a more sophisticated dataset compared with the simulation, thus it requires networks with deeper layers. We employ the SNGAN architecture (Miyato et al.) 2018) in both cGAN and CcGAN consisting of residual blocks for the generator and the discriminator. Moreover, for the generator in cGAN, the regression labels are input into the network by the label embedding (Akata ct al. 2015) and the conditional batch normalization (De Vries et al. 2017). For the discriminator in cGAN, the regression labels are fed into the network by the label embedding and the label projection MMyyato & Koyama, 2018). For CeGAN, the regression labels are fed into networks by our proposed label input method in Section Please refer our codes for more details about the network specifications of cGAN and CcGAN.
S.7.3 TRAINING SETUPS
The cGAN and CcGAN are trained for 30,000 iterations on the training set with the Adam (Kingmal
& Ba 2015) optimizer (with 3 0.5 and 8g 0.999), constant learning rate 10 10-4 and batch size 256. The rule of thumb formulae in Section S are used to select the and SVDL,, where we m, 2. Thus, the three pyram-earameters in this experiments are set as
for HVDL
follows: Æ = 0.0473, & = 0.004, = 50625.
22

---

Published as a conference paper at ICLR 2021
S.7.4 TESTING SETUPS
The RC-49 dataset consists of 899 distinct yaw angles and at each angle there are 49 images (corresponding 49 types of chairs). At the test stage, ask the trained cGAN or CeGAN to generate 200 fake images at each of these 899 yaw angles. Please note that, among these 899 yaw angles, only 450 of them are seen at the training stage so real images at the rest 449 angles are not used in the training.
We evaluate the quality of the fake images from three perspectives, i.e., visual quality, intra-label diversity, and label consistency. One overall metric (Intra-FID) and three separate metrics (NIQE, Diversity, and Label Score) are used. Their details are shown in Supp. S.7.5
S.7.5 PERFORMANCE MEASURES
Before we conduct the evaluation in terms of the four metrics, we first train autoencoder (AE). regression CNN and a classification CNN on all real images in RC-49. The bottleneck dimension of the AE is 512 and the AE is trained to reconstruct the real images in RC-49 with MSE as the loss function. The regression CNN is trained to predict the yaw angle of given image. The classification CNN is trained to predict the chair type of given image. The autoencoder and both two CNNS trained for 200 epochs with batch size 256.
Intra-FID (Miyato & Koyama 2018): We take Intra-FII) as the overall score to evaluate the quality of fake images and we prefer the small Intra-FID score. At each evaluation angle, we compute the FID (Heusel et al. 2017) between 49 real images and 200 fake images in terms of the bottleneck feature of the pre-trained AE The Intra-FID score the average FID over all 899 evaluation angles. Please note that we also try to use the classification CNN to compute the Intra-FID but the Intra-FID scores vary a very wide range and sometimes obviously contradict with the three separate metries.
NIQE (Mittal et al. 2012); NIQE is used to evaluate the visual quality of fake images with the real images as the reference and we prefer the small NIQE score. We train one NIQE model with the 49 real images at each of the 899 angles so we have 899 NIQE models. During evaluation, NIQE score is computed for each evaluation angle based on the NIQE model at that angle. Finally, we report the average and standard deviations of the 899 NIQE scores over the 899 yaw angels. Note that the NIQE implemented by the NIQE module in MATLAB.
Diversity: Diversity is used to evaluate the intra-label diversity and the larger the better. In RC-49, there are 49 chair types. At each evaluation angle, we ask the pre-trained classification to predict the chair types of the 200 fake images and an entropy is computed based on these predicted chair types. The diversity reported in Table is the average of the 899 entropies over all evaluation angles.
Label Score is used to evaluate the label consistency and the smaller the better. We ask the pre-trained regression CNN to predict the yaw angles of all fake images and the predicted angles are then compared with the assigned angles. The Label Score is defined as the average absolute distance between the predicted angles and assigned angles over all fake images, which is equivalent to the Mean Absolute Error (MAE).
Label Score:
S.7.6
EXTRA EXPERIMENTS
S.7.6.1 MORE LINE GRAPHS
S.7.6.2
INTERPOLATION
In Fig. S.7.3 we present some interpolation results of the two CcGAN methods (i.e., HVDL and SVDL). For an input pair (). we fix the noise but perform label-wise interpolations, i.e., varying label y from 4.5 to 85.5. Clearly, all generated images are visually realistic and we can see the chair distribution smoothly changes over continuous angles. Please note that, Fig. S.7.3 is meant to show the smooth change of the chair distribution instead of one single chair so the chair type may change over angles. This confirms CeGAN is capable of capturing the underlying conditional image distribution rather than simply memorizing training data.
23

---

Published as a conference paper at ICLR 2021
-Al iodd
35
30-
24
33-
1.5
25
Agee daya
Angl CTyAM FORE
Argil yoy roac
(a) FID vs Angle
(b) NIQE vs Angle
(c) Diversity vs Angle
Figure S.7.2: Line graphs of FID/NIQE/Diveriity versus yaw angles on RC-49. Figs. (S...( to S.7.2(c) show that two CeGANs consistently outperform cGAN across all angles. The graphs of CeGANs also appear smoother than those of cGAN because of HVDL. and SVDL..
4.5
22.5
31.5
40.5
49.5
58.5
67.5
85.5
HVDL
SVDL
Figure S.7.3: Some example RC-49 fake images from the two CeGAN methods. We fix the noise z but vary the label y.
S.7.6.3 DEGENERATED CcGAN
In this experiment, we consider the extreme case of the proposed CcGAN (degenerated CcGAN), i.e., a + 0 and k + 0 or v + +oo. Then we train the degenerated CeGAN with the same experimental setting as for CcGANs. Some examples from degenerated CoGANs shown in Fig. S.7.4 Since, at each angle, the degenerated CcGAN only uses the images at this angle, it leads to the mode collapse problem (e.g, the row in the yellow rectangle) and bad visual quality (e.g., images in the red rectangle) at some angles.
Note that the degenerated CeGAN is still different from cGAN, since still treat y as continuous scalar instead of a class label here and we use the proposed label input method to incorporate into the generator and the discriminator.
S.7.6.4 CGAN: DIFFERENT NUMBER OF CLASSES
In this experiment, show that cGAN still fails even though we bin 89.9) into other number of classes. We experimented with three different bin setting grouping labels into 90 classes, 150 classes and 210 classes, respectively. Experimental results are shown in Fig. [S7.5 and observe that all three cGANs fail.
S.7.6.5 VARYING SAMPLE SIZE FOR EACH DISTINCT ANGLE
To test cGAN and CcGAN under more challenging scenarios, we vary the sample size each distinct angle in the training set from 45 to 5. We visualize the line graphs of Intra-FID versus the sample size for each distinct training angle in Fig. S.7.6 From this figure, we can see the two CeGAN methods substantially outperform cGAN matter what is the sample size for each distinct angle in the training set. The overall trend in this figure also shows that smaller sample size reduces the performance of both cGAN and CeGAN.
S.7.6.6 VARYING THE STRENGTH OF THE CORRELATION BETWEEN THE IMAGE AND ITS LABEL
To study how the strength of the correlation between the image z and its label y (i.e., the label power) influences the performance of cGAN and CeGAN, in this study, randomly add Gaussian noises with preset standard deviation to the raw regression labels in the training set. The strength of the correlation is controlled by the standard deviation of the Gaussian noises which varies from
24

---

Published as a conference paper at ICLR 2021
4.5
4.5
13.5
13.5
22.5
22.5
31.5
31.5
40.5
49.5
49.5
58.5
58.5
67.5
67.5
76.5
76.5
85.5
85.5
Angle
90 classes
150 classes
210 classes
Angle degenerated CeGAN
Figure S.7.5: Example RC-49 fake images from cGAN when we bin the yaw angle range into different number of classes.
Figure S.7.4: Some example RC-49 fake images from a degenerated CeGAN.
2.0
1.8
Intra-FD
cGAN
CcGAN (HVDL)
1.2
CcGAN (SVDL)
1.0
0.6
0.4
45
35
25
15
Sample Size for Each Distinct Training Angle
Figure S.7.6: Line graphs of Intra-FID versus the sample size for each distinct training angle. The grey vertical dashed line stands for the sample size used the main study of the RC-49 experiment in Section 4.2 Two CcGAN methods substantially outperform cGAN no matter what the sample size for each distinct angle the training set. The overall trend in this figure shows that a smaller sample size deteriorates the performance of both cGAN and CeGAN.
(no Gaussian noise) to 25 (the unit is degree). A large standard deviation corresponds to a weak correlation. The training setup is consistent with the main study in Section 4.2 excepp that we replicate the training set five times and randomly add Gaussian noises to the raw regression labels in the replicated training set. Therefore, each training sample has five noisy labels. We plot the line graphs of Intra-FID versus the standard deviation of Gaussian noise in Fig. S.7.7 From Fig. S.7.7 we can that the performance of two CeGAN methods deteriorates the standard deviation increases; however, the line graph of the performance of cGAN does not have clear increasing or decreasing trend and the Intra-FIDD of cGAN always stays at high level.
25

---

Published as a conference paper at ICLR 2021
cGAN
CcGAN (HVDL) CcGAN (SVDL)
Intra-FID
Standard Deviation Gaussian (degree)
Figure S.7.7: Line graphs of Intra-FID versus the standard deviation of Gaussian noise. The overall trend in the figure shows that the performance of two CcGAN methods deteriorate as the standard deviation increases.
S.8 MORE DETAILS OF THE EXPERIMENT ON THE UTKFACE DATASET IN SECTION
S.8.1 DESCRIPTION THE UTKFACE DATASET
The UTKFace dataset is an age regression dataset Zhang al. 2017). with human face images collected the wild. We use preprocessed version (cropped and aligned), with ages spanning from to 60. After data cleaning (i.e., removing images of very low quality with clearly wrong labels), the overall number of images 14760. Images are resized to 64 x 64. The histogram of UTKFace dataset w.r.t. ages 1-60 is shown in S.8.8
From Fig. S.8.8 we can see UTKFaae dataset is very imbalanced so the samples from the minority age groups unlikely to be chosen each iteration during the GAN training. Consequently, cGAN and CcGAN may not be well-trained at these minority age groups. To increase the chance of drawing these minority samples during training, we randomly replicate samples in the minority age groups ensure that the sample size of each age is more than 200.
800
600
200
Figure S.8.8: The histogram of UTKPace dataset with ages range from to 60.
26

---

Published as a conference paper at ICLR 2021
S.8.2 NETWORK ARCHITECTURES
The network architectures used in this experiment is similar to those in the RC-49 experiment. Please refer to our codes for more details about the network specifications.
S.8.3 TRAINING SETUPS
The cGAN and CeGAN are trained for 40,000 iterations on the training set with the Adam (Kingma & Ba 2015) optimizer (with 31 0.5 and 8 0.999), constant learning rate 10 and batch size 512. The rule of thumb formulae in Section S are used to select the hyper-parameters for HVDL and SVDL, where let m. 1.
S.8.4 PERFORMANCE MEASURES
Similar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE, Diversity, and Label Score. We also train an AE (bottleneck dimension 512), classification CNN, and regression CNN on all images. Please note that, the FTKEFace dataset consists of face images from 5 races based on which we train the classification CNN. The AE and both two CNNs are trained for 200 epochs with batch size 256.
S.8.5
EXTRA EXPERIMENTS
S.8.5.1
MORE LINE GRAPHS
Crarts So0t
2.35
SEl
MW Mommmmm
150
235
10
30
60
20
(a) FID VS Age
(b) NIQE Age
Diversity Age
Figure S.8.9: Line graphs of FID/NIQE/Diversit versus ages UTKFace. Figs, (a) S.8.9(c) show that two CcGANs consistently outperform cGAN across almost all ages. The graphs of CeGANs also appear smoother than those of cGAN because of HVDL and SVDL.
S.8.5.2 INTERPOLATION
To perform label interpolation experiments, we keep the noise vector fixed and vary label from age 3 to age 57 for CcGANs with HVDL and SVDL losses. The interpolation results are illustrated in 1.8.10. As age increases, we observe the synthetic face gradually becomes older in appearance. This observation convincingly shows that both HVDL and SVDL based CcGANs do not simply memorize or overfit to the training set. Indeed, our CcGANs demonstrate continuous control over synthetic images with respect to ages.
3
9
15
21
26
32
39
45
51
57
HVDL
SVDL
Figure 1.8.10: Some examples of generated UTKFace images from CeGAN when the discriminator is trained with HVDL and SVDL, We fix the noise z but vary the label y from 3 to 57.
27

---

Published as a conference paper at ICLR 2021
S.8.5.3 DEGENERATED CcGAN
We consider the extreme cases of the proposed CoGANs on the UTKFace dataset. As shown in Fig. [S.8.I] the degenerated CcGANs fails to generate facial images at some ages (e.g., 51 and 57) because of too small sample sizes.
S.8.5.4 cGAN: DIFFERENT NUMBER OF CLASSES
In the last experiment, we bin samples into different number of classes based on ground-truth labels, in order to increase the number of training samples at each class. Then we train cGAN using samples from the binned classes. We experimented with two different bin setting, i.e., binning image samples into 60 classes and 40 classes, respectively. The results are reported in Fig S.8.12 The results demonstrate cGANs consistently fail to generate diverse synthetic images with labels aligned with their conditional information. Moreover, the image quality is much worse than those from the proposed CcGANs. In conclusion, compared with existing cGANs, our proposed CcGANs have substantially better performance in terms of the image quality and diversity.
3
3
9
9
15
15
21
21
26
26
32
32
39
39
45
45
51
51
57
57
Age
40 classes
60 classes
Age degenerated CcGAN
Figure S.8.12: Example UTKFace fake images from cGAN when we bin the age range into different number of classes.
Some example UTKFace fake images from degenerated CcGAN.
Figure 1.8.11:
S.8.5.5 TRAINING ON IMAGES FOR ODD AGES ONLY AND TESTING ON EVEN-NUMBERED AGES
In this section, we train cGAN and CeGAN on images for odd ages only and test them on even- numbered ages. Since the training set this experiment is half the one in the main study in Section 4.3 we reduce the number of iterations for the GAN training from 40,000 to 20,000 while other settings are unchanged. The quantitative results for cGAN and two CeGAN methods are summarized in Table S.8.3 From Table S.8.3]
two CeGAN methods are still much better than eGAN in terms of all metrics except Label Score, since CeGAN is designed to sacrifice some (not too much) label consistency for much better visual quality and diversity.
28

---

Published as a conference paper at ICLR 2021
Table S.8.3: Training cGAN and CcGAN on images for odd ages only and testing them on even- numbered ages.
Method
Intra-FID
NIQE +
Diversity 1
Label Score
cGAN (30 classes)
4.724 + 1.339
2.763 + 0.384
0.299 + 0.349
9.114 + 7.398
CeGAN (HVDL)
0.724 + 0.161
1.795 + 0.230
1.133 + 0.257
10.341 + 3.931
CeGAN (SVDL)
0.777 + 0.248
1.803 + 0.214
1.257 + 0.112
13.141 + 5.862
S.8.5.6 TRAINING WITH SMALLER SAMPLE SIZES
The histogram in Fig. S.8.8 shows that the UTKFace dataset is highly imbalanced. To balance the training data and also test the performance of cGAN and CcGAN under smaller sample sizes, we vary the maximum sample size for each distinct age in the training from 200 to 50. Note that, in the main study in Section 4.3 we do not restrict the maximum sample size. Since we have much smaller sample size, we reduce the number of iterations for the GAN training from 40,000 20,000 and slightly increase my in Supp. [S.4 from 1 to 2 (we therefore use wider hard/soft vicinity). We visualize the line graphs of Intra-FID versus the maximum sample size for each age of cGAN and clearly that smaller sample size worsens the
CcGAN in Fig, S.8.13 From the figure, performance of both cGAN and CcGAN. Moreover, the Intra-FID scores of cGAN always stay at a high level and are much larger than those of two CeGAN methods.
4-
Intra-FID
3-
cGAN CcGAN (HVDL) CcGAN (SVDL)
200
Maximum Sample Size for Each Age
Figure S.8.13: Line graphs of Intra-FID versus the maximum sample size for each distinct angle in the training set.
S.8.5.7 DIFFAUGMENT CANNOT SAVE CGAN IN THE CONTINUOUS SCENARIO
Diff'Augment (Zhao et al. 2020) is concurrent work which proposes differentiable transformations (i.e., translation, random brightness, contrast, saturation, and cutout) on both real and fake images during the GAN training. This method shows promising results in improving the performance of un- conditional GANs (e.g., StyleGAN2 (Karras et al.) (0200) and class-conditional GANs (e.g., BigGAN (Brock al. 2019)) when training samples different from CeGAN since is designed for the unconditional and slass-conditional scenarios rather than the continuous scenario. Even though incorporating DiffAugment into the cGAN training in the continuous scenario, the two problems (P1) and (P2) discussed in Section I are still unsolved. First, (PI) is still unsolved because DiffAugment does not provide solution better than binning the regression labels into series of disjoint intervals to tackle the problem that some regression labels do not exist in the training set. Second, since DiffAugment is designed for the unconditional and class-conditional scenarios where the number of distinct conditions is always finite and known, DiffAugment doesn't 't provide solution to (P2). Besides these two unsolved problems, another concern of DiffAugment in the continuous scenario is that the ordinal information in the regression labels is not utilized while our CcGAN implicitly uses this ordinal information to construct the soft/hard vicinity.
limited. However, DiffAugment is fundamentally
29

---

Published as a conference paper at ICLR 2021
To support our arguments, incorporate DiffAugment into the cGAN training in the UTKFaae periment while other settings are kept constant. When implementing DiffAugment, we use the official codes from the GitHub repository of DiffAugment The strongest transformation combination (Color + Translation + Cutout) is used the cGAN training. Quantitative results from summarized in T..8e and some example images from GGAN+DiffAugmen are shown in (S.814 Fig. and diversity of the baseline cGAN; however, the performance of cGANeDiffAugment is still much worse than that of the proposed two CeGAN methods. The visual results also support the quantitative evaluations. Therefore, cGAN+DiffAugment still does not solve the two fundamental problems in the continuous scenario, since it is not designed for this purpose.
The quantitative results show that DiftAugment substantially improves the visual quality
Table S.8.4: Average quality of 60,000 fake UTKFaace images from cGAN and CeGAN with standard deviations after the "4" symbol. "1" ("1") indicates lower (higher) values are preferred.
Method
Intra-FID L
NIQE
Diversity 1
Label Score
cGAN (60 classes) cGAN (60 classes) + Difl'Augment
4.516 + 0.965
2.315 + 0.306
0.254 + 0.353
11.087 + 8.119
1.328 1. 0.156
2.077 1. 0.244
1.102 1 0.183
11.212 1 8.329
CoGAN (HVDL)
0.572 + 0.167
1.739 + 0.145
1.338
+ 0.178
9.782 + 7.166
CoGAN (SVDL)
0.547 + 0.181
1.753 + 0.196
1.326
+ 0.198
10.739 + 8.340
3
9
15
21
26
32
39
45
51
57
Figure 1.8.14: Some example UTKFace fake images from CGAN+DiffAugennt Even with the help of Diff'Aummnt, cGAN still has poor visual quality in the continuous scenario.
S.9 POTENTIAL APPLICATIONS AND IMPACTS OF CcGANS
Generally, there are three label scenarios where we can apply CoGANs: Scenario I. mathematically continuous labels (c.g., angles); Scenario II, discrete but ordinal labels (c.g., ages); and Scenario III, discrete, categorical labels but sharing close relationships among different label categories (c.g., fine- grained bird image generation). CeGANs can have potential applications in all three scenarios. For example, in Scenario CcGANs could have potential impacts on autonomous driving which involves predicting the steering angle a continuous scalar) to have better controllability over autonomous cars. In Scenario II, the proposed methods are potentially meaningful in some medical applications. E.g.. in medical experiments, needs to predict the number of cells (i.c., ordinal integers) from a microscopic image. Even with limited microscopic cell images, the proposed CcGAN can generate visually synthetic and diverse microscopic images for the regression model training. In this way, CcGAN may help save tedious efforts of medical reseachers in gathering microscopic images. In Scenario III,, as suggested by AnonReviewer's 5 (Q3), CcGAN could be used on some fine-grained image classification datasets, e.g., on the bird dataset where birds of different categories may share close similarities. The generated bird images can be used to enhance the ine-grained bird image classifiers, and potentially help us better recognize birds and protect the environment. More generally. CeGANs can be potentially used for image generation in regression datasets (associated with scalar labels y). In summary, CcGANs can cover wide range of tasks and applications which could potentially benefit the society.
important task is cell counting, where the cell counting regression
30


Reviews:
[{"review_id": "rP5OPFi61Nb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\nThis paper tries to make the generative adversarial network handle continuous, scalar conditions. Specifically, in order to make it work, the author set an empirical estimate that every small perturbation to the condition y results in negligible changes to the conditional distribution.  The author shows comparative performance improvement over the basic cGAN on circular 2D Gaussians, RC-49, and UTKFace dataset.\n\n##### Strength\nFirst of all, this paper is well-written and easy to follow. This paper tackles one of the important topics in a generative network with appropriate assumptions and shows its relative effectiveness not only on a single dataset but also in various settings. It also provides rich details with the codes in supplementary.\n\n##### Weakness\nFirst of all, although the math and logic are quite easy to follow, it is a bit hard to guess, especially to the person who doesn't have a strong background in this domain, what exact behavior is enforced to the model. Also, along with the first issue, it is hard to predict this method's stability for some non-well curated/designed setting. Please check the question section for the details.\n\n##### Question\n1. As I mentioned in the weakness part, it is quite hard to imagine how this method will behave. Similar to Figure 1 of the original GAN paper (Goodfellow et al., in NeurIPS 2014), or Figure 1 of VEEGAN (Srivastava et al., in NeurIPS 2017), it would be great if the author can visualize how the distribution is matched and what kinds of behaviors are promoted/demoted by the loss.\n\n2. As mentioned in (P1) of this paper, there are many real-world cases that there are zero or only one label exists. To show the effectiveness in such an imbalanced setting, I recommend presenting the result that (i) train people of odd ages only and tested with even-numbered ages and (ii) tested with fewer data in some of the ages (unlike the one in Figure S. VIII.5 or S.VIII.A).\n\n3. Also, I also wonder what happened if the condition is not directly related (to see whether the label power is essential or not). To alleviate such curiosity, I recommend testing it by modifying the label on one of the experiments. For instance, you can try this method on the RC-49 generation, not with the angle but with the chair's volume.\n\n\n\n##### Post-rebuttal \n\nI thank the authors for their thorough comments and detailed explanations for each question. I carefully read the whole, and it helps me understand the entire decision and the processes. However, I would like to suggest two things, regardless of its acceptance, but to make their claim more attractive to general readers: (i) re-sort out the indexes (maybe after rebuttal but before submitting your camera-ready version), and (ii) update figures of the sample with a bit more realistic one (with training the model on larger batch size, for instance). I hope this review phase would make your paper more powerful. (disclaimer: I did not check the soundness of the mathematical equations thoroughly but did check all the rest)", "extra_scores": {}, "extra_text": {}}, {"review_id": "ez8yPte0LMz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Summary:\n\nThis paper proposes a novel continuous conditional GAN which takes continuous scalars (named regression labels) as conditions. There are two problems for continuous conditional GAN: (P1) cGANs with discrete labels are trained to minimize the empirical loss, but this fails for continuous conditions, because there might be few or even zero samples for many labels values. (P2) For continuous labels, the label cannot be embedded by one-hot encoding like discrete labels. The authors propose two solutions for the problems respectively: (S1) The authors give two types of estimate (HVE and SVE) of the joint probability used for calculating the loss, and further formulate new loss functions HVDL and SVDL, based on the estimated joint distribution. (S2) The authors propose new methods of encoding the label input: element-wise addition for the generator, and linear embedding layer for the discriminator. The authors further derive and prove the error bounds for a discriminator trained with HVDL and SVDL. Experiments are conducted on continuous label datasets, and the authors propose a new dataset with continuous labels. Comparisons are made with traditional cGANs with discrete labels.\n\n*** Pros:\n\n1. This paper is the first work on continuous conditional GAN. It solves the two key problems for generalizing traditional cGANs to continuous conditions, and empirically validates the effectiveness of the proposed approach.\n\n2. The proposed approach is theoretically sound. The generalization from traditional cGANs with discrete labels to the novel continuous label conditional GANs is reasonable. The estimations (HVE and SVE) for deriving the new loss functions are clearly explained and mathematically elegant and solid.\n\n3. Error bounds of the discriminator trained with HVDL and SVDL losses are derived and proved. This builds a strong and solid foundation of the proposed approach.\n\n4. Experiments are conducted on various datasets. Reasonable results for generating synthetic data and real-world images are shown and compared with cGANs with discrete labels.\n\n*** Cons:\n\n1. The experiments are conducted on low-resolution images with SNGAN structure. In order to test the generalization ability of the proposed approach, would it be possible to test on other state-of-the-art structures such as BigGAN? And is it able to be trained to generate higher resolution images?\n\n2. Conditional GANs with continuous label conditions is an interesting direction that has been rarely explored before. Since most real-world datasets are with discrete labels, and even for some labels that can be discrete, it might be impossible to label it in a continuous way, and we will assign discrete labels. Can you elaborate more on the potential application scenarios and potential impacts of the continuous cGANs, besides the viewpoint and age examples used in this paper?\n\n*** Justification for rating\n\nI think this paper works on an interesting direction of continuous conditional GANs, which has rarely been explored before. The proposed approach to generalize cGANs to continuous label conditions and the proposed solutions to the key problems are elegant and novel. The authors not only give clear explanations but also provides mathematical error bounds and proves. So I would recommend acceptance.\n\n*** Questions for the rebuttal\n\nPlease address the questions in the Cons section.\n\n*** After rebuttal\n\nThank the authors for the detailed explanation. After reading other reviewers' comments and the author feedback, I would like to keep my rating unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bhkv2V0lwlE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " This work proposes to perform Conditional GAN with regression labels, thus to benefit the model with data-sufficient continuations generation (vs infinite distinct condition for generation).  Inspired by VRM, a CcGAN model is proposed to tackle the challenges in existing methods. (Eq.(6) ---> Eq.(10))  An error bound is also derived for the proposed new discriminator loss. \n\nThe idea to facilitate generation with continuous conditions is interesting and insightful. the experiments are also well-designed and analyzed to support the claim.  My questions are mainly in the following aspects:\n\n1). Comparisons on more challenging data sets are suggested.  Also, as you argue the deficiency of requiring large sample and condition size in baseline models, comparison regarding the different number of, or scare training data are better to be presented.\n\n2) Can you please explain your superiority and connection with [1].\n\n[1]. Zhao, Shengyu, et al. \"Differentiable augmentation for data-efficient gan training.\" arXiv preprint arXiv:2006.10738 (2020). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNsaZb9rFWA", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#########################################\n\nSummary:\n\nThe paper focuses on conditional image generation with continuous label. Current methods utilize categorical conditions, which easily suffers form two problems: (1) the inferior performance when given few images for special category (2) failure of using continue label. In this paper, authors reformulate the class-conditional GANs, and  provide two new objectives (hard vicinal discriminator loss and soft vicinal discriminator loss ) and one generator loss. Both the discriminator loss and the generator loss is extended based on the vicinal risk minimization. Authors design interesting experiments to evaluate the proposed method.\n\n#########################################\n\nPros:\n\n+The paper explores new problem for conditional GANs. Specially the continues label never be studied, which is interesting for me.\n\n+Authors leverage vicinal risk minimization to replace  current empirical generator loss, which is new viewpoint to investigate the conditional image generation.\n\n+The first experiment is interesting, which is suitable to support the proposed method.\n\n+The paper has good motivation, and is easy to follow.\n   \n#########################################\n\nI have a few concerns which is as followiing:\n\n1.  In page 2, authors mention '(1) our experiments in Section 4 show that this approach often makes cGANs collapse;' what is reason why current methods fail? Is it due to the imbalance of data for each category? or Is it normal issue of training GANs and cGANs?\n\n2. I really like 'circular 2-d gaussians' experiment, which is smartly designed the proposed method. For the proposed method (E.q. 11 for generator), it has already  seen  the data which is corresponding to  test label (angle), since the real sample (Figure 1(a)) is localized in any angle and the used label (of the proposed method) for training is noised, which means the noised label is the test label.  Could authors try the real sample like one of Figure (b) which the real sample is not continues.  \n\n3.  I am wondering the proposed method could be used for fine-grand image generation. For example, the bird dataset[1] has 555 category,and some categories are close.\n\n4. In paragraph: How is (P2) solved?. The new method to map the label seems similar to the label embedding. For example, the label embedding of BigGAN is also be updated. But here authors map the label by FC layers, which is corresponding to the learned label embedding. And the combination  of both the image embedding and the class embedding is similar to the one used in SNGAN and BigGAN.\n    \n\n[1] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pages 595–604, 2015. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a novel conditional GAN formulation where continuous scalars (named regression labels) are fed into the GAN as a conditioning variable. Since cGANs with discrete labels are trained to minimize the empirical loss, they fail for continuous conditions, because there might be few or even zero samples for many labels values and also the label cannot be embedded by one-hot encoding like discrete labels. As a solution, the authors propose new methods of encoding the label. 

The paper received a clear accept, two weak accepts and a weak reject. As agreed by all the reviewers, the paper proposes an interesting framework to eliminate some weaknesses of GANs. The rebuttal adequately addresses the reviewer comments and hence the meta reviewer recommends acceptance. 
Decision: Accept (Poster)