CDCGen: Cross-Domain Conditional Generation via
Normalizing Flows and Adversarial Training
Hari Prasanna Das, Ryan Tran, Japjot Singh, Yu-Wen Lin and Costas J. Spanos{hpdas,bobotran,calzoom,yuwen.lin,spanos}@berkeley.edu
Department of Electrical Engineering and Computer Sciences, UC Berkeley
Berkeley, CA, USA

ABSTRACT1

How to generate conditional synthetic data for a domain without
utilizing information about its labels/attributes? Our work presents
a solution to the above question. We propose a transfer learningbased framework utilizing normalizing flows, coupled with bothmaximum-likelihood and adversarial training. We model a source
domain (labels available) and a target domain (labels unavailable)with individual normalizing flows, and perform domain alignment
to a common latent space using adversarial discriminators. Dueto the invertible property of flow models, the mapping has exact
cycle consistency. We also learn the joint distribution of the data
samples and attributes in the source domain by employing an encoder to map attributes to the latent space via adversarial training.
During the synthesis phase, given any combination of attributes,
our method can generate synthetic samples conditioned on themin the target domain. Empirical studies confirm the effectiveness of
our method on benchmarked datasets. We envision our method tobe particularly useful for synthetic data generation in label-scarcesystems by generating non-trivial augmentations via attribute andcomponent transformations. These synthetic samples will introduce
more entropy into the label-scarce domain than their geometricand photometric transformation counterparts, helpful for robust
downstream tasks.

A large majority of the real-world signals obtained are unlabeled,
and require significant human effort or machine intelligence forlabeling. This has led to a surge in popularity of unsupervised learning algorithms. A prominent branch of such algorithms, generativemodeling, has proven to be efficient in transferring knowledge
gained from one (or multiple) domain(s) to other domain(s). Variants of such approaches include cross-domain translation [19, 42],
domain adaptation for classification [17, 44] etc. By jointly modelingthe data samples and their labels/attributes, variations of conditional synthesis methods have been proposed [26, 29, 30], which
during inference phase, can generate synthetic conditional samples.
We combine both the above avenues of cross-domain translation
along with conditional synthesis and propose a framework capableof generating conditional samples for a domain without utilizing
its labels/attributes for training.
Prior works on cross-domain translation involve constructionof a mapping between two (or more) unpaired domains. The translation consistency is maintained by introducing some form of inductive bias terms such as cycle consistency [42], semantic consistency [33], entropic regulation [5] etc. Most of the proposedmodels for domain translation are generative adversarial network(GAN) [14] based and involve many-to-one/one-to-many mappings,
making the cycle consistency only approximate. A recent work,
Alignflow [15] achieves exact cycle consistency by modeling thedomains with normalizing flows via a common latent space. Normalizing flows [10, 22] are a class of generative models which map
an unknown and complex data distribution to a latent space witha simple (e.g. standard gaussian) prior distribution via invertible
mappings. Another benefit with having flow model mappings isthat they offer a rich latent space, which is suitable for a number ofdownstream tasks, such as semi-supervised learning [30], synthetic
data augmentation and adversarial training [4], text analysis andmodel based control etc. Conditional synthesis has been exploredby CGAN [29] by augmenting the conditions with the data andprocessing it via GAN and by ACGAN [30] by introducing an auxilliary classifier for the conditions. This becomes challenging for flow
models which are bijective in nature, and hence indirect methods
must be adopted to jointly model data and the conditions. Liu et al.
[26] propose an encoder-discriminator-classifier-decoder based approach on flow latent space which can generate synthetic samples
for a domain by passing its conditions via encoders to the data via
a flow network. They show improvements in varying the qualityof generated images for handles relating to various features.
We present CDCGen, a generative framework that is capableof transferring knowledge across multiple domains and using it togenerate synthetic samples for domains lacking information about

CCS CONCEPTS• Computing methodologies → Neural networks.

KEYWORDS
Unsupervised Style Transfer, Conditional Synthetic Data Generation, Domain Translation
ACM Reference Format:
Hari Prasanna Das, Ryan Tran, Japjot Singh, Yu-Wen Lin and Costas J.
Spanos. 2018. CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training. In KDD ’21: Knowledge Discovery andData Mining, August 14–18, 2021, Virtual Conference. ACM, New York, NY,
USA, 7 pages. https://doi.org/10.1145/1122445.1122456

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
KDD ’21, August 14–18, 2021, Virtual Conference© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/10.1145/1122445.1122456

INTRODUCTION

KDD ’21, August 14–18, 2021, Virtual ConferenceDas, et al.

Table 1: Comparison of CDCGen with state-of-the-art cross domain translation and conditional syntheis models. Across theboard, CDCGen features all the advantages over other models.
Model

Cross-Domain
Translation

Cycle Consistency

Independent Conditional
Synthesis

Availability of Latent Space Embeddings

XGAN [33]CycleGan [42]Taigman et al. [36]Alignflow [15]CGAN [29]ACGAN [31]CAGlow [26]
CDCGen (ours)

✓
✓
✓
✓
✗
✗
✗
✓

Approximate
Approximate
Approximate
Exact
–
–
–
Exact

✗
✗
✗
✗
✓
✓
✓
✓

✗
✗
✗
✓
✗
✗
✓
✓

labels/attributes. We model the label/attribute scarce domain as
the target, and a related domain with available information about
its labels/attributes as the source. We model the source and target
domain via normalizing flows with a common latent space. For
conditional synthesis, we introduce a variant of ACGAN by using
it on the learned latent space rather than the data space, and train
it with only the data and available labels from the source domain.
The features can be manipulated easily in the latent space, whichis learnt by the conditional synthesis network. During inference
phase, CDCGen offers independently specifying conditions, encoding them to a common latent space and moving through the inverse
flow to generate conditional synthetic samples in the target domain.
Table 1 summarizes the comparison between CDCGen and otherrelated models for different feature availability. CDCGen comes out
to be an amalgamation of all features available among the model
selections.
We establish the CDCGen framework and conduct empiricalevaluations with benchmarked image datasets. CDCGen shows encouraging performance in domain alignment, as well as conditional
generation for all source and target combinations.

The log-likelihood can be expressed as,log 𝑝𝜽 (𝑥) = log 𝑝 (𝑧) + log det

𝜕F (𝑥)
𝜕𝑥𝑇

(2)

𝜕F (𝑥)

is the Jacobian of F at 𝑥. The training of flow models
𝜕𝑥𝑇
is accomplished via maximum-likelihood estimation. Confirmingwith the qualifying properties for a flow as above, different typesof flow models have been introduced to efficiently estimate the
distribution density and generate synthetic samples [3, 9, 10, 22].

where

2.2

Generative Adversarial Networks (GANs)

GANs [14] are a class of implicit generative models which work
based on the principles of a mini-max game. It involves a generator
G which is tasked to generate synthetic samples from standard
noise distribution and a critic C which learns to discriminate thesamples generated by G and samples from original data distribution
𝑝𝑑𝑎𝑡𝑎 . The training objective for a GAN is given by,
min max L (C, G) =E𝑥∼𝑝𝑑𝑎𝑡𝑎 [log C(𝑥)]2

G

PRELIMINARIES

Flow-based generative models and generative adversarial networks
constitute the major building blocks for proposed CDCGen. In thissection, we will briefly describe the functioning of both types ofgenerative models. We will also iterate through the related works
in the space of cross-domain translation and conditional synthesis.

2.1

Flow-based Generative Models

Let X be a high-dimensional random vector with unknown true
distribution 𝑝 (𝑥). The following formulation is directly applicable
to continuous data, and with some pre-processing steps such asdequantization [16, 35, 39] to discrete data. Let Z be the latentvariable with a known standard distribution 𝑝 (𝑧), such as a standard
multivariate gaussian. Using an i.i.d. dataset D, the target is tomodel 𝑝𝜽 (𝑥) with parameters 𝜽 . A flow, F is defined to be aninvertible transformation that maps observed data X to the latentvariable Z. A flow is invertible, so the inverse function T maps Z
to X, i.e.
Z = F (X) = T−1(X) and X = T (Z) = F

−1

(Z)

(1)

C

+E𝑧∼standard noise distribution [log(1 − C(G(𝑧)))]At nash equillibrium, the generator and critic are optimal, and thegenerator is capable of generating samples resembling original data.
GAN based models have been particularly successful in generating
high-fidelity images [2, 21], manipulating features of images togenerate custom samples [32], audio generation [11, 12], video
generation [37] etc. Despite the potential of GANs in generating
qualitative samples, they are hard to train due to the mini-max
optimization. Unlike flow models, they lack a latent space suitable
for a number of downstream applications. Another major problem
with GANs is mode collapse, where the generator starts producing
the same output (or a small set of outputs) over and over again.
A number of remedies have been proposed to tackle this over theyears [1, 28, 34].

3

RELATED WORK

We discuss the related work from two perspectives relevant to
the CDCGen framework, namely cross-domain translation andconditional synthesis.

CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training

3.1

Cross-Domain Translation

Cross-domain translation involves construction of mappings between two or more domains, by training on unpaired data samples
in both the domains. Such a problem is under-constrained andinvolves aligning the domains in feature space via mappings. A
number of research in this space [25, 33, 38, 41, 42] introduce aform of cycle consistency loss which ensures that by translating animage from one domain to another domain via mappings and thenapplying reverse mappings to translate back yields the same image. XGAN [33] uses additional loss terms to incorporate semantic
consistency across domains, to match the subspace for embedding
from multiple domains and prior knowledge via pre-trained models.
However, since all the above models involve GAN based architectures, they lack a latent space embedding useful for downstream
manipulation tasks [22]. Moreover, since the mappings are not guaranteed to be invertible, the cycle consistency is only approximate.
Alignflow [15] involves modeling each of the domains via normalizing flow mappings to a common latent space [6, 7, 10]. It
has a hybrid training objective constituting both maximum likelihood estimation and adversarial training. Moreover, since flows areinvertible mappings, Alignflow achieves exact cycle consistency.
However, flow models, by virtue of the training procedure, face achallenge to align domains which are apart in terms of semantics
and/or style, apparent from the generated samples quality in comparison with GANs. For CDCGen, we use the best of both worlds:
flow model mappings for the domains to a common latent space,along with loss terms useful to align the domains in the embedding
space. CDCGen offers a rich latent space, which is further utilized
for conditional synthesis in label/attribute scarce domains.

3.2

Conditional Synthesis

Conditional generative models have been introduced to generate
desired synthetic data by incorporating conditions information inmodel design. From CGAN [29] which is a modification of conventional GANs and works by feeding the label/attribute information
to the generating block, conditional synthesis has seen different
algorithmic variations [18, 30, 40]. A notable work, ACGAN [30]employs an auxilliary classifier for the discriminator to classify
the class labels. A recent work, CAGlow [26] proposes a variant of
ACGAN with an encoder-decoder network, adding ability to model
unsupervised conditions. Additionally, above works deal with conditional generation in a single domain. We use a variant of ACGAN
over a shared latent space for multiple domains, thereby transferring knowledge from label-rich domains to perform conditional
synthesis in label-scarce domains.

4

THE CDCGEN FRAMEWORKIn this section, we will present the CDCGen framework capableof generating conditional synthetic samples for a domain in an
unsupervised setting. We select a domain with availability of information about the labels/attributes (namely, the source domain)and has shared attributes with the domain for which we don’t haveinformation about labels/attributes (namely, the target domain).
Under this setting, the framework consists of two major networks:
one for domain alignment and one for conditional synthesis. We
consider the case of two domains, but under the assumption of

KDD ’21, August 14–18, 2021, Virtual Conference

having shared attributes between the source and target domains,
the proposed method generalizes to multiple domains seamlessly.

4.1

Domain Alignment

The first step in CDCGen is to align the source and target domains.
Let the source and target domain be denoted by D𝑠 and D𝑡 withunknown marginal density 𝑝𝑠 and 𝑝𝑡 respectively. Both the domains
are mapped via invertible transformations (normalizing flows) F𝑠and F𝑡 to a common latent space 𝑍 , which serves as a shared feature
space for alignment. We assume the shared latent space follows anormal gaussian distribution 𝑝 (𝑧), common for training of mostof the state-of-the-art flow models. The relationship between thesample space and latent space can be represented as,F𝑠F𝑡D𝑠 −−→ 𝑍 ←−− D𝑡
Note that the invertible nature of the flow model is helpful in two
different ways,• It provides a mechanism to translate between source andtarget domains, with invertible mappings F𝑠→𝑡 = F𝑡−1 ◦ F𝑠
and F𝑡 →𝑠 = F𝑠−1 ◦ F𝑡 .
• It helps achieve exact cycle consistency (as introduced in
CycleGAN [42] to ensure accurate representation of the mappings) between the domains, since F𝑠→𝑡 ◦ F𝑡 →𝑠 = F𝑡−1 ◦F𝑠 ◦ F𝑠−1 ◦ F𝑡 = 𝐼 , where 𝐼 is the identity matrix.
We use a hybrid training objective involving both maximum
likelihood estimation and adversarial training. Flow models aretrained with an unsupervised maximum likelihood loss, with anormal gaussian prior on the latent space 𝑍 . Since there are twoflow models involved for the two domains, the maximum likelihood
loss is expressed as,
L𝑀𝐿𝐸 (F𝑠 ) + L𝑀𝐿𝐸 (F𝑡 )For cross-domain mappings, adversarial loss terms are introduced.
These terms introduce inductive bias required for cross domaintranslation [42]. We employ critics C𝑠 and C𝑡 for source and target domains respectively, which distinguish between real samples(sampled from the same domain) vs. generated samples (obtained
via cross-domain mappings). For example, the adversarial loss forsource domain can be expressed as,
L𝐴𝐷𝑉 (C𝑠 , F𝑡 →𝑠 ) =E𝑥𝑠 ∼𝑝𝑠 [log C𝑠 (𝑥𝑠 )]+E𝑥𝑡 ∼𝑝𝑡 [log(1 − C𝑠 (F𝑡 →𝑠 (𝑥𝑡 )))]We also use a domain-adversarial loss [13] which forces the embeddings learnt by the flow models F𝑠 and F𝑡 to lie in the same subspace.
This is achieved by training a classifier C𝐷𝐴𝐿 which takes the latent
space embeddings for each domain and classifies the sample to becoming from D𝑠 or D𝑡 . It is trained in an adversarial manner, witha classification loss function ℓ (·, ·), such as cross-entropy. L𝐷𝐴𝐿
can be expressed as,
L𝐷𝐴𝐿 (F𝑠 , C𝐷𝐴𝐿 ) = E𝑥𝑠 ∼𝑝𝑠 ℓ (D𝑠 , C𝐷𝐴𝐿 (F𝑠 (𝑥𝑠 )))Finally, for domain alignment, the overall loss term is,
L𝐷𝑜𝑚𝑎𝑖𝑛 𝐴𝑙𝑖𝑔𝑛𝑚𝑒𝑛𝑡 (F𝑠 , F𝑡 , C𝑠 , C𝑡 , C𝐷𝐴𝐿 ; 𝜆𝑠 , 𝜆𝑡 , 𝛾𝑠 , 𝛾𝑡 ) =L𝐴𝐷𝑉 (C𝑠 , F𝑡 →𝑠 ) + L𝐴𝐷𝑉 (C𝑡 , F𝑠→𝑡 ) + 𝛾𝑠 L𝐷𝐴𝐿 (F𝑠 , C𝐷𝐴𝐿 )+𝛾𝑡 L𝐷𝐴𝐿 (F𝑡 , C𝐷𝐴𝐿 ) − 𝜆𝑠 L𝑀𝐿𝐸 (F𝑠 ) − 𝜆𝑡 L𝑀𝐿𝐸 (F𝑡 )KDD ’21, August 14–18, 2021, Virtual Conference

Domain Alignment
Networks

Das, et al.

Domain Alignment
Networks

Normalizing Flowfor target domain
Forward flow

Target Domain

Inverse flow

(Attributes unavailable)

Shared latent
space (Z)Normalizing Flowfor target domain
Forward flow

Target Domain

Inverse flow

(Attributes unavailable)

Target Critic

Target Critic

Source Critic

Source Critic
Source Domain(Attributes available)

Normalizing Flowfor source domain

Critic(Real/Generated?)

Inverse flow

Conditional Synthesis
Networks𝜖1
1

Auxiliary
Classifier

Encoder0
Attributes

Source Domain(Attributes available)

Normalizing Flowfor source domain
Inverse flow

Parameter
Sharing

Forward flow

SourceAttributes

Shared latent
space (Z)Forward flow

CustomAttribute
Combinations
Attributes

0

Conditional Synthesis
Networks

𝜖10

Encoder

0001One-hot encoded conditions

Conditional latent space (Z*)(a) CDCGen Training Schematic

One-hot encoded conditions

Conditional latent space (Z*)(b) CDCGen Inference Schematic

Figure 1: Illustration of training and inference methods in CDCGen. The networks inside the dashed box are for domainalignment (Sec 4.1) and those outside are for conditional synthesis (Sec 4.2).
where, hyperparameters 𝜆𝑠 and 𝜆𝑡 dictate the relative contributionof maximum likelihood loss, and 𝛾𝑠 and 𝛾𝑡 correspond to contribution of domain adversarial loss, both as compared to the adversarialloss. The objective is minimized w.r.t. the parameters of the flow
models F𝑠 and F𝑡 and maximized w.r.t. parameters of C𝑠 , C𝑡 and
C𝐷𝐴𝐿 . This procedure is illustrated in the dashed box in Fig. 1(a).

4.2

Conditional Synthesis

For conditional synthesis, we propose a variant of ACGAN [31].
Instead of using class/attribute conditioning on the sample spaceas done in ACGAN, we use it in the shared latent space. Under
the setting of our problem, we don’t have any information about
the labels/attributes in the target domain. So, for the conditional
synthesis part, only the attributes available from the source domain
are used for training.
We denote the available source attributes/conditions as 𝑐𝑠 ∼𝑝 (𝑐𝑠 ), represented as one-hot encodings. Our network consists ofan encoder to model the conditions, a critic to differentiate between
the real and generated latent vectors, and an auxiliary classifier
to classify the encoded conditions. We will introduce each of theabove components and their associated loss functions separately.
Encoder: An encoder network 𝐸 encodes the conditions (𝑐𝑠 , 𝜖)into a latent space 𝑍 ∗ (separate from the shared latent space 𝑍
for aligned domains), where 𝜖 is sampled from standard gaussian
distribution (𝑝 (𝜖)) and is helpful for incorporating stochastic behavior among condition vectors. Let the distribution for the above
mentioned latent space be denoted as 𝑝 ∗ (𝑧). Our objective is tominimize the Jensen-Shannon (JS) divergence between the encoded
distribution 𝑝 ∗ (𝑧) and the shared latent distribution 𝑝 (𝑧) for aligned

domains D𝑠 and D𝑡 . So, the encoder loss is represented as,
L𝐸 = E𝜖∼𝑝 (𝜖),𝑐𝑠 ∼𝑝 (𝑐𝑠 ) [log C(𝐸 (𝑐𝑠 , 𝜖))]where, C is a critic, more about which we describe now.
Critic: A critic C discriminates between the latent vectors coming from generated conditional distribution 𝑝 ∗ (𝑧) and real sharedlatent distribution 𝑝 (𝑧) for aligned domains. This is an adversarial
loss which is trained so as it is unable to distinguish the latentvectors at equilibrium, thus enabling the encoder 𝐸 to generatelatent vectors close to the real shared latent distribution 𝑝 (𝑧). The
loss function for C is,
L𝐶𝑅𝐼𝑇 𝐼𝐶 = E𝑧∼𝑝 ∗ (𝑧) [log C(𝑧)] + E𝑧∼𝑝 (𝑧) [1 − log C(𝑧)]Classifier: A classifier takes the latent vectors (𝑧 ∼ 𝑝 ∗ (𝑧) and𝑧 ∼ 𝑝 (𝑧)) as input and classifies the conditions (𝑐𝑠 ). The classifier
loss is a cross entropy loss between the predicted and true conditions. If the class posterior probabilities are 𝑞(𝑐𝑠 |𝑧), the classifier
loss function can be expressed as,
L𝐶𝐿𝐴𝑆𝑆𝐼 𝐹 𝐼 𝐸𝑅 =E𝑧∼𝑝 ∗ (𝑧),𝑐𝑠 ∼𝑝 (𝑐𝑠 ) 𝑞(𝑐𝑠 |𝑧)+E𝑧∼𝑝 ( 𝑧),𝑐𝑠 ∼𝑝 (𝑐𝑠 ) 𝑞(𝑐𝑠 |𝑧)
The overall loss function for the conditional synthesis part is,
L𝐶𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙 𝑆 𝑦𝑛𝑡ℎ𝑒𝑠𝑖𝑠 =𝛽𝐸 L𝐸 + 𝛽𝐶𝑟 L𝐶𝑅𝐼𝑇 𝐼𝐶+𝛽𝐶𝑙 L𝐶𝐿𝐴𝑆𝑆𝐼 𝐹 𝐼 𝐸𝑅
where 𝛽𝐸 , 𝛽𝐶𝑟 , 𝛽𝐶𝑙 are hyperparameters. The critic and the classifier
networks share their parameters except for their output blocks.
Conditional synthesis procedure is illustrated in Fig. 1(a).

CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training(a) Result with MNIST as source and USPS as target

KDD ’21, August 14–18, 2021, Virtual Conference(b) Result with USPS as source and MNIST as target

Figure 2: Results for domain alignment between source and target domains. The top row has original samples from the source
domain. The middle row is the corresponding latent space mapping and the bottom row is the sample obtained by translating
it to the target domain. The USPS images are slightly blurred due to the upscaling applied as standard pre-processing.

4.3

Inference

CDCGen can generate conditional samples in the target domain,even when the training process does not utilize its class/attributeinformation. To generate samples with conditions 𝑐˜𝑠 , a latent vector𝑧˜ is generated by encoding the one-hot conditions 𝑐˜𝑠 and 𝜖˜ ∼ 𝑝 (𝜖)˜ Then the latent vector 𝑧˜ is
via the encoder network, i.e. 𝑧˜ = 𝐸 (𝑐˜𝑠 , 𝜖).
passed via the inverse flow F𝑡−1 to generate the desired sample in
˜ The inference schematic is illustrated
the target domain, i.e. F𝑡−1 (𝑧).
in Fig. 1(b).

four convolutional layers with stride 2 and channel sizes 64, 128,
256, 512. This is followed by two separate fully-connected layers
for each network head, one for outputting probabilities of real or
fake and the other for classifying the one-hot encoded conditions.
Optimizer: For training the domain alignment network, we use
the Adam optimizer with 𝛽 1 = 0.5, 𝛽 2 = 0.999, and learning rate
1 · 10−6 . For training the conditional synthesis network, we use
the Adam optimizer with 𝛽 1 = 0.5, 𝛽 2 = 0.999, and learning rate
2 · 10−5 .

5

5.1

EXPERIMENTS

In this section, we empirically evaluate CDCGen for synthetic generation in label scarce domains.
Datasets: We perform experiments on 2 standard image datasetsfor digits, namely MNIST [24] and USPS. MNIST contains 60, 000
training and 10, 000 test images with ten classes corresponding todigits from 0 to 9. USPS has 7291 training and 2007 test data withthe same classes as MNIST. To address this imbalance, for eachdomain, we sample 542 images from the original training set foreach class to form the new training set. To form the test set, we
sample 147 images from the original test sets for each class. We
resize all the images to 32 × 32 for training and synthesis.
Source and Target Domain Combinations: We consider twocases, first with MNIST as the source and USPS as the target domain,and second, with the roles interchanged, i.e. USPS as the source and
MNIST as the target. We report results for domain alignment andas well as subsequent conditional synthesis in the target domain,
all while not using any labels from that domain.
Networks: We use architecture from RealNVP [10] for each of
the domain flows (F𝑠 and F𝑡 ). Typical configurations for RealNVP
can be specified as a tuple comprising 𝑁𝑠𝑐𝑎𝑙𝑒𝑠 (number of scales),𝑁𝑐ℎ𝑎𝑛𝑛𝑒𝑙𝑠 (number of channels) in the intermediate layers, and𝑁𝑏𝑙𝑜𝑐𝑘𝑠 (number of residual blocks in the scaling and translation
networks of the coupling layers). For MNIST ↔ USPS case, both F𝑠and F𝑡 are set to RealNVP(2, 64, 8). The critics (C𝑠 and C𝑡 ) used convolutional discriminators from PatchGAN [19] , each with 16 filtersin the critic’s first convolutional layer. For conditional synthesis, weconcatenate the one-hot vector of labels with components of random noise as input to the encoder. The vector then passes throughone fully-connected layer and eight transposed convolutional layerswith upsampling scale 2, 2, 2, 2, 2, 1, 1, 1 and channel sizes 256, 1024,
512, 256, 128, 64, 32, 16 respectively. The supervision block contains

Domain Alignment

In this section we present the results for domain alignment between
source and target combinations. Fig. 2(a) shows the source MNIST
samples and corresponding USPS samples by translating it via theforward source and inverse target flows. The middle sample isvisualization of corresponding latent space sample. Fig. 2(b) depicts
the same with USPS as the source and MNIST as the target. It canbe observed that the class identity is preserved with the translation
with the style adapted for the target domains. The sharpness ofthe translated samples are compromised, which is a result of theflow model assigning some probability mass to all the samples it isfed. This is unlike pure GAN based models which selectively assign
probability mass to meaningful samples.
Another interesting observation is the appearance of digit class
identity in latent space visualizations. This is particularly usefulfrom the perspective of CDCGen, since the conditional synthesis
network works based on the latent space mappings from both thedomains.
We present the t-SNE embeddings for the shared latent space inour proposed domain alignment network for MNIST and USPS in
Fig. 3. It can be observed that the visualization has distinct clusters
for each digit class, but the embeddings from both the source andtarget domain are close and belong to the same cluster for the overall
digit class clustering. The visualization allows us to infer that thelatent space has learned a subspace corresponding to each digit,
and interpolating across this subspace is effectively a conditionalfeature-preserving domain transfer.

5.2

Conditional Synthesis

We trained the conditional synsthesis part of CDCGen (Section 4.2)
with source labels to generate conditional synthetic samples in thetarget domain. Fig. 4(a) shows the samples generated with USPS

KDD ’21, August 14–18, 2021, Virtual ConferenceDas, et al.

Figure 3: t-SNE representation of shared latent space for MNIST ↔ USPS. For each digit, points for USPS are visualized withthe darker colors, and points with lighter colors correspond to MNIST.

(a) Generated samples for USPS as target(b) Generated samples for MNIST as target

Figure 4: Conditional synthetic samples generated by CDCGen. The rows represent conditioned digit classes (0-9) and thecolumns include more samples for each class.
as the target domain and Fig. 4(b) shows the samples generated
with MNIST as the target. Each row corresponds to the digit classes
which are assigned as conditions. It can be observed that CDCGen
is able to generate synthetic samples belonging to the digit class as
conditioned. There are also variations among the samples acrossdifferent columns which shows the stochastic nature of generation
by CDCGen. The compromise in sharpness of the samples generated
can be observed in the generated samples too, and is owed fromthe domain alignment mappings by flow models.

6

CONCLUSIONSCDCGen, a generative framework capable of generating conditional
synthetic samples for domains without the requirement of obtaining its labels/attributes was presented. We also conducted empiricalstudies with standard image datasets to observe feature transfer
and independent conditional generation. In the future, making theconditional generation models across multiple domains can be studied with varying levels of label availability (few-shot learning) fortarget domain. CDCGen can also be adapted for other modalities ofdata including audio and tabular data. It can also be used alongsidereal-world applications [8, 20, 23, 27, 43] where having access todiverse conditional data is important, but is hard to obtain, hence
the need for synthetic data.

ACKNOWLEDGMENTS
This research is funded by the Republic of Singapore’s National
Research Foundation through a grant to the Berkeley Education
Alliance for Research in Singapore (BEARS) for the SingaporeBerkeley Building Efficiency and Sustainability in the Tropics (SinBerBEST) Program. BEARS has been established by the Universityof California, Berkeley as a center for intellectual excellence inresearch and education in Singapore.

REFERENCES[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan.
arXiv preprint arXiv:1701.07875 (2017).
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN
Training for High Fidelity Natural Image Synthesis. In International Conference
on Learning Representations. https://openreview.net/forum?id=B1xsqj09Fm

CDCGen: Cross-Domain Conditional Generation via Normalizing Flows and Adversarial Training[3] Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. 2019. Residual Flows for Invertible Generative Modeling. arXiv preprint
arXiv:1906.02735 (2019).
[4] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas
Usunier. 2017. Parseval networks: Improving robustness to adversarial examples.
In Proceedings of the 34th International Conference on Machine Learning-Volume
70. JMLR. org, 854–863.
[5] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. 2017.
Joint distribution optimal transportation for domain adaptation. In Advances in
Neural Information Processing Systems. 3730–3739.
[6] Hari Prasanna Das, Pieter Abbeel, and Costas J Spanos. 2019. Dimensionality
reduction flows. arXiv preprint arXiv:1908.01686 (2019), 1–10.
[7] Hari Prasanna Das, Pieter Abbeel, and Costas J Spanos. 2019. Likelihood Contribution based Multi-scale Architecture for Generative Flows. arXiv preprint
arXiv:1908.01686 (2019).
[8] Hari Prasanna Das, Ioannis C Konstantakopoulos, Aummul Baneen Manasawala,
Tanya Veeravalli, Huihan Liu, and Costas J Spanos. 2019. A novel graphicallasso based approach towards segmentation analysis in energy game-theoretic
frameworks. In 2019 18th IEEE International Conference On Machine Learning And
Applications (ICMLA). IEEE, 1702–1709.
[9] Laurent Dinh, David Krueger, and Yoshua Bengio. 2014. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 (2014).
[10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016. Density estimation
using Real NVP. CoRR abs/1605.08803 (2016). arXiv:1605.08803[11] Chris Donahue, Julian McAuley, and Miller Puckette. 2018. Adversarial audio
synthesis. arXiv preprint arXiv:1802.04208 (2018).
[12] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. 2019. Gansynth: Adversarial neural audio synthesis.
arXiv preprint arXiv:1902.08710 (2019).
[13] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo
Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016.
Domain-adversarial training of neural networks. The Journal of Machine Learning
Research 17, 1 (2016), 2096–2030.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial
nets. In Advances in neural information processing systems. 2672–2680.
[15] Aditya Grover, Christopher Chute, Rui Shu, Zhangjie Cao, and Stefano Ermon.
2019. AlignFlow: Cycle Consistent Learning from Multiple Domains via Normalizing Flows. arXiv preprint arXiv:1905.12892 (2019).
[16] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. 2019.
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design. arXiv preprint arXiv:1902.00275 (2019).
[17] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko,
Alexei A Efros, and Trevor Darrell. 2017. Cycada: Cycle-consistent adversarial
domain adaptation. arXiv preprint arXiv:1711.03213 (2017).
[18] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. 2018. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition. 7986–7994.
[19] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-toimage translation with conditional adversarial networks. In Proceedings of theIEEE conference on computer vision and pattern recognition. 1125–1134.
[20] Ming Jin, Ruoxi Jia, Hari Prasanna Das, Wei Feng, and Costas Spanos. 2018.
BISCUIT: Building Intelligent System CUstomer Investment Tool. In 10th International Conference on Applied Energy (ICAE).
[21] Tero Karras, Samuli Laine, and Timo Aila. 2018. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948(2018).
[22] Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems.
10215–10224.
[23] Ioannis C Konstantakopoulos, Hari Prasanna Das, Andrew R Barkan, Shiying
He, Tanya Veeravalli, Huihan Liu, Aummul Baneen Manasawala, Yu-Wen Lin,and Costas J Spanos. 2019. Design, benchmarking and explainability analysis of

KDD ’21, August 14–18, 2021, Virtual Conferencea game-theoretic framework towards energy efficiency in smart infrastructure.
arXiv preprint arXiv:1910.07899 (2019).
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradientbased learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–
2324.
[25] Ming-Yu Liu and Oncel Tuzel. 2016. Coupled Generative Adversarial Networks.
arXiv:1606.07536 [cs.CV][26] Rui Liu, Yu Liu, Xinyu Gong, Xiaogang Wang, and Hongsheng Li. 2019. Conditional adversarial generative flow for controllable image synthesis. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition. 7992–8001.
[27] Shichao Liu, Stefano Schiavon, Hari Prasanna Das, Ming Jin, and Costas J Spanos.
2019. Personal thermal comfort models with wearable sensors. Building andEnvironment 162 (2019), 106281.
[28] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. 2016. Unrolled
generative adversarial networks. arXiv preprint arXiv:1611.02163 (2016).
[29] Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784 (2014).
[30] Augustus Odena. 2016. Semi-supervised learning with generative adversarial
networks. arXiv preprint arXiv:1606.01583 (2016).
[31] Augustus Odena, Christopher Olah, and Jonathon Shlens. 2017. Conditional image
synthesis with auxiliary classifier gans. In Proceedings of the 34th International
Conference on Machine Learning-Volume 70. JMLR. org, 2642–2651.
[32] Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv
preprint arXiv:1511.06434 (2015).
[33] Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar
Mosseri, Forrester Cole, and Kevin Murphy. 2020. Xgan: Unsupervised image-toimage translation for many-to-many mappings. In Domain Adaptation for Visual
Understanding. Springer, 33–49.
[34] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford,
and Xi Chen. 2016. Improved techniques for training gans. In Advances in neural
information processing systems. 2234–2242.
[35] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. 2017. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. CoRR abs/1701.05517 (2017). arXiv:1701.05517
http://arxiv.org/abs/1701.05517[36] Yaniv Taigman, Adam Polyak, and Lior Wolf. 2016. Unsupervised cross-domain
image generation. arXiv preprint arXiv:1611.02200 (2016).
[37] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018. Mocogan:
Decomposing motion and content for video generation. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 1526–1535.
[38] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial
Discriminative Domain Adaptation. arXiv:1702.05464 [cs.CV][39] Benigno Uria, Iain Murray, and Hugo Larochelle. 2013. RNADE: The real-valued
neural autoregressive density-estimator. In Advances in Neural Information Processing Systems. 2175–2183.
[40] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan
Catanzaro. 2018. High-resolution image synthesis and semantic manipulation
with conditional gans. In Proceedings of the IEEE conference on computer visionand pattern recognition. 8798–8807.
[41] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE
international conference on computer vision. 2849–2857.
[42] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.
arXiv:1703.10593 [cs.CV][43] Han Zou, Jianfei Yang, Hari Prasanna Das, Huihan Liu, Yuxun Zhou, and Costas J
Spanos. 2019. Wifi and vision multimodal learning for accurate and robust devicefree human activity recognition. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition Workshops. 0–0.
[44] Han Zou, Yuxun Zhou, Jianfei Yang, Huihan Liu, Hari Prasanna Das, and Costas J
Spanos. 2019. Consensus Adversarial Domain Adaptation. In AAAI Conferenceon Artificial Intelligence 2019.