LATENT DOMAIN TRANSFER: CROSSING MODALITIES WITH BRIDGING AUTOENCODERS
Year: 2018

Abstract:
Domain transfer is an exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels. However, most successful applications to date require the two domains to be closely related (e.g., image-to-image, video-video), utilizing similar or shared networks to transform domain-specific properties like texture, coloring, and line shapes. Here, we demonstrate that it is possible to transfer across modalities (e.g., image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (e.g., variational autoencoder and a generative adversarial network). We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space. Through qualitative and quantitative evaluations, we demonstrate that the proposed model preserves both locality and semantic alignment through the transfer process. Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efficient and data efficient retraining of personalized mapping functions.

text: Domain transfer is an exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels. However, most successful applications to date require the two domains to be closely related (e.g., image-to-image, video-video), utilizing similar or shared networks to transform domain-specific properties like texture, coloring, and line shapes. Here, we demonstrate that it is possible to transfer across modalities (e.g., image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (e.g., variational autoencoder and a generative adversarial network). We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space. Through qualitative and quantitative evaluations, we demonstrate that the proposed model preserves both locality and semantic alignment through the transfer process. Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efficient and data efficient retraining of personalized mapping functions.

heading: 1 INTRODUCTION
text: Domain transfer has long captured the imagination of inventors and artists alike. The early precursor of the phonograph, the phonautograph, was actually inspired by the idea of ”words which write themselves”, where the shape of audio waveforms would transform into the shape of writing, capturing the content and character of the speaker’s voice in shape and stroke of the written characters (Feaster, 2009). While perhaps fanciful at the time, modern deep learning techniques have shown similar complex transformations are indeed possible.
Deep learning enables domain transfer by learning a smooth mapping between two domains such that the variations in one domain are reflected in the other. This has been demonstrated to great effect within a data modality, for example transferring between two different styles of image (Isola et al., 2016; Zhu et al., 2017; Li et al., 2018; Li, 2018), video (Wang et al., 2018) and music (Mor et al., 2018). The works have been the basis of interesting creative tools, as small intuitive changes in the source domain are reflected by small intuitive changes in the target domain. Furthermore, the strong conditioning signal of the source domain makes learning transformations easier than learning a full generative model in each domain.
Despite these successes, this line of work in domain transfer has several limitations. The first limitation is that it requires that two domains should be closely related (e.g. image-to-image or videoto-video). This allows the model to focus on transferring local properties like texture and coloring instead of high-level semantics. For example, directly applying these image-to-image transfer such as CycleGAN (Zhu et al., 2017) or its variants to images from distant domains leads to distorted and unrealistic results (Li et al., 2018). This agrees with the findings of Chu et al. (2017) who show that CycleGAN transformations are more akin to adversarial examples than style transfer, as the model
learns to hide information about the source domain in near-imperceptible high-frequency variations of the target domain.
The second limitation is data efficiency. Most conditional GAN techniques, such as Pix2Pix (Isola et al., 2016) and vid2vid (Wang et al., 2018), require very dense supervision from large volumes of paired data. This is usually accomplished by extracting features, such as edges or a segmentation map, and then training the conditional GAN to learn the inverse mapping back to pixels. For many more interesting transformations, no such easy alignment procedure exists, and paired data is scarce. We demonstrate the limitation of existing approaches in Appendix C.
For multi-modal domain transfer, we seek to train a model capable of transferring instances from a source domain (x1) to a target domain (x2), such that local variations in source domain are transferred to local variations in the target domain. We refer to this property as locality. Thus, local interpolation in the source domain would ideally be similar to local interpolation in target domain when transferred.
There are many possible ways that two domains could align such that they maintain locality, with many different alignments of semantic attributes. For instance, for a limited dataset, there is no a priori reason that images of the digit “0” and spoken utterances of the digit “0” would align with each other. Or more abstractly, there may be no agreed common semantics for images of landscapes and passages of music, and it is at the liberty of the user to define such connections based on their own intent. Our goal in modeling is to respect the user’s intent and make sure that the correct semantics (e.g., labels) are shared between the two domains after transfer. We refer to this property as semantic alignment. A user can thus sort a set of data points from in each domain into common bins, which we can use to constrain the cross-domain alignment. We can quantitatively measure the degree of
semantic alignment by using a classifier to label transformed data and measuring the percentage of data points that fall into the same bin for the source and target domain. Our goal can thus be stated as learning transformations that preserve locality and semantic alignment, while requiring as few labels from a user as possible.
To achieve this goal and tackle prior limitations, we propose to abstract the domain domains with independent latent variable models, and then learn to transfer between the latent spaces of those models. Our main contributions include:
• We propose a shared ”bridging” VAE to transfer between latent generative models. Locality and semantic alignment of transformations are encouraged by applying a sliced-wasserstein distance, and a classification loss respectively to the shared latent space.
• We demonstrate with qualitative and quantitative results that our proposed method enables transfer both within a modality (image-to-image) and between modalities (image-to-audio).
• Since we training a smaller secondary model in latent space, we find improvements in training efficiency, measured by both in terms of the amount of required labeled data and well training time.

heading: 2 METHOD
text: Figure 1 diagrams our hierarchical approach. We first independently pre-train separate generative models, either a VAE or GAN, for both the source and target domain. For VAEs, data is encoded from the data domain to the latent space through a learned encoding function z ∼ q(z|x), and decoded back to the data space with a decoder function x̂ ∼ g(x|z). For GANs, we choose latent samples from a spherical Gaussian prior z ∼ p(z) and then use rejection sampling to only select latent samples whose associated data x = g(z) is classified with high confidence by an auxiliary classifier.
We then add the bridging conditional VAE with shared weights, tasked with modeling both latent spaces z1, z2. The VAE has a single shared latent space z′, that corresponds to both domains. Sharing the weights encourages the model to seek common structures between the latent domains, but we also find it helpful to condition both the encoder qshared(z′|z,D) and decoder gshared(z|z′, D), with an additional one-hot domain label,D, to allow the model some flexibility to adapt to variations particular to each domain. While the low-level VAEs have a spherical Gaussian prior, we penalize the KL-Divergence to be less than 1, allowing the models to achieve better reconstructions and retain some structure of the original dataset for the bridging VAE to model. Full architectural and training details can be found in the Appendix.
The conditional bridging VAE objective consists of three types of loss loss terms:
1. Evidence Lower Bound (ELBO). Standard value that is maximized for training a VAE. For each domain d ∈ {1, 2},
LELBOd = −Ez′∼Z′d [log π(zd; g(z ′, D = d)] + βKLDKL (q(z ′|zd, D = d)‖p(z′))
where the likelihood π(z; g) is a spherical Gaussian N (z; g, σ2I), and σ and βKL are hyperparmeters set to 1 and 0.1 respectively to encourage reconstruction accuracy.
2. Sliced Wasserstein Distance (SWD) (Bonneel et al., 2015). The distribution distance between mini-batches of samples from each domain in the shared latent space (z′1, z ′ 2).
LSWD = 1/|Ω| ∑ ω∈Ω W 22 (proj(z ′ 1, ω),proj(z ′ 2, ω))
where Ω is a set of random unit vectors, proj(A, a) is the projection of A on vector a, and W 22 (A,B) is the quadratic Wasserstein distance. 3. Classification Loss (Cls). For each domain d ∈ {1, 2}, we enforce semantic alignment with attribute labels y and a classification loss in the shared latent space:
LClsd = Ez′∈Z′dH(f(z ′), y)
where H is the cross entropy loss, f(z′) is a one-layer linear classifier.
Including terms for both domains, the total training loss is then
L = (LELBO1 + LELBO2 ) + βSWDLSWD + βCls(LCls1 + LCls2 )
Where βSWD and βCls are scalar loss weights. The transfer procedure is illustrated Figure 2 using synthetic data. For reconstructions, data x1 is passed through two encoders, z1 ∼ q(z1|x1), z′ ∼ qshared(z′|z1, D = 1), are reconstructed through two decoders, ẑ1 ∼ gshared(ẑ1|z′, D = 1), x̂1 ∼ g(x̂1|ẑ1). For transformations, the encoding is the same, but decoding uses decoders (and conditioning) from the second domain, ẑ2 ∼ gshared(ẑ2|z′, D = 2), x̂2 ∼ g(x̂2|ẑ2). Further analysis of this example and intuition behind loss terms is summarized in Figure 2 and detailed in Appendix A.

heading: 3 RELATED WORK
text: We discuss two aspects of existing research focuses that are related to our work, followed by how our work differentiates itself from them in order to deal with the challenge identified in this paper.
Latent Generative Models: Deep latent generative models are usually constructed to transfer a simple, tractable distribution p(z) into the approximation of population distribution p∗(x), through an expressive neural network function. Such models include VAE (Kingma & Welling, 2013) and GAN (Goodfellow et al., 2014). GANs are trained with an accompany classifier that attempts to distinguish between samples from the decoder and the true dataset. VAEs, in contrast, are trained with an encoder distribution q(z|x) as an approximation to the posterior p(z|x) using variational approximation through the use of evidence lower bound (ELBO). These classes of models have been thoroughly investigated in many applications and variants (Gulrajani et al., 2017; Li et al., 2017; Bikowski et al., 2018) including conditional generation (Mirza & Osindero, 2014), generation of one domain conditioned on another (Dai et al., 2017; Reed et al., 2016), generation of high-quality images (Karras et al., 2018) and long-range structure in music (Roberts et al., 2018). In terms of overall VAE structure, Zhao et al. (2017) studies options to build hierarchical VAEs.
Domain Transfer: The domain transfer enables transfer between images (Isola et al., 2016; Zhu et al., 2017; Li et al., 2018; Li, 2018), audio (Mor et al., 2018) and video (Wang et al., 2018), intuitively mapping between two domains where the variations in one domain should be reflected in the other. Besides visually appealing results, domain transfer also enables application such as image colorization Zhang et al. (2017). Domain transfer is also proposed to be done through jointly training of generative models (Lu, 2018). Also, the behavior of domain transfer models also attracts attention. For example, Chu et al. (2017) suggests that image transfer does only local, texture level transfer.
To enable transfer between possibly drastically different domains, Our work proposes to use VAE in modeling the latent space of pre-trained generative models, in several aspects differentiating itself from related work. Generally, the modeling of latent space is different from modeling directly the data domains as most of latent generative models naturally do, also, the transfer in a more abstract semantics and between heterogeneous domains differs from most domain transfer method which focusing on locality and similar domains.
More specifically, regarding modeling latent spaces, Zhao et al. (2017) suggests training a hierarchical VAE on a single domain should be done end-to-end, whose extension to multiple domains seems non-trivial and likely to suffer from data efficient issues. Instead, our proposed work, though enabling separation of pre-trained model and conditional shared VAE, apply to domain transfer setting while overcoming this shortcoming. Moreover, regarding shared latent space, Liu et al. (2017) proposes to use shared latent space for two generative models on data domains. It requires joint training of generative models on both domains using dense supervision which is infeasible for drastically different domains. Our work that leverages pre-trained generative model and model the latent space instead of data domains addresses to this limitation.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS
text: While the end goal of our method is to enable creative mapping between datasets with arbitrary alignments, for quantitative studies we restrict ourselves to three domains where there exist a somewhat natural alignment to compare against:
1. MNIST (LeCun, 1998), which contains images of hand-written digits of 10 classes from “0” to “9”.
2. Fashion MNIST (Xiao et al., 2017), which contains fashion related objects such as shoes, tshirts, categorized into 10 classes. The structure of data and the size of images are identical to MNIST.
3. SC09, a subset of Speech Commands Dataset 1, which contains the record of audio of humans saying digits from “0” to “’9”.
For MNIST and Fashion MNIST, we prepare VAE with MLP encoder and decoder following setting up in Engel et al. (2018). More specifically, we use stacks of fully-connected linear layers activated by ReLU, together with a “Gated Mixing Layer”. The full network architecture of the bridging VAE is detailed in Appendix B. For SC09 we use the publicly available WaveGAN (Donahue et al., 2018)2. We would like to emphasize that we only use class level supervision for enforcing semantic alignment with the latent classifier.
We examine three scenarios of domain transfer:
1. MNIST ↔ MNIST. We first train two lower-level VAEs from different initial conditions. The bridging autoencoder is then tasked with transferring between latent spaces while maintaining the digit class from source to target.
2. MNIST↔ Fashion MNIST. In this scenario, we specify a global one-to-one mapping between 10 digit classes and 10 fashion object classes (See Table 6 in Appendix for details).
1Available at https://ai.googleblog.com/2017/08/launching-speech-commands-dataset. html. License: https://creativecommons.org/licenses/by/4.0/
2Available at https://github.com/chrisdonahue/wavegan
The bridging autoencoder is tasked with preserving this mapping as it transfers between images of digits and clothing.
3. MNIST↔ SC09. For the speech dataset, we first train a GAN to generate audio waveforms (Donahue et al., 2018) of spoken digits. We chose to use a WaveGAN because we wanted a global latent variable for the full waveform (as opposed to a distributed latent code as in Engel et al. (2017)). It also gives us an opportunity to explore transferring between different classes of models. The bridging autoencoder is then tasked with transferring between a VAE of written digits and a GAN of spoken digits.

heading: 4.2 RECONSTRUCTION AND DOMAIN TRANSFER
text: For reconstructions and domain transfer, we present both qualitative and quantitative results. Quantitative measurements of semantic alignment are performed with pre-trained classifiers in each data domain. Given that the datasets have pre-aligned classes, when evaluating transferring from data xd1 to xd2 , the reported accuracy is the portion of instances that xd1 and xd2 have the same predicted class.
Qualitative reconstruction results are shown in Figure 3 and the quantitative reconstruction accuracies are given in Table 1. For domain transfer, qualitative results are shown in Figure 4 and the quantitative transfer accuracies are given in Table 2. 3
For both reconstruction and domain transfer, we see very good semantic alignment in the MNIST → MNIST and MNIST → Fashion MNIST, and reasonably good alignment in MNIST → SC09. The lower performance on SC09 is likely due to the comparatively poor performance of the base generative model, and would improve with a better audio generation model. For reference, the WaveGAN has to model a much larger signal than the MNIST VAEs (16000 dimensions vs. 768 dimensions).

heading: 4.3 INTERPOLATION
text: Interpolation can act as a good proxy for locality and local smoothness of the latent transformations, as by definition good interpolations require that small changes in the source domain are reflected by small changes in the target domain. We show inter-class and inter-class interpolation in Figure 5 and Figure 6 respectively. Particularly, we are interested in two comparing three rows of interpolations: (1) the interpolation in the source domain’s latent space, which acts a baseline for smoothness of interpolation for a pre-trained generative model, (2) transfer fixed points to the target domain’s latent space and interpolate in that space, and (3) transfer all points of the source interpolation to the target domain’s latent space, which shows how the transferring warps the latent space. We use spherical interpolation (e.g., √ pv1 + √ (1− p)v2)since we are interpolating in the Gaussian latent space.
Note in Figure 5 that the second and third rows have comparably smooth trajectories, reflecting that locality has been preserved. For inter-class interpolation in Figure 5 interpolation is smooth within a class, but between classes the second row blurs pixels to create blurry combinations of digits, while the full transformation in the third row makes sudden transitions between classes. This is expected from our training procedure as the bridging autoencoder is modeling the marginal posterior of each latent space, and thus always stays on the manifold of the actual data during interpolation.

heading: 4.4 DATA AND COMPUTATION EFFICIENCY, AND ABLATION ANALYSIS
text: Since our method is a semi-supervised method, we want to know how effectively our method leverages the labeled data. In Table 3 we show for the MNIST→MNIST setting the performance measured by transfer accuracy with respect to the number of labeled data points. Labels are distributed
3In Figure 3 and 4 we show the spectrum of audio samples for demonstration purpose. The corresponding audio samples themselves are available here: https://drive.google.com/drive/u/8/folders/ 12u6fKvg0St6gjQ_c2bThX9B2KRJb7Cvk
evenly among classes. The accuracy of transformations grows monotonically with the number of labels and reaches over 50% with as few as 10 labels per a class. Without labels, we also observe accuracies greater than chance due to unsupervised alignment introduced by the SWD penalty in the shared latent space.
Besides data efficiency, pre-training the base generative models has computational advantages. For large generative models that take weeks to train, it would be infeasible to retrain the entire model for each new cross-domain mapping. The bridging autoencoder avoids this cost by only retraining the latent transfer mappings. As an example from these experiments, training the bridging autoencoder for MNIST↔ SC09 takes about one hour on a single GPU, while retraining the SC09 WaveGAN takes around four days.
Finally, we perform an ablation study to confirm the benefits of each architecture component to transfer accuracy. For consistency, we stick to the MNIST → MNIST setting with fully labeled data. In Table 4, we see that the largest contribution to performance is the giving the bridging VAE a domain conditioning signal, allowing it to share weights between domains, but also adapt to the specific structure of each domain. Further, the increased overlap in the shared latent space induced by the SWD penalty is reflected in the greater transfer accuracies.

heading: 5 CONCLUSION
text: We have demonstrated an approach to learn mappings between disparate domains by bridging the latent codes of each domain with a shared autoencoder. We find bridging VAEs are able to achieve high transfer accuracies, smoothly map interpolations between domains, and even connect different model types (VAEs and GANs). Here, we have restricted ourselves to datasets with intuitive classlevel mappings for the purpose of quantitative comparisons, however, there are many interesting creative possibilities to apply these techniques between domains without a clear semantic alignment. As a semi-supervised technique, we have shown bridging autoencoders to require less supervised labels, making it more feasible to learn personalized cross-modal domain transfer based on the creative guidance of individual users.

heading: APPENDIX A TRAINING TARGET DESIGN
text: We want to archive following three goals for the proposed VAE for latent spaces:
1. It should be able to model the latent space of both domains, including modeling local changes as well.
2. It should encode two latent spaces in a way to enable domain transferability. This means encoded z1 and z2 in the shared latent space should occupy overlapped spaces.
3. The transfer should be kept in the same class. That means, regardless of domain, zs for the same class should occupy overlapped spaces.
With these goals in mind, we propose to use an optimization target composing of three kinds of losses. In the following text for notational convenience, we denote approximated posterior Z ′d , q(z′|zd, D = d), zd ∼ q(zd|xd), xd ∼ p(xd) for d ∈ {1, 2}, the process of sampling z′d from domain d.
1. Modeling two latent spaces with local changes. VAEs are often used to model data with local changes in mind, usually demonstrated with smooth interpolation, and we believe this property also applies when modeling the latent space of data. Consider for each domain d ∈ {1, 2}, the VAE is fit to data to maximize the ELBO (Evidence Lower Bound)
LELBOd = Ez′∼Z′d [log π(zd; g(z ′, D = d)]− βKLDKL (q(z′|zd, D = d)‖p(z′))
where both q and g are fit to maximize LELBOd . Notably, the latent space zs are continuous, so we choose the likelihood π(z; g) to be the product ofN (z; g, σ2I), where we set σ to be a constant that effectively sets log π(z; g) = ||z − g||2, which is the L2 loss in Figure 1 (a). Also, DKL is denoted as KL loss in Figure 1 (a).
2. Cross-domain overlapping in shared latent space. Formally, we propose to measure the cross-domain overlapping through the distance between following two distributions as a proxy: the distribution of z′ from source domain (e.g., z′1 ∼ Z ′1) and that from the target domain (e.g., z′2 ∼ Z ′1). We use Wasserstein Distance (Arjovsky et al., 2017) to measure the distance of two sets of samples (this notion straightforwardly applies to the mini-batch setting) S′1 and S ′ 2, where S ′ 1 is sampled from the source domain z′1 ∼ Z ′1 and S′1 from the target domain z′2 ∼ Z ′d. For computational efficiency and inspired by Deshpande et al. (2018), we use SWD, or Sliced Wasserstein Distance (Bonneel et al., 2015) between S′1 and S ′ 2 as a loss term to encourage cross-domain overlapping in shared latent space. This means in practice we introduce the loss term
LSWD = 1 |Ω| ∑ ω∈Ω W 22 (proj(S ′ 1, ω),proj(S ′ 2, ω))
where Ω is a set of random unit vectors, proj(A, a) is the projection ofA on vector a, andW 22 (A,B) is the quadratic Wasserstein distance, which in the one-dimensional case can be easily solved by monotonically pairing points in A and B, as proven in Deshpande et al. (2018).
3. Intra-class overlapping in shared latent space. We want that regardless of domain, zs for the same class should occupy overlapped spaces, so that instance of a particular class should retain its label through the transferring. We therefore introduce the following loss term for both domain d ∈ {1, 2}
LClsd = Ez′∈Z′dH(f(z ′), lx′)
where H is the cross entropy loss, f(z′) is a one-layer linear classifier, and lx′ is the one-hot representation of label of x′ where x′ is the data associated with z′. We intentionally make classifier f as simple as possible in order to encourage more capacity in the VAE instead of the classifier. Notably, unlike previous two categories of losses that are unsupervised, this loss requires labels and is thus supervised.
In Figure 7 we show the intuition to design and the contribution to performance from each loss terms.

heading: APPENDIX B MODEL ARCHITECTURE
text: The model architecture of our proposed VAE is illustrated in Figure B. The model relies on Gated Mixing Layers, or GML. We find empirically that GML improves performance by a large margin than linear layers, for which we hypothesize that this is because both the latent space (z1, z2) and the shared latent space z′ are Gaussian space, GML helps optimization by starting with a good initialization. We also explore other popular network components such as residual network and batch normalization, but find that they are not providing performance improvements. Also, the condition is fed to encoder and decoder as a 2-length one hot vector indicating one of two domains.
For all settings, we use the dimension of shared latent space 100, βSWD = 1.0 and βCLs = 0.05,
Specifically, for MNIST↔MNIST and MNIST↔ Fashion MNIST, we use the dimension of shared latent space 8, 4 layers of FC (fully connected layers) of size 512 with ReLU, βKL = 0.05, βSWD = 1.0 and βCLs = 0.05; while for MNIST↔ SC09, we use the dimension of shared latent space 16, 8 layers of FC (fully connected layers) of size 1024 with ReLU βKL = 0.01, βSWD = 3.0 and βCLs = 0.3. The difference is due to that GAN does not provide posterior, so the latent space points estimated by the classifier is much harder to model.
For optimization, we use Adam optimizer(Kingma & Ba, 2014) with learning rate 0.001, beta1 = 0.9 and beta2 = 0.999. We train 50000 batches with batch size 128. We do not employ any other tricks for VAE training.

heading: APPENDIX C COMPARISON TO EXISTING APPROACHES
text: We compare our results with two existing approaches, Pix2Pix (Isola et al., 2016) on the left and CycleGAN (Zhu et al., 2017) on the right, on the same MNIST↔ Fashion MNIST transfer settings used in Figured 4. We show qualitative results from applying Pix2Pix and CycleGAN in Figure 9, which can be compared with Figured 4, as well as quantitative results in Table 5. Both qualitative and quantitative results shows the limitation of existing methods and our proposed approach’s advantage over them.

heading: APPENDIX D SUPPLEMENTARY FIGURES

Review summaries:
The authors demonstrate that it is possible to transfer across modalities (e.g., image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (e.g., variational autoencoder and a generative adversarial network). Some detailed comments are listed as follows, 

In this paper, the authors study an interesting problem, i.e., heterogeneous domain transfer such as knowledge transfer between an image domain and a speech/audio domain. In particular, the proposed solution contains two major steps: (i) pre-train each domain via VAE or GAN, and (ii) train a conditional VAE in semi-supervised manner in order to bridge two domains (see Section 2.2). Experiments on three public datasets (including three cross-domain settings) show the effectiveness of the proposed two-step solution.