Latent Domain Transfer: Crossing modalities with Bridging Autoencoders

Domain transfer is an exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains , preserving local variations and capturing many aspects of variation without labels . However , most successful applications to date require the two domains to be closely related ( e.g. , image-to-image , video-video ) , utilizing similar or shared networks to transform domain-specific properties like texture , coloring , and line shapes . Here , we demonstrate that it is possible to transfer across modalities ( e.g. , image-to-audio ) by first abstracting the data with latent generative models and then learning transformations between latent spaces . We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion , and even between different types of models ( e.g. , variational autoencoder and a generative adversarial network ) . We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space . Through qualitative and quantitative evaluations , we demonstrate that the proposed model preserves both locality and semantic alignment through the transfer process . Finally , the hierarchical structure decouples the cost of training the base generative models and semantic alignments , enabling computationally efficient and data efficient retraining of personalized mapping functions . 1 INTRODUCTION . Domain transfer has long captured the imagination of inventors and artists alike . The early precursor of the phonograph , the phonautograph , was actually inspired by the idea of ” words which write themselves ” , where the shape of audio waveforms would transform into the shape of writing , capturing the content and character of the speaker ’ s voice in shape and stroke of the written characters ( Feaster , 2009 ) . While perhaps fanciful at the time , modern deep learning techniques have shown similar complex transformations are indeed possible . Deep learning enables domain transfer by learning a smooth mapping between two domains such that the variations in one domain are reflected in the other . This has been demonstrated to great effect within a data modality , for example transferring between two different styles of image ( Isola et al. , 2016 ; Zhu et al. , 2017 ; Li et al. , 2018 ; Li , 2018 ) , video ( Wang et al. , 2018 ) and music ( Mor et al. , 2018 ) . The works have been the basis of interesting creative tools , as small intuitive changes in the source domain are reflected by small intuitive changes in the target domain . Furthermore , the strong conditioning signal of the source domain makes learning transformations easier than learning a full generative model in each domain . Despite these successes , this line of work in domain transfer has several limitations . The first limitation is that it requires that two domains should be closely related ( e.g . image-to-image or videoto-video ) . This allows the model to focus on transferring local properties like texture and coloring instead of high-level semantics . For example , directly applying these image-to-image transfer such as CycleGAN ( Zhu et al. , 2017 ) or its variants to images from distant domains leads to distorted and unrealistic results ( Li et al. , 2018 ) . This agrees with the findings of Chu et al . ( 2017 ) who show that CycleGAN transformations are more akin to adversarial examples than style transfer , as the model learns to hide information about the source domain in near-imperceptible high-frequency variations of the target domain . The second limitation is data efficiency . Most conditional GAN techniques , such as Pix2Pix ( Isola et al. , 2016 ) and vid2vid ( Wang et al. , 2018 ) , require very dense supervision from large volumes of paired data . This is usually accomplished by extracting features , such as edges or a segmentation map , and then training the conditional GAN to learn the inverse mapping back to pixels . For many more interesting transformations , no such easy alignment procedure exists , and paired data is scarce . We demonstrate the limitation of existing approaches in Appendix C. For multi-modal domain transfer , we seek to train a model capable of transferring instances from a source domain ( x1 ) to a target domain ( x2 ) , such that local variations in source domain are transferred to local variations in the target domain . We refer to this property as locality . Thus , local interpolation in the source domain would ideally be similar to local interpolation in target domain when transferred . There are many possible ways that two domains could align such that they maintain locality , with many different alignments of semantic attributes . For instance , for a limited dataset , there is no a priori reason that images of the digit “ 0 ” and spoken utterances of the digit “ 0 ” would align with each other . Or more abstractly , there may be no agreed common semantics for images of landscapes and passages of music , and it is at the liberty of the user to define such connections based on their own intent . Our goal in modeling is to respect the user ’ s intent and make sure that the correct semantics ( e.g. , labels ) are shared between the two domains after transfer . We refer to this property as semantic alignment . A user can thus sort a set of data points from in each domain into common bins , which we can use to constrain the cross-domain alignment . We can quantitatively measure the degree of semantic alignment by using a classifier to label transformed data and measuring the percentage of data points that fall into the same bin for the source and target domain . Our goal can thus be stated as learning transformations that preserve locality and semantic alignment , while requiring as few labels from a user as possible . To achieve this goal and tackle prior limitations , we propose to abstract the domain domains with independent latent variable models , and then learn to transfer between the latent spaces of those models . Our main contributions include : • We propose a shared ” bridging ” VAE to transfer between latent generative models . Locality and semantic alignment of transformations are encouraged by applying a sliced-wasserstein distance , and a classification loss respectively to the shared latent space . • We demonstrate with qualitative and quantitative results that our proposed method enables transfer both within a modality ( image-to-image ) and between modalities ( image-to-audio ) . • Since we training a smaller secondary model in latent space , we find improvements in training efficiency , measured by both in terms of the amount of required labeled data and well training time . 2 METHOD . Figure 1 diagrams our hierarchical approach . We first independently pre-train separate generative models , either a VAE or GAN , for both the source and target domain . For VAEs , data is encoded from the data domain to the latent space through a learned encoding function z ∼ q ( z|x ) , and decoded back to the data space with a decoder function x̂ ∼ g ( x|z ) . For GANs , we choose latent samples from a spherical Gaussian prior z ∼ p ( z ) and then use rejection sampling to only select latent samples whose associated data x = g ( z ) is classified with high confidence by an auxiliary classifier . We then add the bridging conditional VAE with shared weights , tasked with modeling both latent spaces z1 , z2 . The VAE has a single shared latent space z′ , that corresponds to both domains . Sharing the weights encourages the model to seek common structures between the latent domains , but we also find it helpful to condition both the encoder qshared ( z′|z , D ) and decoder gshared ( z|z′ , D ) , with an additional one-hot domain label , D , to allow the model some flexibility to adapt to variations particular to each domain . While the low-level VAEs have a spherical Gaussian prior , we penalize the KL-Divergence to be less than 1 , allowing the models to achieve better reconstructions and retain some structure of the original dataset for the bridging VAE to model . Full architectural and training details can be found in the Appendix . The conditional bridging VAE objective consists of three types of loss loss terms : 1 . Evidence Lower Bound ( ELBO ) . Standard value that is maximized for training a VAE . For each domain d ∈ { 1 , 2 } , LELBOd = −Ez′∼Z′d [ log π ( zd ; g ( z ′ , D = d ) ] + βKLDKL ( q ( z ′|zd , D = d ) ‖p ( z′ ) ) where the likelihood π ( z ; g ) is a spherical Gaussian N ( z ; g , σ2I ) , and σ and βKL are hyperparmeters set to 1 and 0.1 respectively to encourage reconstruction accuracy . 2 . Sliced Wasserstein Distance ( SWD ) ( Bonneel et al. , 2015 ) . The distribution distance between mini-batches of samples from each domain in the shared latent space ( z′1 , z ′ 2 ) . LSWD = 1/|Ω| ∑ ω∈Ω W 22 ( proj ( z ′ 1 , ω ) , proj ( z ′ 2 , ω ) ) where Ω is a set of random unit vectors , proj ( A , a ) is the projection of A on vector a , and W 22 ( A , B ) is the quadratic Wasserstein distance . 3 . Classification Loss ( Cls ) . For each domain d ∈ { 1 , 2 } , we enforce semantic alignment with attribute labels y and a classification loss in the shared latent space : LClsd = Ez′∈Z′dH ( f ( z ′ ) , y ) where H is the cross entropy loss , f ( z′ ) is a one-layer linear classifier . Including terms for both domains , the total training loss is then L = ( LELBO1 + LELBO2 ) + βSWDLSWD + βCls ( LCls1 + LCls2 ) Where βSWD and βCls are scalar loss weights . The transfer procedure is illustrated Figure 2 using synthetic data . For reconstructions , data x1 is passed through two encoders , z1 ∼ q ( z1|x1 ) , z′ ∼ qshared ( z′|z1 , D = 1 ) , are reconstructed through two decoders , ẑ1 ∼ gshared ( ẑ1|z′ , D = 1 ) , x̂1 ∼ g ( x̂1|ẑ1 ) . For transformations , the encoding is the same , but decoding uses decoders ( and conditioning ) from the second domain , ẑ2 ∼ gshared ( ẑ2|z′ , D = 2 ) , x̂2 ∼ g ( x̂2|ẑ2 ) . Further analysis of this example and intuition behind loss terms is summarized in Figure 2 and detailed in Appendix A . 3 RELATED WORK . We discuss two aspects of existing research focuses that are related to our work , followed by how our work differentiates itself from them in order to deal with the challenge identified in this paper . Latent Generative Models : Deep latent generative models are usually constructed to transfer a simple , tractable distribution p ( z ) into the approximation of population distribution p∗ ( x ) , through an expressive neural network function . Such models include VAE ( Kingma & Welling , 2013 ) and GAN ( Goodfellow et al. , 2014 ) . GANs are trained with an accompany classifier that attempts to distinguish between samples from the decoder and the true dataset . VAEs , in contrast , are trained with an encoder distribution q ( z|x ) as an approximation to the posterior p ( z|x ) using variational approximation through the use of evidence lower bound ( ELBO ) . These classes of models have been thoroughly investigated in many applications and variants ( Gulrajani et al. , 2017 ; Li et al. , 2017 ; Bikowski et al. , 2018 ) including conditional generation ( Mirza & Osindero , 2014 ) , generation of one domain conditioned on another ( Dai et al. , 2017 ; Reed et al. , 2016 ) , generation of high-quality images ( Karras et al. , 2018 ) and long-range structure in music ( Roberts et al. , 2018 ) . In terms of overall VAE structure , Zhao et al . ( 2017 ) studies options to build hierarchical VAEs . Domain Transfer : The domain transfer enables transfer between images ( Isola et al. , 2016 ; Zhu et al. , 2017 ; Li et al. , 2018 ; Li , 2018 ) , audio ( Mor et al. , 2018 ) and video ( Wang et al. , 2018 ) , intuitively mapping between two domains where the variations in one domain should be reflected in the other . Besides visually appealing results , domain transfer also enables application such as image colorization Zhang et al . ( 2017 ) . Domain transfer is also proposed to be done through jointly training of generative models ( Lu , 2018 ) . Also , the behavior of domain transfer models also attracts attention . For example , Chu et al . ( 2017 ) suggests that image transfer does only local , texture level transfer . To enable transfer between possibly drastically different domains , Our work proposes to use VAE in modeling the latent space of pre-trained generative models , in several aspects differentiating itself from related work . Generally , the modeling of latent space is different from modeling directly the data domains as most of latent generative models naturally do , also , the transfer in a more abstract semantics and between heterogeneous domains differs from most domain transfer method which focusing on locality and similar domains . More specifically , regarding modeling latent spaces , Zhao et al . ( 2017 ) suggests training a hierarchical VAE on a single domain should be done end-to-end , whose extension to multiple domains seems non-trivial and likely to suffer from data efficient issues . Instead , our proposed work , though enabling separation of pre-trained model and conditional shared VAE , apply to domain transfer setting while overcoming this shortcoming . Moreover , regarding shared latent space , Liu et al . ( 2017 ) proposes to use shared latent space for two generative models on data domains . It requires joint training of generative models on both domains using dense supervision which is infeasible for drastically different domains . Our work that leverages pre-trained generative model and model the latent space instead of data domains addresses to this limitation .

Summary: In this paper, the authors study an interesting problem, i.e., heterogeneous domain transfer such as knowledge transfer between an image domain and a speech/audio domain. In particular, the proposed solution contains two major steps: (i) pre-train each domain via VAE or GAN, and (ii) train a conditional VAE in semi-supervised manner in order to bridge two domains (see Section 2.2). Experiments on three public datasets (including three cross-domain settings) show the effectiveness of the proposed two-step solution.