Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings

1 INTRODUCTION Joint image-text representations find application in cross-domain tasks such as imageconditioned text generation ( captioning ; Mao et al. , 2015 ; Karpathy & Fei-Fei , 2017 ; Xu et al. , 2018 ) and text-conditioned image synthesis ( Reed et al. , 2016 ) . Yet , image and text distributions follow distinct generative processes , making joint generative modeling of the two distributions challenging . Current state-of-the-art models for learning joint image-text distributions encode the two domains in a common shared latent space in a fully supervised setup ( Gu et al. , 2018 ; Wang et al. , 2019 ) . While such approaches can model supervised information in the shared latent space , they do not preserve domain-specific information . However , as the domains under consideration , e.g . images and texts , follow distinct generative processes , many-to-many mappings naturally emerge – there are many likely captions for a given image and vice versa . Therefore , it is crucial to also encode domain-specific variations in the latent space to enable many-to-many mappings . State-of-the-art models for cross-domain synthesis leverage conditional variational autoencoders ( VAEs , cVAEs ; Kingma & Welling , 2014 ) or generative adversarial networks ( GANs ; Goodfellow et al. , 2014 ) for learning conditional distributions . However , such generative models ( e.g. , Wang et al. , 2017 ; Aneja et al. , 2019 ) enforce a Gaussian prior in the latent space . Gaussian priors can result in strong regularization or posterior collapse as they impose stringent constraints while modeling complex distributions in the latent space ( Tomczak & Welling , 2018 ) . This severely limits the accuracy and diversity of the cross-domain generative model . Recent work ( Ziegler & Rush , 2019 ; Bhattacharyya et al. , 2019 ) has found normalizing flows ( Dinh et al. , 2015 ) advantageous for modeling complex distributions in the latent space . Normalizing flows can capture a high degree of multimodality in the latent space through a series of transformations from a simple distribution to a complex data-dependent prior . Ziegler & Rush ( 2019 ) apply normalizing flow-based priors in the latent space of unconditional variational autoencoders for discrete distributions and character-level modeling . We propose to leverage normalizing flows to overcome the limitations of existing cross-domain generative models in capturing heterogeneous distributions and introduce a novel semi-supervised Latent Normalizing Flows for Many-to-Many Mappings ( LNFMM ) framework . We exploit normalizing flows ( Dinh et al. , 2015 ) to capture complex joint distributions in the latent space of our model ( Fig . 1 ) . Moreover , since the domains under consideration , e.g . images and texts , have different generative processes , the latent representation for each distribution is modeled such that it contains both shared cross-domain information as well as domain-specific information . The latent dimensions constrained by supervised information from paired data model the common ( semantic ) information across images and texts . The diversity within the image and text distributions , e.g . different visual or textual styles , are encoded in the residual latent dimensions , thus preserving domain-specific variation . We can hence synthesize diverse samples from a distribution given a reference point in the other domain in a many-to-many setup . We show the benefits of our learned many-to-many latent spaces for real-world image captioning and text-to-image synthesis tasks on the COCO dataset ( Lin et al. , 2014 ) . Our model outperforms the current state of the art for image captioning w.r.t . the Bleu and CIDEr metrics for accuracy as well as on various diversity metrics . Additionally , we also show improvements in diversity metrics over the state of the art in text-to-image generation . 2 RELATED WORK . Diverse image captioning . Recent work on image captioning introduces stochastic behavior in captioning and thus encourages diversity by mapping an image to many captions . Vijayakumar et al . ( 2018 ) sample captions from a very high-dimensional space based on word-to-word Hamming distance and parts-of-speech information , respectively . To overcome the limitation of sampling from a high-dimensional space , Shetty et al . ( 2017 ) ; Dai et al . ( 2017 ) ; Li et al . ( 2018 ) build on Generative Adversarial Networks ( GANs ) and modify the training objective of the generator , matching generated captions to human captions . While GAN-based models can generate diverse captions by sampling from a noise distribution , they suffer on accuracy due to the inability of the model to capture the true underlying distribution . Wang et al . ( 2017 ) ; Aneja et al . ( 2019 ) , therefore , leverage conditional Variational Autoencoders ( cVAEs ) to learn latent representations conditioned on images based on supervised information and sequential latent spaces , respectively , to improve accuracy and diversity . Without supervision , cVAEs with conditional Gaussian priors suffer from posterior collapse . This results in a strong trade-off between accuracy and diversity ; e.g . Aneja et al . ( 2019 ) learn sequential latent spaces with a Gaussian prior to improve diversity , but suffer on perceptual metrics . Moreover , sampling captions based only on supervised information limits the diversity in the captions . In this work we show that by learning complex multimodal priors , we can model text distributions efficiently in the latent space without specific supervised clustering information and generate captions that are more diverse and accurate . Diverse text-to-image synthesis . State-of-the-art methods for text-to-image synthesis are based on conditional GANs ( Reed et al. , 2016 ) . Much of the research for text-conditioned image generation has focused on generating high-resolution images similar to the ground truth . Zhang et al . ( 2017 ; 2019b ) introduce a series of generators in different stages for high-resolution images . AttnGAN ( Xu et al. , 2018 ) and MirrorGAN ( Qiao et al. , 2019 ) aim at synthesizing fine-grained image features by attending to different words in the text description . Dash et al . ( 2017 ) condition image generation on class information in addition to texts . Yin et al . ( 2019 ) use a Siamese architecture to generate images with similar high-level semantics but different low-level semantics.In this work , we instead focus on generating diverse images for a given text with powerful latent semantic spaces , unlike GANs with Gaussian priors , which fail to capture the true underlying distributions and result in mode collapse . Normalizing flows & Variational Autoencoders . Normalizing flows ( NF ) are a class of density estimation methods that allow exact inference by transforming a complex distribution to a simple distribution using the change-of-variables rule . Dinh et al . ( 2015 ) develop flow-based generative models with affine transformations to make the computation of the Jacobian efficient . Recent works ( Dinh et al. , 2017 ; Kingma & Dhariwal , 2018 ; Ardizzone et al. , 2019 ; Behrmann et al. , 2019 ) extend flow-based generative models to multi-scale architectures to model complex dependencies across dimensions . Vanilla Variational Autoencoders ( VAEs ; Kingma & Welling , 2014 ) consider simple Gaussian priors in the latent space . Simple priors can provide very strong constraints , resulting in poor latent representations ( Hoffman & Johnson , 2016 ) . Recent work has , therefore , considered modeling complex priors in VAEs . Particularly , Wang et al . ( 2017 ) ; Tomczak & Welling ( 2018 ) propose mixtures of Gaussians with predefined clusters , Chen et al . ( 2017 ) use neural autoregressive model priors , and van den Oord et al . ( 2017 ) use discrete models in the latent space , which improves results for image synthesis . Ziegler & Rush ( 2019 ) learn a prior based on normalizing flows to model multimodal discrete distributions of character-level texts in the latent spaces with nonlinear flow layers . However , this invertible layer is difficult to be optimized in both directions . Bhattacharyya et al . ( 2019 ) learn conditional priors based on normalizing flows to model conditional distributions in the latent space of cVAEs . In this work , we learn a conditional prior using normalizing flows in the latent space of our variational inference model , modeling joint complex distributions in the latent space , particularly of images and texts for diverse cross-domain many-to-many mappings . 3 METHOD . To learn joint distributions pµ ( xt , xv ) of texts and images that follow distinct generative processes with ground-truth distributions pt ( xt ) and pv ( xv ) , respectively , in a semi-supervised setting , we formulate a novel joint generative model based on variational inference : Latent Normalizing Flows for Many-to-Many Mappings ( LNFMM ) . Our model defines a joint probability distribution over the data { xt , xv } and latent variables z with a distribution pµ ( xt , xv , z ) = pµ ( xt , xv|z ) pµ ( z ) , parameterized by µ . We maximize the likelihood of pµ ( xt , xv ) using a variational posterior qθ ( z|xt , xv ) , parameterized by variables θ . As we are interested in jointly modeling distributions with distinct generative processes , e.g . images and texts , the choice of the latent distribution is crucial . Mapping to a shared latent distribution can be very restrictive ( Xu et al. , 2018 ) . We begin with a discussion of our variational posterior qθ ( z|xt , xv ) and its the factorization in our LNFMM model , followed by our normalizing flow-based priors , which enable qθ ( z|xt , xv ) to be complex and multimodal , allowing for diverse many-to-many mappings . Factorizing the latent posterior . We choose a novel factorized posterior distribution with both shared and domain-specific components . The shared component zs is learned with supervision from paired image-text data and encodes information common to both domains . The domain-specific components encode information that is unique to each domain , thus preserving the heterogeneous structure of the data in the latent space . Specifically , consider zt and zv as the latent variables to model text and image distributions . Recall from above that zs denotes the shared latent variable for supervised learning , which encodes information shared between the data points xt and xv . Given this supervised information , the residual information specific to each domain is encoded in z′t and z ′ v . This leads to the factorization of the variational posterior of our LNFMM model with zt = [ zs z′t ] and zv = [ zs z′v ] , log qθ ( zs , z ′ t , z ′ v|xt , xv ) = log qθ1 ( zs|xt , xv ) + log qθ2 ( z′t|xt , zs ) + log qθ3 ( z′v|xv , zs ) . ( 1 ) Next , we derive our LNFMM model in detail . Since directly maximizing the log-likelihood of pµ ( xt , xv ) with the variational posterior is intractable , we derive the log-evidence lower bound for learning the posterior distributions of the latent variables z = { zs , z′t , z′v } . 3.1 DERIVING THE LOG-EVIDENCE LOWER BOUND . Maximizing the marginal likelihood pµ ( xt , xv ) given a set of observation points { xt , xv } is generally intractable . Therefore , we develop a variational inference framework that maximizes a variational lower bound on the data log-likelihood – the log-evidence lower bound ( ELBO ) with the proposed factorization in Eq . ( 1 ) , log pµ ( xt , xv ) ≥ Eqθ ( z|xt , xv ) [ log pµ ( xt , xv|z ) ] + Eqθ ( z|xt , xv ) [ log pφ ( z ) − log qθ ( z|xt , xv ) ] , ( 2 ) where z = { zs , z′t , z′v } are the latent variables . The first expectation term is the reconstruction error . The second expectation term minimizes the KL-divergence between the variational posterior qθ ( z|xt , xv ) and a prior pφ ( z ) . Taking into account the factorization in Eq . ( 1 ) , we now derive the ELBO for our LNFMM model . We first rewrite the reconstruction term as Eqθ ( zs , z′t , z′v|xt , xv ) [ log pµ ( xt|zs , z′t , z′v ) + log pµ ( xv|zs , z′t , z′v ) ] , ( 3 ) which assumes conditional independence given the domain-specific latent dimensions z′t , z ′ v and the shared latent dimensions zs . Thus , the reconstruction term can be further simplified as Eqθ1 ( zs|xt , xv ) qθ2 ( z′t|xt , zs ) [ log pµ ( xt|zs , z′t ) ] + Eqθ1 ( zs|xt , xv ) qθ3 ( z′v|xv , zs ) [ log pµ ( xv|zs , z′v ) ] . ( 4 ) Next , we simplify the KL-divergence term on the right of Eq . ( 2 ) . We use the chain rule along with Eq . ( 1 ) to obtain DKL ( qθ ( zs , z ′ t , z ′ v|xt , xv ) ∥∥ pφ ( zs , z′t , z′v ) ) = DKL ( qθ1 ( zs|xt , xv ) ∥∥ pφs ( zs ) ) + DKL ( qθ2 ( z ′ t|xt , zs ) ∥∥ pφt ( z′t|zs ) ) +DKL ( qθ3 ( z′v|xv , zs ) ∥∥ pφv ( z′v|zs ) ) . ( 5 ) This assumes a factorized prior of the form pφ ( zs , z′t , z ′ v ) = pφs ( zs ) pφt ( z ′ t|zs ) pφv ( z′v|zs ) , consistent with our conditional independence assumptions , given that information specific to each distribution is encoded in { z′t , z′v } . The final ELBO can then be expressed as log pµ ( xt , xv ) ≥ Eqθ1 ( zs|xt , xv ) qθ2 ( z′t|xt , zs ) [ log pµ ( xt|zs , z′t ) ] +Eqθ1 ( zs|xt , xv ) qθ3 ( z′v|xv , zs ) [ log pµ ( xv|zs , z′v ) ] −DKL ( qθ1 ( zs|xt , xv ) ∥∥ pφs ( zs ) ) −DKL ( qθ2 ( z ′ t|xt , zs ) ∥∥ pφt ( z′t|zs ) ) −DKL ( qθ3 ( z′v|xv , zs ) ∥∥ pφv ( z′v|zs ) ) . ( 6 ) In the standard VAE formulation ( Kingma & Welling , 2014 ) , the priors corresponding to pφt ( z ′ t|zs ) and pφv ( z ′ v|zs ) are modeled as standard normal distributions . However , Gaussian priors limit the expressiveness of the model in the latent space since they result in strong constraints on the posterior ( Tomczak & Welling , 2018 ; Razavi et al. , 2019 ; Ziegler & Rush , 2019 ) . Specifically , optimizing with a Gaussian prior pushes the posterior distribution towards the mean , limiting diversity and hence generative power ( Tomczak & Welling , 2018 ) . This is especially true for complex multimodal image and text distributions . Furthermore , alternatives like Gaussian mixture model-based priors ( Wang et al. , 2017 ) also suffer from similar drawbacks and additionally depend on predefined heuristics like the number of components in the mixture model . Analogously , the VampPrior ( Tomczak & Welling , 2018 ) depends on a predefined number of pseudo-inputs to learn the prior in the latent space . Similar to Ziegler & Rush ( 2019 ) ; Bhattacharyya et al . ( 2019 ) , which learn priors based on exact inference models , we propose to learn the conditional priors pφt ( z ′ t|zs ) and pφv ( z′v|zs ) jointly with the variational posterior in Eq . ( 1 ) using normalizing flows .

Summary: This paper addresses the problem of many-to-many cross domain mapping tasks (such as captioning or text-to-image synthesis). It proposes a double variational auto-encoder architecture mapping data to a factored latent representation with both shared and domain-specific components. The proposed model makes use of normalizing flow-based priors to enrich the latent representation and of an invertible network for ensuring the consistency of the shared component across the two autoencoders. Experiments are thorough and demonstrate results that are competitive or better than the state-of-the-art.