LATENT NORMALIZING FLOWS FOR MANY-TO-MANY CROSS-DOMAIN MAPPINGS
Year: 2020

Abstract:
Learned joint representations of images and text form the backbone of several important cross-domain tasks such as image captioning. Prior work mostly maps both domains into a common latent representation in a purely supervised fashion. This is rather restrictive, however, as the two domains follow distinct generative processes. Therefore, we propose a novel semi-supervised framework, which models shared information between domains and domain-specific information separately. The information shared between the domains is aligned with an invertible neural network. Our model integrates normalizing flow-based priors for the domain-specific information, which allows us to learn diverse many-to-many mappings between the two domains. We demonstrate the effectiveness of our model on diverse tasks, including image captioning and text-to-image synthesis.

text: 1 INTRODUCTION
Joint image-text representations find application in cross-domain tasks such as imageconditioned text generation (captioning; Mao et al., 2015; Karpathy & Fei-Fei, 2017; Xu et al., 2018) and text-conditioned image synthesis (Reed et al., 2016). Yet, image and text distributions follow distinct generative processes, making joint generative modeling of the two distributions challenging.
Current state-of-the-art models for learning joint image-text distributions encode the two domains in a common shared latent space in a fully supervised setup (Gu et al., 2018; Wang et al., 2019). While such approaches can model supervised information in the shared latent space, they do not preserve domain-specific information. However, as the domains under consideration, e.g. images and texts, follow distinct generative processes, many-to-many mappings naturally emerge – there are many likely captions for a given image and vice versa. Therefore, it is crucial to also encode domain-specific variations in the latent space to enable many-to-many mappings.
State-of-the-art models for cross-domain synthesis leverage conditional variational autoencoders (VAEs, cVAEs; Kingma & Welling, 2014) or generative adversarial networks (GANs; Goodfellow et al., 2014) for learning conditional distributions. However, such generative models (e.g., Wang et al., 2017; Aneja et al., 2019) enforce a Gaussian prior in the latent space. Gaussian priors can result in strong regularization or posterior collapse as they impose stringent constraints while modeling complex distributions in the latent space (Tomczak & Welling, 2018). This severely limits the accuracy and diversity of the cross-domain generative model.
Recent work (Ziegler & Rush, 2019; Bhattacharyya et al., 2019) has found normalizing flows (Dinh et al., 2015) advantageous for modeling complex distributions in the latent space. Normalizing flows can capture a high degree of multimodality in the latent space through a series of transformations from a simple distribution to a complex data-dependent prior. Ziegler & Rush (2019) apply normalizing flow-based priors in the latent space of unconditional variational autoencoders for discrete distributions and character-level modeling.
We propose to leverage normalizing flows to overcome the limitations of existing cross-domain generative models in capturing heterogeneous distributions and introduce a novel semi-supervised Latent Normalizing Flows for Many-to-Many Mappings (LNFMM) framework. We exploit normalizing flows (Dinh et al., 2015) to capture complex joint distributions in the latent space of our model (Fig. 1). Moreover, since the domains under consideration, e.g. images and texts, have different generative processes, the latent representation for each distribution is modeled such that it contains both shared cross-domain information as well as domain-specific information. The latent dimensions constrained by supervised information from paired data model the common (semantic) information across images and texts. The diversity within the image and text distributions, e.g. different visual or textual styles, are encoded in the residual latent dimensions, thus preserving domain-specific variation. We can hence synthesize diverse samples from a distribution given a reference point in the other domain in a many-to-many setup. We show the benefits of our learned many-to-many latent spaces for real-world image captioning and text-to-image synthesis tasks on the COCO dataset (Lin et al., 2014). Our model outperforms the current state of the art for image captioning w.r.t. the Bleu and CIDEr metrics for accuracy as well as on various diversity metrics. Additionally, we also show improvements in diversity metrics over the state of the art in text-to-image generation.

heading: 2 RELATED WORK
text: Diverse image captioning. Recent work on image captioning introduces stochastic behavior in captioning and thus encourages diversity by mapping an image to many captions. Vijayakumar et al. (2018) sample captions from a very high-dimensional space based on word-to-word Hamming distance and parts-of-speech information, respectively. To overcome the limitation of sampling from a high-dimensional space, Shetty et al. (2017); Dai et al. (2017); Li et al. (2018) build on Generative Adversarial Networks (GANs) and modify the training objective of the generator, matching generated captions to human captions. While GAN-based models can generate diverse captions by sampling from a noise distribution, they suffer on accuracy due to the inability of the model to capture the true underlying distribution. Wang et al. (2017); Aneja et al. (2019), therefore, leverage conditional Variational Autoencoders (cVAEs) to learn latent representations conditioned on images based on supervised information and sequential latent spaces, respectively, to improve accuracy and diversity. Without supervision, cVAEs with conditional Gaussian priors suffer from posterior collapse. This results in a strong trade-off between accuracy and diversity; e.g. Aneja et al. (2019) learn sequential latent spaces with a Gaussian prior to improve diversity, but suffer on perceptual metrics. Moreover, sampling captions based only on supervised information limits the diversity in the captions. In this work we show that by learning complex multimodal priors, we can model text distributions efficiently in the latent space without specific supervised clustering information and generate captions that are more diverse and accurate.
Diverse text-to-image synthesis. State-of-the-art methods for text-to-image synthesis are based on conditional GANs (Reed et al., 2016). Much of the research for text-conditioned image generation has focused on generating high-resolution images similar to the ground truth. Zhang et al. (2017; 2019b) introduce a series of generators in different stages for high-resolution images. AttnGAN (Xu et al., 2018) and MirrorGAN (Qiao et al., 2019) aim at synthesizing fine-grained image features by attending to different words in the text description. Dash et al. (2017) condition image generation on class information in addition to texts. Yin et al. (2019) use a Siamese architecture to generate images with similar high-level semantics but different low-level semantics.In this work, we instead focus on generating diverse images for a given text with powerful latent semantic spaces, unlike GANs with Gaussian priors, which fail to capture the true underlying distributions and result in mode collapse.
Normalizing flows & Variational Autoencoders. Normalizing flows (NF) are a class of density estimation methods that allow exact inference by transforming a complex distribution to a simple distribution using the change-of-variables rule. Dinh et al. (2015) develop flow-based generative models with affine transformations to make the computation of the Jacobian efficient. Recent works (Dinh et al., 2017; Kingma & Dhariwal, 2018; Ardizzone et al., 2019; Behrmann et al., 2019) extend flow-based generative models to multi-scale architectures to model complex dependencies across dimensions. Vanilla Variational Autoencoders (VAEs; Kingma & Welling, 2014) consider simple Gaussian priors in the latent space. Simple priors can provide very strong constraints, resulting in poor latent representations (Hoffman & Johnson, 2016). Recent work has, therefore, considered modeling complex priors in VAEs. Particularly, Wang et al. (2017); Tomczak & Welling (2018)
propose mixtures of Gaussians with predefined clusters, Chen et al. (2017) use neural autoregressive model priors, and van den Oord et al. (2017) use discrete models in the latent space, which improves results for image synthesis. Ziegler & Rush (2019) learn a prior based on normalizing flows to model multimodal discrete distributions of character-level texts in the latent spaces with nonlinear flow layers. However, this invertible layer is difficult to be optimized in both directions. Bhattacharyya et al. (2019) learn conditional priors based on normalizing flows to model conditional distributions in the latent space of cVAEs. In this work, we learn a conditional prior using normalizing flows in the latent space of our variational inference model, modeling joint complex distributions in the latent space, particularly of images and texts for diverse cross-domain many-to-many mappings.

heading: 3 METHOD
text: To learn joint distributions pµ(xt, xv) of texts and images that follow distinct generative processes with ground-truth distributions pt(xt) and pv(xv), respectively, in a semi-supervised setting, we formulate a novel joint generative model based on variational inference: Latent Normalizing Flows for Many-to-Many Mappings (LNFMM). Our model defines a joint probability distribution over the data {xt, xv} and latent variables z with a distribution pµ(xt, xv, z) = pµ(xt, xv|z)pµ(z), parameterized by µ. We maximize the likelihood of pµ(xt, xv) using a variational posterior qθ(z|xt, xv), parameterized by variables θ. As we are interested in jointly modeling distributions with distinct generative processes, e.g. images and texts, the choice of the latent distribution is crucial. Mapping to a shared latent distribution can be very restrictive (Xu et al., 2018). We begin with a discussion of our variational posterior qθ(z|xt, xv) and its the factorization in our LNFMM model, followed by our normalizing flow-based priors, which enable qθ(z|xt, xv) to be complex and multimodal, allowing for diverse many-to-many mappings.
Factorizing the latent posterior. We choose a novel factorized posterior distribution with both shared and domain-specific components. The shared component zs is learned with supervision from paired image-text data and encodes information common to both domains. The domain-specific components encode information that is unique to each domain, thus preserving the heterogeneous structure of the data in the latent space. Specifically, consider zt and zv as the latent variables to model text and image distributions. Recall from above that zs denotes the shared latent variable for supervised learning, which encodes information shared between the data points xt and xv . Given this supervised information, the residual information specific to each domain is encoded in z′t and z ′ v . This leads to the factorization of the variational posterior of our LNFMM model with zt = [zs z′t] and zv = [zs z′v],
log qθ(zs, z ′ t, z ′ v|xt, xv) = log qθ1(zs|xt, xv) + log qθ2(z′t|xt, zs) + log qθ3(z′v|xv, zs). (1)
Next, we derive our LNFMM model in detail. Since directly maximizing the log-likelihood of pµ(xt, xv) with the variational posterior is intractable, we derive the log-evidence lower bound for learning the posterior distributions of the latent variables z = {zs, z′t, z′v}.

heading: 3.1 DERIVING THE LOG-EVIDENCE LOWER BOUND
text: Maximizing the marginal likelihood pµ (xt, xv) given a set of observation points {xt, xv} is generally intractable. Therefore, we develop a variational inference framework that maximizes a variational lower bound on the data log-likelihood – the log-evidence lower bound (ELBO) with the proposed factorization in Eq. (1),
log pµ(xt, xv) ≥ Eqθ(z|xt,xv) [ log pµ(xt, xv|z) ] + Eqθ(z|xt,xv) [ log pφ(z)− log qθ(z|xt, xv) ] , (2)
where z = {zs, z′t, z′v} are the latent variables. The first expectation term is the reconstruction error. The second expectation term minimizes the KL-divergence between the variational posterior qθ(z|xt, xv) and a prior pφ(z). Taking into account the factorization in Eq. (1), we now derive the ELBO for our LNFMM model. We first rewrite the reconstruction term as
Eqθ(zs,z′t,z′v|xt,xv) [ log pµ(xt|zs, z′t, z′v) + log pµ(xv|zs, z′t, z′v) ] , (3)
which assumes conditional independence given the domain-specific latent dimensions z′t, z ′ v and the shared latent dimensions zs. Thus, the reconstruction term can be further simplified as
Eqθ1 (zs|xt,xv)qθ2 (z′t|xt,zs) [ log pµ(xt|zs, z′t) ] + Eqθ1 (zs|xt,xv)qθ3 (z′v|xv,zs) [ log pµ(xv|zs, z′v) ] . (4)
Next, we simplify the KL-divergence term on the right of Eq. (2). We use the chain rule along with Eq. (1) to obtain
DKL ( qθ(zs, z ′ t, z ′ v|xt, xv) ∥∥ pφ(zs, z′t, z′v)) = DKL(qθ1(zs|xt, xv)∥∥ pφs(zs))+ DKL ( qθ2(z ′ t|xt, zs)
∥∥ pφt(z′t|zs))+DKL(qθ3(z′v|xv, zs)∥∥ pφv (z′v|zs)).(5) This assumes a factorized prior of the form pφ(zs, z′t, z ′ v) = pφs(zs)pφt(z ′ t|zs)pφv (z′v|zs), consistent with our conditional independence assumptions, given that information specific to each distribution is encoded in {z′t, z′v}. The final ELBO can then be expressed as log pµ(xt, xv) ≥ Eqθ1 (zs|xt,xv)qθ2 (z′t|xt,zs) [ log pµ(xt|zs, z′t)
] +Eqθ1 (zs|xt,xv)qθ3 (z′v|xv,zs) [ log pµ(xv|zs, z′v) ] −DKL ( qθ1(zs|xt, xv)
∥∥ pφs(zs)) −DKL ( qθ2(z ′ t|xt, zs) ∥∥ pφt(z′t|zs))−DKL(qθ3(z′v|xv, zs)∥∥ pφv (z′v|zs)). (6)
In the standard VAE formulation (Kingma & Welling, 2014), the priors corresponding to pφt(z ′ t|zs) and pφv (z ′ v|zs) are modeled as standard normal distributions. However, Gaussian priors limit the expressiveness of the model in the latent space since they result in strong constraints on the posterior (Tomczak & Welling, 2018; Razavi et al., 2019; Ziegler & Rush, 2019). Specifically, optimizing with a Gaussian prior pushes the posterior distribution towards the mean, limiting diversity and hence generative power (Tomczak & Welling, 2018). This is especially true for complex multimodal image and text distributions. Furthermore, alternatives like Gaussian mixture model-based priors (Wang et al., 2017) also suffer from similar drawbacks and additionally depend on predefined heuristics like the number of components in the mixture model. Analogously, the VampPrior (Tomczak & Welling, 2018) depends on a predefined number of pseudo-inputs to learn the prior in the latent space. Similar to Ziegler & Rush (2019); Bhattacharyya et al. (2019), which learn priors based on exact inference models, we propose to learn the conditional priors pφt(z ′ t|zs) and pφv (z′v|zs) jointly with the variational posterior in Eq. (1) using normalizing flows.

heading: 3.2 VARIATIONAL INFERENCE WITH NORMALIZING FLOW-BASED PRIORS
text: Normalizing flows are exact inference models, which can map simple distributions to complex densities through a series of Kt invertible mappings,
fφt = f Kt φt ◦ fKt−1φt ◦ · · · ◦ f 1 φt .
This allows us to transform a simple base density ∼ p( ) to a complex multimodal conditional prior pφt(z ′ t|zs) (and correspondingly to pφv (z′v|zs)). The likelihood of the latent variables under the base density can be obtained using the change-of-variables formula. A composition of invertible mappings f iφt , parameterized by parameters φt, is learned such that = f −1 φt
(z′t|zs). The loglikelihood with Jacobian J iφt = ∂f i φt/∂fi−1φt , assuming f 0 φt
= I is the identity, can be expressed as
log pφt(z ′ t|zs) = log p ( f−1φt (z ′ t|zs) ) − log ∣∣∣det ∂z′t∂ ∣∣∣ = log p ( f−1φt (z ′ t|zs) ) − Kt∑ i=1 log ∣∣det J iφt∣∣. (7)
Using data-dependent and non-volume preserving transformations, multimodal priors can be jointly learned in the latent space, allowing for more complex posteriors and better solutions of the evidence lower bound. Using Eq. (7), the ELBO with normalizing flow-based priors can be expressed by rewriting the KL-divergence terms in Eq. (6) as (analogously for the other term)
DKL ( qθ2(z ′ t|xt, zs) ∥∥ pφt(z′t|zs)) =− Eqθ2 (z′t|xt,zs)[ log p(f−1φt (z′t|zs))− Kt∑ i=1 log ∣∣det J iφt∣∣]
+ Eqθ2 (z′t|xt,zs) [ log qθ2(z ′ t|xt, zs) ] .
(8)
Next, we describe our complete model for learning joint distributions with latent normalizing flows using Eqs. (6) and (8), which enables many-to-many mappings between domains.
3.3 LATENT NORMALIZING FLOW MODEL FOR MANY-TO-MANY MAPPINGS
We illustrate our complete model in Fig. 2. It consists of two domain-specific encoders to learn the domainspecific latent posterior distributions qθ2(z ′ t|xt, zs) and qθ3(z ′ v|xt, zs). As the shared latent variable zs encodes information common to both domains, it holds that qθ1(zs|xt, xv) = qθ1(zs|xt) = qθ1(zs|xv) for a matching pair of data points (xt, xv). Therefore, each encoder must be able to model the common supervised information independently for every matching pair (xt, xv). We enforce this by splitting the output dimensions of each encoder into zv = [zs z′v] and zt = [zs z ′ t], respectively (cf. Eq. 1), and constraining the supervised latent dimensions to encode the same information. We assume that the shared latent code zs has d′ dimensions. We propose to learn the posterior distribution qθ1(zs|xt, xv) as the shared latent space between two domain-specific au-
toencoders. One simple method to induce sharing is by minimizing the mean-squared error between the encodings. However, this is not ideal given that xt and xv follow different highly multimodal generative processes. We, therefore, learn an invertible mapping fφs : Rd
′ → Rd′ with invertible neural networks such that the d′-dimensional latent code zs can be transformed between the domains v and t. Let zv with d ≥ d′ dimensions be the encoded latent variable for distribution pv(xv). A bijective mapping fφs : (zv)d′ 7→ (zt)d′ is learned with an invertible mapping analogous to Eq. (7) as
log qθ1(zs|xt, xv) = log qθ1 ( (zv)d′ |xt, xv ) = log qθ1 ( fφs((zv)d′)|xt ) + K∑ i=1 log ∣∣det J iφs ∣∣. (9)
Here, (·)d′ denotes restricting the latent code to the d′-dimensional shared part. fφs is an invertible neural network with affine coupling layers (Dinh et al., 2015), i.e. fφs = f K φs ◦ fK−1φs ◦ · · · ◦ f 1 φs
. This makes it easy to compute Jacobians J iφs = ∂f i φs/∂fi−1φs formulated as triangular matrices for the layers of the invertible neural network. Again, we assume that f0φs = I is the identity.
The number of dimensions for domain-specific information, i.e. the dimensionality of z′t and z ′ v , can be different for the two domains, depending on the complexity of each distribution. Note that by conditioning on the dimensions with a supervision signal, we minimize the redundancy in the dimensions for unsupervised information without disentangling the dimensions. The multimodal conditional prior in qθ2(z ′ t|zs, xt) and qθ3(z′v|zs, xv) is modeled with non-volume-preserving normalizing flow models with Eq. (7) (Dinh et al., 2017), parameterized by φt and φv , respectively.
The data log-likelihood terms in Eqs. (4) and (6) are defined as the (negative) reconstruction errors. Let gθv : xv 7→ zv with θv = {θ1, θ3} and gθt : xt 7→ zt with θt = {θ1, θ2} denote the image and text encoders, respectively. Further, hωv : zv 7→ xv and hωt : zt 7→ xt are the decoders with parameters ωv and ωt corresponding to images and texts, respectively, with x̃v = hωv (gθv (xv)) and x̃t = hωt (gθt(xt)) denoting the decoded image and text samples. For texts, we consider the output probability of the jth word (xt)j of the ground-truth sentence xt given the previous reconstructed words (x̃t)0:j−1 from the text decoder, and define the reconstruction error as Lrect (xt, x̃t) = − ∑ j log pµ ( (xt)j |(x̃t)0:j−1 ) . For images, the reconstruction loss between the input image xv and reconstructed image x̃v from the image decoder is taken as Lrecv (xv, x̃v) = ‖xv − x̃v‖, where ‖ · ‖ is either the `1 or `2 norm. Furthermore, in Eq. (9) we define log qθ1 (fφs((zv)d′)|xt) as the cost of mapping (zv)d′ to the latent space of texts under the transformation fφs ; we use the mean-squared error between the encoded text representations (zt)d′ and the transformed image representation fφs ((zv)d′) of the paired data (xt, xv) (see Ardizzone et al., 2019).
Starting from the ELBO on the right-hand side of Eq. (6), the learned latent priors in Eq. (8), the invertible mapping from Eq. (9), and plugging in the reconstruction terms as just defined, the overall objective of our semi-supervised generative model framework to be minimized is given by
Lµ(xt,xv) = λ1DKL(qθ1 ( zs|xt, xv) ∥∥ pφs(zs))+ λ2DKL(qθ2(z′t|xt, zs)∥∥ pφt(z′t|zs)) + λ3DKL ( qθ3(z ′ v|xv, zs)
∥∥ pφv (z′v|zs))+ λ4Lrect (xt, x̃t) + λ5Lrecv (xv, x̃v). (10) Here, µ = {θ, φs, φv, φt, ωv, ωt} are all parameters to be learned; λi, i = {1, . . . , 5} are regularization parameters. Recall that x̃t and x̃v are decoded text and image samples, respectively. We assume the prior pφs(zs) on the shared latent space to be uniform. The strength of this uniform prior on the shared dimensions can be controlled with the regularization parameter λ1.
Our model allows for bi-directional many-to-many mappings. In detail, given a data point xv from the image domain with latent encoding zv , we first map it to the text domain through the invertible transformation zs = fφs((zv)d′). We can now generate diverse texts by sampling from the learned latent prior pφt(z ′ t|zs). A similar procedure is followed for sampling images given text through the learned prior pφv (z ′ v|zs). For conditional generation tasks, as we do not have to sample from the supervised latent space, we find a “uniform” prior pφs(zs) to be advantageous in practice as it loosens the constraints on the decoders. Alternatively, a more complex flow-based prior could also be used here to enable sampling of the shared semantic space. We show the effectiveness of our joint latent normalizing flow-based priors on real-world tasks, i.e. diverse image captioning and text-to-image synthesis.

heading: 4 EXPERIMENTS
text: To validate our method for learning many-to-many mappings to provide latent joint distributions, one of the important real-world tasks is that of image-to-text or text-to-image synthesis. To that end, we perform experiments on the COCO dataset (Lin et al., 2014). It contains 82,783 training and 40,504 validation images, each with five captions. Following Wang et al. (2016); Mao et al. (2015) for image captioning, we use 118,287 data points for training and evaluate on 1,000 test images. For text-to-image synthesis, the training set contains 82,783 images and 40,504 validation data points at test time (Reed et al., 2016; Huang et al., 2017). Architecture details can be found in the Appendix.

heading: 4.1 IMAGE CAPTIONING
text: We evaluate our approach against methods that generate diverse captions for a given image. We compare against methods based on (conditional) variational autoencoders, AG-CVAE (Wang et al., ∗Our implementation
LNFMM 97.0 4741 0.60 0.37 0.51
Table 3. Diversity evaluation on at most the best-5 sentences after consensus re-ranking
Image Caption Image Caption
• Two elephants standing next to each other in a river. • A herd of elephants in a grassy area of water. • Two elephants walking through a river while standing in the water. • Two elephants are walking and baby in the water. • A group of elephants walking around a watering hole.
• A living room filled with furniture and a large window. • A room with a couch and a large wooden table. • A living room filled area with furniture and a couch and chair. • A room with a couch, chair, and a lamp. • This is a room with furniture on the floor.
Table 5. Example captions generated by our model
2016), which uses additional supervision based on information about objects in the images for a Gaussian mixture model in the latent space, and Seq-CVAE (Aneja et al., 2019), which models sequential latent spaces with Gaussian priors. We also include Div-BS (Vijayakumar et al., 2018) based on beam search and POS (Aneja et al., 2018), which uses additional supervision from images. Additionally, we include different ablations to show the effectiveness of various components of our approach. LNFMM-MSE does not contain the invertible neural network fφs , i.e. we directly minimize the mean squared error between (zt)d′ and (zv)d′ in Eq. (10). We fix the image encodings of a VGG-16 encoder in LNFMM (pre-trained) for comparison to image captioning methods with pre-trained image features. Furthermore, LNFMM-TXT contains the latent flow for unsupervised information fφt only for the text distribution. In this setting, we have zv = zs and thus zv is transformed through the invertible neural network as fφs(zv). To show that our framework can be applied in the semi-supervised setting, we also consider limited labeled data with only 30% of the paired training data for supervision. The remaining training data is included as unpaired data for modeling domain-specific information.
Evaluation. We evaluate the accuracy with Bleu (B) 1-4 (Papineni et al., 2002), CIDEr (C; Vedantam et al., 2015), ROUGE (R; Lin, 2004), METEOR (M; Denkowski & Lavie, 2014), and SPICE (S; Anderson et al., 2016). For evaluating diversity, we consider the metrics of Wang et al. (2017); Aneja et al. (2019). Uniqueness is the percentage of unique captions generated on the test set. Novel sentences are the captions that were never observed in the training data. m-Bleu-4 computes Bleu-4 for each diverse caption with respect to remaining diverse captions per image. The Bleu-4 obtained is averaged across all images. Div-n is the ratio of distinct n-grams to the total number of words generated per set of diverse captions.
Results. In Table 1 we show the caption evaluation metrics in the oracle setting, i.e. taking the maximum score for each accuracy metric over all the candidate captions. We consider 100 samples z, consistent with previous methods. The cVAE baseline with an image-conditioned Gaussian prior does not perform well on all metrics, showing the inability of the Gaussian prior to model meaningful latent spaces representative of the multimodal nature the underlying data distribution. The overall trend across metrics is that our LNFMM model improves the upper bound on Bleu and CIDEr while being comparable on the Rouge and Spice metrics.
Comparing the accuracy of the baseline LNFMM-MSE with LNFMM, we can conclude that learning the shared posterior distribution of zs with our invertible mapping is better than directly minimizing the mean squared error in the latent space due to differences in the complexity of the distributions. Also note that LNFMM (pre-trained) with fixed image-encoded representations has better performance compared to AG-CVAE and Seq-CVAE, in particular. This highlights that the LNFMM
learns representations in the latent space that are representative of the underlying data distribution even for pre-trained features.
Table 2 considers a more realistic setting (as ground-truth captions are not always available) where, instead of comparing against the reference captions of the test set, reference captions for images from the training set most similar to the test image are retrieved. The generated captions are then ranked with the CIDEr score (Mao et al., 2015). While Div-BS has very good accuracy across metrics due to the wide search space, our LNFMM model gives state-of-the-art accuracy on various Bleu metrics and especially the CIDEr score, which is known to correlate well with human evaluations. More interestingly, compared to AG-CVAE with conditional Gaussian mixture priors based on object (class) information, our LNFMM model, which does not encode any additional supervised information in the latent space, outperforms the former on all accuracy metrics by a large margin. Moreover, the recent GXN (Gu et al., 2018) and M3DGAN (Ma et al., 2019) also study bi-directional synthesis with joint models in Gaussian latent spaces. Ma et al. (2019) additionally model attention in the latent space. From Table 4, we see that our method considerably outperforms the competing methods, validating the importance of complex priors in the latent space for image-text distributions. This again highlights that the complex joint distribution of images and texts captured by our LNFMM model is more representative of the ground-truth data distribution. We additionally experiment with limited labelled training data (30% labeled), which shows that our approach copes well with limited paired data. We finally compare LMFMM against LNFMM-TXT to show the importance of joint learning of image and text generative models. With a generative model only for texts, the joint distribution cannot be captured effectively in the latent space.
With diversity being an important goal here, we show in Table 3 that our LNFMM method improves diversity across all metrics, with a 6.5% improvement in unique captions generated in the test set and 4741/5000 captions not previously seen in the training set. Our generated captions for a given image also show more diversity with low mutual overlap (mBLEU) compared to the state of the art. We, moreover, observe high n-gram diversity for the generated captions of each image. Div-BS with high accuracy has limited diversity as it can repeat the n-grams in different captions. POS and AG-CVAE, due to guided supervision in the latent space, offer diversity but model only syntactic or semantic diversity, respectively (Wang & Chan, 2019). Our proposed LNFMM model in Table 5 shows a range of diverse captions with different semantics and syntactic structure. Therefore, we conclude that the proposed LNFMM can effectively learn semantically meaningful joint latent representations without any additional object or text-guided supervision. The data-dependent learnt priors are thus promising for synthesizing captions with high human-correlated accuracy as well as diversity.

heading: 4.2 TEXT-TO-IMAGE SYNTHESIS
text: Given a text description, we are now interested in generating diverse images representative of the domain-specific structure of images. To that end, we include a discriminator to improve the image quality of our image decoder. Note that this does not affect the joint latent space of the LNFMM model. We evaluate our method against state-of-the-art approaches such as AttnGAN (Xu et al., 2018), HD-GAN (Zhang et al., 2018b), StackGAN (Zhang et al., 2017), and GAN-INT-CLS (Reed et al., 2016). While our main goal is to encourage text-conditioned diversity in the generated samples, the current state-of-the-art for text-to-image generation aims at improving the realism of the
generated images. Note that various GANs can be integrated with the image decoder of our framework as desired.
Evaluation. As we are interested in modeling diversity, we study the diversity in generated images using the Inference via Optimization (IvOM; Srivastava et al., 2017) and LPIPS (Zhang et al., 2018a) metrics against the state-of-the-art AttnGAN. Given the text, for each matching image, IvOM finds the closest image the model is capable of generating. Thus, it shows whether the model can match the diversity of the ground-truth distribution. LPIPS (Zhang et al., 2018a) evaluates diversity by computing pairwise perceptual similarities using a deep neural network. Additionally, we also report the Inception score (Salimans et al., 2016).
Results. In Table 6, our method improves over AttnGAN for both IvOM and LPIPS scores, showing that our method can effectively model the image semantics conditioned on the texts in the latent space, as well as generate diverse images for a given caption. Note that AttnGAN uses extra supervision to improve the inception score. However, it is unclear if this improves the visual quality of the generated images as pointed out by
Zhang et al. (2018b). We improve the IS over HD-GAN, which does not use additional supervision. Qualitative examples in Fig. 3 show that our LNFMM model generates diverse images, e.g., closeup images of food items as well as different orientations of the baseball player in the field. In Fig. 4 we additionally see that given a caption, images generated by our LNFMM model capture detailed semantics of the test images compared to that of AttnGAN, showing the representative power of our latent space.

heading: 5 CONCLUSION
text: We present a novel and effective semi-supervised LNFMM framework for diverse bi-directional many-to-many mappings with learnt priors in the latent space, which enables modeling joint imagetext distributions. Particularly, we model domain-specific information conditioned on the shared information between the two domains with normalizing flows, thus preserving the heterogeneous structure of the data in the latent space. Our extensive experiments with bi-directional synthesis show that our latent space can effectively model data-dependent priors, which enable highly accurate and diverse generated samples of images or texts.
Acknowledgement. This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1.

heading: A APPENDIX
text: A.1 NETWORK ARCHITECTURE
We provide the details of the network architecture of Fig. 2.
Image pipeline. The network consists of an image encoder built upon VGG-16. 4096-dimensional activations of input images are extracted from the fully-connected layer of VGG-16. This is followed by a 2048-dimensional fully-connected layer with ReLU activations. We then project it to the latent space with a 1056-dimensional fully-connected layer. The image pipeline has d′ = dim zs = 992 and dim z′v = 64 dimensions. In the image decoder, we leverage the architecture of Zhang et al. (2019a) to synthesize images of 64 × 64 or 256 × 256 dimensions. The input to the decoder has dimensionality 1056. For the image generation experiments, we additionally apply the discriminator of Zhang et al. (2019b) to the output of the image decoder.
Text pipeline. We use a bidirectional GRU with two layers and a hidden size of 1024 as text encoder. This outputs 1024-dimensional latent representations for sentences. For text, dim z′t = 32. The text decoder is a LSTM with one layer and hidden size of 512.
Flow modules. Our network consists of two flow modules for conditional priors on image and text domains and an invertible neural network to exchange supervised information.
Invertible neural network for supervision (fφs ): This invertible neural network consists of 12 flow layers and input dimensionality of 992. Each flow consists of conditional affine coupling layers followed by a switch layer (Dinh et al., 2015).
Latent flow for conditional prior on images (fφv ): We map the 64 dimensions of the image encodings from the image encoder to a Gaussian with normalizing flows with 16 layers of flow and 512 hidden channels. Each flow consists of conditional affine coupling layers followed by a switch layer (Dinh et al., 2017).
Latent flow for conditional prior on texts (fφt ): We map the 32 dimensions of the text encodings from the text encoder to a Gaussian with normalizing flows with 16 layers of flow and 1024 hidden channels. Each flow consists of a conditional activation normalization layer followed by conditional affine coupling layers. Invertible 1× 1 convolutions are applied to the output of the affine coupling layers, which is followed by a switch layer (Kingma & Dhariwal, 2018).
A.2 DIVERSITY IN IMAGE CAPTIONING
We show more qualitative examples of the captions generated by our LNFMM model in Table 7. The example captions show syntactic as well as semantic diversity.
A.3 TEXT-TO-IMAGE SYNTHESIS
We additionally show diverse images generated by our LNFMM model in Fig. 6. Our generated images can successfully capture the text semantics and also exhibit image specific diversity, e.g. in style and orientation of objects. Furthermore, to show that the latent space captures the joint distribution, we show the images generated by our model with IvOM by finding a zv conditioned on the input text that is most likely to have generated the test image. We show the images generated for the zv most likely to have generated the image. Our generated samples in Fig. 7 capture the details in the images, showing that our LNFMM model learns powerful latent representations.
A.4 VISUALIZATION OF LATENT SPACES
In Table 8 we show the t-SNE (van der Maaten & Hinton, 2008) visualization of the latent space of texts conditioned on an image. In the examples shown, we observe global patterns like ‘table with foods’ across all captions of an image and local patterns in clusters like ‘table with several’ or ‘a dinner table with’ showing image-conditioned representation as well as domain-specific variation.
Similarly, in Table 9 we can observe clusters in image space conditioned on texts. Here again, we can observe global patterns like ‘close-up of food’ and local patterns like ‘pizza’, ‘food with toppings, or ‘sandwiches’.
A.5 VALIDATION WITH HUMAN EVALUATION
In addition to validating the performance of our LNFMM method on image captioning for accuracy in Table 1 and for diversity Table 3 with various measures, we conduct a human evaluation of the quality of captions generated by our LNFMM approach against Div-BS (Vijayakumar et al., 2018) for accuracy and diversity. We presented five captions each for a set of images and for each method to four human annotators. The human annotators assessed the captions for accuracy and diversity on a scale from 1 to 10. Here, accuracy is defined as how well the captions describe the details in a given image and diversity can be syntactic diversity or semantic diversity.
In Fig. 8 (left), the score for each image is the average score given by the four annotators and Fig. 8 (right) shows the scores for the captions of each image from each annotator. Here, our findings are similar to those in Table 1 and Table 3. The captions generated by Div-BS were assessed to have diversity scores in the range of [5, 8], while the diversity scores fell into the range [7, 10] for our LNFMM method. Furthermore, the accuracy of the captions of each image generated with our LNFMM was found to be better compared to Div-BS, thereby showing that LNFMM can generate captions that are both diverse and accurate.

Review summaries:
This paper addresses the problem of many-to-many cross domain mapping tasks (such as captioning or text-to-image synthesis). It proposes a double variational auto-encoder architecture mapping data to a factored latent representation with both shared and domain-specific components. The proposed model makes use of normalizing flow-based priors to enrich the latent representation and of an invertible network for ensuring the consistency of the shared component across the two autoencoders. Experiments are thorough and demonstrate results that are competitive or better than the state-of-the-art.

The paper introduces a variational model for text to image and image to text mappings. The novelty consists in separating the modeling of text and image latent representations on one hand and the modeling of a shared content representation on the other hand. Priors for text, image and shared representations are generated through an invertible – flow model. The motivation for this is to allow for complex priors. Training for the shared component is supervised using aligned text and image data, while training for the residual text and image components is unsupervised. Experiments are performed for text and image generation, using training data from the COCO dataset.