Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings
Authors: Shweta Mahajan
Iryna Gurevych
Stefan Roth
Venue:  (2020)

Abstract: Learned joint representations of images and text form the backbone of several important cross-domain tasks such as image captioning. Prior work mostly maps both domains into a common latent representation in a purely supervised fashion. This is rather restrictive, however, as the two domains follow distinct generative processes. Therefore, we propose a novel semi-supervised framework, which models shared information between domains and domain-specific information separately. 
The information shared between the domains is aligned with an invertible neural network. Our model integrates normalizing flow-based priors for the domain-specific information, which allows us to learn diverse many-to-many mappings between the two domains. We demonstrate the effectiveness of our model on diverse tasks, including image captioning and text-to-image synthesis.

Published as a conference paper at ICLR 2020
LATENT NORMALIZING FLOWS FOR MANY-TO-MANY CROSS-DOMAIN MAPPINGS
Shweta Mahajan, Iryna Gurevych, Stefan Roth Department of Computer Science, TU Darmstadt, Germany
ABSTRACT
Learned joint representations of images and text form the backbone of several im- portant cross-domain tasks such as image captioning. Prior work mostly maps both domains into common latent representation in purely supervised fash- ion. This is rather restrictive. however, as the two domains follow distinct gen- erative processes. Therefore, we propose novel semi-supervised framework, which models shared information between domains and domain-specine informa- tion separately. The information shared between the domains aligned with an invertible neural network. Our model integrates normalizing flow-based priors for the domain-specite information, which allows us to learn diverse many-to-many mappings between the two domains. We demonstrate the effectiveness of our model on diverse tasks, including image captioning and text-to-image synthesis.
INTRODUCTION
Joint image-text representations find applica- tion in cross-domain tasks such as image- conditioned text generation (captioning; MMo
Data Ditribution
Genersted Semples
Space
al.] 2013 Karpathy & Fei-Fei 2017, Xu et at. 2018) 20.18) and text-conditions image synthe- sis (Reed al.) 2016). Yet, image and text dis-
group paople mamming arouna onwannd
grouo poops starding in annou avna gman propes standing poew.
tributions follow distinct generative processes, making joint generative modeling of the two distributions challenging.
coup peoper
Figure 1. Joint multimodal latent representation im- ages and texts of our LNFMM model for diverse many- to many mappings
Current state-of-the-art models for learning joint image-text distributions encode the two domains a common shared latent space in a fully supervised setup (Gu et al. 2018 Wang et al,, 2019). While such approaches model supervised information in the shared latent space, they do not preserve domain-specific information. However, as the domains under consideration, e.g. im- ages and texts, follow distinct generative processes, many-to-many mappings naturally emerge there are many likely captions for given image and vice versa. Therefore, it is crucial to also encode domain-specific variations in the latent space enable many-to-many mappings.
State-of-the-art models for cross-domain synthesis leverage conditional variational autoencoders (VAEs, eVAEs; Kingma & Welling 2014) generative adversarial networks (GANs; Goodtellow 2014) for learning conditional distributions. However, such generative models (e.g., Wang 2017 Aneja et al. (019)) enforce Gaussian prior in the latent space. Gaussian priors can result in strong regularization or posterior collapse as they impose stringent constraints while mod- eling complex distributions the latent space (Tomezak Welling 2018). This severely limits the accuracy and diversity of the cross-domain generative model.
et al.
al.
Recent work (Ziegler & Rush 2019 Bhattacharyya et al. 2019) has found normalizing flows (Dinh) et al.) 2015) advantageous for modeling complex distributions in the latent space. Normalizing flows can capture a high degree multimodality in the latent space through series of transformations from simple distribution to complex data-dependent prior. Ziegler & Rush (2019) apply malizing flow-based priors in the latent space of unconditional variational autoencoders for discrete distributions and character-level modeling.

---

Published as a conference paper at ICLR 2020
We propose to leverage normalizing flows to overcome the limitations of existing dommin generative models in capturing heterogeneous distributions and introduce a novel semi-supervised Latent Normalizing Flows for Many-to-Mant Mappings (LNFMM) framework. We exploit normal- izing flows (Dinh et al. 2015) to capture complex joint distributions in the latent space of our model (Fig. D. Moreover, since the domains under consideration, e-g. images and texts, have different gen- erative processes, the latent representation for each distribution is modeled such that it contains both shared cross-domain information as well as domain-specific information. The latent dimensions constrained by supervised information from paired data model the common (semantic) information across images and texts. The diversity within the image and text distributions, e-g. different visual or textual styles, are encoded in the residual latent dimensions, thus preserving domain-speciife vari- ation. We can hence synthesize diverse samples from distribution given a reference point in the other domain in a many-to-many setup. We show the benefits of our learned many-to-many latent spaces for real-world image captioning and text-to-image synthesis tasks on the COCO dataset (Lin) ct al. 2014). Our model outperforms the current state of the art for image captioning w.r.t. the Bleu and CIDEr metrics for accuracy as well as on various diversity metrics. Additionally, we also show improvements in diversity metrics over the state of the art in text-to-image generation.
2 RELATED WORK
Diverse image captioning. Recent work on image captioning introduces stochastic behavior in captioning and thus encourages diversity by mapping an image to many captions. Vijayakumar et al. (2018) sample captions from a very high-dimensional space based on word-to-word Hamming distance and pfrts-of-speech information, respectively. To overcome the limitation of sampling from a high-dimensional space, Shetty et al. (2017); Dai et al. (201)); Li et al. (2018) build on Gener- ative Adversarial Networks (GANs) and modify the training objective of the generator, matching generated captions to human captions. While GAN-based models can generate diverse captions by sampling from a noise distribution, they suffer on accuracy due to the inability of the model to cap- ture the true underlying distribution. [Wang et al. (2017); Aneeaa et al. (2019), therefore, leverage conditional Variational Autoencoders (cVAEs) to learn latent representations conditioned on images based on supervised information and sequential latent spaces, respectively, to improve accuracy and diversity. Without supervision, cVAES with conditional Gaussian priors suffer from posterior col- lapse. This results in strong trade-off between accuracy and diversity; e-g. [Aneja et al. (2019) learn sequential latent spaces with a Gaussian prior to improve diversity, but suffer on perceptual metrics. Moreover, sampling captions based only on supervised information limits the diversity in the captions. In this work we show that by learning complex multimodal priors, we can model text distributions efficiently in the latent space without specific supervised clustering information and generate captions that are more diverse and accurate.
Diverse text-to-image synthesis. State-of-the-an methods for text-to-image synthesis are based on conditional GANs (Reed et al. (0166). Much of the research for text-conditioned image generation has focused on generating high-resolution images similar to the ground truth. Zhang et al. 20117
2019b) introduce a series of generators in different stages for high-resolution images. AttnGAN (Xu et al. 2018)) and MirrorGAN (Qiao et al.) 2019) aim at synthesizing fine-grat -ne-graieed image features by attending to different words in the text description. Dash et al. (2017) condition image generation on
class information addition to texts. Yin et al. (2019) use a Siamese architecture to generate images with similar high-level semantics but different low-level semantics. In this work, we instead focus on generating diverse images for a given text with powerful latent semantic spaces, unlike GANs with Gaussian priors, which fail to capture the true underlying distributions and result in mode collapse.
Normalizing flows & Variational Autoencoders. Normalizing flows (NF) are a class of density estimation methods that allow exact inference by transforming a complex distribution to a simple distribution using the change-of-variables rule. Dinh et al. (2015) develop flow-based generative models with affine transformations to make the computation of the Jacobian efficient. Recent works (Dinh et al. 2017 Kingma& & Dhariwal 2018 Ardizzone et al., 2019 Behrmann et al. 2019) extend flow-based generative models to multi-scale architectures to model complex dependencies across dimensions. Vanilla Variational Autoencoders (VAEs; Kingma & Welling 2014) consider simple Gaussian priors in the latent space. Simple priors can provide very strong constraints, resulting in poor latent representations (Hofiman & Johnson| 2016). Recent work has, therefore, considered modeling complex priors in VARS. Particularly, Wang et al. (2017); Tomezak & Welling (2018)
2

---

Published as a conference paper at ICLR 2020
propose mixtures of Gaussians with predefined clusters, Chen et al. 2017
neural autoregressive
model priors, and van den Oord et al. (0117) use discrete models in the latent space, which improves results for image synthesis. Ziegler & Rush (2019) learn prior based on normalizing flows to model multimodal discrete distributions of character-level texts in the latent spaces with nonlinear flow layers. However, this invertible layer is difficult to be optimized in both directions. [Bhattacharyya et al. (2019) learn conditional priors based on normalizing flows to model conditional distributions the latent space of cAEEs. In this work, we learn a conditional prior using normalizing flows in the latent space of our variational inference model, modeling joint complex distributions in the latent space, particularly of images and texts for diverse domss- many-to -o--mnny mappings.
3 METHOD
To learn joint distributions pyu (1 F.y) of texts and images that follow distinct generative processes with ground-truth distributions pr(Iz) and (x,)),, respectively, in a semi-supervised setting, we formulate novel joint generative model based variational inference: Latent Normalizing Flows for Many-to-Many Mappings (LNFMM). Our model defines joint probability distribution over the data [e) and latent variables with a distribution (x Ze. 2) = Py ( 2e (z),, parame- terized by Js. We maximize the likelihood of (=t, F.) using a variational posterior go(=|<r,, zw), parameterized by variables 0. As we are interested in jointly modeling distributions with distinct generative processes, e.g. images and texts, the choice of the latent distribution is crucial. Mapping to a shared latent distribution can be very restrictive (Xu et al. 2018). We begin with a discussion of our variational posterior qo (2|22, x.)) and its the factorization in our LNFMM model, followed by our normalizing www-base priors, which enable go (2(222, x.) to be complex and multimodal, allowing for diverse many-to- to-many mappings.
Factorizing the latent posterior. We choose novel factorized posterior distribution with both shared and commann-specife components. The shared component 2, is learned with supervision from paired image-text data and encodes information common to both domains. The domain-specific components encode information that is unique to each domain, thus preserving the heterogeneous structure of the data in the latent space. Specifically, consider and 2, as the latent variables to model text and image distributions. Recall from above that 2g denotes the shared latent variable for supervised learning, which encodes information shared between the data points I; and zg. Given this supervised information, the residual information specific to each domain is encoded in 2 and 2/ This leads to the factorization of the variational posterior of our LNFMM model with 22 [a, x/1 and žn = [ =
log (z(, t/s 20, x.) = log qo, (Zgl z.) + log gos (2//ar, (ifae,,z) z.) + log gos (2// =e).
Next, we derive LNFMM model in detail. Since directly maximizing the log-likelihood of Pa (2,,,, with the variational posterior intractable, we derive the log-evidence lower bound for learning the posterior distributions of the latent variables = = { 2/ 2/)
3.1 DERIVING THE LOG-EVIDENCE LOWER BOUND
Maximizing the marginal likelihood Pn (It, uw) given a set of observation points (vr, } is gen- erally intractable. Therefore, we develop variational inference framework that maximizes vari- ational lower bound on the data log-lik likeliiol - the log-evidence lower bound (ELBO) with the proposed factorization in Eq. (
log Pa (Zt, 2 Eg log Pa (2,, [o]=]] + (go((le (og log (0 xw)]
(2)
where = (2,, 2 the latent variables. The first expectation term is the reconstruction
error. The second expectation term minimizes the KL--divegeecce between the variational posterior go (2|22,, x.) and prior P²(2). Taking into account the factorization in Eq. (I). now derive the
ELBO for our LNFMM model. We first rewrite the reconstruction term as
Egg (R, af, a.)) [lo p. (xt 2.. 2/, 2{)) + log (xv) 2., 24, =[)]
3

---

Published as a conference paper at ICLR 2020
which assumes conditional independence given the domain -specffe latent dimensions z/1 2/ and the shared latent dimensions 2g. Thus, the reconstruction term can be further simplified as
Ego, (e (=|] [..).) 1og z{]] + Ege, , (aa (e welgg (1g (e))))
[og (zn//s =(] (4)
Next, we simplify the KI-diieegence term on the right of Eq. 2). We use the chain rule along with Eq. ① obtain
Dx (g0 2/2 Z0 Fe)
(2 po(2si ((, z()) DKL (90, (2g (2,,, 4o) Po. (2+))+
(5)
DKL (go, (2/l 24) Po. (z)lz=)) + DKL(gp, (Z 20) || po. (c|12=))
This assumes a factooried prior of the form Pa (Ze, z4, x Pa. (z+ (=1) )o.. (2/)lz)), z.). consistent with our conditional independence assumptions, given that information specific to each distribution is encoded in (= =(). The final ELBO can then be expressed as
log Py (Ite, N..) 2 Egg, (salee. da,, (x) =.)) [log P/ (aral Ras =()]
+Ego, (2 =..)) log Py (Nrel Rss =1)]] - DxL (20, (2slar,, Po. (2s)) (6)
-DaL (go, (2/ (2/1 2x) Po. (=]]==)) DKL (go, (2' (r 2x) Po. (=).)=))
In the standard VAE formulation (Kingma & Welling 2014), the priors corresponding to Po. (z) 2s) and Po. (2y, ) are modeled as standard normal distributions. However, Gaussian priors limit the expressiveness of the model in the latent space since they result in strong constraints on the posterior (Tomczak & Welling 2018 Razavi et al. 2019: Ziegler & Rush Rush|20119). Specifically, optimizing with Gaussian prior pushes the posterior distribution towards the mean, limiting diversity and hence generative power (Tomezak & Welling 2018). This is especially true for complex multimodal image and text distributions. Furthermore, alternatives like Gaussian mixture model-based priors (Wang) iet al. (217)) also suffer from similar drawbacks and additionally depend on predefined heuristics like the number of components in the mixture model. Analogously, the VampPrior (Tomczak & Welling, 2018) depends on a predefined number of pseudo-input to learn the prior in the latent space. Similar to Ziegler & Rush| (2019); [Bhattacharyya et al. (2019), which learn priors based on exact inference models, we propose to learn the conditional priors Pa, (=(l=x) and pa. (=(lzs) jointly with the variational posterior in Eq. T using normalizing flows.
3.2 VARIATIONAL INFERENCE WITH NORMALIZING FLOW-BASED PRIORS
Normalizing flows are exact inference models, which can map simple distributions to complex den- sities through a series of K, invertible mappings,
So. - s pk:-1 o.... of f2.
This allows us to transform a simple base density ~ p(c) to a complex multimodal conditional prior Pa. ==]]==) (and correspondingly to Po. Po. (z)).z)), The likelihood of the latent variables under formula. A composition of invertible
the base density can be obtained using the change-of-variabee mappings f parameterized by parameters di. is learned such that fØr (=(l==). The log-
likelihood with Jacobian S2. ol0=// assuming for = I is the identity, can be expressed as
log Po. (=fl==) = log p( ST) (z((za)) - log det dz Je
K.
= log p( ( (2(lex)) log det J
Using data-dependent and non-volume preserving transformations, multimodal priors can be jointly learned in the latent space, allowing for more complex posteriors and better solutions of the evidence lower bound. Using Eq. (7). the ELBO with normalizing flow-based priors can be expressed by rewriting the Ki-diieeee terms in Eq. (6) as (analogously for the other term)
Kr
DEL. (g0, (26 2x) Po. (z]1 = (eeg(t( [logp(ss (x(les))
log det J...] (8)
+ Egn, (2() log go, (2f1er, 2=)]
4

---

Published as a conference paper at ICLR 2020
Next, we describe complete model for learning joint distributions with latent normalizing flows using Eqs. (6) and which enables many-to-man tan-too-myy mappings between domains.
3.3 LATENT NORMALIZING FLOW MODEL FOR MANY-TO-MANY MAPPINGS
We illustrate complete model in Fig. 2
It consists
of two domain-specific encoders to learn the domain- specific latent posterior distributions go, (2/]111; =x) and qe, (z', [rr 2z). As the shared latent variable 2, en- codes information common to both domains, it holds that qo, (2,
K.
Image
Image Decoder
qo, (zg(24) = qo, (z) for match- ing pair of data points (Zt, Therefore, each encoder must be able to model the common supervised informa- tion independently for every matching pair (Ir, K.)). We enforce this by splitting the output dimensions of each encoder into Zy = [z+ = and 2; = [zs zíI, respectively (ef. Eq. D and constraining the supervised latent dimen- sions to encode the same information. We assume that the shared latent code 2g has d' dimensions. We pro- pose to learn the posterior distribution qu, (=. 2i.) as the shared latent space between two domain-specific au- toencoders. One simple method to induce sharing is by minimizing the mean -quarrd error between the encodings. However, this is not ideal given that 2, and 2y follow different highly multimodal
-logp(x.)
fow layers
fe.
-(s)
X
flow
Patelia
Figure 2. Our LNFMM architecture
generative processes. We, therefore, learn an invertible mapping fe. R - Rd with invertible neural networks such that the d' dd-dmenssionl latent code 2, can be transformed between the domains v and I. Let 2, with d 2 d' dimensions the encoded latent variable for distribution pa (z.)). bijective mapping Jo. (Zu) + (z is learned with invertible mapping analogous to Eq.
as
log qo, (zalze, Fz, x.) = log qo, ((zv) xw) = log qo, (fo, ((z)) (xx)) + Σ log det J
(9)
Here, (+)ar denotes restricting the latent code the dimdienssoa shared part. f0. is invertible neural network with affine coupling layers (Dinh al. 2015). i.e. fo, f f pk-1
fo.- This makes it casy to compute Jacobians J /ars, formulated as triangular matrices for the layers of the invertible neural network. Again, assume that f8. / is the identity.
The number of dimensions for domain-specine information, i.e. the dimensionality of 2/ and 2'g can be different for the two domains, depending on the complexity of each distribution. Note that the dimensions with supervision signal, minimize the redundancy in the dimensions for unsupervised information without disentangling the dimensions. The multimodal conditional prior in qo, (z/l 2., 24) and (2, 2s 2.p) 2.,
by conditioning
is modeled with non-volumee-reserving malizing flow models with Eq. (7) (Dinh et al. 2017). parameterized by é; and é., respectively.
The data lig-likelihood terms in Eqs. (4) and (6) are defined the (negative) reconstruction er- rors. Let 9a. Ip + 2, with Og = 10,, 0g) and ge,
+ with 0r = {0,, 02) de- note the image and text encoders, respectively. Further, hw. 2g F> E. and h : 2 H zt are the decoders with parameters We and We corresponding to images and texts, respectively, with J, hw,, (go_ (x.)) and ã, = hws, (go, (ar)) denoting the decoded image and text samples. For texts, we consider the output probability of the jth word (I*), of the ground-truth sentence given the previous reconstructed words (#t)0:g-1 from the text decoder, and define the recon- struction error as (Ie ăz) = - E, log ((xr)) loss between the input image T, and reconstructed image H, from the image decoder taken as Lee T.) || Zpll. where || is either the ε or l2 norm. Furthermore, in Eq. (9) we define log qo. (fe, (a as the cost of mapping (zv)ar to the latent space of texts under the transformation fer;
For images, the reconstruction
the mean- meaa-squared error between the encoded text representations (2x ) and the transformed image representation fo. ((z)))) of the paired data (I,, x.,) (see Ardizzone et al. 2019).
5

---

Published as a conference paper at ICLR 2020
Method
B4
B-3
B-2
B-1
C
R
M
S
CVAE (baseline)"
0.309
0.376
0.527
0.696
0.950
0.538
0.252
0.176
Div-BS Vijayakunarr al. 2018
0.402
0,555
0.698
0. 0.846
1.448
0.666
0.372
0.290
POS DSSlipaide of
0.550
0.672
0.787
0.909
1.661
0,725
0:409
0.311
VAL Wang
2017
0.557
0.654
0.767
0.883
1.517
0.690
0.345
0.277
Soq-CVAE (Aieja et 2019
0,575
0.691
0.803
0,922
1,695
0.733
0,410
0,320
INFMM-MSE (pre-triined)
0.6066
0.686
0.798
0.915
1.682
0.723
0.400
0.306
LNFMM (pre trained)
0.600
0.695
0.804
0.917
1.697
0.729
0.400
0.311
LNFMM
0.597
0.920
1.705
0.729
0.402
0.316
Table 1. Oracle performance for captioning on the COCO dataset with different metrics
Method
B-4
B-3
B-2
B-1
C
R
M
S
Div-BS Vijavkumerr al 2018
0.325
0.430
0.569
0.734
1.034
0.538
0.255
0.187
POS (Deshpunde et a. 2019
0,316
0.425
0.569
0.739
1.045
0.532
0.255
0.188
AG-OVAE Wang ot al
0.311
0.559
0.732
1.001
0.528
0.245
0.179
0.318
0.433
0.582
0.747
1.055
0.538
0.247
0.188
LNFMM-IXT (semi-supervised, 30% labelod)
0.276
0,384
0.529
0.706
0.973
0,511
0.241
0.171
LNFMM- oseii-speervised, 30% labeled)
0.277
0.388
0.531
0.704
0.910
0.509
0.231
LNFMM (semi-superrissd. 30% labeled)
0.300
0.413
0.559
0.729
0.984
0.538
0.242
0.172
Table 2. Consensus re-ranking for captioning the COCO dataset using CIDEr
Starting from the ELBO on the right-hand side of Eq. (6). the learned latent priors in Eq. (8) the invertible mapping from Eq. (9), and plugging the reconstruction terms as just defined, the overall objective of our semi- cupi-supereised generative model framework to be minimized is given by
(u( = - DxI (zglae,
(z+)) + A2 (g0z (=(( , 2,) Po. (=1lzs)) Aa (yos (2 =.) (z/. A4 (x), (x,, + As (I., T.).
(10)
Here, Jl {0, sh. &u., the, Wa, wi) } are all parameters to be learned; A, (l...., 5] are regulariza- tion parameters. Recall that 3g and é. are decoded text and image samples. respectively. We assume the prior Po, (2x) the shared latent space to be uniform. The strength of this uniform prior on the shared dimensions can be controlled with the regularization parameter Aj.
Our model allows for bi-directional many-t mappings. In detail, given data point z from the image domain with latent encoding že, we first map it to the text domain through the invertible transformation 2, fo. ((=w)rr). We can now generate diverse texts by sampling from the learned latent prior Ps, [=]]=a). A similar procedure is followed for sampling images given text through the learned prior Pa. (=(1z.). For conditional generation tasks, as do have sample from the supervised latent space, we find "uniform" prior Ps, (z2) to be advantageous in practice as loosens the constraints on the decoders. Alternatively, a more complex flow-based prior could also be used here to enable sampling of the shared semantic space. We show the effectiveness of our joint latent normalizing flow-based priors on real-world tasks, i.e. diverse image captioning and text-to-in text-to-image synthesis.
4 EXPERIMENTS
To validate our method for learning many-to-many mappings to provide latent joint distributions, one of the important real-world tasks is that of image-to-text text-to-image synthesis. To that end, we perform experiments on the COCO dataset (Lin et al.] 2014). It contains 82,783 training and 40,504 validation images, each with five captions. Following Wang al. (2016); Mao et al. (2015) for image captioning, we use 1 118,287 data points for training and evaluate on 1,000 test images. For text-to-image synthesis, the training set contains 82,783 images and 40,504 validation data points test time (Reed et al. 2016 Huang et al. 2017). Architecture details can be found in the Appendix.
4.1 IMAGE CAPTIONING
We evaluate our approach against methods that generate diverse captions for given image. We compare against methods based on (conditional) variational autoencoders, AG-CVAE (Wang et al.)
"Our implementation

---

Published as a conference paper at ICLR 2020
Method
Unique f
Novel f
mBLEU
Div-I
Div-2
Mothod
B-1
B-4
CIDEr
Div-RS
3421
MAD-GAN
0.652
0.238
POS
91.5
3446
0.67
0.23
0.33
GXN
0.571
0.149
0.611
AG-CVAE
47.4
3069
0.70
0.23
0.32
Sey CVAE
4215
LNFMM
0.747
0.315
1.055
LNFMM
97.0
4741
0.60
0.37
0.51
Table Comparison to the state of
art for bi-directional generation
Table 3. Diversity evaluation on at most the best-5 sentences after consensus re-ranking
Image
Caption
Image
Caption
Two elephunts standing next to each other river.
living room filled with fumi- and large window.
A herd of elephants in grassy of water.
room with couch and a large weoden table.
Two elephants walking through a river while standing in the water.
A living room filled area with furniture and couch and chair. room with couch, chair, and lamp.
Two elephants walking and baby water.
group of elephants walking
This is with furniture the floor.
around a watering hole.
Table 5. Example captions generated by our model
2016), which uses additional supervision based on information about objects in the images for a Gaussian mixture model in the latent space, and Seq-CVAE (Aneja et al. 2019). which models sequential latent spaces with Gaussian priors. We also include Div-BS Vijayakumar et al., 2018) based on beam search and POS (Aneja et al., 2018), which uses additional supervision from images. Additionally, we include different ablations to show the effectiveness of various components of our approach. LNFMM-M minimize the mean squared error between (zt)a' and (z.)ar in Eq. (10). We fix the image encodings of a VGG- 16 encoder in LNFMM (pre-trained) for comparison to image captioning methods with pre-trained image features. Furthermore, LNFMM-TXX contains the latent flow for unsupervised information fo. only for the text distribution. In this setting, we have &n = 2, and thus 2, is transformed through the invertible neural network fe, (2.). To show that our framework can be applied in the semi-supervised setting, we also consider limited labeled data with only 30% of the paired training data for supervision. The remaining training data is included unpaired data for modeling domain-specific information.
does not contain the invertible neural network fo,, i.e. we directly
Evaluation. We evaluate the accuracy with Bleu (B) 1-4 (Papineni et al., 2002), CIDEr (C; Vedan-
tam al. 2015), ROUGE (R: Lin 2004), METEOR (M: Denkowski & Lavie 2014), and SPICE (S:[Anderson et al. 2016). For evaluating diversity, we consider the metrics of Wang et al. (017);
Aneja et al. (2019). Uniqueness is the percentage unique captions generated on the test set. Novel sentences are the captions that were never observed in the training data. m- B--Bleu-4 computes Blcu-4 for each diverse caption with respect to remaining diverse captions per image. The Bleu-4 obtained is averaged across all images. Div-n is the ratio of distinct ntgrams to the total number of words generated per set of diverse captions.
Results. In Table D we show the caption evaluation metrics in the oracle setting, i.e. taking the maximum score for each accuracy metric over all the candidate captions. We consider 100 samples 2, consistent with previous methods. The eVAE baseline with an image-conditioned Gaussian prior does not perform well on all metrics, showing the inability of the Gaussian prior to model meaningful latent spaces representative of the multimodal nature the underlying data distribution. The overall trend across metrics is that LNFMM model improves the upper bound Bleu and CIDEr while being comparable on the Rouge and Spice metrics.
Comparing the accuracy of the baseline LNFMM-MSE with LNFMM, we can conclude that learn- ing the shared posterior distribution of 2, with our invertible mapping is better than directly mini- mizing the mean squared error in the latent space due to differences in the complexity of the distri- butions. Also note that LNFMM (pre-trained) with fixed image-encoded representations has better performance compared to AG-CVAE and Seq-CVAE, in particular. This highlights that the LNFMM
7

---

Published as a conference paper at ICLR 2020
Text
Sumple
Sample
Sample
Sample
Ground-truth
AttiGAN
Our LNFMM
close up a pizza with loppings
baseball
player
swinging
bat during
game
Figure 3. Example images generated by our LNFMM model high- lighting diversity
Figure Text-conditioned samples clos- est to test image with IoVM
learns representations in the latent space that are representative of the underlying data distribution even for pre-trained features.
realistic setting (as ground-truth captions are not always available) where, instead of comparing against the reference captions of the test set, reference captions for images from the training set most similar the test image are retrieved. The generated captions are then ranked with the CIDEr score (Mao et al.) (015). While Div-BS has very good accuracy across metrics due the wide search space, our LNFMM model gives state-of-the-art accuracy on various Bleu metrics and especially the CIDEr score, which is known to correlate well with human evaluations. More in- terestingly, compared to AG-CVAE with conditional Gaussian mixture priors based on object (class) information, our LNFMM model, which does not encode any additional supervised information in the latent space, outperforms the former on all accuracy metrics by a large margin. Moreover, the re- cent GXN (Gu et al. 2018) and M DGDGAN (Ma et al. 2019) also study bi-directional synthesis with joint models in Gaussian latent spaces. Ma et al. (2019) additionally model attention in the latent space. From Table 4 we that our method considerably outperforms the competing methods, val- idating the importance of complex priors in the latent space for image-text distributions. This again highlights that the complex joint distribution of images and texts captured by our LNFMM model is more representative of the ground-truth data distribution. We additionally experiment with limited labelled training data (30% labeled). which shows that our approach copes well with limited paired data. We finally compare LMFMM against LNFMM-TH TXXT to show the importance of joint learning of image and text generative models. With a generative model only for texts, the joint distribution cannot be captured effectively in the latent space.
Table 2 considers
With diversity being an important goal here, we show in Table 3 that our LNFMM method improves diversity across all metrics, with a 6.5% improvement in unique captions generated in the test set and 4741/5000 captions not previously seen in the training set. Our generated captions for given image also show more diversity with low mutual overlap (mBLEU) compared to the state of the art. We, moreover, observe high n n-gram diversity for the generated captions of each image. Div-BS with high accuracy has limited diversity it can repeat the n-grams in different captions. POS and AG-CVAE, due to guided supervision in the latent space, offer diversity but model only syntactic or semantic diversity, respectively (Wang & Chan 2019). Our proposed LNFMM model in Table 5 shows range of diverse captions with different semantics and syntactic structure. Therefore, we conclude that the proposed LNFMM can effectively learn semantically meaningful joint latent representations without any additional object or text-guided supervision. The data-dependent learnt priors are thus promising for synthesizing captions with high human-correlated accuracy well as diversity.
4.2 TEXT-TO-IMAGE SYNTHESIS
Given a text description, we are now interested in generating diverse images representative of the domain-specific structure images. To that end, we include a discriminator to improve the image quality of our image decoder. Note that this does not affect the joint latent space of the LNFMM model. We evaluate our method against state-of-the-an approaches such as AttiGAN [Xu et al, 2018), HD-GAN (Zhang al. 2018b), StackGAN (Zhang al. 2017), and GAN-INT-CLS (Reed) ct al. 2016). While our main goal is to encourage text-conditioned diversity in the generated sam- ples, the current state-of-the-art for text-to-image generation aims at improving the realism of the
8

---

Published as a conference paper at ICLR 2020
generated images. Note that various GANs can be integrated with the image decoder of our frame- work as desired.
Evaluation. As we are interested in modeling diversity, we study the diversity in generated images using the Inference via Optimization (IvOM; Srivastava et al. (017)) and LPIPS (Zhng et al. 201 Saa metrics against the state-of-the-an art AttnGAN. Given the text, for each matching image, IvOM finds the closest image the model capable of generating. Thus, shows whether the model can match the diversity of the ground-truth distribution. L.IPS (Zhang al., 2018a) evaluates diversity by computing pairwise perceptual similarities using a deep neural network. Additionally, we also report the Inception score (Salimans al. 2016
Results. In Table 6 our method improves over At- tnGAN for both IvOM and LPIPS scores, showing that our method can effectively model the image semantics conditioned on the texts in the latent space, as well as generate diverse images for a given caption. Note that AttnGAN uses extra supervision to improve the incep- tion score. However, it is unclear if this improves the visual quality of the generated images as pointed out by Zhang al. (2018b). We improve the IS over HD-GAN, which does not use additional supervision. Qualitative examples in Fig. show that our LNFMM model generates diverse images, e.g., close- up images of food items well different orientations of the baseball player in the field. In Fig. 4 we additionally see that given a caption, images generated by our LNFMM model capture detailed semantics of the test images compared to that of AttnGAN, showing the representative power of our latent space.
Method
IS 7
IvOM 4
LPIPS
AttiGAN
25.89.10.47
1.101"
0.472"
GAN-INT-CLS
7,88+: 0.07
Stack-GAN
8.45-=0.03
HD-GAN
11.86=0.18
LNFMM
::0.18
0.430
Table 6. Evaluation on text-to-image synthesis
5 CONCLUSION
We present a novel and effective semi-supervvsed LNFMM framework for diverse bi-directional many-to-many mappings with learnt priors the latent space, which enables modeling joint image- text distributions. Particularly, we model domain-specinf information conditioned on the shared information between the two domains with normalizing flows, thus preserving the heterogeneous structure of the data in the latent space. Our extensive experiments with bi-directional synthesis show that our latent space can effectively model data-dependent priors, which enable highly accurate and diverse generated samples of images texts.
This work has been supported by the German Research Foundation as part of the Re- scarch Training Group Adaptive Preparation Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1.
Akknowledgement.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic proposi- tional image caption evaluation. In ECCV, 2016.
Jyoti Aneja, Aditya Deshpande, and Alexander G. Schwing. Convolutional image captioning. In CVPR, 2018.
Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander G. Schwing, Sequential latent spaces for modeling the intention during diverse image captioning. ICCV, 2019.
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In ICLR, 2019.
Jens Behrmann, Will Grathwohl, Ricky Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In ICML, 2019.
Apratim Bhattacharyya, Michael Hanselmann, Mario Fritz, Bernt Schiele, and Christoph-Nikolas Strachle. Conditional flow variational autoencoders for structured sequence prediction. In BDL@ NewrlPS, 2019.

---

Published as a conference paper at ICLR 2020
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR, 2017.
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descrip- tions via a conditional GAN. In ICCV, 2017.
Ayushman Dash, John Cristian Borges Gamboa, Sheraz Ahmed, Marcus Liwicki, and Muham- mad Zeshan Afzal. TAC-GAN Text conditioned auxiliary classifier generative adversarial net- work. In AAAI, 2017.
Michael J. Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In WMT@ ACL, 2014.
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G Schwing, and David Forsyth. Fast, diverse and accurate image captioning guided by part-of-speech. In CVPR, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. In ICLR, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. Look, imagine and match: Im- proving textual-visual cross-modal retrieval with generative models. In CVPR, 2018.
Matthew D. Hoffman and Matthew J. Johnson. ELBO surgery: yet another way to up the variational evidence lower bound. In NIPS Workshop on Advances in Approximate Bayesian Inference, 2016.
Xun Huang, Yixuan Li, Omid Poursaeed, John E. Hopcroft, and Serge J. Belongie. Stacked genera- tive adversarial networks. In CVPR, 2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. TPAMI, 39(4):664 676, 2017.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In NeurlPS, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. ICLR, 2014.
Dianqi Li, Qiuyuan Huang, Xiaodong He, Lei Zhang, and Ming-Ting Sun. Generating diverse and accurate visual captions by comparative adversarial learning. vrxiiv/884.0000 008.0 [cs.CV), 2018.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. ACL, 2004.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitncc.. Microsoft COCO: common objects context. In ECCV, 2014.
Shuang Ma, Daniel J. McDuff, and Yale Song. M3D-GAN; multi-modal multi-domain translation with universal attention. In CVPR, 2019.
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. Deep captioning with multimodal recurrent neural networks (m-RNN). In ICLR, 2015.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In ACI, 2002.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. MinrorGAN; Learning ext-to--mage generation by redescription. CVPR, 2019.
10

---

Published as a conference paper at ICLR 2020
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In ICLR, 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML,, 2016.
Tim Salimans, lan J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, 2016.
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In ICCV, 2017.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles A. Sutton. VEE- GAN: Reducing mode collapse in GANs using implicit variational learning, In NIPS, 2017.
Jakub M. Tomezak and Max Welling. VAE with a VampPrior. In AISTATS, 2018.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation leamn ing. In NewrIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 955v) 2605, 2008.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In CVPR, 2015.
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In AAAI, 2018.
Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive Gaussian encoding space. In NIPS, 2017.
Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. TPAMI, 41(2):394 407, 2019.
Qingzhong Wang and Antoni B. Chan. Describing like humans: on diversity in image captioning. In CVPR, 2019.
Zhuhao Wang, Fei Wu, Weiming Lu, Jun Xiao, Xi Li, Zitong Zhang, and Yueting Zhuang. Diverse image captioning via grouptalk. In IJCAI, 2016.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-graing grained text to image generation with attentional generative adversarial net- works. In CVPR, 2018
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disentan- gling for text-to-image generation. CVPR, 2019.
Han Zhang, Tao Xu, and Hongsheng Li. StacKGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML, 2019a.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N. Metaxas. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. TPAMI, 41(8):1947-1962, 2019b.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018a.
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with hierarchically-neste adversarial network. In CVPR, 2018b.
Zachary M. Ziegler and Alexander M. Rush. Latent normalizing flows for discrete sequences. In ICML., 2019.
11

---

Published as a conference paper at ICLR 2020
X
X
Figure 5. Inference model of our approach (right) in comparison to the conditional variational autoencoder of AG-CVAE (left: Wang et al. 2017). AG-CVAE models only supervised information in the latent dimensions. Our model encodes domain-specific variations in the conditional priors z and E/_
A APPENDIX
A.1 NETWORK ARCHITECTURE
We provide the details of the network architecture of Fig.
Image pipeline. The network consists of an image encoder built upon VGG-16. 4096-dimensional activations of input images are extracted from the culnecco layer of VGG-16. This is followed by a -ineesninnii tully-comnected layer with ReLU activations. We then project it to the latent space with and dim 2 = 64 dimensions. In the image decoder, we leverage the architecture of Zhang et al. (019a) to synthesize images of 64 64 256 256 dimensions. The input to the decoder has dimensionality 1056. For the image generation experiments, we additionally apply the discriminator of Zhang et al. 2019b) the output of the image decoder.
955--dimensional fully-connected layer. The image pipeline has d' dim 2g 992
Text pipeline. We use a bidirectional GRU with two layers and a hidden size of 1024 as text encoder. This outputs 1024-dimensional latent representations for sentences. For text, dim 2/ 32. The text decoder is a LSTM with one layer and hidden size of 512.
Flow modules. Our network consists of two flow modules for conditional priors on image and text domains and an invertible neural network to exchange supervised information.
Invertible neural network for supervision (fe.): J: This invertible neural network consists of 12 flow layers and input dimensionality of 992. Each flow consists of conditional affine coupling layers followed by a switch layer (Dinh et al. 2015).
Latent flow for conditional prior images (f We map the 64 dimensions of the image encodings from the image encoder to a Gaussian with normalizing flows with 16 layers of flow and 512 hidden channels. Each flow consists of conditional affine coupling layers followed by switch layer (Dinh) et al. 2017)-
Latent flow for conditional prior texts (fo.): We map the 32 dimensions of the text encodings from the text encoder to a Gaussian with normalizing flows with 16 layers of flow and 1024 hidden channels. Each flow consists of a conditional activation normalization layer followed by conditional affine coupling layers. Invertible 1 x convolutions are applied to the output of the affine coupling layers, which followed by switch layer (Kingma & Dhariwal 2018).
A.2 DIVERSITY IN IMAGE CAPTIONING
We show more qualitative examples of the captions generated by our LNFMM model Table / The example captions show syntactic well semantic diversity.
12

---

Published as a conference paper at ICLR 2020
Image
Caption
Image
Caption
A woman bolding
brella while slanding in the rain.
A woman standing in fromt of refrigerator
A woman is holding um- breila the street
Two people standing to- gether in a large kitchen
A woman walking a street umbrella in
Two people standing
kitchen counter.
A woman is holding an um- brella while walking in the rain
A family preparing food kitchen.
A few people standing in the kitchen table.
A woman walking down a
street while holding an brella
man holding
large clock tower in the middle of a building.
Atrinc
racket the tennis court.
A tennis player about to hit
A tall building with a clock in front of building.
tennis ball
A män is playing tennis with
A clock tower in the sky with
the tennis A man standing on tennis holding racket
clock lop.
A tall building with clock top.
A prepares hit ball with tennis racket.
A tall clock lower with
clock tower on it.
Table Example captions generated by our LNFMM model
Toxt
Sample #1
Sample #2
Sample #3
Sample #4
Ground-tuth
LNFMM (Ours)
A person surting in ocean.
skier skiing down a snow stope.
A clophant is shown walking the grass.
Figure Example images generated by our LNFMM model high- lighting diversity
Figure Text-conditioned samples closest to test image with IvOM by our LNFMM
A.3 TEXT-TO-IMAGE SYNTHESIS
We additionally show diverse images generated by LNFMM model in Fig. Our generated images can successfully capture the text semanties and also exhibit image specific diversity, e.g. in style and orientation of objects. Furthermore, show that the latent space captures the joint distri- bution, we show the images generated by our model with IvOM by finding a 2, conditioned on the input text that is most likely to have generated the test image. We show the images generated for the 2o most likely to have generated the image. Our generated samples in Fig. capture the details in images, showing that our LNFMM model learns powerful latent representations.
A.4 VISUALIZZATION OF LATENT SPACES
In Table we show the t-SNE (yan der Maaten & Hinton 2008) visualization of the latent space of texts conditioned on image. In the examples shown, we observe global patterns like 'table with foods' across all captions of an image and local patterns in clusters like 'table with several' or dinner table with' showing image-conditioned representation well as domain-specific variation.
13

---

Published as a conference paper at ICLR 2020
Image
Latent Space
Captions
table set with various different foods.
table with several plates of food items it.
table with several plates and various
food
buflet table with plates full of laod
items,
long table topped with food and uten- sils.
wooden table topped with plates filled
with food.
dinner
many dishes and
food.
table with plates food.
dinner table with many dishes and meal.
baschall player hirting basehall baseball ficld.
baseball player sliding plate.
player swinging standing
field.
husehall player swinging a hat next
plate.
batter swinging at a ball during a game.
swinging baveball
game.
bascball player standing to hit ball.
baseball player hitting ball during game,
bascball player swinging
ball.
basehall players
umpire waiting
for a pitch.
bascball player cateher bascball
game.
a baseball player holding bat in the bat- box.
Table 8. t-SNE visualization of the latent space of texts conditioned image. Example captions clustered using k-mmans. Captions are color-coded based on corresponding clusters
Similarly, in Table@wee we can observe clusters in image space conditioned on texts. Here again, can observe global patterns like 'close-up food' and local patterns like 'pizza', 'food with toppings, or 'sandwiches'.
A.5 VALIDATION WITH HUMAN EVALUATION
In addition to validating the performance of our LNFMM method on image captioning for accuracy in Table [ and for diversity Table 3 with various measures, conduct human evaluation of the quality of captions generated by our LNFMM approach against Div-BS /Vijayakumar et al. 2018) for accuracy and diversity. We presented five captions each for set of images and for each method to four human annotators. The human annotators assessed the captions for accuracy and diversity on a scale from I to 10. Here, accuracy is defined as how well the captions describe the details in given image and diversity can be syntactic diversity or semantic diversity.
In Fig. (left)) the score for each image the average score given by the four annotators and Fig. (right) shows the scores for the captions of each image from each annotator. Here, our findings are similar to those in Table and Table 3 The captions generated by Div-BS were assessed to
have diversity scores in the range of [5, 8). while the diversity scores fell into the range [7, 10] for our LNFMM method. Furthermore, the accuracy of the captions of each image generated with our LNFMM was found to be better compared to Div-BS, thereby showing that LNFMM can generate captions that are both diverse and accurate.
14

---

Published as a conference paper at ICLR 2020
Latent Space
Images
A pizza sitting on pan table.
A man playing with flying disc the grass.
Table t-SNE visualization of the latent space of images conditioned caption. Example images clustered using k-means. Images are color-coded based on corresponding clusters
10
10
8
8
Diversity
6
Diversity
6
4
4
2
2
Div.Ei (Vijayakumar al.)
Div.185 (Vijayakumar et. al.)
VFMM (Ours)
NFMM (Durs)
0
0
2
4
6
8
10
0
2
4
6
10
Accuracy
Accuracy
Figure 8. Comparison of accuracy and diversity scores of captions for images generated with LNFMM (Ours)
DIV-BS /Vijayakumar al.] (018): Scores for captions of image averaged across annotators (left) and scores for all captions of an image for each annotator (right). To make coinciding scores be casier to see, small random jitter is added to each human assessment.
15


Reviews:
[{"review_id": "rJed5iPm5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a variational model for text to image and image to text mappings. The novelty consists in separating the modeling of text and image latent representations on one hand and the modeling of a shared content representation on the other hand. Priors for text, image and shared representations are generated through an invertible – flow model. The motivation for this is to allow for complex priors. Training for the shared component is supervised using aligned text and image data, while training for the residual text and image components is unsupervised. Experiments are performed for text and image generation, using training data from the COCO dataset.\nThe proposed model presents several innovations: separate unsupervised modeling of text and image and joint supervised modeling of shared latent variables, the use of three normalizing flows for the priors respectively associated to these variables. The intuition behind the model is well introduced. However, the technical description of the model itself is somewhat imprecise. Particularly section 3.3 describing the shared component and the global model should be carefully checked. Both descriptions are too imprecise e.g. the d’ dimensional component of z_v, z_t are not introduced; the derivation or explanation of eq. (10)  is not provided, J_phi in eq (9) not defined, etc. There are some typos or erros, check eq (7), (8), q_phi I instead of q_theta in §3.3.\nThe experiments compare the model with several different baselines and are quite extensive. Please indicate whether you performed all the tests yourself or picked the numbers in the literature. A better description of the baselines characteristics and of the model variants (MSE, TXT) and  their relations with the proposed model, in this paragraph, would help appreciate the results.\nThe proposed model seems to compare well with different baselines, but the presentation of the experiments is not that clear. For example, the ablation study in Table 1 basically shows that the Phi_s flow component behaves similarly to the complete flow model. This MSE variant is not used anymore in the other comparisons, why? Same remark for the other baselines, why some are used and some not in the different tests? The same remark hold for the text to image experiments.\n Overall, there are interesting new ideas, a new model, insufficient model description and experiments details. \n\n\n----- After rebuttal -----------\n\nThe authors made an effort to clarify and correct the technical errors. I still have some concerns with confusing notations.  But I keep my score.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkesJ76W5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a model for joint image-text representations \n\nThe paper proposes a model fro cross-domain generative tasks, specifically image captioning and text-to-image synthesis. The proposed Latent Normalizing Flows for Many-to-Many Mappings uses normalizing flows to model complex joint distributions. The latent representation consist of domain-specific representation and cross-domain information shared across image and text using invertible metrics. \n\nNovelty: \n\n- The paper explores an interesting area of learning representations for cross-domain tasks such as image captioning and text-to-image synthesis.\n- The model is well explained.  Section 3 explains the model formulation nicely, has consistent notation and slowly  builds up to the final formulation by explaining each component concisely.\n- Recent methods like VQ-VAE [1]  have shown promising results for image generation. The related work doesn't provide any discussion regarding that. \n\nExperiments / Analysis:\n\n- The model contains exhaustive experiments for both image-captioning and image generation. The model performs on-par or beat state of the art methods on both perceptual and diversity metrics. On diversity metrics, the model performs much better than other recent methods like Seq-CVAE (which arrived on ArXiv only a few weeks prior to the submission deadline) and POS.\n- The model also shows results on text-to-image synthesis comparing with multiple baselines and various diversity metrics and inception score.\n- While the proposed methods beats existing diversity and perceptual metrics, it'd be good to also run a human study since these metrics are only a proxy to human judgement.\n- Apart from showing empirical result, the paper can benefit from providing Insights what the domain specific representation has learnt and what the cross-domain representation learnt.\n- Can the model benefit from training on unaligned image and textual data to learn better domain specific representations? \n\nClarity: \n- Ablations not clear: It's not a 100% clear from the paper what LNFMM-MSE and LNFMM-TXT mean. \"LNFMM-TXT contains unsupervised dimensions only for the text distribution and\nall encoded image features are used for supervision, i.e.without fφv\" What does this sentence mean?  Similarly, it's not clear what \"LNFMM (semi-supervised, 30% labeled)\" mean?\n- It's also not clear why the authors call the approach a semi supervised setup? For instance,  the paper relies on supervision from  paired image-caption data to train the model. \n\n[1] Neural Discrete Representation Learning; Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu\n\n\n**Update after rebuttal**\nThank you for clarifications to my questions regarding the ablations and  using unaligned image and textual data to learn better domain specific representations. The visualizations in Appendix A.4 are also somewhat helpful in understanding what the representations have learnt. After reading the rebuttal, and considering that they will run the human study to further validate their approach in the final manuscript, I am happy to raise the score. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxBYSQ0KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper addresses the problem of many-to-many cross domain mapping tasks (such as captioning or text-to-image synthesis). It proposes a double variational auto-encoder architecture mapping data to a factored latent representation with both shared and domain-specific components. The proposed model makes use of normalizing flow-based priors to enrich the latent representation and of an invertible network for ensuring the consistency of the shared component across the two autoencoders. Experiments are thorough and demonstrate results that are competitive or better than the state-of-the-art.\n\nDecision:\nThis work is a good example of meticulous and well-executed neural network engineering. It combines well-known ideas (variational auto-encoders, normalizing flow priors, invertible networks) into an effective and working solution for a complicated problem. The model is shown to bring improvements in the state-of-the-art over several metrics and benchmarks. The manuscript is well written and easy to follow (provided some technical familiarity with variational inference). It includes all the necessary details for understanding the method. Related works appear to have been discussed and compared properly, although I cannot assess if important works on cross-domain mapping are missing. For these reasons, I recommend this work for acceptance without reservation.\n\nAdditional feedback:\n- Above Eqn 5: K- divergence --> KL divergence\n- The code could have been cleaned up and better organized, for easier reproducibility and reuse. \n\n---\n\nPost-rebuttal update: Thank you for your answers. I still recommend your work for acceptance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of many-to-many cross-domain mapping tasks with a double variational auto-encoder architecture, making use of the normalizing flow-based priors.

Reviewers and AC unanimously agree that it is a well written paper with a solid approach to a complicated real problem supported by good experimental results. There are still some concerns with confusing notations, and with human study to further validate their approach, which should be addressed in a future version.

I recommend acceptance.
Decision: Accept (Poster)