Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper introduces a variational model for text to image and image to text mappings. The novelty consists in separating the modeling of text and image latent representations on one hand and the modeling of a shared content representation on the other hand. Priors for text, image and shared representations are generated through an invertible – flow model. The motivation for this is to allow for complex priors. Training for the shared component is supervised using aligned text and image data, while training for the residual text and image components is unsupervised. Experiments are performed for text and image generation, using training data from the COCO dataset.
The proposed model presents several innovations: separate unsupervised modeling of text and image and joint supervised modeling of shared latent variables, the use of three normalizing flows for the priors respectively associated to these variables. The intuition behind the model is well introduced. However, the technical description of the model itself is somewhat imprecise. Particularly section 3.3 describing the shared component and the global model should be carefully checked. Both descriptions are too imprecise e.g. the d’ dimensional component of z_v, z_t are not introduced; the derivation or explanation of eq. (10)  is not provided, J_phi in eq (9) not defined, etc. There are some typos or erros, check eq (7), (8), q_phi I instead of q_theta in §3.3.
The experiments compare the model with several different baselines and are quite extensive. Please indicate whether you performed all the tests yourself or picked the numbers in the literature. A better description of the baselines characteristics and of the model variants (MSE, TXT) and  their relations with the proposed model, in this paragraph, would help appreciate the results.
The proposed model seems to compare well with different baselines, but the presentation of the experiments is not that clear. For example, the ablation study in Table 1 basically shows that the Phi_s flow component behaves similarly to the complete flow model. This MSE variant is not used anymore in the other comparisons, why? Same remark for the other baselines, why some are used and some not in the different tests? The same remark hold for the text to image experiments.
 Overall, there are interesting new ideas, a new model, insufficient model description and experiments details. 


----- After rebuttal -----------

The authors made an effort to clarify and correct the technical errors. I still have some concerns with confusing notations.  But I keep my score.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary:

The paper proposes a model for joint image-text representations 

The paper proposes a model fro cross-domain generative tasks, specifically image captioning and text-to-image synthesis. The proposed Latent Normalizing Flows for Many-to-Many Mappings uses normalizing flows to model complex joint distributions. The latent representation consist of domain-specific representation and cross-domain information shared across image and text using invertible metrics. 

Novelty: 

- The paper explores an interesting area of learning representations for cross-domain tasks such as image captioning and text-to-image synthesis.
- The model is well explained.  Section 3 explains the model formulation nicely, has consistent notation and slowly  builds up to the final formulation by explaining each component concisely.
- Recent methods like VQ-VAE [1]  have shown promising results for image generation. The related work doesn't provide any discussion regarding that. 

Experiments / Analysis:

- The model contains exhaustive experiments for both image-captioning and image generation. The model performs on-par or beat state of the art methods on both perceptual and diversity metrics. On diversity metrics, the model performs much better than other recent methods like Seq-CVAE (which arrived on ArXiv only a few weeks prior to the submission deadline) and POS.
- The model also shows results on text-to-image synthesis comparing with multiple baselines and various diversity metrics and inception score.
- While the proposed methods beats existing diversity and perceptual metrics, it'd be good to also run a human study since these metrics are only a proxy to human judgement.
- Apart from showing empirical result, the paper can benefit from providing Insights what the domain specific representation has learnt and what the cross-domain representation learnt.
- Can the model benefit from training on unaligned image and textual data to learn better domain specific representations? 

Clarity: 
- Ablations not clear: It's not a 100% clear from the paper what LNFMM-MSE and LNFMM-TXT mean. "LNFMM-TXT contains unsupervised dimensions only for the text distribution and
all encoded image features are used for supervision, i.e.without fφv" What does this sentence mean?  Similarly, it's not clear what "LNFMM (semi-supervised, 30% labeled)" mean?
- It's also not clear why the authors call the approach a semi supervised setup? For instance,  the paper relies on supervision from  paired image-caption data to train the model. 

[1] Neural Discrete Representation Learning; Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu


**Update after rebuttal**
Thank you for clarifications to my questions regarding the ablations and  using unaligned image and textual data to learn better domain specific representations. The visualizations in Appendix A.4 are also somewhat helpful in understanding what the representations have learnt. After reading the rebuttal, and considering that they will run the human study to further validate their approach in the final manuscript, I am happy to raise the score. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Summary:
This paper addresses the problem of many-to-many cross domain mapping tasks (such as captioning or text-to-image synthesis). It proposes a double variational auto-encoder architecture mapping data to a factored latent representation with both shared and domain-specific components. The proposed model makes use of normalizing flow-based priors to enrich the latent representation and of an invertible network for ensuring the consistency of the shared component across the two autoencoders. Experiments are thorough and demonstrate results that are competitive or better than the state-of-the-art.

Decision:
This work is a good example of meticulous and well-executed neural network engineering. It combines well-known ideas (variational auto-encoders, normalizing flow priors, invertible networks) into an effective and working solution for a complicated problem. The model is shown to bring improvements in the state-of-the-art over several metrics and benchmarks. The manuscript is well written and easy to follow (provided some technical familiarity with variational inference). It includes all the necessary details for understanding the method. Related works appear to have been discussed and compared properly, although I cannot assess if important works on cross-domain mapping are missing. For these reasons, I recommend this work for acceptance without reservation.

Additional feedback:
- Above Eqn 5: K- divergence --> KL divergence
- The code could have been cleaned up and better organized, for easier reproducibility and reuse. 

---

Post-rebuttal update: Thank you for your answers. I still recommend your work for acceptance.
review_title: Official Blind Review #2

Meta-review: This paper addresses the problem of many-to-many cross-domain mapping tasks with a double variational auto-encoder architecture, making use of the normalizing flow-based priors.

Reviewers and AC unanimously agree that it is a well written paper with a solid approach to a complicated real problem supported by good experimental results. There are still some concerns with confusing notations, and with human study to further validate their approach, which should be addressed in a future version.

I recommend acceptance.
Decision: Accept (Poster)