Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: This paper uses visual representation learned over monolingual corpora with image annotations, which overcomes the lack of large-scale bilingual sentence-image pairs for multimodal NMT. Their approach enables visual information to be integrated into large-scale text-only NMT. Experiments on four widely used translation datasets show that the proposed approach achieves significant improvements over strong baselines.

Strengths:
- This paper is well motivated and well written. I especially like how they use external paired sentence-image data from Multi30k to learn weak pairs for sentences in machine translation.
- Experimental results are convincing. I like how low-resource translation is included as a priority in their experiments.

Weaknesses:
- Do you have any explanations as to why the number of images, if too large, actually hurts translation performance? Is it because more images also leads to a higher chance of noisy images?
- It would be nice to have an experiment that varies the size of the external paired sentence-image dataset and tested the impact on performance.
- Please comment on the extra computation required for obtaining image data for MT sentences and for learning image representations.
- Why are there missing BLEU scores and the number of parameters in Table 1?

### Post rebuttal ###
Thank you for your detailed answers to my questions.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper provides an approach to use visual information to improve text only neural machine translation systems. The approach creates a "topic word to images" map using an existing image aligned translation corpora. Given a source sentence, the model extracts relevant images, extracts their Resnet features and fuses them with the features generated from the word sequence. The decoder uses these fused representation to generate the target sentence. Overall, I like the approach, seems like it can be easily augmented to existing NMT systems. 

One of the claims of the paper was to be able to use monolingual image aligned data. However image captioning datasets are not mentioned. It would make sense to use image captioning data to create the image lookup. Also, what will be the performance of a standard image captioning system on the task ? I believe it will not be great, but I think for completeness, you should add such a baseline.

Minor comments: 
1. What is M in Algorithm 1 ? 
2. First paragraph in related work is very unrelated to the current subject, please remove.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors propose to augment NMT with a grounded inventory of images.  The intuition is clear and the premise is very tempting.  The key architectural choice is to allow the transformer to use language embeddings to attend into a topic-image lookup table.  The proportion is learned to balance how much signal comes from each source.    Figure 4, attempts to investigate the importance of this sharing and its effects on performance.

While reviewing this paper I went back and read the EN-DE evaluation data for the last few years trying to see how often I could reason that images would help and I came up severely lacking.  For example, "The old system of private arbitration courts is off the table" from DE-EN 2016 Dev doesn't seem like it should benefit from this architecture.  It's then hard for me to square that with the +VR gains seen throughout this work on non-grounded datasets.  I trust that the authors did in fact achieve these results but I cannot figure out how or why.  This is all further confused by the semantic topics used for clustering the images which ignores stop words and therefore spatial relations or any grammatical nuances.  

In contrast, it does make sense that Multi30K would benefit from this architecture.  As a minor note, were different feature extractors compared? The recent flurry of papers on multimodal transformers indicate that deeper resnet stacks correspond to improved downstream performance.  Is that also true in this domain?
review_title: Official Blind Review #1

Meta-review: This paper proposes using visual representations learned in a monolingual setting with image annotations into machine translation. Their approach obviates the need to have bilingual sentences aligned with image annotations, a very restricted resource. An attention layer allows the transformer to incorporate a topic-image lookup table. Their approach achieves significant improvements over strong baselines. The reviewers and the authors engaged in substantive discussions. This is a strong paper which should be included in ICLR. 


Decision: Accept (Spotlight)