Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: For the task of translating a sentence describing an image from one language to another language with the image as additional input for the translation task, the paper uses multimodal attention. For the multimodal attention, the paper explores to use Multimodal Compact Bilinear Pooling (MCB) [Fukui 2016].

Strength:
-	Using MCB for this task seems to makes sense has not previously explored to my knowledge and slightly improves performances.
-	The paper evaluates the task and ablations on the Multi30k dataset.

Main Weaknesses:
Discussion and comparison to related work: 
1.	There has been a large number of works looking at the multimodal translation problem, e.g. [Elliott 2015], [Iacer 2016], but the paper reads like, it is the first work looking at this problem. Specifically, the model from [Iacer 2016] is very similar to this work, apart from MCB.
2.	Please cite prior work more precisely: The work misses the citation for tensor sketch algorithm from [Pham and Pagh 2013]; specifically also in Figure 1, where the visualization and algorithm seems to be based on Fukui  2016.
3.	 [Iacer 2016] also reports the results of using Moses, a statistical machine translation pipeline, which does not use the image an achieves 52. Meteor, higher than any result reported in this work.
4.	See for https://staff.fnwi.uva.nl/s.c.frank/mmt_wmt_slides.pdf for many more results on the same dataset and task, many approaches achieving > 50 METEOR.


Further Weaknesses:
1.	Please cite the actual publications not the arXives, whenever available.

While the paper integrates MCB [Fukui] in multimodal translation, which has not been done before to my knowledge, the paper significantly lacks coverage and comparison to related work, making it not acceptable in this form. Most notably, the approach is very similar to [Iacer 2016], apart from using MCB, but the paper does not cite [Iacer 2016].


References:
[Pham and Pagh 2013] Ninh Pham and Rasmus Pagh. 2013. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’13, pages 239–247, New York, NY, USA. ACM. 
[Elliott 2015] Elliott, Desmond, Stella Frank, and Eva Hasler. "Multilingual Image Description with Neural Sequence Models." arXiv preprint arXiv:1510.04709 (2015).

[Iacer 2016] Calixto, Iacer, Desmond Elliott, and Stella Frank. "Dcu-uva multimodal mt system report." Proceedings of the First Conference on Machine Translation, Berlin, Germany. 2016.

review_title: Sensible idea, but comparison to related work insufficient

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper investigates the problem of combining variable-length information from two different modalities. The specific method considered in the paper is compact bilinear pooling, which is compared against simpler methods in the context of multimodal machine translation. Two versions of the algorithm are considered, which differ in whether the information extracted from text influences the computation of attention weights for the elements of the representation of the image.

As mentioned by another review, a major issue of the paper is that that the prior work by [Calixto 2016] is not mentioned. Besides the statement "To our knowledge, there is currently no multimodal translation architecture that convincingly surpass [sic] a monomodal attention baseline" contradicts the results reported in [Calixto 2016]. They do report an improvement over the text-only NMT. This makes it hard to trust the results of this paper.

The writing of the paper could be improved. A lot of space is used to explain the basic model, but the proposed methods are explained extremely briefly. A few sentences explaining the compact bilinear pooling could help. The Algorithm 1 is not very helpful without any explanation. Most importantly, the explanation of the pre-attention mechanism, which is perhaps is the main novelty, is very vague.

Typos and minor writing issues:
- bottom of page 2: c_t^t is rather confusing notation
- bottom of page 2: "in a multiplicative but" - a word is missing
- beginning of Section 3: I believe it should be \alpha instead of \epsilon, and it makes sense to say "learning rate \alpha" to prevent confusion

Pros: the idea of pre-attention seems novel
Cons: results are not consistent with the prior work (which has not been mentioned), writing is not clear

[Calixto 2016] Calixto, Iacer, Desmond Elliott, and Stella Frank. "Dcu-uva multimodal mt system report." Proceedings of the First Conference on Machine Translation, Berlin, Germany. 2016.
review_title: results are not consistent with prior work

Decision: Reject