Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Can you please tell how exactly you performed model selection? If you tuned the hyperparameters using the development (validation) set, then you should only use the model that performed best on it. From Table 1 it seems like this model is G+O+TXT, and the test scores of this model are only marginally better than the baseline (0.6 METEOR, 0.2 BLEU).
review_title: Development and test set scores

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an approach to the task of multimodal machine translation, namely to the case when an image is available that corresponds to both source and target sentences. 

The idea seems to be to use a latent variable model and condition it on the image. In practice from Equation 3 and Figure 3 one can see that the image is only used during training to do inference. That said, the approach appears flawed, because the image is not really used for translation.

Experimental results are weak. If the model selection was done properly, that is using the validation set, the considered model would only bring 0.6 METEOR and 0.2 BLEU advantage over the baseline. In the view of the overall variance of the results, these improvements can not be considered significant. 

The qualitative analysis in Subsection 4.4 appears inconclusive and unconvincing.

Overall, there are major issues with both the approach and the execution of the paper.


review_title: There are major issues

final_score: rating: null
review_content: I like your underlying motivation of using "a latent
variable containing an underlying semantic extracted from texts and images".  Do you have evidence that your particular model is indeed capturing useful semantics, for example with some qualitative demonstrations?
review_title: Latent representations

final_score: rating: 3: Clear rejection
review_content: This paper proposes a multimodal neural machine translation that is based upon previous work using variational methods but attempts to ground semantics with images. Considering way to improve translation with visual information seems like a sensible thing to do when such data is available. 

As pointed out by a previous reviewer, it is not actually correct to do model selection in the way it was done in the paper. This makes the gains reported by the authors very marginal. In addition, as the author's also said in their question response, it is not clear if the model is really learning to capture useful image semantics. As such, it is unfortunately hard to conclude that this paper contributes to the direction that originally motivated it.




review_title: Promising research direction but not quite there

final_score: rating: 3: Clear rejection
review_content: I have problems understanding the motivation of this paper. The authors claimed to have captured a latent representation of text and image during training and can translate better without images at test time, but didn't demonstrate convincingly that images help (not to mention the setup is a bit strange when there are no images at test time). What I see are only speculative comments: "we observed some gains, so these should come from our image models". The qualitative analysis doesn't convince me that the models have learned latent representations; I am guessing the gains are due to less overfitting because of the participation of images during training. 

The dataset is too small to experiment with NMT. I'm not sure if it's fair to compare their models with NMT and VNMT given the following description in Section 4.1 "VNMT is fine-tuned by NMT and our models are fine-tuned with VNMT". There should be more explanation on this.

Besides, I have problems with the presentation of this paper.
(a) There are many symbols being used unnecessary. For example: f & g are used for x (source) and y (target) in Section 3.1. 
(b) The ' symbol is not being used in a consistent manner, making it sometimes hard to follow the paper. For example, in section 3.1.2, there are references about h'_\pi obtained from Eq. (3) which is about h_\pi (yes, I understand what the authors mean, but there can be better ways to present that).
(c) I'm not sure if it's correct in Section 3.2.2 h'_z is computed from \mu and \sigma. So how \mu' and \sigma' are being used ?
(d) G+O-AVG should be something like G+O_{AVG}. The minus sign makes it looks like there's an ablation test there. Similarly for other symbols.

Other things: no explanations for Figure 2 & 3. There's a missing \pi symbol in Appendix A before the KL derivation.

review_title: Unclear motivation & unconvincing results

Meta-review: The area chair agrees with the reviewers that this paper is not of sufficient quality for ICLR. The experimental results are weak (there might be even be some issues with the experimental methodology) and it is not at all clear whether the translation model benefits from the image data. The authors did not address the final reviews.
Decision: Reject