Title: NEURAL MACHINE TRANSLATION WITH LATENT SE-MANTIC OF IMAGE AND TEXT

Abstract: Although attention-based Neural Machine Translation have achieved great success, attention-mechanism cannot capture the entire meaning of the source sentence because the attention mechanism generates a target word depending heavily on the relevant parts of the source sentence. The report of earlier studies has introduced a latent variable to capture the entire meaning of sentence and achieved improvement on attention-based Neural Machine Translation. We follow this approach and we believe that the capturing meaning of sentence benefits from image information because human beings understand the meaning of language not only from textual information but also from perceptual information such as that gained from vision. As described herein, we propose a neural machine translation model that introduces a continuous latent variable containing an underlying semantic extracted from texts and images. Our model, which can be trained endto-end, requires image information only when training. Experiments conducted with an English-German translation task show that our model outperforms over the baseline.

1. INTRODUCTION:

Neural machine translation (NMT) has achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015) . In contrast to statistical machine translation, which requires huge phrase and rule tables, NMT requires much less memory. However, the most standard model, NMT with attention (Bahdanau et al., 2015) entails the shortcoming that the attention mechanism cannot capture the entire meaning of a sentence because it generates a target word while depending heavily on the relevant parts of the source sentence (Tu et al., 2016) . To overcome this problem, Variational Neural Machine Translation (VNMT), which outperforms NMT with attention introduces a latent variable to capture the underlying semantic from source and target (Zhang et al., 2016) . We follow the motivation of VNMT, which is to capture underlying semantic of a source.

Image information is related to language. For example, we human beings understand the meaning of language by linking perceptual information given by the surrounding environment and language (Barsalou, 1999) . Although it is natural and easy for humans, it is difficult for computers to understand different domain's information integrally. Solving this difficult task might, however, bring great improvements in natural language processing. Several researchers have attempted to link language and images such as image captioning by Xu et al. (2015) or image generation from sentences by Reed et al. (2016) . They described the possibility of integral understanding of images and text. In machine translation, we can expect an improvement using not only text information but also image information because image information can bridge two languages.

As described herein, we propose the neural machine translation model which introduces a latent variable containing an underlying semantic extracted from texts and images. Our model includes an explicit latent variable z, which has underlying semantics extracted from text and images by introducing a Variational Autoencoder (VAE) (Kingma et al., 2014; Rezende et al., 2014) . Our model, which can be trained end-to-end, requires image information only when training. As described herein, we tackle the task with which one uses a parallel corpus and images in training, while using a source corpus in translating. It is important to define the task in this manner because we rarely have a corresponding image when we want to translate a sentence. During translation, our model generates a semantic variable z from a source, integrates variable z into a decoder of neural machine translation system, and then finally generates the translation. The difference between our model and VNMT is that we use image information in addition to text information.

ℎ " ℎ " ℎ # ℎ $ ℎ % ℎ # ℎ $ ℎ % ℎ " ' ( # ( " ( $ ( % ) # ) " ) $ ℎ * & ℎ * + log / # 0 1 0 " 0 # 0 $ ) # ) " ) $ ℎ 2 & ℎ 2

For experiments, we used Multi30k (Elliott et al., 2016) , which includes images and the corresponding parallel corpora of English and German. Our model outperforms the baseline with two evaluation metrics: METEOR (Denkowski & Lavie, 2014) and BLEU (Papineni et al., 2002) . Moreover, we obtain some knowledge related to our model and Multi30k. Finally, we present some examples in which our model either improved, or worsened, the result.

Our paper contributes to the neural machine translation research community in three ways.

• We present the first neural machine translation model to introduce a latent variable inferred from image and text information. We also present the first translation task with which one uses a parallel corpus and images in training, while using a source corpus in translating.

• Our translation model can generate more accurate translation by training with images, especially for short sentences.

• We present how the translation of source is changed by adding image information compared to VNMT which does not use image information.

2. BACKGROUND:

Our model is the extension of Variational Neural Machine Translation (VNMT) (Zhang et al., 2016) .

Our model is also viewed as one of the multimodal translation models. In our model, VAE is used to introduce a latent variable. We describe the background of our model in this section.

2.1. VARIATIONAL NEURAL MACHINE TRANSLATION:

The VNMT translation model introduces a latent variable. This model's architecture shown in Figure 1 excludes the arrow from π. This model involves three parts: encoder, inferrer, and decoder. In the encoder, both the source and target are encoded by bidirectional-Recurrent Neural Networks (bidirectional-RNN) and a semantic representation is generated. In the inferrer, a latent variable z is modeled from a semantic representation by introducing VAE. In the decoder, a latent variable z is integrated in the Gated Recurrent Unit (GRU) decoder; also, a translation is generated.

Our model is followed by architecture, except that the image is also encoded to obtain a latent variable z.

2.2. MULTIMODAL TRANSLATION:

Multimodal Translation is the task with which one might one can use a parallel corpus and images.

The first papers to study multimodal translation are Elliott et al. (2015) and Hitschler & Riezler (2016) . It was selected as a shared task in Workshop of Machine Translation 2016 (WMT161 ). Although several studies have been conducted (Caglayan et al., 2016; Huang et al., 2016; Calixto et al., 2016; Libovický et al., 2016; Rodríguez Guasch & Costa-jussà, 2016; Shah et al., 2016) , they do not show great improvement, especially in neural machine translation (Specia et al., 2016) . Here, we introduce end-to-end neural network translation models like our model. Caglayan et al. (2016) integrate an image into an NMT decoder. They simply put source context vectors and image feature vectors extracted from ResNet-50's 'res4f relu' layer (He et al., 2016) into the decoder called multimodal conditional GRU. They demonstrate that their method does not surpass the text-only baseline: NMT with attention. Huang et al. (2016) integrate an image into a head of source words sequence. They extract prominent objects from the image by Region-based Convolutional Neural Networks (R-CNN) (Girshick, 2015) .

Objects are then converted to feature vectors by VGG-19 (Simonyan & Zisserman, 2014) and are put into a head of source words sequence. They demonstrate that object extraction by R-CNN contributes greatly to the improvement. This model achieved the highest METEOR score in NMTbased models in WMT16, which we compare to our model in the experiment. We designate this model as CMU. Caglayan et al. (2016) argue that their proposed model did not achieve improvement because they failed to benefit from both text and images. We assume that they failed to integrate text and images because they simply put images and text into neural machine translation despite huge gap exists between image information and text information. Our model, however, presents the possibility of benefitting from images and text because text and images are projected to their common semantic space so that the gap of images and text would be filled.

2.3. VARIATIONAL AUTO ENCODER:

VAE was proposed in an earlier report of the literature Kingma et al. (2014) ; Rezende et al. (2014) . Given an observed variable x, VAE introduces a continuous latent variable z, with the assumption that x is generated from z. VAE incorporates p θ (x|z) and q ϕ (z|x) into an end-to-end neural network. The lower bound is shown below.

EQUATION

3. NEURAL MACHINE TRANSLATION WITH LATENT SEMANTIC OF IMAGE AND TEXT:

We propose a neural machine translation model which explicitly has a latent variable containing an underlying semantic extracted from both text and image. This model can be seen as an extension of VNMT by adding image information.

Our model can be drawn as a graphical model in Figure 3 . Its lower bound is

EQUATION

where x, y, π, z respectively denote the source, target, image and latent variable, and p θ and q ϕ respectively denote the prior distribution and the approximate posterior distribution. It is noteworthy in Eq. ( 2) that we want to model p(z|x, y, π), which is intractable. Therefore we model q ϕ (z|x, y, π) We model all distributions in Eq. ( 2) by neural networks. Our model architecture is divisible into three parts: 1) encoder, 2) inferrer, and 3) decoder.

3.1. ENCODER:

In the encoder, the semantic representation h e is obtained from the image, source, and target. We propose several methods to encode an image. We show how these methods affect the translation result in the Experiment section. This representation is used in the inferrer. This section links to the green part of Figure 1 .

3.1.1. TEXT ENCODING:

The source and target are encoded in the same way as Bahdanau et al. (2015) . The source is converted to a sequence of 1-of-k vector and is embedded to d emb dimensions. We designate it as the source sequence. Then, a source sequence is put into bidirectional RNN. Representation h i is obtained by concatenating ⃗ h i and ⃗

h i : ⃗ h i = RNN( ⃗ h i-1 , E wi ), ⃗ h i = RNN( ⃗ h i+1 , E wi ), h i = [ ⃗ h i ; ⃗ h i ]

, where E wi is the embedded word in a source sentence,

h i ∈ R d h , and ⃗ h i , ⃗ h i ∈ R d h 2 . It is conducted through i = 0 to i = T f ,

where T f is the sequence length. GRU is implemented in bidirectional RNN so that it can attain long-term dependence. Finally, we conduct mean-pooling to h i and obtain the source representation vector as

h f = 1 T f ∑ T f i h i .

The exact same process is applied to target to obtain target representation h g .

3.1.2. IMAGE ENCODING AND SEMANTIC REPRESENTATION:

We use Convolutional Neural Networks (CNN) to extract feature vectors from images. We propose several ways of extracting image features.

Global (G):

The image feature vector is extracted from the image using a CNN. With this method, we use a feature vector in the certain layer as π. Then π is encoded to the image representation vector h π simply by affine transformation as

EQUATION

)

Global and Objects (G+O) First we extract some prominent objects from images in some way.

Then, we obtain fc7 image feature vectors π from the original image and extracted objects using a CNN. Therefore π takes a variable length. We handle π in two ways: average and RNN encoder.

In average (G+O-AVG), we first obtain intermediate image representation vector h ′ π by affine transformation in Eq. (3). Then, the average of h ′ π becomes the image representation vector:

h π = ∑ l i h ′ π i l

, where l is the length of h ′ π .

In RNN encoder (G+O-RNN), we first obtain h ′ π by affine transformation in Eq. ( 3). Then, we encode h ′ π in the same way as we encode text in Section 3.1.1 to obtain h π . Global and Objects into source and target (G+O-TXT) Thereby, we first obtain h ′ π by affine transformation in Eq. ( 3). Then, we put sequential vector h ′ π into the head of the source sequence and target sequence. In this case, we set d π to be the same dimension as d emb . In fact, the source sequence including h ′ π is only used to model q ϕ (z|x, y, π). Context vector c (Eq. ( 15)) and p θ (z|x) are computed by a source sequence that does not include h ′ π . We encode the source sequence including h ′ π as Section 3.1.1 to obtain h f and h g . In this case, h π is not obtained. Image information is contained in h f and h g .

All representation vectors h f , h g and h π are concatenated to obtain a semantic representation vector as

h e = [h f ; h g ; h π ],

where h e ∈ R de=2×d h +dπ (in G+O-TXT:

h e = [h f ; h g ],

where h e ∈ R de=2×d h ). It is an input of the multimodal variational neural inferrer.

3.2. INFERRER:

We model the posterior q ϕ (z|x, y, π) using a neural network and also the prior p θ (z|x) by neural network. This section links to the black and grey part of Figure 1 .

3.2.1. NEURAL POSTERIOR APPROXIMATOR:

Modeling the true posterior p θ (z|x, y, π) is usually intractable. Therefore, we consider modeling of an approximate posterior q ϕ (z|x, y, π) by introducing VAE. We assume that the posterior q ϕ (z|x, y, π) has the following form:

EQUATION

The mean µ and standard deviation σ of the approximate posterior are the outputs of neural networks.

Starting from the variational neural encoder, a semantic representation vector h e is projected to latent semantic space as

EQUATION

where

W (1) z ∈ R dz×(de) b (1) z ∈ R dz . g(•)

is an element-wise activation function, which we set as tanh(•). Gaussian parameters of Eq. ( 4) are obtained through linear regression as

EQUATION

)

where µ, log σ 2 ∈ R dz .

3.2.2. NEURAL PRIOR MODEL:

We model the prior distribution p θ (z|x) as follows:

EQUATION

µ ′ and σ ′ are generated in the same way as that presented in Section 3.2.1, except for the absence of y and π as inputs. Because of the absence of representation vectors, the dimensions of weight in equation ( 5) for prior model are W

′ (1) z ∈ R dz×d h , b ′ (1) z ∈ R dz .

We use a reparameterization trick to obtain a representation of latent variable z: h ′ z = µ + σϵ, ϵ ∼ N (0, I). During translation, h ′ z is set as the mean of p θ (z|x). Then, h ′ z is projected onto the target space as

EQUATION

)

h ′ e is then integrated into the neural machine translation's decoder.

3.3. DECODER:

This section links to the orange part of Figure 1 . Given the source sentence x and the latent variable z, decoder defines the probability over translation y as

EQUATION

How we define the probability over translation y is fundamentally the same as VNMT, except for using conditional GRU instead of GRU. Conditional GRU involves two GRUs and an attention mechanism. We integrate a latent variable z into the second GRU. We describe it in the appendix.

3.4. MODEL TRAINING:

Monte Carlo sampling method is used to approximate the expectation over the posterior Eq. ( 2),

E q ϕ (z|x,y,π) ≈ 1 L ∑ L l=1 log p θ (y|x, h (l) z )

, where L is the number of samplings. The training objective is defined as

EQUATION

where

h z = µ + σ • ϵ, ϵ ∼ N (0, I).

The first term, KL divergence, can be computed analytically and is differentiable because both distributions are Gaussian. The second term is also differentiable. We set L as 1. Overall, the objective L is differentiable. Therefore, we can optimize the parameter θ and variational parameter ϕ using gradient ascent techniques.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:

We used Multi30k (Elliott et al., 2016) as the dataset. Multi30k have an English description and a German description for each corresponding image. We handle 29,000 pairs as training data, 1,014 pairs as validation data, and 1,000 pairs as test data.

Before training, punctuation normalization and lowercase are applied to both English and German sentences by Moses (Koehn et al., 2007 ) scripts2 . Compound-word splitting is conducted only to German sentences using Sennrich et al. (2016) 3 . Then we tokenize sentences 2 and use them as training data. We produce vocabulary dictionaries from training data. The vocabulary becomes 10,211 words for English and 13,180 words for German after compound-word splitting.

Image features are extracted using VGG-19 CNN (Simonyan & Zisserman, 2014) . We use 4096dimensional fc7 features. To extract the object's region, we use Fast R-CNN (Girshick, 2015) . Fast R-CNN is trained on ImageNet and MSCOCO dataset4 .

All weights are initialized by N (0, 0.01I). We use the adadelta algorithm as an optimization method. The hyperparameters used in the experiment are presented in the Appendix. All models are trained with early stopping. When training, VNMT is fine-tuned by NMT model and our models are finetuned using VNMT. When translating, we use beam-search. The beam-size is set as 12. Before evaluation, we restore split words to the original state and de-tokenize 2 generated sentences.

We implemented proposed models based on dl4mt5 . Actually, dl4mt is fundamentally the same model as Bahdanau et al. (2015) , except that its decoder employs conditional GRU 6 . We implemented VNMT also with conditional GRU so small difference exists between our implementation and originally proposed VNMT which employs normal GRU as a decoder. We evaluated results based on METEOR and BLUE using MultEval7 .

4.2. RESULT:

Table 1 presents experiment results. It shows that our models outperforms the baseline in both METEOR and BLEU. Figure 4 shows the plot of METEOR score of baselines and our models models in validation. Figure 5 shows the plot of METEOR score and the source sentence length. 1 shows that G scores the best in proposed models. In G, we simply put the feature of the original image. Actually, proposed model does not benefit from R-CNN, presumably because we can not handle sequences of image features very well. For example, G+O-AVG uses the average of multiple image features, but it only makes the original image information unnecessarily confusing.

Figure 4 shows that G and G+O-AVG outperforms VNMT almost every time, but all model scores increase suddenly in the 17,000 iteration validation. We have no explanation for this behavior. Figure 4 also shows that G and G+O-AVG scores fluctuate more moderately than others. We state that G and G+O-AVG gain stability by adding image information. When one observes the difference between the test score and the validation score for each model, baseline scores decrease more than proposed model scores. Especially, the G score increases in the test, simply because proposed models produce a better METEOR score on average, as shown in Figure 4 .

Figure 5 shows that G and G+O-AVG make more improvements on baselines in short sentences than in long sentences, presumably because q ϕ (z|x, y, π) can model z well when a sentence is short. Image features always have the same dimension, but underlying semantics of the image and text differ. We infer that when the sentence is short, image feature representation can afford to approximate the underlying semantic, but when a sentence is long, image feature representation can not approximate the underlying semantic.

Multi30k easily becomes overfitted, as shown in Figure 8 and 9 in the appendix. This is presumably because 1) Multi30k is the descriptions of image, making the sentences short and simple, and 2) Multi30k has 29,000 sentences, which could be insufficient. In the appendix, we show how the parameter setting affects the score. One can see that decay-c has a strong effect. Huang et al. (2016) states that their proposed model outperforms the baseline (NMT), but we do not have that observation. It can be assumed that their baseline parameters are not well tuned.

4.4. QUALITATIVE ANALYSIS:

We presented the top 30 sentences, which make the largest METEOR score difference between G and VNMT, to native speakers of German and get the overall comments. They were not informed of our model training with image in addition to text. These comments are summarized into two general remarks. One is that G translates the meaning of the source material more accurately than VNMT.

The other is that our model has more grammatical errors as prepositions' mistakes or missing verbs compared to VNMT. We assume these two remarks are reasonable because G is trained with images which mainly have a representation of noun rather than verb, therefore can capture the meaning of materials in sentence.

Figure 6 presents the translation results and the corresponding image which G translates more accurately than VNMT in METEOR. Figure 7 presents the translation results and the corresponding image which G translates less accurately than VNMT in METEOR. Again, we note that our model does not use image during translating. In Figure 6 , G translates "a white and black dog" correctly while VNMT translates it incorrectly implying "a white dog and a black dog". We assume that G correctly translates the source because G captures the meaning of material in the source. In Figure 7 , G incorrectly translates the source. Its translation result is missing the preposition meaning "at", which is hardly represented in image.We present more translation examples in appendix.

Source a woman holding a white and black dog. Truth eine frau hält einen weiß-schwarzen hund. VNMT eine frau hält einen weißen und schwarzen hund. Our Model (G) eine frau hält einen weiß-schwarzen hund. 

5. CONCLUSION:

As described herein, we proposed the neural machine translation model that explicitly has a latent variable that includes underlying semantics extracted from both text and images. Our model outperforms the baseline in both METEOR and BLEU scores. Experiments and analysis present that our model can generate more accurate translation for short sentences. In qualitative analysis, we present that our model can translate nouns accurately while our model make grammatical errors.

A DERIVATION OF LOWER BOUNDS:

The lower bound of our model can be derived as follows: 

p(y|x) = ∫ p(

EQUATION

EQUATION

where E ∈ R d emb ×dt signifies the target word embedding, s ′ j ∈ R d h denotes the hidden state, r ′ j ∈ R d h and o ′ j ∈ R d h respectively represent the reset and update gate activations. d t stands for the dimension of target; the unique number of target words.

[W ′ , W ′ r , W ′ o ] ∈ R d h ×d emb , [U ′ , U ′ r , U ′ o ] ∈ R d h ×d h

are the parameters to be learned.

Context vector c j is obtained as

EQUATION

)

EQUATION

)

EQUATION

where [U att , W catt , W att ] ∈ R d h ×d h are the parameters to be learned.

The second GRU computes s j from s ′ j , c j and h ′ e as

EQUATION

)

EQUATION

EQUATION

EQUATION

where s j ∈ R d h stands for the hidden state, r j ∈ R d h and o j ∈ R d h are the reset and update gate activations.

[W, W r , W o ] ∈ R d h ×d h , [U, U r , U o ] ∈ R d h ×d h , [V, V r , V o ] ∈ R d h ×dz

are the parameters to be learned. We introduce h ′ e obtained from a latent variable here so that a latent variable can affect the representation s j through GRU units.

Finally, the probability of y is computed as

EQUATION

where L u ∈ R dt×d emb , L s ∈ R d emb ×d h and L c ∈ R d emb ×d h are the parameters to be learned.

C TRAINING DETAIL:

C.1 HYPERPARAMETERS We found that Multi30k dataset is easy to overfit. Figure 8 and Figure 9 present training cost and validation METEOR score graph of the two experimental settings of the NMT model. Table 3 presents the hyperparameters which were used in the experiments. Large decay-c ans small batchsize give the better METEOR scores in the end. Training is stopped if there is no validation cost improvements over the last 10 validations. We present some selected translations from VNMT and our proposed model (G). As of translation 3 to 5 our model give the better METEOR scores than VNMT and as of translation 6 to 8 VNMT give the better METEOR scores than our models. 

:

& ℎ $ &

http://www.statmt.org/wmt16/

https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/{normalize-punctuation, lowercase, tokenizer, detokenizer}.perl

3 https://github.com/rsennrich/subword-

nmt 4 https://github.com/rbgirshick/fast-rcnn/tree

/coco 5 https://github.com/nyu-dl/dl4mt-

tutorial6 The architecture is described at https://github.com/nyu-dl/dl4mt-tutorial/blob/master/docs/cgru.pdf

https://github.com/jhclark/multeval, we use meteor1.5 instead of meteor1.4, which is the default of MultEval.

Figures and Tables:

Figure 1: Architecture of Proposed Model. Green dotted lines denote that π and encoded y are used only when training.

Figure 2: VNMT

Figure 4: METEOR score to the validation data which are calculated for each 1000 iterations.

Figure 6: Translation 1

Figure 7: Translation 2

Figure 8: NMT Training Cost

Figure 10: Word Length Histogram of the Multi30k Test Dataset

Figure 12: Validation cost

Figure 13: Translation 3

Figure 14: Translation 4

Figure 15: Translation 5

Figure 16: Translation 6

Figure 17: Translation 7

Evaluation Result on Multi30k dataset (English-German). The scores in parentheses are computed with '-norm' parameter. NMT is dl4mt's NMT (in the session3 directory). The score of the CMU is from(Huang et al., 2016).

Table content: <table><tr><td/><td colspan="2">METEOR ↑</td><td colspan="2">BLEU ↑</td></tr><tr><td/><td>val</td><td>test</td><td>val</td><td>test</td></tr><tr><td>NMT</td><td colspan="4">51.5 (55.8) 50.5 (54.9) 35.8 33.1</td></tr><tr><td>VNMT</td><td colspan="4">52.2 (56.3) 51.1 (55.3) 37.0 34.9</td></tr><tr><td>CMU</td><td>-(-)</td><td>-(54.1)</td><td>-</td><td>-</td></tr><tr><td>Our Model G</td><td colspan="4">50.6 (54.8) 52.4 (56.0) 34.5 36.5</td></tr><tr><td colspan="5">G+O-AVG 51.8 (55.8) 51.8 (55.8) 35.7 35.8</td></tr><tr><td colspan="5">G+O-RNN 51.8 (56.1) 51.0 (55.4) 35.9 34.9</td></tr><tr><td colspan="5">G+O-TXT 52.6 (56.8) 51.7 (56.0) 36.6 35.1</td></tr><tr><td>4.3 QUANTITATIVE ANALYSIS</td><td/><td/><td/><td/></tr><tr><td>Table</td><td/><td/><td/><td/></tr></table>

presents parameters that we use in the experiments.

Table content: <table/>

Hyperparameters. The name is the variable name of dl4mt except for dimv and dim pic, which are the dimension of the latent variables and image embeddings. We set dim (number of LSTM unit size) and dim word (dimensions of word embeddings) 256, batchsize 32, maxlen (max output length) 50 and lr (learning rate) 1.0 for all models. decay-c is weights on L2 regularization.

Table content: <table><tr><td/><td colspan="3">dimv dim pic decay-c</td></tr><tr><td>NMT</td><td>-</td><td>256</td><td>0.001</td></tr><tr><td>VNMT</td><td>256</td><td>256</td><td>0.0005</td></tr><tr><td>Our Model G</td><td>256</td><td>512</td><td>0.001</td></tr><tr><td colspan="2">G+O-AVG 256</td><td>256</td><td>0.0005</td></tr><tr><td colspan="2">G+O-RNN 256</td><td>256</td><td>0.0005</td></tr><tr><td colspan="2">G+O-TXT 256</td><td>256</td><td>0.0005</td></tr></table>

Hyperparameters using the experiments in the Figure8 and 9Figure10presents the English word length histogram of the Multi30k test dataset. Most sentences in the Multi30k are less than 20 words. We assume that this is one of the reasons why Multi30k is easy to overfit.

Table content: <table><tr><td colspan="2">dim dim word lr</td><td colspan="3">decay-c maxlen batchsie</td></tr><tr><td>1 256 256</td><td colspan="2">1.0 0.0005</td><td>30</td><td>128</td></tr><tr><td>2 256 256</td><td colspan="2">1.0 0.001</td><td>50</td><td>32</td></tr></table>

Peer review (iclr2017):
This paper proposes a multimodal neural machine translation that is based upon previous work using variational methods but attempts to ground semantics with images. Considering way to improve translation with visual information seems like a sensible thing to do when such data is available. As pointed out by a previous reviewer, it is not actually correct to do model selection in the way it was done in the paper. This makes the gains reported by the authors very marginal. In addition, as the author*s also said in their question response, it is not clear if the model is really learning to capture useful image semantics. As such, it is unfortunately hard to conclude that this paper contributes to the direction that originally motivated it.