An empirical study on the effectiveness of images in Multimodal Neural Machine Translation	In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multimodal translation task (English, image to German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.

Conclusion and future work	We have tried different attention mechanism and tweaks for the image modality. We showed improvements and encouraging results overall on the Flickr30K Entities dataset. Even though we identified some flaws of the current attention mechanisms, we can conclude pretty safely that images are an helpful resource for the machine in a translation task. We are looking forward to try out richer and more suitable features for multimodal translation (ie. dense captioning features). Another interesting approach would be to use visually grounded word embeddings to capture visual notions of semantic relatedness.

Qualitative results	For space-saving and ergonomic reasons, we only discuss about the hard stochastic and soft attention, the latter being a generalization of the local attention. As we can see in Figure FIGREF44 , the soft attention model is looking roughly at the same region of the image for every decoding step INLINEFORM0 . Because the words "hund"(dog), "wald"(forest) or "weg"(way) in left image are objects, they benefit from a high gating scalar. As a matter of fact, the attention mechanism has learned to detect the objects within a scene (at every time-step, whichever word we are decoding as shown in the right image) and the gating scalar has learned to decide whether or not we have to look at the picture (or more accurately whether or not we are translating an object). Without this scalar, the translation scores undergo a massive drop (as seen in  caglayan2016does) which means that the attention mechanisms don't really understand the more complex relationships between objects, what is really happening in the scene.

Surprisingly, the gating scalar happens to be really low in the stochastic attention mechanism: a significant amount of sentences don't have a summed gating scalar INLINEFORM1 0.10. The model totally discards the image in the translation process. It is also worth to mention that we use a ResNet trained on 1.28 million images for a classification tasks. The features used by the attention mechanism are strongly object-oriented and the machine could miss important information for a multimodal translation task. We believe that the robust architecture of both encoders INLINEFORM0 combined with a GRU layer and word-embeddings took care of the right translation for relationships between objects and time-dependencies. Yet, we noticed a common misbehavior for all our multimodal models: if the attention loose track of the objects in the picture and "gets lost", the model still takes it into account and somehow overrides the information brought by the text-based annotations. The translation is then totally mislead. We illustrate with an example: The monomodal translation has a sentence-level BLEU of 82.16 whilst the soft attention and hard stochastic attention scores are of 16.82 and 34.45 respectively.

Acknowledgements	This work was partly supported by the Chist-Era project IGLU with contribution from the Belgian Fonds de la Recherche Scientique (FNRS), contract no. R.50.11.15.F, and by the FSO project VCYCLE with contribution from the Belgian Waloon Region, contract no. 1510501.

Hard Stochastic attention	This model is a stochastic and sampling-based process where, at every timestep INLINEFORM0 , we are making a hard choice to attend only one annotation. This corresponds to one spatial location in the image. Hard attention has previously been used in the context of object recognition  ,  and later extended to image description generation  . In the context of multimodal NMT, we can follow  icml2015xuc15 because both our models involve the same process on images. The mechanism INLINEFORM0 is now a function that returns a sampled intermediate latent variables INLINEFORM1 based upon a multinouilli distribution parameterized by INLINEFORM2 : DISPLAYFORM0  where INLINEFORM0 an indicator one-hot variable which is set to 1 if the INLINEFORM1 -th annotation (out of INLINEFORM2 ) is the one used to compute the context vector INLINEFORM3 : DISPLAYFORM0   Context vector INLINEFORM0 is now seen as the random variable of this distribution. We define the variational lower bound INLINEFORM1 on the marginal log evidence INLINEFORM2 of observing the target sentence INLINEFORM3 given modality annotations INLINEFORM4 .

These vector representations, also called context vectors, are computed in order the capture the most relevant information in a modality to output the best translation of a sentence. To investigate the effectiveness of information obtained from images, a multimodal machine translation shared task  has been addressed to the MT community. The best results of NMT model were those of  huang2016attention who used LSTM fed with global visual features or multiple regional visual features followed by rescoring. Recently,  CalixtoLC17b proposed a doubly-attentive decoder that outperformed this baseline with less data and without rescoring. Our paper is structured as follows. In section SECREF2 , we briefly describe our NMT model as well as the conditional GRU activation used in the decoder. We also explain how multi-modalities can be implemented within this framework. In the following sections ( SECREF3 and SECREF4 ), we detail three attention mechanisms and explain how we tweak them to work as well as possible with images. Finally, we report and analyze our results in section SECREF5 then conclude in section SECREF6 .

Neural Machine Translation	In this section, we detail the neural machine translation architecture by  BahdanauCB14, implemented as an attention-based encoder-decoder framework with recurrent neural networks (§ SECREF2 ). We follow by explaining the conditional GRU layer (§ SECREF8 ) - the gating mechanism we chose for our RNN - and how the model can be ported to a multimodal version (§ SECREF13 ).

Text-based NMT	Given a source sentence INLINEFORM0 , the neural network directly models the conditional probability INLINEFORM1 of its translation INLINEFORM2 . The network consists of one encoder and one decoder with one attention mechanism. The encoder computes a representation INLINEFORM3 for each source sentence and a decoder generates one target word at a time and by decomposing the following conditional probability : DISPLAYFORM0  Each source word INLINEFORM0 and target word INLINEFORM1 are a column index of the embedding matrix INLINEFORM2 and INLINEFORM3 . The encoder is a bi-directional RNN with Gated Recurrent Unit (GRU) layers  ,  , where a forward RNN INLINEFORM4 reads the input sequence as it is ordered (from INLINEFORM5 to INLINEFORM6 ) and calculates a sequence of forward hidden states INLINEFORM7 . A backward RNN INLINEFORM8 reads the sequence in the reverse order (from INLINEFORM9 to INLINEFORM10 ), resulting in a sequence of backward hidden states INLINEFORM11 . We obtain an annotation for each word INLINEFORM12 by concatenating the forward and backward hidden state INLINEFORM13 . Each annotation INLINEFORM14 contains the summaries of both the preceding words and the following words.

The representation INLINEFORM15 for each source sentence is the sequence of annotations INLINEFORM16 . The decoder is an RNN that uses a conditional GRU (cGRU, more details in § SECREF8 ) with an attention mechanism to generate a word INLINEFORM0 at each time-step INLINEFORM1 . The cGRU uses it's previous hidden state INLINEFORM2 , the whole sequence of source annotations INLINEFORM3 and the previously decoded symbol INLINEFORM4 in order to update it's hidden state INLINEFORM5 : DISPLAYFORM0  In the process, the cGRU also computes a time-dependent context vector INLINEFORM0 . Both INLINEFORM1 and INLINEFORM2 are further used to decode the next symbol. We use a deep output layer  to compute a vocabulary-sized vector : DISPLAYFORM0  where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 are model parameters. We can parameterize the probability of decoding each word INLINEFORM4 as: DISPLAYFORM0  The initial state of the decoder INLINEFORM0 at time-step INLINEFORM1 is initialized by the following equation : DISPLAYFORM0  where INLINEFORM0 is a feedforward network with one hidden layer.

Conditional GRU	The conditional GRU consists of two stacked GRU activations called INLINEFORM0 and INLINEFORM1 and an attention mechanism INLINEFORM2 in between (called ATT in the footnote paper). At each time-step INLINEFORM3 , REC1 firstly computes a hidden state proposal INLINEFORM4 based on the previous hidden state INLINEFORM5 and the previously emitted word INLINEFORM6 : DISPLAYFORM0   Then, the attention mechanism computes INLINEFORM0 over the source sentence using the annotations sequence INLINEFORM1 and the intermediate hidden state proposal INLINEFORM2 : DISPLAYFORM0  Finally, the second recurrent cell INLINEFORM0 , computes the hidden state INLINEFORM1 of the INLINEFORM2 by looking at the intermediate representation INLINEFORM3 and context vector INLINEFORM4 : DISPLAYFORM0

Multimodal NMT	Recently,  CalixtoLC17b proposed a doubly attentive decoder (referred as the "MNMT" model in the author's paper) which can be seen as an expansion of the attention-based NMT model proposed in the previous section. Given a sequence of second a modality annotations INLINEFORM0 , we also compute a new context vector based on the same intermediate hidden state proposal INLINEFORM1 : DISPLAYFORM0  This new time-dependent context vector is an additional input to a modified version of REC2 which now computes the final hidden state INLINEFORM0 using the intermediate hidden state proposal INLINEFORM1 and both time-dependent context vectors INLINEFORM2 and INLINEFORM3 : DISPLAYFORM0   The probabilities for the next target word (from equation EQREF5 ) also takes into account the new context vector INLINEFORM0 : DISPLAYFORM0  where INLINEFORM0 is a new trainable parameter. In the field of multimodal NMT, the second modality is usually an image computed into feature maps with the help of a CNN. The annotations INLINEFORM0 are spatial features (i.e. each annotation represents features for a specific region in the image) . We follow the same protocol for our experiments and describe it in section SECREF5 .

Attention-based Models	We evaluate three models of the image attention mechanism INLINEFORM0 of equation EQREF11 . They have in common the fact that at each time step INLINEFORM1 of the decoding phase, all approaches first take as input the annotation sequence INLINEFORM2 to derive a time-dependent context vector that contain relevant information in the image to help predict the current target word INLINEFORM3 . Even though these models differ in how the time-dependent context vector is derived, they share the same subsequent steps. For each mechanism, we propose two hand-picked illustrations showing where the attention is placed in an image.

Soft attention	Soft attention has firstly been used for syntactic constituency parsing by  NIPS2015Vinyals but has been widely used for translation tasks ever since. One should note that it slightly differs from  BahdanauCB14 where their attention takes as input the previous decoder hidden state instead of the current (intermediate) one as shown in equation EQREF11 . This mechanism has also been successfully investigated for the task of image description generation  where a model generates an image's description in natural language. It has been used in multimodal translation as well  , for which it constitutes a state-of-the-art. The idea of the soft attentional model is to consider all the annotations when deriving the context vector INLINEFORM0 . It consists of a single feed-forward network used to compute an expected alignment INLINEFORM1 between modality annotation INLINEFORM2 and the target word to be emitted at the current time step INLINEFORM3 .

The inputs are the modality annotations and the intermediate representation of REC1 INLINEFORM4 : DISPLAYFORM0  The vector INLINEFORM0 has length INLINEFORM1 and its INLINEFORM2 -th item contains a score of how much attention should be put on the INLINEFORM3 -th annotation in order to output the best word at time INLINEFORM4 . We compute normalized scores to create an attention mask INLINEFORM5 over annotations: DISPLAYFORM0   Finally, the modality time-dependent context vector INLINEFORM0 is computed as a weighted sum over the annotation vectors (equation ). In the above expressions, INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are trained parameters.

Local Attention	In this section, we propose a local attentional mechanism that chooses to focus only on a small subset of the image annotations. Local Attention has been used for text-based translation  and is inspired by the selective attention model of  gregor15 for image generation. Their approach allows the model to select an image patch of varying location and zoom. Local attention uses instead the same "zoom" for all target positions and still achieved good performance. This model can be seen as a trade-off between the soft and hard attentional models. The model picks one patch in the annotation sequence (one spatial location) and selectively focuses on a small window of context around it. Even though an image can't be seen as a temporal sequence, we still hope that the model finds points of interest and selects the useful information around it. This approach has an advantage of being differentiable whereas the stochastic attention requires more complicated techniques such as variance reduction and reinforcement learning to train as shown in section SECREF22 . The soft attention has the drawback to attend the whole image which can be difficult to learn, especially because the number of annotations INLINEFORM0 is usually large (presumably to keep a significant spatial granularity).

DISPLAYFORM0  The learning rules can be derived by taking derivatives of the above variational free energy INLINEFORM0 with respect to the model parameter INLINEFORM1 : DISPLAYFORM0  In order to propagate a gradient through this process, the summation in equation EQREF26 can then be approximated using Monte Carlo based sampling defined by equation EQREF24 : DISPLAYFORM0  To reduce variance of the estimator in equation EQREF27 , we use a moving average baseline estimated as an accumulated sum of the previous log likelihoods with exponential decay upon seeing the INLINEFORM0 -th mini-batch: DISPLAYFORM0

More formally, at every decoding step INLINEFORM0 , the model first generates an aligned position INLINEFORM1 . Context vector INLINEFORM2 is derived as a weighted sum over the annotations within the window INLINEFORM3 where INLINEFORM4 is a fixed model parameter chosen empirically. These selected annotations correspond to a squared region in the attention maps around INLINEFORM7 . The attention mask INLINEFORM8 is of size INLINEFORM9 . The model predicts INLINEFORM10 as an aligned position in the annotation sequence (referred as Predictive alignment (local-m) in the author's paper) according to the following equation: DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are both trainable model parameters and INLINEFORM2 is the annotation sequence length INLINEFORM3 . Because of the sigmoid, INLINEFORM4 . We use equation EQREF18 and EQREF19 respectively to compute the expected alignment vector INLINEFORM5 and the attention mask INLINEFORM6 . In addition, a Gaussian distribution centered around INLINEFORM7 is placed on the alphas in order to favor annotations near INLINEFORM8 : DISPLAYFORM0  where standard deviation INLINEFORM0 . We obtain context vector INLINEFORM1 by following equation .

Image attention optimization	Three optimizations can be added to the attention mechanism regarding the image modality. All lead to a better use of the image by the model and improved the translation scores overall. At every decoding step INLINEFORM0 , we compute a gating scalar INLINEFORM1 according to the previous decoder state INLINEFORM2 : DISPLAYFORM0  It is then used to compute the time-dependent image context vector : DISPLAYFORM0    icml2015xuc15 empirically found it to put more emphasis on the objects in the image descriptions generated with their model. We also double the output size of trainable parameters INLINEFORM0 , INLINEFORM1 and INLINEFORM2 in equation EQREF18 when it comes to compute the expected annotations over the image annotation sequence. More formally, given the image annotation sequence INLINEFORM3 , the tree matrices are of size INLINEFORM4 , INLINEFORM5 and INLINEFORM6 respectively. We noticed a better coverage of the objects in the image by the alpha weights. Lastly, we use a grounding attention inspired by  delbrouck2017multimodal.

The mechanism merge each spatial location INLINEFORM0 in the annotation sequence INLINEFORM1 with the initial decoder state INLINEFORM2 obtained in equation EQREF7 with non-linearity : DISPLAYFORM0   where INLINEFORM0 is INLINEFORM1 function. The new annotations go through a L2 normalization layer followed by two INLINEFORM2 convolutional layers (of size INLINEFORM3 respectively) to obtain INLINEFORM4 weights, one for each spatial location. We normalize the weights with a softmax to obtain a soft attention map INLINEFORM5 . Each annotation INLINEFORM6 is then weighted according to its corresponding INLINEFORM7 : DISPLAYFORM0   This method can be seen as the removal of unnecessary information in the image annotations according to the source sentence. This attention is used on top of the others - before decoding - and is referred as "grounded image" in Table TABREF41 .

Experiments	For this experiments on Multimodal Machine Translation, we used the Multi30K dataset  which is an extended version of the Flickr30K Entities. For each image, one of the English descriptions was selected and manually translated into German by a professional translator. As training and development data, 29,000 and 1,014 triples are used respectively. A test set of size 1000 is used for metrics evaluation.

Training and model details	All our models are build on top of the nematus framework  . The encoder is a bidirectional RNN with GRU, one 1024D single-layer forward and one 1024D single-layer backward RNN. Word embeddings for source and target language are of 620D and trained jointly with the model. Word embeddings and other non-recurrent matrices are initialized by sampling from a Gaussian INLINEFORM0 , recurrent matrices are random orthogonal and bias vectors are all initialized to zero. To create the image annotations used by our decoder, we used a ResNet-50 pre-trained on ImageNet and extracted the features of size INLINEFORM0 at its res4f layer  . In our experiments, our decoder operates on the flattened 196 INLINEFORM1 1024 (i.e INLINEFORM2 ). We also apply dropout with a probability of 0.5 on the embeddings, on the hidden states in the bidirectional RNN in the encoder as well as in the decoder. In the decoder, we also apply dropout on the text annotations INLINEFORM3 , the image features INLINEFORM4 , on both modality context vector and on all components of the deep output layer before the readout operation.

We apply dropout using one same mask in all time steps  . We also normalize and tokenize English and German descriptions using the Moses tokenizer scripts  . We use the byte pair encoding algorithm on the train set to convert space-separated tokens into subwords  , reducing our vocabulary size to 9226 and 14957 words for English and German respectively. All variants of our attention model were trained with ADADELTA  , with mini-batches of size 80 for our monomodal (text-only) NMT model and 40 for our multimodal NMT. We apply early stopping for model selection based on BLEU4 : training is halted if no improvement on the development set is observed for more than 20 epochs. We use the metrics BLEU4  , METEOR  and TER  to evaluate the quality of our models' translations.

Quantitative results	We notice a nice overall progress over  CalixtoLC17b multimodal baseline, especially when using the stochastic attention. With improvements of +1.51 BLEU and -2.2 TER on both precision-oriented metrics, the model shows a strong similarity of the n-grams of our candidate translations with respect to the references. The more recall-oriented metrics METEOR scores are roughly the same across our models which is expected because all attention mechanisms share the same subsequent step at every time-step INLINEFORM0 , i.e. taking into account the attention weights of previous time-step INLINEFORM1 in order to compute the new intermediate hidden state proposal and therefore the new context vector INLINEFORM2 . Again, the largest improvement is given by the hard stochastic attention mechanism (+0.4 METEOR): because it is modeled as a decision process according to the previous choices, this may reinforce the idea of recall. We also remark interesting improvements when using the grounded mechanism, especially for the soft attention. The soft attention may benefit more of the grounded image because of the wide range of spatial locations it looks at, especially compared to the stochastic attention.

This motivates us to dig into more complex grounding techniques in order to give the machine a deeper understanding of the modalities. Note that even though our baseline NMT model is basically the same as  CalixtoLC17b, our experiments results are slightly better. This is probably due to the different use of dropout and subwords. We also compared our results to  caglayan2016does because our multimodal models are nearly identical with the major exception of the gating scalar (cfr. section SECREF4 ). This motivated some of our qualitative analysis and hesitation towards the current architecture in the next section.

Figure FIGREF47 shows the attention maps for both mechanism. Nevertheless, one has to concede that the use of images indubitably helps the translation as shown in the score tabular.

Introduction	In machine translation, neural networks have attracted a lot of research attention. Recently, the attention-based encoder-decoder framework  ,  has been largely adopted. In this approach, Recurrent Neural Networks (RNNs) map source sequences of words to target sequences. The attention mechanism is learned to focus on different parts of the input sentence while decoding. Attention mechanisms have shown to work with other modalities too, like images, where their are able to learn to attend the salient parts of an image, for instance when generating text captions  . For such applications, Convolutional Neural Networks (CNNs) such as Deep Residual  have shown to work best to represent images. Multimodal models of texts and images empower new applications such as visual question answering or multimodal caption translation. Also, the grounding of multiple modalities against each other may enable the model to have a better understanding of each modality individually, such as in natural language understanding applications. In the field of Machine Translation (MT), the efficient integration of multimodal information still remains a challenging task. It requires combining diverse modality vector representations with each other.

Question: What misbehavior is identified?
Answer: if the attention loose track of the objects in the picture and "gets lost", the model still takes it into account and somehow overrides the information brought by the text-based annotations