Question: Why is global attention mechanism preferred?
Evidence: The model used for translation is the one implemented by Bahdanau et al. Bahdanau2014. A bidirectional LSTM encoder first takes the source sentence and encodes it into a context vector which acts as input for the decoder. The decoder is attention-based where the hidden states of the decoder get as input the weighted sum of all the hidden layer outputs of the encoder alongwith the output of the previous hidden layer and the previously decoded word. This provides a contextual reference into the source language sentence BIBREF4 .
Neural Machine Translation models directly compute the probability of the target language sentence given the source language sentence, word by word for every time step. The model with a basic decoder without the attention module computes the log probability of target sentence given source sentence as the sum of log probabilities of every word given every word before that. The attention-based model, on the other hand, calculates: DISPLAYFORM0 
where INLINEFORM0 is the number of words in the target sentence, INLINEFORM1 is the target sentence, INLINEFORM2 is the source sentence, INLINEFORM3 is the fixed length output vector of the encoder and INLINEFORM4 is the weighted sum of all the hidden layer outputs of the encoder at every time step. Both the encoder's output context vector and the weighted sum (known as attention vector) help to improve the quality of translation by enabling selective source sentence lookup.
The decoder LSTM computes: DISPLAYFORM0 
where the probability is computed as a function of the decoder's output in the previous time step INLINEFORM0 , the hidden layer vector of the decoder in the current timestep INLINEFORM1 and the context vector from the attention mechanism INLINEFORM2 . The context vector INLINEFORM3 for time step INLINEFORM4 is computed as a weighted sum of the output of the entire sentence using a weight parameter INLINEFORM5 : DISPLAYFORM0 
where INLINEFORM0 is the number of tokens in the source sentence, INLINEFORM1 refers to the value of the hidden layer of the encoder at time step INLINEFORM2 , and INLINEFORM3 is the alignment parameter. This parameter is calculated by means of a feed forward neural network to ensure that the alignment model is free from the difficulties of contextualization of long sentences into a single vector. The feed forward network is trained along with the neural translation model to jointly improve the performance of the translation. Mathematically, DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 is the softmax output of the result of the feedforward network, INLINEFORM1 is the hidden state value of the decoder at timestep INLINEFORM2 and INLINEFORM3 is the encoder's hidden layer annotation at timestep INLINEFORM4 . A concatenation of the forward and the reverse hidden layer parameters of the encoder is used at each step to compute the weights INLINEFORM5 for the attention mechanism. This is done to enable an overall context of the sentence, as opposed to a context of only all the previous words of the sentence for every word in consideration. Fig. FIGREF12 is the general architecture of the neural translation model without the Bidirectional LSTM encoder.
A global attention mechanism is preferred over local attention because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence. Using local attention mechanism with a monotonic context lookup, where the region around INLINEFORM0 source word is looked up for the prediction of the INLINEFORM1 target word, is impractical because of the structural discordance between the English and Tamil sentences (see Figs. FIGREF37 and FIGREF44 ). The use of gaussian and other such distributions to facilitate local attention would also be inefficient because the existence of various forms of translations for the same source sentence involving morphological and structural variations that don't stay uniform through the entire corpus BIBREF5 .
The No Peepholes (NP) variant of the LSTM cell, formulated in Greff et al. greff2015lstm is used in this experiment as it proved to give the best results amongst all the variants of an LSTM cell. It is specified by means of a gated mechanism designed to ensure that the vanishing gradient problem is prevented. LSTM maintains its hidden layer in two components, the cell vector INLINEFORM0 and the actual hidden layer output vector INLINEFORM1 . The cell vector is ensured to never reach zero by means of a weighted sum of the previous layer's cell vector INLINEFORM2 regulated by the forget gate INLINEFORM3 and an activation of the weighted sum of the input INLINEFORM4 in the current timestep INLINEFORM5 and the previous timestep's hidden layer output vector INLINEFORM6 . The combination is similarly regulated by the input gate INLINEFORM7 . The hidden layer output is determined as an activation of the cell gate, regulated by the output gate INLINEFORM8 . The interplay between these two vectors ( INLINEFORM9 and INLINEFORM10 ) at every timestep ensures that the problem of vanishing gradients doesn't occur. The three gates are also formed as a sigmoid of the weighted sum of the previous hidden layer output INLINEFORM11 and the input in the current timestep INLINEFORM12 . The output generated out of the LSTM's hidden layer is specified as a weighted softmax over the hidden layer output INLINEFORM13 . The learnable parameters of an LSTM cell are all the weights INLINEFORM14 and the biases INLINEFORM15 . DISPLAYFORM0 
The LSTM specified by equations 7 through 11 is the one used for the decoder of the model. The encoder uses a bidirectional RNN LSTM cell in which there are two hidden layer components INLINEFORM0 and INLINEFORM1 that contribute to the output INLINEFORM2 of each time step INLINEFORM3 . Both the components have their own sets of LSTM equations in such a way that INLINEFORM4 for every timestep is computed from the first timestep till the INLINEFORM5 token is reached and INLINEFORM6 is computed from the INLINEFORM7 timestep backwards until the first token is reached. All the five vectors of the two components are all exactly the same as the LSTM equations specified with one variation in the computation of the result. DISPLAYFORM0 
A global attention mechanism is preferred over local attention because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence.
Answer: Because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence.