# Named Entity Correction in Neural Machine Translation Using the Attention Alignment Map
## 3 Proposed Method  
### AAM in Sequence-to-Sequence Models  
In this subsection, we describe the seq2 seq model [13] used in our method and explain how to obtain an AAM from it. The seq2 seq model consists of an LSTM-based [11] encoder-decoder and an attention mechanism. The encoder encodes a sequence of input tokens \(\mathbf{\hat{x}}=(\mathbf{\hat{x}_{1}},\mathbf{\hat{x}_{2}},\ldots,\mathbf{ \hat{x}_{n}})\), represented as dense vectors, into a context vector \(c\), which is a fixed  
Figure 2: Framework of the transformer model. The red box is the encoder, and the blue one is the decoder. The outputs of the encoder are fed into the encoder-decoder attention layers. All residual connections and normalization layers are omitted.  
length vector. We used a bidirectional LSTM (BiLSTM) [40; 41; 42; 43] as the encoder to capture bidirectional context information of the input sentences.  
\[\overrightarrow{h_{i}} =f(\hat{x}_{i},\overrightarrow{h_{i-1}}), \tag{8}\] \[\overleftarrow{h_{i}} =f(\hat{x}_{i},\overleftarrow{h_{i+1}}), \tag{9}\]  
where \(f\) are stacked unidirectional LSTM cells and \(\overrightarrow{h_{i}}\in\mathbb{R}^{d}\) and \(\overleftarrow{h_{i}}\in\mathbb{R}^{d}\) are the hidden states of the top forward LSTM cell and the top backward cell, respectively. Moreover, \(i\) indicates the encoding steps, and \(d\) is the number of hidden units of the top LSTM cell in the encoder.  
\[c=[\overrightarrow{h_{n}};\overleftarrow{h_{1}}] \tag{10}\]  
Hidden states at the last encoding step for both directions \(\overrightarrow{h_{n}}\) and \(\overleftarrow{h_{1}}\) are concatenated to obtain \(c\in\mathbb{R}^{2d}\). Stacked unidirectional LSTM cells are used for the decoder. Once the encoder produces context vector \(c\), the bottom LSTM cell of the decoder is initialized with \(c\).  
\[s_{j=1}=c, \tag{11}\]  
where \(s_{j}\in\mathbb{R}^{2d}\) are the hidden states of the bottom LSTM cell in the decoder, and subscript \(j\) denotes the index of the decoding step. Next, the decoder starts the process of decoding:  
\[p(y_{j})=g(y_{j-1},\ s_{j},\ o_{j})\,. \tag{12}\]  
Each decoding step computes the probability of the next token using three components. The first is the previously generated token \(y_{j-1}\); the second is the current hidden state \(s_{j}\); the third is an attention output vector \(o_{j}\). An attention output allows the decoder to retrieve hidden states of encoder \(\{h_{1},\cdots,h_{n}\}\), where \(h_{i}=[\overrightarrow{h_{i}};\overleftarrow{h_{i}}]\).  
\[o_{j} =\sum_{i=1}^{n}a_{ij}h_{i}\,, \tag{13}\] \[a_{ij} =\frac{exp(e_{ij})}{\sum_{k=1}^{n}exp(e_{kj})}\,,\] (14) \[e_{ij} =v_{a}^{T}\tanh(W^{a}s_{j-1}+U^{a}h_{i}+b^{a})\,. \tag{15}\]  
Attention scores \(e_{ij}\) indicate how related \(s_{j-1}\) is to \(h_{i}\). Here, \(W^{a}\in\mathbb{R}^{d_{a}\times 2d}\), \(U^{a}\in\mathbb{R}^{d_{a}\times 2d}\), \(v_{a}\in\mathbb{R}^{d_{a}}\), and \(b^{a}\in\mathbb{R}^{d_{a}}\) are trainable parameters, where \(d_{a}\) is the hidden dimension of the attention mechanism. Attention weights are computed by the softmax function across the attention scores. The attention output \(o_{j}\) is a weighted sum of hidden states from the encoder \(\{h_{1},\cdots,h_{n}\}\). It tells the decoder where to focus on the input sentence when the decoder generates the next token.  
In the seq2 seq model, an attention alignment map (AAM) \(\mathbf{A}\in\mathbb{R}^{n\times m}\), where \(n\) and \(m\) represent the sequence lengths of the input and output sentences, can be easily computed by stacking up the results of (14) while the model generates the translation. Figure 1 illustrates the framework of the seq2 seq model used in this paper.