# Named Entity Correction in Neural Machine Translation Using the Attention Alignment Map
## 3 Proposed Method
### Transformer  
In this subsection, we describe the transformer model [16] used in our method and explain how to obtain an AAM from it. The transformer also consists of an encoder and a decoder. Unlike the seq2 seq model, it introduces a position encoding [16; 44] to add positional information into the model, because it does not have any recurrent units that would model positional information automatically. \[x^{\prime}_{i} =\bar{x}_{i}+PE(i)\,, \tag{16}\] \[y^{\prime}_{j} =\bar{y}_{j}+PE(j)\,. \tag{17}\]  
Here, \(PE(k)\in\mathbb{R}^{d_{model}}\) produces a position encoding vector that corresponds to position \(k\), \(d_{model}\) is the dimensionality of the model, and \(i\) and \(j\) are the positional indices of the input and output sentence, respectively. The positional encoding vectors are added to a sequence of input tokens \(\mathbf{\tilde{x}}=(\bar{x}_{1},\bar{x}_{2},\dots,\bar{x}_{n})\) represented as dense vectors as in (2). The sum of embedding vectors and positional encoding \(\mathbf{x}^{\prime}=(x^{\prime}_{1},x^{\prime}_{2},\dots,x^{\prime}_{n})\in \mathbb{R}^{d_{model}\times n}\) are fed into the bottom encoding layer. The encoder is a stack of encoding layers, where each encoding layer is composed of a self-attention layer and a feedforward layer. The self-attention layer of the bottom encoding layer takes \(\mathbf{x}^{\prime}\), and the others receive the outputs of the encoding layer right below them. Self-attention layers allow the model to refer to other tokens in the input sequence. \[A^{enc}_{h}=\text{softmax}\Bigg{(}\frac{Q_{h}K^{T}_{h}}{\sqrt{d_{k}}}\Bigg{)} \tag{18}\] \[\text{Attention}(Q_{h},K_{h},V_{h})=A^{enc}_{h}V_{h}\] (19) \[\text{head}_{h}=\text{Attention}(Q_{h},K_{h},V_{h})\] (20) \[\text{MultiHead}(Q,K,V)=\] (21) \[\text{Concat}(\text{head}_{1},\cdots,\text{head}_{h_{n}})\,W^{O}\,.\]  
The \"multihead scaled dot-product attention\" is computed by the above equations and was proposed by [PERSON] et al. [16]. Here, \(Q_{h}\), \(K_{h}\), and \(V_{h}\) are linear transformations of its input. \(Q_{h}=\mathbf{z_{e}^{T}}\cdot W_{h}^{Q}\), \(K_{h}=\mathbf{z_{e}^{T}}\cdot W_{h}^{K}\), and \(V_{h}=\mathbf{z_{e}^{T}}\cdot W_{h}^{V}\), where \(\mathbf{z_{e}}\in\mathbb{R}^{d_{model}\times n}\) is the input of each attention layer in the encoder. \(A^{enc}_{h}\) denotes an AAM of the encoder self-attention layer on the \(h\)th head. Moreover, \(W_{h}^{Q}\in\mathbb{R}^{d_{model}\times d_{k}}\), \(W_{h}^{K}\in\mathbb{R}^{d_{model}\times d_{k}}\), \(W_{h}^{V}\in\mathbb{R}^{d_{model}\times d_{k}}\), and \(W^{O}\in\mathbb{R}^{h_{d}d_{k}\times d_{model}}\) are trainable parameters, and \(d_{k}=d_{v}=d_{model}/h_{n}\), where \(h_{n}\) is the number of heads. \[\text{FFN}(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}\,. \tag{22}\]  
The outputs of the self-attention layers pass through the feedforward network. Each position is processed independently and identically. Here, \(W_{1}\in\mathbb{R}^{d_{model}\times d_{ff}}\), \(W_{2}\in\mathbb{R}^{d_{ff}\times d_{model}}\), \(b_{1}\in\mathbb{R}^{d_{ff}}\), and \(b_{2}\in\mathbb{R}^{d_{model}}\) are learnable parameters, where \(d_{ff}\) is the dimensionality of the inner linear transformation. The final output of the encoder is considered as contextual representation \(\mathbf{c}=(c_{1},c_{2},\dots,c_{n})\in\mathbb{R}^{d_{model}\times n}\) as in (5). It is fed into the encoder-decoder attention layers of the decoder. The decoder has a stack of decoding layers, where each decoding layer consists of a self-attention layer, encoder-decoder attention, and a feedforward network. By analogy to the encoder, the bottom decoding layer takes the sum of embedding vectors and positional encoding \(\mathbf{y}^{\prime}=(y^{\prime}_{1},y^{\prime}_{2},\dots,y^{\prime}_{m})\in \mathbb{R}^{d_{model}\times m}\), as in (17), and the others receive the outputs of the decoding layer right below them. Self-attention layers in the decoder are similar to those in the encoder. However, the model can only retrieve the earlier positions at the current step.