# Named Entity Correction in Neural Machine Translation Using the Attention Alignment Map
## 2 Machine Translation  
### Neural Machine Translation  
NMT maps a source sentence to a target sentence with neural networks. In a probabilistic representation, the NMT model is required to map a given source sentence \(\mathbf{X}=\left[\mathbf{x}_{1}\,\mathbf{x}_{2}\,\cdots\,\mathbf{x}_{n}\right] \in\mathbb{B}^{n_{x}\times n}\) to a target sentence \(\mathbf{Y}=\left[\mathbf{y}_{1}\,\mathbf{y}_{2}\,\cdots\,\mathbf{y}_{m} \right]\in\mathbb{B}^{n_{t}\times m}\), where \(\mathbb{B}=\left\{0,1\right\}\), a binary domain space, \(v_{s}\) and \(v_{t}\) are the source (input) and target (output) vocabulary size, and \(n\) and \(m\) represent the sequence lengths of the input and output sentences, respectively. A vocabulary is usually defined by a set of tokens, which is a minimum processing unit for natural language processing (NLP) models. From a linguistic point of view, words or characters are the most popular mapping units for the tokens, depending on the grammar of the source and target languages. Each element of the encoding vector is assigned a positive integer index that uniquely identifies a single token in the corresponding vocabulary, so we can construct source vectors \(\mathbf{x}_{k}\in\mathbb{B}^{n_{x}}\) by the following one-hot representation:  
\[x_{k,i}=\left\{ \begin{array}{ll}1&\text{if $i=\text{index of $k$th token}$}\\ 0&\text{otherwise}\end{array} \right., \tag{1}\]  
where \(x_{k,i}\) is the \(i\)th element of \(\mathbf{x}_{k}\). We can also construct a one-hot representation for the target vector \(\mathbf{y}_{k}\) in a similar manner as well. The one-hot representation is extremely sparse, and the dimensions of input and target vector, \(n\) and \(m\), may become too large to handle for the large vocabulary sizes. The embedding method, a general approach in natural language processing, is introduced to produce dense vector representations for the one-hot encoding vectors [33; 34; 35; 36]. For given dimensions of the source and target, \(d_{s}\) and \(d_{t}\) with \(d_{s}\ll v_{s}\) and \(d_{t}\ll v_{t}\), linear embeddings from a higher dimensional binary space to a lower dimensions real domain are defined as follows:  
\[\mathbf{E}^{s}\in\mathbb{R}^{d_{s}\times v_{s}}\,,\qquad\mathbf{ E}^{t}\in\mathbb{R}^{d_{t}\times v_{t}}\,,\] \[\tilde{\mathbf{x}}_{i}=\mathbf{E}_{s}\mathbf{x}_{i}\in\mathbb{R} ^{d_{s}}\,,\qquad\tilde{\mathbf{y}}_{j}=\mathbf{E}_{t}\mathbf{y}_{i}\in \mathbb{R}^{d_{t}}\,, \tag{2}\]  
where \(\mathbb{R}\) is the real number space and \(\mathbf{E}^{s}\) and \(\mathbf{E}^{t}\) are the source and target embedding matrices, respectively. Applying the linear embedding in (2), dense representation for the source and the target sentences are obtained by multiplying \(\mathbf{E}_{s}\) and \(\mathbf{E}_{t}\) to \(\mathbf{X}\) and \(\mathbf{Y}\),  
\[\tilde{\mathbf{X}} = \left[\tilde{\mathbf{x}}_{1}\,\tilde{\mathbf{x}}_{2}\,\cdots\, \tilde{\mathbf{x}}_{n}\right]\,=\,\mathbf{E}_{s}\mathbf{X}\in\mathbb{R}^{d_{s} \times n} \tag{3}\] \[\tilde{\mathbf{Y}} = \left[\tilde{\mathbf{y}}_{1}\,\tilde{\mathbf{y}}_{2}\,\cdots\, \tilde{\mathbf{y}}_{n}\right]\,=\,\mathbf{E}_{t}\mathbf{Y}\in\mathbb{R}^{d_{t} \times n}\,. \tag{4}\]  
This linear transformation is one of the Word2 Vec methods [33]. In our paper, we use this embedding for all the input and target one-hot vectors. The target of machine translation is finding a mapping that maximizes the conditional probability \(p(\mathbf{Y}|\mathbf{X})\). The direct approximation of \(p(\mathbf{Y}|\mathbf{X})\) is intractable due to the high dimensionality, so most of recent NMT models are based on an encoder-decoder architecture [34]. The encoder reads an input sentence \(\tilde{\mathbf{X}}\) in a dense representation and encodes it into an intermediate, contextual representation \(\mathbf{C}\). \[\mathbf{C}=\text{Encoder}(\tilde{\mathbf{X}})\,, \tag{5}\]  
where \"Encoder\" is a neural network model for deriving contextual representation. After the encoding process, the decoder starts generating a translated sentence.