## 3 Spelling Correction
### Encoder-Decoder Neural Network Model  
The encoder-decoder model, also known as Sequence to sequence (Seq2 seq) [26; 29], is an approach of Neural Networks that consists of two sequential Recurrent Neural Networks (RNNs). The first RNN is responsible for encoding an input of variable-length symbol sequences into a fixed-length representation, while the second RNN decodes this fixed-length representation into another variable-length symbol sequences. A potential problem with the traditional Seq2 seq approach is that a neural network needs to be able to compress all the information needed from an input sentence to decode it later. This can make it difficult for the model to process very long sentences, especially those that are longer than training sentences [30]. According to [30], Attention mechanism allows the model to learn how to generate a context vector that best represents the input sentence at each stage of decoding; this means that the model learns to focus and pay more attention to the relevant parts of the input sequence. [PERSON]'s Attention (additive style) [30], and [PERSON]'s Attention (multiplicative style) [31] are considered the major approaches in the Seq2 seq models. Both mechanisms follow the same application, slightly changing the score vectors calculation:  
\[score(h_{t},h_{s})= \begin{cases}h_{t}^{\top}W_{a}h_{s}&\text{Multiplicative}\\ v_{a}^{\top}tanh(W_{a}h_{t}+U_{a}h_{s})&\text{Additive}\end{cases}  \tag{1}\]  
where \(h_{t}\) is the encoder output, \(h_{s}\) is the hidden state and \(v_{a}\), \(W_{a}\), \(U_{a}\) are weights' matrices. Thus, after the encoder produces hidden states of each element in the input sequence, the steps are defined: (i) the alignment scores (attention weights) are calculated between the hidden state of the previous decoder and the hidden states of each encoder, where \(T_{x}\) denotes the length of source:  
\[\alpha_{ts}=\frac{exp(score(h_{t},h_{s}))}{\sum_{k=1}^{T_{x}}exp(score(h_{t},h_ {k}))} \tag{2}\]  
(ii) the context vector is calculated by multiplying the alignment scores by their respective hidden states of the encoder:  
\[c_{t}=\sum_{s}\alpha_{ts}h_{s} \tag{3}\](iii) the context vector is concatenated with the previous decoder output (attention vector), feeding the decoder at the current time-step to produce a new output:  
\[a_{t}=f(c_{t},h_{t})=tanh(W_{c}[c_{t};h_{t}]) \tag{4}\]  
These steps repeat themselves for each decoder time-step until the decoding process ends (\"<EOS>\" special character or maximum length). Most recently, a mechanism, called Multi-Head Attention, was proposed in a model architecture composed entirely of attention layers, the Transformer model [22], which makes use of Scaled Dot-Product Attention to relate different positions of a single sentence and calculate a representation. In this way, the Attention mechanism has the function of mapping vectors of a query and a set of key-value pairs, in an output. The output is calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a query compatibility function with the corresponding key. Then, the input data consist of queries and keys of dimension \(d_{k}\) and values of dimension \(d_{v}\). To obtain the weights of the values, a softmax function is applied to the dot product of the query with all the keys, divided each by \(\sqrt{d_{k}}\). The output matrix is defined as bellow:  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{5}\]  
where the attention function computes a set of queries simultaneously, packed into a \(Q\) matrix. The keys and values are also packed into \(K\) and \(V\) matrices, respectively.