# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 2 Materials and Methods
### Encoder-Decoder Architecture  
The encoder-decoder architecture is a common and popular design in recent Neural Machine Translation literature [16]. The model is composed of two parts: (1) An encoder that takes the input \(\mathbf{X}\) (in this case, a phrase), and produces an intermediate representation \(\mathbf{Z}\) (or code) that highlights its main features; and (2) a decoder that processes that set of features and produces the required output \(\mathbf{Y}\) (in this case, a normalized phrase). \(\mathbf{Z}\) is a matrix of size \(\mathbf{Z}\in\mathbb{M}_{f\times l}\), where \(f\) represents the selected number of features to encode for each input value. Figure 4 shows a basic diagram of this model. Figure 3: Sample entries from the preprocessed dataset. This is a simplified representation of the encoder-decoder architecture, involving that the size of the input, the intermediate representation and the output is the same. This can be the case, for example, of many image processing tasks. Nevertheless, in many NLP problems, they can have different sizes. In our case, the intermediate code always has the same length, \(l\), as the input, but the length of the output can be different. The end of the output is determined by the production of an \(<\)eos\(>\) symbol. Thus, a more precise diagram of the model is presented in Figure 5.  
There is a trend in using Long Short-Term Memory (LSTM) neural networks as encoders and decoders (for example, [PERSON] et al. [17]) due to their ability to capture long dependencies among the elements of a sequence. Our proposed model uses an LSTM network as decoder. However, different encoders have been analyzed, including the proposed one, and their performances have been tested and compared. Besides, some additional techniques that are common in the deep learning field were applied to improve the effectiveness of the system, such as batch processing (processing the input in batches of a certain size), dropout [18, 19] (randomly removing some neurons with a given probability), weight normalization [20] (regularizing the weights of the neuronal layers), gradient clipping [21] (limiting the norm of the gradient), and decaying learning rate combined with the Adam optimizer [22] (progressively reducing the learning rate used in the backpropagation algorithm). Attention Mechanisms  
The basic encoder-decoder model is a very powerful and useful architecture, but some key issues arise when it is put into practice. Two of them stand out and are worth mentioning: (1) As shown in Figure 4, at each time step, the decoder only works with the code produced at that moment, hindering the usage of long-term dependencies; and (2) output and input need to have the same length, as previously mentioned, limiting its application to many practical problems. We overcome these two restrictions by making use of attention mechanisms [23]. The idea behind them, depicted in Figure 5, is simple: First, produce the codes of the whole input sequence at once and, at each time step, let the decoder choose the most interesting elements of the input based on the latest output. Figure 4: A basic encoder–decoder architecture. Blue: Encoder. Red: Decoder. Given an input sequence, \(\mathbf{X}=x_{1}x_{2}\dots\), the system produces an output sequence, \(\mathbf{Y}=y_{1}y_{2}\dots\) The intermediate code is \(\mathbf{Z}=z_{1}z_{2}\dots\) There are two special symbols: \(<\)sos\(>\) start of sequence; \(<\)eos\(>\) end of sequence. This can be expressed in mathematical terms as follows. Suppose that \(\mathbf{Z}=z_{1}z_{2}\ldots z_{l}\) is the obtained intermediate representation. The attention mechanism consists of a fully connected neural network with one hidden layer, taking as input \(\mathbf{Z}\) and the vector \(h_{t}\) of the hidden state of the LSTM decoder at each time step \(t\). This network produces a vector of _interesting_ features \(a\in\mathbb{M}_{f\times 1}\).