# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 3 Experimental Results
### Proposed Methods and Number of Parameters  
As stated before, the presented encoder-decoder architecture for text normalization was implemented in Python, using Pytorch and OpenNMT. In order to analyze whether the proposed CFE achieves a significant improvement, different existing encoders were taken into account. These alternative encoders (and their aliases) are the following:  
* A 3 layer bidirectional LSTM network.
* A 4 layer fully CNN encoder, where the \(i\)-th element is an embedding of the \(i\)-th input, i.e., the encoded value only depends on the \(i\)-th input value.
* A traditional CNN with dilated convolutions. This is similar to the proposed CFE, but without considering causality.
* The proposed Causal Feature Extractor encoder.  
In all these cases, the only modification on the architecture resides in the the encoder. The decoder and attention mechanism remain always the same, that is, as they were described in the previous section. The hyperparameters of the models were manually tuned by trial and error, trying to obtain the best results. After that, the results presented here are averaged over five repetitions of the same models trained with different random seeds. Table 2 presents the selected hyperparameters.  
The number of parameters of the four models used in the experiments are shown in Table 3. These values correspond to the number of trainable parameters, i.e., the weights of the neural networks of each model, considering the encoder and the whole model. In general, the more parameters, the greater the complexity of the model is (and the greater the memory and time requirements are). So, for a similar performance, simpler models are usually preferred.  
It can be observed that LSTM requires nearly 10 times more parameters than FE and CFE, while FCNN requires 2.5 times more parameters. This translates into a lower efficiency and speed of convergence of these models. In any case, the rest of the system (attention and decoder) has a considerably larger number of parameters, with about 6.3 million values that have to be trained. To initialize these parameters, the uniform method of [PERSON] [28] was used.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Description** & **Value** \\ \hline \(b\) & Batch size & 128 \\ \(f\) & Number of features produced by the encoder & 256 \\ \(s\) & Size of the internal hidden vectors of LSTM & 128 \\ \(d\) & Number of columns of the context matrix & 10 \\ \(ml\) & Number of neurons of the intermediate dense networks & 256 \\ \(w\) & Width of the convolutional filters & 5 \\ \(rf\) & Receptive field to be considered in the input & 10 \\ \(lr\) & Initial learning rate of Adam algorithm & 0.001 \\ \(\beta\) & Multiplier used in the decay of the learning rate & 0.85 \\ \(step\) & Number of iterations before applying the decay & 400 \\ \(p_{d}\) & Probability of disabling a neuron in the dropout & 0.5 \\ \(p_{t}\) & Probability of substituting a weight by expected value & 0.4 \\ \(clip\) & Upper limit of the norm of the gradient in the clipping & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameters of the models used in the experiments.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Number of Parameters of** & **LSTM** & **FCNN** & **FE** & **CFE** \\ \hline Encoder (millions) & 1.102 & 0.285 & 0.111 & 0.111 \\ Total (millions) & 7.380 & 6.653 & 6.479 & 6.479 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Number of internal parameters of the models compared (in millions), considering only the encoder and the entire model (encoder + attention mechanism + decoder).