# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 3 Experimental Results
### First Experiment  
The purpose of this first experiment is to compare the accuracy achieved by the different methods in a reduced setup using the shortest entries containing: 50,000 training samples, 10,000 test samples, and 10,000 validation samples, as presented in Table 1. These values have been chosen so that the number of samples is large enough to train an accurate normalization model, but also small enough to require a reasonable training time that allows multiple repetitions. Specifically, training is limited to only 1 h in all cases. The obtained results are shown in Table 4. Observe that these results are averaged over 5 runs and extracted from the test set, except from the results concerning the training speed, which are taken from the training logs. From left to right, the columns of Table 4 indicate the following parameters:  
* Negative Log-Likelihood Loss (NLLLoss). It is the measure optimized by the neural networks during training, since it is the common measure used in classification problems.
* Character Error Rate (CER). It is defined as the mean Levenshtein distance [29] between the prediction and the expected value, that is, the minimum number of character insertions/deletions/substitutions required to change one sentence into the other, for all the test samples.
* Accuracy. It is a basic and well-known measure, defined as the percentage of correct output values, measured at a character level.
* Number of iterations performed during the training phase in the duration of the experiment (in this case 1 h).
* Rate. Number of iterations per second, on average, achieved during training.  
In order to get an in-depth view of the differences in the training process, Figure 9 shows the evolution of the NLLLoss of the validation samples for each model during training. Table 5 shows the resulting \(p\)-values after running the approximate randomization test for each pair of models.  
In this first experiment, CFE is clearly able to achieve the best results in terms of accuracy, CER and NLLLoss of the test set. It is interesting to observe that, although LSTM, FE and CFE tend to converge to the same NLLLoss values on the validation set, as shown in Figure 9, the differences are more prominent on the test set. Thus, the proposed method has a greater capacity for generalization on previously unobserved samples. Moreover, the statistical tests in Table 5 prove that these differences are significant. The probability that the results from FE and CFE are equivalent is below 2%. FCNN was unable to provide correct results, producing a very large character error rate. Concerning the computational efficiency, FE was able to execute almost 2 iterations per second. CFE is a 10% slower, but it is faster than the remaining methods.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Validation**} \\ \hline
**Encoder** & **NLLLoss** & **CER (\%)** & **Acc (\%)** & **No. Iters** & **Rate** \\ \hline LSTM & 1.352 & 3.13 & 95.87 & 3620 & 1.005 \\ FCNN & 5.035 & 70.61 & 28.40 & 4370 & 1.214 \\ FE & 1.042 & 2.52 & 96.46 & 6980 & 1.939 \\ CFE & 0.952 & 2.24 & 96.83 & 6300 & 1.750 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results obtained by the four models for the first experiment (E1). Encoder: Name of the encoder used. NLLLoss: Negative log-likelihood loss. CER: Character error rate. Acc (%): Accuracy. No. iters: Number of iterations. Rate: Iterations per second.