# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 3 Experimental Results
### Attention Matrices  
In order to analyze whether the CFE encoder makes a better usage of the attention mechanisms than its non-causal counterpart, it is interesting to observe some actual examples and the attention matrices that they generate. These matrices are a representation of the decoder focus of interest while it was processing the input: The \(i\)-th row represents the \(i\)-th predicted character, and the \(j\)-th column is the model focus while predicting that character, i.e., the values of \(\alpha^{\prime}_{i}\) (see Equation (2)).  
The first case, shown in Table 9, is an example extracted from the test set of the first experiment, E1. The input phrase is \"_23 Aug 2013_\". Regarding what would be ideally expected from the attention matrix, it should show three different phases: (1) First, it outputs the day while focusing on its digits; (2) then, the attention is moved towards the month; and (3) it finishes by looking at the year. Figure 12 shows the four attention matrices obtained.  
Figure 11: Evolution of the training (red) and validation (blue) errors (NLLLoss) for the proposed CFE encoder using dataset E3. In this case, the horizontal scale represents the number of iterations. The training time was 22 h.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline  & & **Test** & \\ \hline
**Encoder** & **NLLLoss** & **CER (\%)** & **Acc (\%)** \\ \hline CFE & 1.701 & 5.44 & 92.74 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Results obtained by the proposed CFE encoder for the third experiment (E3). Encoder: Name of the encoder used. NLLLoss: Negative log-likelihood loss. CER: Character error rate. Acc (%): Accuracy.  
The matrix obtained by CFE (Figure 12d) is the one that most closely resembles what one would expect from a useful attention mechanism. It clearly presents the three phases of this prediction, in which the system selects the day, the month and the year. FE and LSTM also follow this scheme, although not so clearly defined.  
The second prediction selected to exemplify the use of the attention mechanism is taken from the test set of the second experiment, E2. It corresponds to the input sentence \"_Belpiela is a community in Tamale Metropolitan District in the Northern Region of Ghana.\"_ This sample has been specifically selected because it is a longer case where the input and output are identical; thus, the ideally expected attention matrices should resemble an identity matrix. Table 10 shows the predictions obtained by the four models, whereas Figure 13 depicts the corresponding attention matrices.  
\begin{table}
\begin{tabular}{l l l} \hline \hline Input & \multicolumn{2}{l}{_Belpiela is a community in Tamale Metropolitan District in the Northern Region of Ghana._} \\ \hline Output & \multicolumn{2}{l}{_Belpiela is a community in Tamale Metropolitan District in the Northern Region of Ghana._} \\ \hline LSTM & ✗ & _Belpiela is a community in Tamale Metropolitan Disire_ \\  & & _egion te i_ \(\quad\)\(\quad\)\(\quad\)\(\quad\)\(\quad\)\(\quad\)\(\  
Again, CFE is the only model that is able to produce a valid attention matrix, clearly resembling an identity. For this reason, it is the only method that was able to predict the correct output in this case. FE has a similar shape, but it produces some gaps which lead to a repetition of the word \"Region\" in the output.