# Text Normalization Using Encoder-Decoder Networks Based on the Causal Feature Extractor
## 2 Materials and Methods
### Statistical Significance Test  
When comparing the performance of various models, a critical aspect is to ensure that the obtained differences are statistically significant. It must be proved that those differences are significant, and not a mere product of the implicit variance of the training process. This is typically performed using some statistical test that asserts that the differences are actual differences up to some confidence level of probability, usually 95%.  
In this paper, we have applied the approximate randomization test [24], also known as random permutation test. This statistical test measures the probability of the outputs of two different models of being indistinguishable, i.e., the probability that, by just looking at the predictions, we cannot tell whether they come from different models. The main reasons for using this method are: (1) It is computationally efficient; (2) it is distribution-free, meaning that it does not make any assumptions on the measured distribution; and (3) it is model-free, that is, the only required resources to perform the test are the actual predictions, making it suitable for any type of model.  
Let us assume that the predictions of two different models are the ordered sets \(A=\{a_{i}\}_{i=1}^{n}\) and \(B=\{b_{i}\}_{i=1}^{n}\), and we have a function \(e\) that measures the similarity of the predictions with respect to the expected values, \(Y\); for example, in our case \(e\) is the accuracy measure, defined as the percentage of correctly predicted characters with respect to the length of the output. Then, we can define the function:  
\[t(A,B)=e(A,Y)-e(B,Y) \tag{3}\]  
We want to estimate the probability of obtaining an error bigger than \(t(A,B)\), assuming that both sets of predictions are indistinguishable, that is \(P(X\geq t(A,B)H_{0})\), where \(H_{0}\) is the null hypothesis (i.e., both models are not significantly different).  
The algorithm to approximate this value consists in repeating many times the following process: Randomly swap each element of the first set with its counterpart in the second set; and count the number of times that the total error difference, measured by \(t\), is greater or equal than the original one, that is, \(t(A,B)\). Figure 8 shows the pseudocode of this algorithm. A small \(p\)-value, e.g., below 0.05, 0.02 or 0.01, indicates that the null hypothesis has to be rejected, so the models are significantly different.  
In this test, the estimation of the \(p\)-value has an error itself, which is given by \(\sqrt{p(1-p)/R}\), where \(p\) is the obtained \(p\)-value, and \(R\) is the number of iterations. If this error is too large, the \(p\)-value is unreliable; hence, the number of repetitions has to be computed to reduce the error [25]. In order to force the upper bound of the confidence interval of the estimated \(p\) being below the decision threshold, we need to find an \(R\) such that \(l^{2}\alpha(1-\alpha)/P^{2}\leq R\), where \(l\) is the confidence interval. Using \(P\) at \(\alpha=0.05\) and requesting a 95% confidence interval, we get \(R\geq 7600\). Consequently, that is the number of repetitions used in the tests.