## 7 Evaluation
### Deeper Insight into Errors  
As highlighted in the introduction, creating a high-quality speech corpus is inherently linked to manual labor. Normalization is a crucial step in the corpus creation process. For this reason, in the present research, we are interested in the types of errors that occur in automatic normalization and whether they can be efficiently corrected manually. To this end, we designed an evaluation tool that gives us more detailed results. The total number of errors on the word level, which was represented before with WER, is now split into the following categories:  
* Missing conversion--the literal and standard forms of a word differ, but the normalization algorithm did not convert the word, i.e., the word remained in literal form. * Wrong conversions--the literal and standard forms of a word are different, but the normalization algorithm made a conversion to yet another word form. * Unwarranted conversions--the literal and standard forms of a word are the same, but the normalization algorithm made a conversion, although none was needed. * Inserted words--the normalization algorithm produced more words in its output than were given in the input. * Deleted words--the normalization algorithm produced fewer words in its output than there were given in the input. In the context of the evaluation tool, we use the term \"conversion\" since it can be used to evaluate other types of text conversion, not only normalization. Table 3 shows examples of each error type where the literal form, the standard form, and an example of erroneous conversion from the system are given. Additionally, our tool also provides the percentage of correct words in the evaluated segments and the number of words that did not need conversion, i.e., they are the same in literal and standard form. To use our tool, we must provide the literal form file, the standard form file, and one or more files with results to be evaluated; this is in contrast to the WER calculation, which needs only the standard form and the evaluated segments. If several evaluated segments are provided, e.g., from different systems, a statistical significance test can be performed. We implemented bootstrap resampling (BR) and approximate randomization tests (AR), as they are the most often used tests in machine translations [43; 44], and we wanted to explore whether they behave similarly as in machine translation. Our implementation follows the presented algorithms in [45], whereby we implemented the approximate randomization test as a two-ended test. We also modified the final \(p\)-value calculation to resemble the calculation used in SacreBLEU [46], a tool for evaluating machine translation results, which was also used in our experiments. While the calculation used in [45] uses the count of samples exceeding the reference value and the number of trials, we incremented both numbers by 1 in our tool (see the algorithms below). Statistical tests are performed by making individual comparisons of the first system and each of the remaining systems. In the context of the statistical test, the first system is called the reference system. \(p\)-value results are provided for both tests for all remaining systems. The approximate randomization test is shown in Algorithm 1. As input, we have the number of trials \(n\) and all system outputs to be evaluated--test files. The first test file is treated as the reference system with a total number of \(S_{X}\) errors. Any other system with \(S_{Y}\) errors is then compared to the reference system, with _Diff_ being the actual difference in performance. Tuples are pairs of the same segment (line) in both files. Shuffling a tuple means swapping the segments between both files. Each tuple is shuffled with a probability of 0.5. After shuffling, we have two new files with total error counts \(S_{A}\) and \(S_{B}\). If this difference is larger than or equal to the actual difference, we increase the counter \(c\) by 1. To implement a two-sided test, we use absolute values for both differences.