# Evaluation of English-Slovak Neural and Statistical Machine Translation
## 4 Results
### MT Text Identification  
Based on the previous results (Section 4.1), we further examined the MT texts in which the highest differences in the BLEU_n scores given to the MT systems (GT_SMT, GT_NMT, mt@ec_SMT, and eTranslation_NMT) used for translation from English into Slovak were found. To identify these texts, residuals were used [39; 40]. To identify extreme values (Figure 3a,b, Figure 4a,b, Figure 5a,b, and Figure 6a,b) we used a rule of \(\pm\) 2\(\sigma\), i.e., values outside the interval were considered as extremes. Figure 2: Mean estimations of the points and intervals for the automatic metrics: (**a**) BLEU_3 and (**b**) BLEU_4. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**(a)**} & \multicolumn{4}{c}{**(b)**} \\ \hline BLEU\_3 & GT\_SMT & GT\_NMT & EC\_SMT & EC\_NMT & BLEU\_4 & GT\_SMT & GT\_NMT & EC\_SMT & EC\_NMT \\ \hline GT\_SMT & & 0.00000 & 0.07270 & 0.00233 & GT\_SMT & & 0.00004 & 0.46602 & 0.12269 \\ GT\_NMT & 0.00000 & & 0.00000 & 0.12649 & GT\_NMT & 0.00004 & & 0.00000 & 0.14572 \\ EC\_SMT & 0.07270 & 0.00000 & & 0.00000 & EC\_SMT & 0.46602 & 0.00000 & & 0.00030 \\ EC\_NMT & 0.00233 & 0.12649 & 0.00000 & & EC\_NMT & 0.12269 & 0.14572 & 0.00030 & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Multiple comparisons—Bonferroni adjustment: (**a**) BLEU_3 and (**b**) BLEU_4. In the case of BLEU_1, we identified five texts (ID_163, ID_183, ID_244, ID_267, and ID_283) that showed a significant accuracy of GT_NMT compared to GT_SMT; vice-versa, only four texts (ID_156, ID_221, ID_247, and ID_280) showed a significant accuracy of GT_SMT compared to GT_NMT (Figure 3a). In the case of the EC's tool, we achieved different results. Based on BLEU_1, we identified six texts (ID_151, ID_156, ID_163, ID_180, ID_221, and ID_280) that showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT; vice-versa, only two texts (ID_169 and ID_235) showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT (Figure 3b). In the case of BLEU_2, we identified four texts (ID_183, ID_240, ID_244, and ID_283) that showed a significant accuracy of GT_NMT compared to GT_SMT; vice-versa, only two texts (ID_156 and ID_192) showed a significant accuracy of GT_SMT compared to GT_NMT (Figure 4a). In the case of the EC's tool, we achieved the following results: We identified eight texts (ID_142, ID_156, ID_165, ID_166, ID_180, ID_240, ID_279, and ID_280) that showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT; vice-versa, only three texts (ID_178, ID_223, and ID_274) showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT (Figure 4b). Based on BLEU_3, we identified seven texts (ID_165, ID_180, ID_183, ID_205, ID_232, ID_240, and ID_242) that showed a significant accuracy of GT_NMT compared to GT_SMT; vice-versa, only two texts (ID_156 and ID_192) showed a significant accuracy of GT_SMT compared to GT_NMT (Figure 5a). In the case of the EC's tool, we identified six texts (ID_156 and ID_192) and (ID_280) that showed a significant accuracy of GT_NMT compared to GT_NMT compared to mt@ec_SMT (Figure 5b). Figure 4: Visualization of BLEU_2 residuals for the MT outputs: (**a**) Google Translate; (**b**) the EC’s tool. Figure 3: Visualization of BLEU_1 residuals for the machine translation (MT) outputs: (**a**) Google Translate; (**b**) the European Commission’s (EC’s) tool. (ID_142, ID_156, ID_165, ID_166, ID_205, and ID_240) that showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT; vice-versa, only two texts (ID_223 and ID_274) showed a significant accuracy of eTranslation_NMT compared to mt@ec_SMT (Figure 5b). In the case of BLEU_4, we identified seven texts (ID_162, ID_165, ID_180, ID_183, ID_205, ID_232, and ID_240) that showed a significant accuracy of GT_NMT compared to GT_SMT; vice-versa, only one text (ID_156) showed a significant accuracy of GT_SMT compared to GT_NMT (Figure 6a).