# Evaluation of English-Slovak Neural and Statistical Machine Translation
## 3 Materials and Methods
Evaluation of the texts using automatic metrics of accuracy from the Python Natural Language Toolkit library, or NLTK, which provides an implementation of the BLEU score using the sentence_bleu function. In our research, we applied automatic metrics of accuracy [14]. The metrics of accuracy (e.g., precision, recall, BLEU) are based on the closeness of the MT output (\(h\)) with the reference (\(r\)) in terms of n-grams; their lexical overlap is calculated in (A) the number of common words (\(h\cap r\)), (B) the length (number of words) of the MT output, and (C) the length (number of words) of the reference. The higher the values of these metrics, the better the translation quality. BLEU (Bilingual Evaluation Understudy) [14], which is used in our research, is a geometric mean of n-gram precisions (\(p=A/B\)), and the second part is a brevity penalty (BP), i.e., a length-based penalty to prevent very short sentences as compensation for inappropriate translation. BLEU represents two features of MT quality--adequacy and fluency [7]. \[ \begin{array}{c}\mathit{BLEU\_n}=exp\sum_{n=1}^{N}w_{n}\log p_{n}\times BP, \\ p_{n}=\mathit{precision}_{n}=\frac{\sum_{S\in C}\sum_{n-gram\in S}count_{ matched}(n-gram)}{\sum_{S\in C}\sum_{n-gram\in S}count(n-gram)}, \\ \text{and}\ =\ \left\{\begin{array}{ll}1,&if\ h>r\\ e^{1-\frac{r}{r}},&if\ h\leq r\end{array}\right.,\end{array} \end{array}\]  
where \(S\) indicates hypothesis (\(h\)) and reference (\(r\)) in the complete corpus \(C\). 7. Comparison of the translation quality based on the system (GT, EC) and translation technology (SMT, NMT). We verified the quality of the machine translations using the BLEU_n (\(n\) = 1, 2, 3, and 4) automatic evaluation metrics. We tested the differences in MT quality--represented by the score of the BLEU_n automatic metrics--between the translations generated from Google Translate (GT_SMT and GT_NMT) and the European Commission's MT tool (EC_mt@ec and EC_e-translation). This resulted in the following global null hypotheses:  
The quality of machine translation (BLEU_n, \(n\) = 1, 2, 3, and 4) does not depend on the MT system (GT or EC) or the translation technology (SMT or NMT). To test for differences between dependent samples (BLEU_n: EC_SMT, GT_SMT, EC_NMT, and GT_NMT), we used adjusted univariate tests for repeated measures due to the failure of the sphericity assumption (Mauchley sphericity test--BLEU_1: \(W\) = 0.831, _Chi-Square_ = 29.149, \(df\) = 5, \(p\) < 0.001; BLEU_2: \(W\) = 0.725, _Chi-Square_ = 50.651, \(df\) = 5, \(p\) < 0.001; BLEU_3: \(W\) = 0.804, _Chi-Square_ = 34.495, \(df\) = 5, \(p\) < 0.001; BLEU_4: \(W\) = 0.618, _Chi-Square_ = 76.023, \(df\) = 5, \(p\) < 0.001). For all BLEU_n automatic metrics, the test is significant (\(p\) < 0.001), i.e., the assumption is violated. If the assumption of covariance matrix sphericity is not met, the type I error rate increases. We adjustedthe degrees of freedom using the Greenhouse-Geisser adjustment (\(df1=(T-1)\), \(df2=(T-1)(D-1)\)) for the F-test that was used, thus achieving the declared level of significance. \[adj.df1=\tilde{\epsilon}(T-1),\]  
\[adj.df2=\tilde{\epsilon}(T-1)(D-1),\]  
where \(T\) is the number of dependent samples (BLEU_n scores of the examined translations) and \(D\) is the number of cases (documents). After rejecting the global null hypotheses for individual BLEU_n, the Bonferroni adjustment was used as a post hoc test [37,38]. 8. Identification of the texts with the greatest distance between SMT and NMT [39]. We used residuals to compare scores of the BLEU_n automatic metrics of MT_SMT with MT_NMT at the document level. In our case, the residual analysis was defined as follows: \(\left(residual\ value\right)_{i}=\left(BLEU\_n\ score\ of\ NMT\ text\right)_{i}-\left(BLEU\_n\ score\ of\ SMT\ text\right)\)\({}_{i}i=1,2,\dots,D\), where \(D\) is the number of examined texts in the dataset, _NMT_ is a neural machine translation, and _SMT_ is a statistical machine translation.