# Evaluation of English-Slovak Neural and Statistical Machine Translation
## 3 Materials and Methods  
The aim of this study is to evaluate SMT (Google Translate and mt@ec, the European Commission's MT tool) and NMT systems (Google Translate and eTranslation, formerly mt@ec) for the translation direction of English to Slovak. Slovak, one of the official EU languages, is a synthetic language containing an inflected morphology and a loose word order [31]. The linguistic evaluation was performed using state-of-the-art automatic evaluation metrics (BLEU_n). We compared the differences between the NMT and SMT on a specific dataset. Two different MT systems were used for this purpose. The reference translation was obtained by two professional translators using our online system, OSTPERE (Online System for Translation, Post-Editing, Revision, and Evaluation), which was originally created for post-editing machine-translated documents [32; 33; 34]. The online system was built on a PHP platform and uses a MySQL database to store the text documents and their translations. The dataset consisted of 160 journalistic texts written in English. We translated all of the examined texts using both Google Translate (SMT and NMT) and the European Commission's MT tool (mt@ec and eTranslation). The composition of the created dataset is depicted in Table 1, and the dataset is available (Supplementary Data S1) [35]. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Feature Type** & **GT\_SMT** & **GT\_NMT** & **mt@ec\_SMT** & **E-translation\_NMT** & **Reference** \\ \hline Average sentence length (words) & 21.74 & 19.54 & 18.06 & 20.15 & 18.66 \\ Average word length (characters) & 5.32 & 5.43 & 5.6 & 5.4 & 5.53 \\ Frequency of long sentences (w \(\geq\) 10) & 84.83\% & 83.97\% & 73.42\% & 83.44\% & 81.87\% \\ Frequency of short sentences (w \(<\) 10) & 15.17\% & 16.03\% & 26.58\% & 16.56\% & 18.13\% \\ Frequency of nouns & 29.94\% & 31.00\% & 31.34\% & 29.40\% & 29.67\% \\ Frequency of adjectives & 9.23\% & 10.06\% & 10.25\% & 9.68\% & 10.25\% \\ Frequency of adverbs & 3.77\% & 3.30\% & 3.76\% & 3.56\% & 3.51\% \\ Frequency of verbs & 16.16\% & 15.02\% & 14.67\% & 16.00\% & 15.91\% \\ Frequency of pronominals & 7.32\% & 7.64\% & 6.88\% & 7.67\% & 8.67\% \\ Frequency of participles & 2.42\% & 1.81\% & 2.95\% & 2.40\% & 1.65\% \\ Frequency of morphemes & 1.65\% & 2.36\% & 1.61\% & 2.34\% & 3.07\% \\ Frequency of abbreviation & 0.52\% & 0.36\% & 0.88\% & 0.49\% & 0.25\% \\ Frequency of numbers & 1.74\% & 1.97\% & 1.97\% & 1.81\% & 1.33\% \\ Frequency of undefinable POSs & 0.42\% & 0.36\% & 0.39\% & 0.33\% & 0.32\% \\ Frequency of particules & 1.87\% & 2.20\% & 2.25\% & 2.01\% & 2.37\% \\ Frequency of foreign words & 3.19\% & 2.56\% & 2.21\% & 2.44\% & 1.83\% \\ Frequency of interjections & 0.00\% & 0.00\% & 0.04\% & 0.03\% & 0.00\% \\ Frequency of numerals & 3.10\% & 2.52\% & 2.74\% & 2.70\% & 2.78\% \\ Frequency of prepositions and conjunctions & 18.68\% & 18.83\% & 18.08\% & 19.13\% & 18.38\% \\ \hline \end{tabular}
\end{table} 
Table 1: Lexico-grammatical dataset composition. As the focus was on the evaluation of different MT systems and architectures, we had to create an approach to the experiment. We proceeded based on a methodology consisting of the following steps:  
1. Obtaining the unstructured text data (source text) of a journalistic style in English. 2. Text preparation--removing the document formatting. 3. Machine translation using various systems: 1. Google Translate--statistical machine translation, 2. Google Translate--neural machine translation, 3. mt@ec--statistical machine translation, 4. eTranslation--neural machine translation. 4. Human translation of the documents using the online system OSTPERE. 5. Text alignment--the segments of source texts were aligned with the generated MT system output, where each source text segment had its corresponding MT outputs and human translation output; this was done using the HunAlign tool [36]. 6.