# Readability Metrics for Machine Translation in Dutch: Google vs. Azure & IBM
## 3 Methods
### Construction of the Manually Classified Dataset  
Figure 2 graphically displays the dataset creation. Because this study originates from research on translating question and answering datasets, the SQUAD 2.0 dataset is chosen [38]. The SQUAD dataset contains a variety of subjects. The dataset is divided into paragraphs, questions, and answers. In May 2020, a translation request was executed at three MT cloud providers: Microsoft Azure, IBM Cloud and Google Cloud. The titles, paragraphs, questions, and answers are translated from English to Dutch. Figure 1: Global research design. The first part is dataset creation, and the second part is analysis. A random selection of paragraphs from the SQUAD dataset is taken for selecting sentences. Then, the original English paragraph and the Dutch translated paragraphs are divided with a sentence splitter of the NLTK package in Python (sent_tokenize) [39]. The paragraphs without an equal count of sentences compared with the source text were ruled out. The reason for ruling these sentences out is that it is more difficult to select them automatically. From the rest of the sentences, 146 sentences are selected. Because the SQUAD dataset also contains questions, 67 questions were added as sentences to the dataset. These questions are randomly chosen from all the SQUAD questions. Next, the machine-translated sentences are classified in the best-worst scaling [40]. Sentences are positioned as best translated (1) to least well translated (3). The task presented the sentences in random order, and the ids of the different sentences were hashed so that a classifying participant could never see the MT provider behind the sentence. Thereby, the annotator could give extra information about the rating process. Table 1 shows an example of the task. As shown in Table 1, the annotators could give extra information about their best-worst scaling. In the task. 'No extra information' was default selected. Sometimes the annotators were confused or had doubts about the selection and could specify that further under the best-worst scale. They could have doubts about two or all translated sentences. After classifying the sentences on the best-worst scale by one human rater (author), a ranking is created, visible in Table 2. Remarkably, Google translations score far better than the other providers. The same findings were also with the other annotators. Google translations are hardly scaled as worst. \begin{table}
\begin{tabular}{p{42.7 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Position** & **Provider** & **Sentences** & **Drag \& Drop** \\ \hline \multirow{5}{*}{1} & Original & By the early 20 th century balloon, or airship, guns, for land and naval use were attracting attention. & \\ \cline{2-3}  & Google & Tegen het begin van de 20e eeuw trokken de ballon, of het luchtschip, kanonnen voor land- en marinegebruik de aandacht. & \(\ddagger\) \\ \cline{2-3}  & Azure & Door het begin van de 20e eeuw ballon, of luchtschip, geweren, voor land en marine gebruik waren het aantrekken van de aandacht. & \(\ddagger\) \\ \cline{2-3}  & IBM & Door de vroege 20e-eeuwse ballon, of luchtschip, wapens, voor land en marine gebruik trok aandacht. & \(\ddagger\) \\ \cline{2-3}  & No extra information & \\ \(\oslash\) I’ve doubts between position 1 and 2 & \\ \(\oslash\) I’ve doubts between position 2 and 3 & \\ \(\oslash\) They are all equal to me & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Example of different machine-translated texts with the original. In this example, Google is selected as best. Figure 2: Dataset creation. First selecting the sentences, second translating the sentences from English to Dutch, third a human best-worst scale rating, fourth putting the texts into T-Scan and last combining the translations, T-Scan output and best-worst scale into one dataset. To ensure the classification is correct, other annotators are added to expose a kappa score to test the correctness of the human-annotated dataset. Five different annotators checked the classifications of the first annotator.