# Low-Resource Neural Machine Translation Improvement Using Source-Side Monolingual Data
## 5 Methodology
### Experimental Architecture
#### 5.4.2 Self-Learning
_BLEU_ is defined as

\[BLEU=BP4\times\left(\prod_{n=1}^{4}\times pn\right)\]

where \(pn\) measures the modified n-gram precision between a document with candidate translations and a set of human-authored reference documents, and the brevity penalty \(BP\) downscales the score for outputs shorter than the reference. _BLEU_ measures the closeness of the machine translation to the human reference translation taking into consideration the translation length, word choice, and word order. #### 5.5.2 Translation Edit Rate (TER)

TER is a character-based automatic metric for measuring the number of edit operations needed to transform the machine-translated output into a human-translated reference [41]. TER is defined as the minimum number of edits needed to change a hypothesis (i.e., translation) so that it exactly matches one of the references, normalized by the average length of the references. Because we were concerned with the minimum number of edits needed to modify the hypothesis, we measured only the number of edits to the closest reference specifically [41]:

\[TER=\frac{\#edit\ average}{\#of\ referencewords}.\]

Possible edits include insertion, deletion, and substitution of single words as well as shifts of word sequences. The dataset TER score is the total number of edits divided by the total number of words and multiplied by 100. TER ranges from 0 to infinity. The greater the score, the farther the translation is from the reference. #### 5.5.3 Character-Level F-Score (chrF)

chrF is a tool for the automatic evaluation of machine translation output based on character n-gram precision and recall, enhanced with word n-grams [42]. It calculates the F-score averaged on all the character and word n-grams, with character n-gram order set to 6 and word n-gram order set to 2. chrF score calculated as

\[(1+\beta 2)\frac{CHRP\times CHRR}{\beta 2\times CHRP+CHRR},\]

where _CHRP_ is the percentage of n-grams in the hypothesis, which have a counterpart in the reference (character n-gram precision), and _CHRR_ is the percentage of character n-grams in the reference, which is also present in the hypotheses (character n-gram recall). _CHRP_ and _CHRR_ are averaged over all n-grams, and \(\beta\) is the parameter to assign \(\beta\) times more importance to recall than to precision. If \(\beta=1\), the precision and recall have the same importance. ## 6 Results

In this section, we discuss the results of our experiments. In Section 6.1, we show the results of our baseline NMT models in both Wolaytta-English and English-Wolaytta translations. In Section 6.2, we present the results of the self-trained NMT models using both synthetic and authentic datasets. Finally, in Section 6.3, we give the results of our final fine-tuned NMT model on the authentic dataset. ### Baseline

As discussed in Section 5.4.1, to select the best-performing model, we trained and evaluated three baseline models on the authentic parallel dataset. Table 4 shows the BLEU, chrF, and TER scores of the three baseline models in Wolaytta-English translation. It can be seen in Table 4 that the best-performing baseline model is the transformer model. Similarly, Table 5 shows that for the English-Wolaytta translation, the transformer model outperformed the other models. In Section 7, we discuss the performance of the baseline models in detail. \begin{table}
\begin{tabular}{c c c c c} \hline
**Model (Wolaytta–English)** & **BLEU (\%)** & **chrF** & **TER** & **Loss** \\ \hline LSTM & 6.0 & 23.9 & 87.4 & 5.38 \\ Bi-LSTM & 6.7 & 24.7 & 86.6 & **5.30** \\
**Transformer** & **12.2** & **31.8** & **80.2** & 5.69 \\ \hline \end{tabular}
\end{table} 
Table 4: BLEU, chrF, and TER scores of baseline models in Wolaytta–English translation. ### Self-Learning

As considered in Section 5.4.2, we selected the outperforming NMT model from our baseline models and trained it on the combination of mixed (parallel and synthetic) dataset.