# Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation
## 5 Question 1: Which mPLM Is Best for QE Tasks?
### Experimental Results for Question 1  
5.3.1. Sub-Task 1  
To check which model out of various mPLMs performs well for the QE task, we raise question 1, and proceed with finetuning using mPLMs. The experimental results for the QE of sub-task 1 (i.e., the direct assessment at the sentence-level) are shown in Table 2.  
As a result of the experiment, XLM-TLM showed the highest performance for sub-task 1 with a Pearson correlation coefficient of 0.442. In terms of the minimum and average performances, this system consistently demonstrated the highest performance compared to the other models. To investigate the cause of this result, we need to focus on the input data of XLM-TLM in the pretraining process. The XLM-TLM model utilizes parallel data during pretraining and can refer to the context of either side when predicting the source- and target-side masked words. Likewise, in the QE field, the concatenating sentences of the source and target language are provided as an input to the model. This is similar to the form of the input for the XLM-TLM model in that it provides sentences in both languages as the input, while the other models use the mono data of multiple languages. According to [PERSON] and [PERSON] [8], when predicting a masked word during XLM-TLM learning, the model can be encouraged to align the source and target language representations by attending the translated sentence along with the surrounding masked word. Therefore, when using the aligned representation derived between the source and target languages in the XLM-TLM model for QE, it is possible to infer what part of the translated sentence is wrong. The model with the second highest average performance is the mBERT model. This model provided approximately 0.012 less \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Pearson**} & \multicolumn{3}{c}{**MAE**} & \multicolumn{3}{c}{**RMSE**} \\ \cline{2-10}  & **Max** & **Min** & **Average** & **Min** & **Max** & **Average** & **Min** & **Max** & **Average** \\ \hline XLM-R-base & 0.380 & 0.280 & 0.328 & 0.459 & 0.479 & 0.473 & 0.648 & 0.679 & 0.665 \\ \hline XLM-R-large & 0.338 & 0.242 & 0.298 & 0.480 & 0.520 & 0.495 & 0.685 & 0.713 & 0.698 \\ \hline mBERT & 0.407 & 0.322 & 0.382 & 0.452 & 0.468 & 0.458 & 0.642 & 0.672 & 0.655 \\ \hline mBART & 0.402 & 0.306 & 0.351 & 0.465 & 0.534 & 0.490 & 0.642 & 0.729 & 0.677 \\ \hline XLM-CLM & 0.296 & 0.168 & 0.253 & 0.474 & 0.516 & 0.489 & 0.683 & 0.703 & 0.691 \\ \hline XLM-MLM & 0.219 & 0.192 & 0.206 & 0.493 & 0.526 & 0.503 & 0.693 & 0.728 & 0.708 \\ \hline XLM-MLM-17 & 0.318 & 0.143 & 0.253 & 0.465 & 0.525 & 0.490 & 0.670 & 0.731 & 0.696 \\ \hline XLM-MLM-100 & 0.256 & 0.191 & 0.232 & 0.482 & 0.536 & 0.498 & 0.690 & 0.702 & 0.695 \\ \hline XLM-TLM & 0.442 & 0.336 & 0.394 & 0.451 & 0.683 & 0.517 & 0.631 & 0.805 & 0.681 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: mPLM finetuning results for the test set of the WMT20 sub-task 1.  
than that of the first-ranked model and demonstrates a comparable performance. mBART did not show a strong performance in the regression task, but the maximum value only showed a difference of about 0.005 compared to the mBERT model. Both models apply various noising schemes during pretraining, and it can be predicted that this strategy will help improve their performance. In the case of XLM-R-large, many research groups that participated in WMT20 used this model; however, for sub-task 1, it was not ranked high. When comparing the average Pearson correlation coefficients of the models based on XLM, XLM-MLM-17 was 0.021 higher than that of XLM-MLM-100, and XLM-MLM, which learned only English and German, showed the lowest performance. XLM-MLM-17 and XLM-MLM-100 are approximately twice the size of XLM-MLM considering the number of layers and hidden states, etc. and the languages were also expanded to 17 and 100 languages, respectively. It can be inferred that the number of languages and model capacity helped to improve the performance for QE.