# Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation
## 5 Question 1: Which mPLM Is Best for QE Tasks?
### Experimental Results for Question 2  
In order to check whether the order of the input sentence affects the performance while performing QE finetuning, we raise question 2 and compare the sentence order with the reversed sentence order when constructing the input sequence. The experimental results for sub-task 1 are shown in Table 4. As a result of this experiment, it can be observed that the model performance changes by simply reversing the order of the input sentence. In this table, we denote Avg Diff as the difference between the average Pearson correlation coefficients of the original input and reverse orders. As can be seen from the Avg Diff values, when the input sentence order was reversed, the average Pearson correlation coefficient of XLM-R-large improved by +0.032, while that of XLM-MLM-100 improved by +0.008. However, for all other models, the performance deteriorated when the order of the input sentences was reversed. Likewise, in Figure 1, it was confirmed that the overall reversed order input sentences in sub-task 1 did not help to improve the performance of the model. \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Pearson**} & \multicolumn{3}{c}{**MAE**} & \multicolumn{3}{c}{**RMSE**} \\ \cline{2-10}  & **Max** & **Min** & **Average** & **Min** & **Max** & **Average** & **Min** & **Max** & **Average** \\ \hline XLM-R-base & 0.456 & 0.438 & 0.448 & 0.146 & 0.156 & 0.150 & 0.189 & 0.204 & 0.195 \\ \hline XLM-R-large & 0.507 & 0.489 & 0.498 & 0.141 & 0.155 & 0.145 & 0.178 & 0.204 & 0.186 \\ \hline mBERT & 0.435 & 0.389 & 0.417 & 0.149 & 0.182 & 0.160 & 0.189 & 0.230 & 0.204 \\ \hline mBART & 0.475 & 0.452 & 0.463 & 0.142 & 0.148 & 0.144 & 0.179 & 0.195 & 0.184 \\ \hline XLM-CLM & 0.309 & 0.275 & 0.298 & 0.158 & 0.161 & 0.159 & 0.196 & 0.200 & 0.198 \\ \hline XLM-MLM & 0.358 & 0.303 & 0.334 & 0.156 & 0.160 & 0.158 & 0.194 & 0.199 & 0.197 \\ \hline XLM-MLM-17 & 0.433 & 0.408 & 0.415 & 0.149 & 0.157 & 0.154 & 0.188 & 0.192 & 0.190 \\ \hline XLM-MLM-100 & 0.421 & 0.381 & 0.409 & 0.152 & 0.164 & 0.158 & 0.190 & 0.207 & 0.198 \\ \hline XLM-TLM & 0.522 & 0.498 & 0.510 & 0.152 & 0.222 & 0.177 & 0.199 & 0.273 & 0.227 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: mPLM finetuning results for the test set of the WMT20 sub-task 2. Conversely, in the case of sub-task 2, the result of reversing the input sentences provided a better overall performance. As can be seen in Table 5 and Figure 1, only two models of XLM-CLM and XLM-MLM-17 declined in performance based on the average Pearson correlation coefficient, while all other models consistently exhibited improved performances. In particular, the range of performance fluctuations was high in both XLM-TLM and mBERT. These two models also showed the highest variation in sub-task 1, and it can, therefore, be said that these models respond most sensitively to the input sentence order. The models with the lowest performance fluctuations were XLM-MLM-17 and mBART. In the case of mBART, there was little change in performance even in sub-task 1, and there was no significant change in the performance in response to the varied input structure.