# Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation
## 5 Question 1: Which mPLM Is Best for QE Tasks?
### Model Details  
We conducted a finetuning performance comparison using a total of 9 models including XLM-R base, XLM-R large, mBERT, mBART, XLM-CLM, XLM-MLM, XLM-MLM-17, XLM-MLM-100, and XLM-TLM. English-German was used as the language pair for this experiment, and performance comparisons were conducted for each mPLM at sub-task 1 and sub-task 2 sentence-levels. These models are described as follows:  
* **XLM-R-base**: Pretraining was performed with 220M parameters, 12 layers, 8 heads, and 768 hidden states. * **XLM-R-large**: Pretraining was performed using 550M parameters. The hidden states were expanded to 1024, and 24 layers, and 16 heads were used, which is twice the scale of the base model. * **mBERT**: The model parameters of mBERT were 110M, 12 layers, 768 hidden states, and 12 heads. * **mBART**: mBART was pretrained with 610M parameters, 24 layers, 1024 hidden states, and 16 heads. * **XLM-CLM**: A pretrained CLM for English and German. In total, 6 layers, 1024 hidden states, and 8 heads were used. * **XLM-MLM**: A pretrained MLM for English and German. In total, 6 layers, 1024 hidden states, and 8 heads were used. * **XLM-MLM-17**: Pretraining was conducted by expanding the MLM into 17 languages. It was trained using 570M parameters, 16 layers, 1280 hidden states, and 16 heads. * **XLM-MLM-100**: Pretraining was conducted by expanding the MLM into 100 languages. It was trained using 570M parameters, 16 layers, 1280 hidden states, and 16 heads. * **XLM-TLM**: TLM was performed for 15 languages. In total, 12 layers, 1024 hidden states, and 8 heads were used. We performed finetuning using the pretrained model released in HuggingFace's transformers library [42]. We did not proceed with additional pretraining and data augmentation so that the pure performances of the mPLMs could be objectively evaluated and compared in the QE task. In preprocessing, we performed subword tokenization using the tokenizer provided for each model in HuggingFace. For the model input, we added segment embeddings for mBERT, listing tokens separated by 0 and 1 to give a distinction between sentence 1 and sentence 2. XLM has added a position embedding that gives a number corresponding to \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Sub-Task 1**} & \multicolumn{3}{c}{**Sub-Task 2**} \\ \cline{2-7}  & **Train** & **Dev** & **Test** & **Train** & **Dev** & **Test** \\ \hline \# Instance & 7000 & 1000 & 1000 & 7000 & 1000 & 1000 \\ \hline \# SRC Token & 98,127 & 14,102 & 14,043 & 114,980 & 16,519 & 16,371 \\ \hline \# MT Token & 97,453 & 14,003 & 14,019 & 112,342 & 16,160 & 16,154 \\ \hline Average Score & \(-0.008\) & \(-0.049\) & 0.040 & 0.318 & 0.312 & 0.312 \\ \hline Median Score & 0.162 & 0.211 & 0.319 & 0.3 & 0.295 & 0.286 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the QE dataset. We denote the number of instances in each dataset as # Instance. # SRC Token and # MT Token refer to the number of tokens in source- and target-side sentences for each dataset, respectively. the token index for each source sentence and MT output, as well as a language embedding that is segmented by a unique number for each language. As a training procedure for finetuning, we first load mPLMs to initialize the parameters. After that, additional embeddings for each model are put as input to the model along with the sentences concatenated with the source and target sentences. We put the output corresponding to the position of the [CLS] token among the last hidden states as an input to the linear classifier and measured the loss between the predicted value and the label. We use the mean squared error (MSE) loss as the loss function. We found that the model has a diverse range of performance fluctuations depending on the seed value, and we attempted to reduce the effect of the seed value on the general performance of the model.