# Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation
## 3 Multilingual Pretrained Language Models for QE
### Multilingual BERT  
BERT [37] is built on a transformer [38] architecture, which consists solely of an encoder structure.  
BERT performs a self-supervised learning process for large-scale mono-lingual corpus. Because the self-supervised learning process performs supervision on raw text on its own, it does not require labeled data, so it can utilize large amounts of raw data. After performing user-defined problems such as masked language model (MLM) and next sentence prediction (NSP) on unlabeled raw data, transfer learning is performed for downstream tasks. More specifically, the user generates arbitrary tasks and labels for raw text to learn language information, and uses the representations obtained through this process as initialization values for downstream tasks. For the case of BERT, MLM, and NSP are used as pretraining schemes.  
MLM is a procedure of randomly masking tokens in the original sentence with [MASK] tokens. The objectives is to correctly predict these masked tokens based on left and right context of the sentence. In particular, the last hidden vector corresponding to the mask token goes through softmax and returns as the word with the highest probability in the vocabulary. In the process of masking, 15% of the original sentences are randomly sampled, then among them, 80% of these selected tokens are replaced by [MASK], 10% are replaced by random tokens in the vocabulary, and 10% remain unchanged. Through this masking process, a defective sentence \(\bar{X}=\{\bar{x}_{1},\bar{x}_{2},\ldots,\bar{x}_{n}\}\) is generated from an unlabeled monolingual sentence \(X=\{x_{1},x_{2},\ldots,x_{n}\}\). In the training process, \(\bar{X}\) is fed into a BERT model, which is parameterized by \(\theta\), and the model is then trained to return \(X\). This task can be described by Equation (1).  
\[\max_{\theta}\sum_{(X,\bar{X})\in D}\sum_{i=1}^{n}\log P(x_{i}\mid\bar{x}_{1}, \ldots,\bar{x}_{n},\theta) \tag{1}\]  
This equation indicates that a model is trained to predict an original token \(x_{i}\) by considering a defective sentence \(\bar{X}\). By referring to nearby context while restoring a [MASK] token, a model can be trained using bidirectional contextual representation.  
NSP is a binary classification task that aims to train by understanding sentence relationships. In the training process, two sentences are concatenated to construct inputs, and these sentences are then selected from an unlabeled monolingual corpus based on a probability. Successive sentences are selected for half of the time, while randomly picked sentences are chosen otherwise. The main objective of NSP is to distinguish whether these input sentences are successive or not. Through this training process, a model can obtain an improved understanding of relationships between sentences.  
Multilingual BERT (mBERT) [7] is a BERT-based multilingual model. The same pretraining schemes as BERT (MLM and NSP) are adopted for mBERT. However, unlike BERT, mBERT is trained with a multilingual unlabeled corpus, which is comprised of 104 languages.  
The way we adapt mBERT to a QE task is as follows. For the assessment of an entire sentence, we leverage the first hidden representation obtained from the mBERT model. By applying a linear classification head without the activation function, we can obtain the final prediction score of the sentence. Therefore, the sentence assessment score \(score_{sentence}\) is derived from an encoded representation of the input sentence, \(H=\{h_{1},h_{2},\ldots,h_{m}\}\), as shown in Equation (2).  
\[score_{sentence}=W\cdot h_{1}+b \tag{2}\]  
In Equation (2), \(W\in\mathbb{R}^{1\times hiden}\) and \(b\in\mathbb{R}^{1\times 1}\) are trainable parameters where _hidden_ indicates the hidden layer size of pretrained mBERT. During the QE training process, the mean squared error (MSE) loss between \(score_{sentence}\) and the label score is considered.