# Comparative Analysis of Current Approaches to Quality Estimation for Neural Machine Translation
## 3 Multilingual Pretrained Language Models for QE
### Cross-Lingual Language Model  
XLM [8] is a transformer-based model that extends existing language model pretraining methods, which mainly focus on a monolingual language representation, to themultiple language representation. XLM is pretrained through MLM and CLM by leveraging a multilingual unlabeled corpus. To achieve a better multilingual language understanding, TLM, which is a pretraining scheme utilizing a parallel corpus, is applied. Unlike mBERT, NSP is not considered during pretraining.  
CLM is a pretraining scheme in which the objective is to model the probability of a word given the previous words in a sentence. This can be described as in Equation (3).  
\[\max_{\theta}\sum_{X\in D}\sum_{i=1}^{n}\log P(x_{i}\mid x_{t<i},\theta) \tag{3}\]  
It can be said that the goal of CLM is to maximize the probability of a token based on preceding tokens. Through this process, a model can obtain an improved language understanding.  
TLM is an extension of MLM and improves cross-lingual understanding by utilizing parallel data in the pretraining phase. The source and target sentences of a parallel corpus are first connected, and then some tokens in these sentences are replaced with [MASK] tokens. The training objective of TLM predicts masked tokens the same as in mBERT. However, masked tokens can be predicted by referring to the surrounding context of the masked tokens, as well as sentences from other languages concatenated. It is characterized by TLM that by predicting masked tokens by referencing both languages simultaneously, a representation containing information between languages can be obtained.  
This can be described as shown in Equation (4).  
\[\max_{\theta}\sum_{(X,Y,\bar{X},\bar{Y})\in D}\left[\sum_{i\in M_{x}}\log P(x_ {i}\mid\bar{X}:\bar{Y},\theta)+\sum_{j\in M_{y}}\log P(y_{j}\mid\bar{X}:\bar{ Y},\theta)\right] \tag{4}\]  
In Equation (4), \(\bar{X}:\bar{Y}\) indicates corrupted input data where \(\bar{X}\) is a source sentence component and \(\bar{Y}\) is a target sentence component. \(M_{x}\) and \(M_{y}\) are index sets that consist of the indices indicating masked tokens in the source and target sentences, respectively. When predicting a masked word in a source sentence during the training process, a model can refer to the nearby source language context, as well as target sentence. This can encourage the model to acquire a better understanding of multilingual representation. Additionally, to obtaining decent multilingual representation, distinct language embeddings, and respective position embeddings are applied to each language.  
XLM utilizes Wikipedia data for the pretraining of various languages. As the amount of established Wikipedia data differs for each language, bias towards high-resource languages can be obtained if such data are utilized without any preprocessing. To alleviate the data imbalance problem, different sampling ratios are applied in the training process. The applied sampling ratios are determined using a multinomial distribution, which is denoted in Equation (5).  
\[q_{i}=\frac{p_{i}^{\alpha}}{\sum_{j=1}^{N}p_{j}^{\alpha}}\ \ \text{where}\ \ \ p_{i}=\frac{n_{i}}{\sum_{j=1}^{N}n_{j}} \tag{5}\]  
Here, \(q_{i}\) indicates a sampling ratio for the \(i^{th}\) language data, with amount \(n_{i}\), among the total dataset that comprises \(N\) languages. \(\alpha\) is a hyperparameter that is set to 0.7 for the pretraining of XLM, such that the sampling ratio is increased for low-resource languages and decreased for high-resource languages.  
For the XLM-based QE model, the overall training process is similar to Section 3.1, except that positional embeddings that encode absolute positions and language embeddings that indicate the language of each token are applied.