SlowGuess/NAACL_2021_OCR huggingface dataset:
INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi†‡*, Li Dong‡, Furu Wei‡, Nan Yang‡, Saksham Singhal‡, Wenhui Wang‡, Xia Song‡, Xian-Ling Mao†, Heyan Huang†, Ming Zhou‡
$^{\dagger}$ Beijing Institute of Technology
{czw,maoxl,hhy63}@bit.edu.cn
{lidong1, fuwei, nanya, saksingh, wenwan, xiao}@microsoft.com
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pretraining task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.
Learning cross-lingual language representations plays an important role in overcoming the language barrier of NLP models. The recent success of cross-lingual language model pre-training (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020a; Chi et al., 2020; Liu et al., 2020) significantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and question answering.
State-of-the-art cross-lingual pre-trained models are typically built upon multilingual masked language modeling (MMLM; Devlin et al. 2019; Conneau et al. 2020a), and translation language modeling (TLM; Conneau and Lample 2019). The goal of both pretext tasks is to predict masked tokens given input context. The difference is that
MMLM uses monolingual text as input, while TLM feeds bilingual parallel sentences into the model. Even without explicit encouragement of learning universal representations across languages, the derived models have shown promising abilities of cross-lingual transfer.
In this work, we formulate cross-lingual pretraining from a unified information-theoretic perspective. Following the mutual information maximization principle (Hjelm et al., 2019; Kong et al., 2020), we show that the existing pretext tasks can be viewed as maximizing the lower bounds of mutual information between various multilingual-multi-granularity views.
Specifically, MMLM maximizes mutual information between the masked tokens and the context in the same language while the anchor points across languages encourages the correlation between cross-lingual contexts. Moreover, we present that TLM can maximize mutual information between the masked tokens and the parallel context, which implicitly aligns encoded representations of different languages. The unified information-theoretic framework also inspires us to propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCo). The model learns to distinguish the translation of an input sentence from a set of negative examples. In comparison to TLM that maximizes token-sequence mutual information, XLCo maximizes sequence-level mutual information between translation pairs which are regarded as cross-lingual views of the same meaning. We employ the momentum contrast (He et al., 2020) to realize XLCo. We also propose the mixup contrast and conduct the contrast on the universal layer to further facilitate the cross-lingual transferability.
Under the presented framework, we develop a cross-lingual pre-trained model (INFOXLM) to leverage both monolingual and parallel corpora. We jointly train INFOXLM with MMLM, TLM
*Contribution during internship at Microsoft Research. Contact person: Li Dong and Furu Wei.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576-3588 June 6-11, 2021. ©2021 Association for Computational Linguistics
and XLCo. We conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference (Connieu et al., 2018), cross-lingual question answering (Lewis et al., 2020), and cross-lingual sentence retrieval (Artetxe and Schwenk, 2019). Experimental results show that INFOXLM outperforms strong baselines on all the benchmarks. Moreover, the analysis indicates that INFOXLM achieves better cross-lingual transferability.
2.1 Cross-Lingual LM Pre-Training
Multilingual BERT (mBERT; Devlin et al. 2019) is pre-trained with the multilingual masked language modeling (MMLM) task on the monolingual text. mBERT produces cross-lingual representations and performs cross-lingual tasks surprisingly well (Wu and Dredze, 2019). XLM (Conneau and Lample, 2019) extends mBERT with the translation language modeling (TLM) task so that the model can learn cross-lingual representations from parallel corpora. Unicoder (Huang et al., 2019) tries several pre-training tasks to utilize parallel corpora. ALM (Yang et al., 2020) extends TLM to code-switched sequences obtained from translation pairs. XLM-R (Conneau et al., 2020a) scales up MMLM pre-training with larger corpus and longer training. LaBSE (Feng et al., 2020) learns cross-lingual sentence embeddings by an additive translation ranking loss.
In addition to learning cross-lingual encoders, several pre-trained models focus on generation. MASS (Song et al., 2019) and mBART (Liu et al., 2020) pretrain sequence-to-sequence models to improve machine translation. XNLG (Chi et al., 2020) focuses on the cross-lingual transfer of language generation, such as cross-lingual question generation, and abstractive summarization.
2.2 Mutual Information Maximization
Various methods have successfully learned visual or language representations by maximizing mutual information between different views of input. It is difficult to directly maximize mutual information. In practice, the methods resort to a tractable lower bound as the estimator, such as InfoNCE (Oord et al., 2018), and the variational form of the KL divergence (Nguyen et al., 2010). The estimators are also known as contrastive learning (Arora et al., 2019) that measures the representation similarities
between the sampled positive and negative pairs. In addition to the estimators, various view pairs are employed in these methods. The view pair can be the local and global features of an image (Hjelm et al., 2019; Bachman et al., 2019), the random data augmentations of the same image (Tian et al., 2019; He et al., 2020; Chen et al., 2020), or different parts of a sequence (Oord et al., 2018; Henaff, 2020; Kong et al., 2020). Kong et al. (2020) show that learning word embeddings or contextual embeddings can also be unified under the framework of mutual information maximization.
3 Information-Theoretic Framework for Cross-Lingual Pre-Training
In representation learning, the learned representations are expected to preserve the information of the original input data. However, it is intractable to directly model the mutual information between the input data and the representations. Alternatively, we can maximize the mutual information between the representations from different views of the input data, e.g., different parts of a sentence, a translation pair of the same meaning.
In this section, we start from a unified information-theoretic perspective, and formulate cross-lingual pre-training with the mutual information maximization principle. Then, under the information-theoretic framework, we propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCo). Finally, we present the pre-training procedure of our INFOXLM.
The goal of multilingual masked language modeling (MMLM; Devlin et al. 2019) is to recover the masked tokens from a randomly masked sequence. For each input sequence of MMLM, we sample a text from the monolingual corpus for pretraining. Let  $(c_{1},x_{1})$  denote a monolingual text sequence, where  $x_{1}$  is the masked token, and  $c_{1}$  is the corresponding context. Intuitively, we need to maximize their dependency (i.e.,  $I(c_{1};x_{1})$ ), so that the context representations are predictive for masked tokens (Kong et al., 2020).
For the example pair  $(c_{1},x_{1})$ , we construct a set  $\mathcal{N}$  that contains  $x_{1}$  and  $|\mathcal{N}| - 1$  negative samples drawn from a proposal distribution  $q$ . According to the InfoNCE (Oord et al., 2018) lower bound, we
$$
\begin{array}{l} I (c _ {1}; x _ {1}) \\ \geqslant \underset {q (\mathcal {N})} {E} \left[ \log \frac {f _ {\boldsymbol {\theta}} \left(c _ {1} , x _ {1}\right)}{\sum_ {x ^ {\prime} \in \mathcal {N}} f _ {\boldsymbol {\theta}} \left(c _ {1} , x ^ {\prime}\right)} \right] + \log | \mathcal {N} | \tag {1} \\ \end{array}
$$
where  $f_{\theta}$  is a function that scores whether the input  $c_{1}$  and  $x_{1}$  is a positive pair.
Given context  $c_{1}$ , MMLM learns to minimize the cross-entropy loss of the masked token  $x_{1}$ :
$$
\mathcal {L} _ {\mathrm {M M L M}} = - \log \frac {\exp \left(g _ {\boldsymbol {\theta} _ {T}} \left(c _ {1}\right) ^ {\top} g _ {\boldsymbol {\theta} _ {E}} \left(x _ {1}\right)\right)}{\sum_ {x ^ {\prime} \in \mathcal {V}} \exp \left(g _ {\boldsymbol {\theta} _ {T}} \left(c _ {1}\right) ^ {\top} g _ {\boldsymbol {\theta} _ {E}} \left(x ^ {\prime}\right)\right)} \tag {2}
$$
where  $\mathcal{V}$  is the vocabulary,  $g_{\theta_E}$  is a look-up function that returns the token embeddings,  $g_{\theta_T}$  is a Transformer that returns the final hidden vectors in position of  $x_{1}$ . According to Equation (1) and Equation (2), if  $\mathcal{N} = \mathcal{V}$  and  $f_{\pmb{\theta}}(c_1,x_1) = \exp (g_{\pmb{\theta}_T}(c_1)^\top g_{\pmb{\theta}_E}(x_1))$ , we can find that MMLM maximizes a lower bound of  $I(c_{1};x_{1})$
Next, we explain why MMLM can implicitly learn cross-lingual representations. Let  $(c_{2},x_{2})$  denote a MMLM instance that is in different language as  $(c_{1},x_{1})$ . Because the vocabulary, the position embedding, and special tokens are shared across languages, it is common to find anchor points (Pires et al., 2019; Dufter and Schütze, 2020) where  $x_{1} = x_{2}$  (such as subword, punctuation, and digit) or  $I(x_{1},x_{2})$  is positive (i.e., the representations are associated or isomorphic). With the bridge effect of  $\{x_{1},x_{2}\}$ , MMLM obtains a v-structure dependency “ $c_{1}\rightarrow \{x_{1},x_{2}\} \leftarrow c_{2}$ ”, which leads to a negative co-information (i.e., interaction information)  $I(c_{1};c_{2};\{x_{1},x_{2}\})$  (Tsujishita, 1995). Specifically, the negative value of  $I(c_{1};c_{2};\{x_{1},x_{2}\})$  indicates that the variable  $\{x_1,x_2\}$  enhances the correlation between  $c_{1}$  and  $c_{2}$  (Fano, 1963).
In summary, although MMLM learns to maximize  $I(c_{1}, x_{1})$  and  $I(c_{2}, x_{2})$  in each language, we argue that the task encourages the cross-lingual correlation of learned representations. Notice that for the setting without word-piece overlap (Artetxe et al., 2020; Conneau et al., 2020b; K et al., 2020), we hypothesize that the information bottleneck principle (Tishby and Zaslavsky, 2015) tends to transform the cross-lingual structural similarity into isomorphic representations, which has similar bridge effects as the anchor points. Then we can explain how the cross-lingual ability is spread out as above. We leave more discussions about the setting without word-piece overlap for future work.
3.2 Translation Language Modeling
Similar to MMLM, the goal of translation language modeling (TLM; Conneau and Lample 2019) is also to predict masked tokens, but the prediction is conditioned on the concatenation of a translation pair. We try to explain how TLM pre-training enhances cross-lingual transfer from an information-theoretic perspective.
Let  $c_{1}$  and  $c_{2}$  denote a translation pair of sentences, and  $x_{1}$  a masked token taken in  $c_{1}$ . So  $c_{1}$  and  $x_{1}$  are in the same language, while  $c_{1}$  and  $c_{2}$  are in different ones. Following the derivations of MMLM in Section 3.1, the objective of TLM is maximizing the lower bound of mutual information  $I(c_{1},c_{2};x_{1})$ . By re-writing the above mutual information, we have:
$$
I \left(c _ {1}, c _ {2}; x _ {1}\right) = I \left(c _ {1}; x _ {1}\right) + I \left(c _ {2}; x _ {1} \mid c _ {1}\right) \tag {3}
$$
The first term  $I(c_{1};x_{1})$  corresponds to MMLM, which learns to use monolingual context. In contrast, the second term  $I(c_{2};x_{1}|c_{1})$  indicates crosslingual mutual information between  $c_{2}$  and  $x_{1}$  that is not included by  $c_{1}$ . In other words,  $I(c_{2};x_{1}|c_{1})$  encourages the model to predict masked tokens by using the context in a different language. In conclusion, TLM learns to utilize the context in both languages, which implicitly improves the crosslingual transferability of pre-trained models.
3.3 Cross-Lingual Contrastive Learning
Inspired by the unified information-theoretic framework, we propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCo). The goal of XLCo is to maximize mutual information between the representations of parallel sentences  $c_{1}$  and  $c_{2}$ , i.e.,  $I(c_{1},c_{2})$ . Unlike maximizing token-sequence mutual information in MMLM and TLM, XLCo targets at cross-lingual sequence-level mutual information.
We describe how the task is derived as follows. Using InfoNCE (Oord et al., 2018) as the lower bound, we have:
$$
I \left(c _ {1}; c _ {2}\right) \geqslant E _ {q (\mathcal {N})} \left[ \log \frac {f _ {\boldsymbol {\theta}} \left(c _ {1} , c _ {2}\right)}{\sum_ {c ^ {\prime} \in \mathcal {N}} f _ {\boldsymbol {\theta}} \left(c _ {1} , c ^ {\prime}\right)} \right] + \log | \mathcal {N} | \tag {4}
$$
where  $\mathcal{N}$  is a set that contains the positive pair  $c_{2}$  and  $|\mathcal{N}| - 1$  negative samples. In order to maximize the lower bound of  $I(c_{1};c_{2})$ , we need to design the function  $f_{\theta}$  that measures the similarity between the input sentence and the proposal distribution
$q(\mathcal{N})$ . Specifically, we use the following similarity function  $f_{\theta}$ :
$$
f _ {\boldsymbol {\theta}} \left(c _ {1}, c _ {2}\right) = \exp \left(g _ {\boldsymbol {\theta}} \left(c _ {1}\right) ^ {\top} g _ {\boldsymbol {\theta}} \left(c _ {2}\right)\right) \tag {5}
$$
where  $g_{\theta}$  is the Transformer encoder that we are pre-training. Following (Devlin et al., 2019), a special token [CLS] is added to the input, whose hidden vector is used as the sequence representation. Additionally, we use a linear projection head after the encoder in  $g_{\theta}$ .
Momentum Contrast Another design choice is how to construct  $\mathcal{N}$ . As shown in Equation (4), a large  $|\mathcal{N}|$  improves the tightness of the lower bound, which has been proven to be critical for contrastive learning (Chen et al., 2020).
In our work, we employ the momentum contrast (He et al., 2020) to construct the set  $\mathcal{N}$ , where the previously encoded sentences are progressively reused as negative samples. Specifically, we construct two encoders with the same architecture which are the query encoder  $g_{\theta_Q}$  and the key encoder  $g_{\theta_K}$ . The loss function of XLCo is:
$$
\mathcal {L} _ {\mathrm {X L C o}} = - \log \frac {\exp \left(g _ {\boldsymbol {\theta} _ {Q}} \left(c _ {1}\right) ^ {\top} g _ {\boldsymbol {\theta} _ {K}} \left(c _ {2}\right)\right)}{\sum_ {c ^ {\prime} \in \mathcal {N}} \exp \left(g _ {\boldsymbol {\theta} _ {Q}} \left(c _ {1}\right) ^ {\top} g _ {\boldsymbol {\theta} _ {K}} \left(c ^ {\prime}\right)\right)} \tag {6}
$$
During training, the query encoder  $g_{\theta_Q}$  encodes  $c_{1}$  and is updated by backpropagation. The key encoder  $g_{\theta_K}$  encodes  $\mathcal{N}$  and is learned with momentum update (He et al., 2020) towards the query encoder. The negative examples in  $\mathcal{N}$  are organized as a queue, where a newly encoded example is added while the oldest one is popped from the queue. We initialize the query encoder and the key encoder with the same parameters, and pre-fill the queue with a set of encoded examples until it reaches the desired size  $|\mathcal{N}|$ . Notice that the size of the queue remains constant during training.
Mixup Contrast For each pair, we concatenate it with a randomly sampled translation pair from another parallel corpus. For example, consider the pairs  $\langle c_1, c_2 \rangle$  and  $\langle d_1, d_2 \rangle$  sampled from two different parallel corpora. The two pairs are concatenated in a random order, such as  $\langle c_1 d_1, c_2 d_2 \rangle$ , and  $\langle c_1 d_2, d_1 c_2 \rangle$ . The data augmentation of mixup encourages pre-trained models to learn sentence boundaries and to distinguish the order of multilingual texts.
Contrast on Universal Layer As a pre-training task maximizing the lower bound of sequence-level mutual information, XLCo is usually jointly learned with token-sequence tasks, such as MMLM, and TLM. In order to make XLCo more compatible with the other pretext tasks, we propose to conduct contrastive learning on the most universal (or transferable) layer in terms of MMLM and TLM.
In our implementations, we instead use the hidden vectors of [CLS] at layer 8 to perform contrastive learning for base-size (12 layers) models, and layer 12 for large-size (24 layers) models. Because previous analysis (Sabet et al., 2020; Dufter and Schütze, 2020; Conneau et al., 2020b) shows that the specific layers of MMLM learn more universal representations and work better on cross-lingual retrieval tasks than other layers. We choose the layers following the same principle.
The intuition behind the method is that MMLM and TLM encourage the last layer to produce language-distinguishable token representations because of the masked token classification. But XLCo tends to learn similar representations across languages. So we do not directly use the hidden states of the last layer in XLCo.
3.4 Cross-Lingual Pre-Training
We pretrain a cross-lingual model INFOXLM by jointly maximizing the lower bounds of three types of mutual information, including monolingual token-sequence mutual information (MMLM), cross-lingual token-sequence mutual information (TLM), and cross-lingual sequence-level mutual information (XLCo). Formally, the loss of cross-lingual pre-training in INFOXLM is defined as:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {M M L M}} + \mathcal {L} _ {\mathrm {T L M}} + \mathcal {L} _ {\mathrm {X L C O}} \tag {7}
$$
where we apply the same weight for the loss terms.
Both TLM and XLCo use parallel data. The number of bilingual pairs increases with the square of the number of languages. In our work, we set English as the pivot language following (Conneau and Lample, 2019), i.e., we only use the parallel corpora that contain English.
In order to balance the data size between highresource and low-resource languages, we apply a multilingual sampling strategy (Conneau and Lample, 2019) for both monolingual and parallel data. An example in the language  $l$  is sampled with the probability  $p_l \propto (n_l / n)^{0.7}$ , where  $n_l$  is the number
of instances in the language  $l$ , and  $n$  refers to the total number of data. Empirically, the sampling algorithm alleviates the bias towards high-resource languages (Conneau et al., 2020a).
In this section, we first present the training configuration of INFOXLM. Then we compare the finetuning results of INFOXLM with previous work on three cross-lingual understanding tasks. We also conduct ablation studies to understand the major components of INFOXLM.
Corpus We use the same pre-training corpora as previous models (Conneau et al., 2020a; Conneau and Lample, 2019). Specifically, we reconstruct CC-100 (Conneau et al., 2020a) for MMLM, which remains 94 languages by filtering the language code larger than 0.1GB. Following (Conneau and Lample, 2019), for the TLM and XLCo tasks, we employ 14 language pairs of parallel data that involves English. We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), and WikiMatrix (Schwenk et al., 2019). The size of parallel corpora is about 42GB. More details about the pre-training data are described in the appendix.
Model Size We follow the model configurations of XLM-R (Conneau et al., 2020a). For the Transformer (Vaswani et al., 2017) architecture, we use 12 layers and 768 hidden states for INFOXLM (i.e., base size), and 24 layers and 1,024 hidden states for INFOXLMLarge (i.e., large size).
Hyperparameters We initialize the parameters of INFOXLM with XLM-R. We optimize the model with Adam (Kingma and Ba, 2015) using a batch size of 2048 for a total of 150K steps for INFOXLM, and 200K steps for INFOXLM LARGE. The same number of training examples are fed to three tasks. The learning rate is scheduled with a linear decay with 10K warmup steps, where the peak learning rate is set as 0.0002 for INFOXLM, and 0.0001 for INFOXLM LARGE. The momentum coefficient is set as 0.9999 and 0.999 for INFOXLM and INFOXLM LARGE, respectively. The length of the queue is set as 131,072. The training procedure takes about 2.3 days × 2 Nvidia DGX-2 stations for INFOXLM, and 5 days × 16
Nvidia DGX-2 stations for INFOXLM LARGE. Details about the pre-training hyperparameters can be found in the appendix.
We conduct experiments over three cross-lingual understanding tasks, i.e., cross-lingual natural language inference, cross-lingual sentence retrieval, and cross-lingual question answering.
Cross-Linguational Language Inference The Cross-Linguational Language Inference corpus (XNLI; Conneau et al. 2018) is a widely used cross-lingual classification benchmark. The goal of NLI is to identify the relationship of an input sentence pair. We evaluate the models under the following two settings. (1) Cross-Linguual Transfer: fine-tuning the model with English training set and directly evaluating on multilingual test sets. (2) Translate-Train-All: fine-tuning the model with the English training data and the pseudo data that are translated from English to the other languages.
Cross-Linguual Sentence Retrieval The goal of the cross-lingual sentence retrieval task is to extract parallel sentences from bilingual comparable corpora. We use the subset of 36 language pairs of the Tatoeba dataset (Artetxe and Schwenk, 2019) for the task. The dataset is collected from Tatoeba<sup>1</sup>, which is an open collection of multilingual parallel sentences in more than 300 languages. Following (Hu et al., 2020), we use the averaged hidden vectors in the seventh Transformer layer to compute cosine similarity for sentence retrieval.
Cross-Lingual Question Answering We use the Multilingual Question Answering (MLQA; Lewis et al. 2020) dataset for the cross-lingual QA task. MLQA provides development and test data in seven languages in the format of SQuAD v1.1 (Rajpurkar et al., 2016). We follow the fine-tuning method introduced in (Devlin et al., 2019) that concatenates the question-passage pair as the input.
We compare INFOXLM with the following pretrained Transformer models: (1) Multilingual BERT (MBERT; Devlin et al. 2019) is pre-trained with MMLM on Wikipedia in 102 languages; (2) XLM (Conneau and Lample, 2019) pretrains both MMLM and TLM tasks on Wikipedia in 100
https://tatoeba.org/eng/