Multilingual Pre-training with Self-supervision from Global Co-occurrence
Year: 2023

Abstract:
Global co-occurrence information is the primary source of structural information on multilingual corpora, and we find that analogical/parallel compound words across languages have similar co-occurrence counts/frequencies (normalized) giving weak but stable selfsupervision for cross-lingual transfer. Following the observation, we aim at associating contextualized representations with relevant (contextualized) representations across languages with the help of co-occurrence counts. The result is MLM-GC (MLM with Global Cooccurrence) pre-training that the model learns local bidirectional information from MLM and global co-occurrence information from a logbilinear regression. Experiments show that MLM-GC pre-training substantially outperforms MLM pre-training for 4 downstream cross-lingual tasks and 1 additional monolingual task, showing the advantages of forming isomorphic spaces across languages.

heading: 1 Introduction
text: Empirical studies (Lample et al., 2018a; Conneau et al., 2020a,c) show multilinguality and crosslinguality emerge from MLM pre-training on multilingual corpora without any supervision. The model is trained/pre-trained as a generator that yields masked token probabilities over the vocabulary. To improve cross-lingual transfer, we present MLM-GC (MLM with Global Co-occurrence) with the combined objective of the generator and a global log-bilinear regression for multilingual pretraining. Our starting point is from two observations on multilingual MLM pre-training.
Language’s structural information is every property of an individual language that is invariant to the script of the language. Conneau et al. (2020c); Karthikeyan et al. (2020); Sinha et al. (2021); Pires et al. (2019) show that structural similarities across languages can contribute to cross-lingual transfer.
Co-occurrence information or n-gram is the primary source of structural information available to all methods. Some methods like span-based masking (Devlin et al., 2019; Joshi et al., 2020; Levine et al., 2021) now exist to leverage this information for new masking schemes in monolingual MLM pre-training, aiming at improving context understanding. However, in multilingual MLM pre-training, the question still remains as to how meaning is generated from these statistics on multilingual corpora, how the structural similarities could be learned from that meaning across languages, and how cross-lingual transfer might be improved from that meaning.
Furthermore, GloVe (Pennington et al., 2014) prove that leveraging global co-occurrence information can search for relevant information on monolingual embedding space. Inspired by GloVe, we assume that global co-occurrence information can also be used to search for relevant information across languages on multilingual corpora. This assumption underlies Zipf’s law (Ha et al., 2002; Søgaard, 2020) that analogical words and compound words across languages might have similar frequencies/counts on the multilingual corpora. Our empirical studies further justify our assumption that analogical/parallel compound words across languages have similar co-occurrence counts (normalized). Meanwhile, in multilingual MLM pre-training, one of the ultimate goals is to form contextualized representations. Then, global co-occurrence information might be used to regularize representation learning in multilingual MLM pre-training, which allows for better contextualized representations in cross-lingual transfer.
In this work, we present MLM-GC to utilize global co-occurrence information. MLM-GC builds on MLM with an extra objective of global log-bilinear regression that minimizes the error between dot products of neighboring contextualized representations and the matrix of global co-
7526
occurrence counts. Since MLM only needs to predict masked tokens, we only consider the contextualized representations of the masked tokens and their neighbors, factorizing relevant global cooccurrence counts from the matrix. The model is pre-trained to learn bidirectional information from MLM and the global co-occurrence information from the global log-bilinear regression. On multilingual corpora, MLM-GC pre-training can improve cross-lingual transfer because analogical/parallel compound words across languages might have similar co-occurrence counts allowing for cross-lingual transfer, which is justified in our empirical studies on translation pairs.
We have three contributions. 1) We present MLM-GC pre-training for multilingual tasks. The model is additionally supervised by co-occurrence counts on multilingual corpora. 2) MLM-GC pre-training outperforms MLM pre-training on 4 multilingual/cross-lingual tasks. The objective of MLM-GC can be adapted to encoder-decoderbased MLM models, e.g., MASS (Song et al., 2019) and encoder-based MLM models, e.g., XLM (Lample and Conneau, 2019). MLM-GC pre-training can also work on monolingual corpora for language understanding tasks. 3) MLM-GC pre-training can help the model to form isomorphic embedding spaces across languages, which is potentially useful for cross-lingual and multilingual tasks. Our empirical study shows that analogical compound words across languages have similar co-occurrence counts (normalized) contributing to structural similarities across languages for cross-lingual transfer.

heading: 2 Related Work and Comparison
text: Structural Similarity and Zipf’s Law Zipf’s law (Zipf, 1949, 2013; Søgaard, 2020) indicates that words or phrases appear with different frequencies, and one may suggest analogical words or phrases appear with relatively similar frequencies in other languages. In multilingual MLM pre-training, Conneau et al. (2020c); Karthikeyan et al. (2020); Pires et al. (2019); Karthikeyan et al. (2020); Sinha et al. (2021) shed light on studying structural information and find that structural similarities across languages are essential for multilinguality, where in this paper, structural similarities mean similar counts as Zipf’s law indicated. We follow this line, consider structural similarities from co-occurrence counts, and provide an empirical study to observe how the model learns
structural similarities from global co-occurrence counts on multilingual corpora. Meanwhile, GloVe (Pennington et al., 2014) report that co-occurrence counts can provide regularities for embeddings to understand word analogies for monolingual tasks. We extend the scope of GloVe to contextualized representations and multilingual tasks, helping the model form isomorphic spaces across languages in multilingual MLM pre-training.
N-gram, Co-occurrence, and Regularity in MLM pre-training Studying co-occurrence or n-gram is not a novel idea in MLM pre-training. Whole Word Masking (Devlin et al., 2019), SpanBERT (Joshi et al., 2020), and PMI-Masking (Levine et al., 2021) suggest n-gram spans across several sub-tokens for masking to improve context understanding in monolingual tasks because the model may only learn from easier multi-tokens instead of usefully hard context, where easier multitokens are in a subset of the context and result in sub-optimization. In contrast, we show that cooccurrence counts can refine contextualized representations for improving context understanding and allow for better cross-lingual transfer, suggesting a new objective for MLM pre-training instead of a new masking scheme to capture global cooccurrence information in multilingual pre-training. On the other hand, for cross-lingual transfer, the contextualized representations could be further regularized and refined by aligning cherry-picked pairs after MLM pre-training on multilingual corpora (Ren et al., 2019; Chaudhary et al., 2020; Wang et al., 2020; Cao et al., 2020; Aldarmaki and Diab, 2019; Artetxe et al., 2020; Ai and Fang, 2021). Compared to that, MLM-GC pre-training does not require dictionaries, translation tables, or statistical machine translation models.

heading: 3 Approach

heading: 3.1 Global Regression Modeling in Monolingual Embedding Space
text: GloVe (Pennington et al., 2014) presents a logbilinear regression model:
L = V∑
i,j=1
f(Xwiwj )(E T wiEwj − logXwiwj )2,
(1)
where f(x) =
{ (x/xmax) α, x < xmax
1, otherwise , V is
the vocabulary, Ew is the embedding of token w, X
stands for the matrix of token-token co-occurrence counts, entries Xwiwj tabulate the number of times token wj occurs in the context of token wi, and xmax is empirically set to 100. The model is able to distinguish relevant embeddings from irrelevant embeddings and discriminate between the two relevant embeddings.

heading: 3.2 Global Co-occurrence Modeling for Contextualized Representations
text: In MLM pre-training, when wt at the position t is replaced by the artificial masking token [M]t, the final hidden state or the contextualized representation H[M]t of position t is factorized from the final sequence representation of the input sentence to predict wt (with a softmax operation). We further factorize a neighboring contextualized representation Hwk (of wt) at position k for the neighboring token wk. Note that wk could be masked (if spanbased masking strategies are applied, e.g., MASK (Song et al., 2019) ) or unmasked (e.g., XLM (Lample and Conneau, 2019)), and we test both scenarios in our experiments. Then, similar to global regression modeling in monolingual embedding space, we consider a regression model 1:
L[M]twk = f(Xwtwk)(HT[M]tHwk − logXwtwk) 2. (2) For all the neighboring tokens wt±n of the input sentence at position [t − n, ..., t) ∪ (t, ..., t + n], i.e., excluding position t, we have the model LGC . Then, we employ the new global log-bilinear regression model in MLM pre-training. Formally, given the factorized Hwt±n and H[M]t from H and Xwtwt±n from X , we have the model:
LGC = 1
2n
∑
n
f(Xwtwt±n)( HT[M]tHwt±n√
d − logXwtwt±n)2,
(3) where d is the model dimension. Compared to Eq.1, we add scaling √ d and weight 12n to make
training stable, where √ d is inspired by scaled dotproduct attention (Vaswani et al., 2017) to prevent the dot products get large. They serve as principled hyper-parameters.
To obtain the matrix of token-token cooccurrence counts on multilingual corpora for multilingual tasks, we follow GloVe’s suggestion that a
1We provide an alternative in Appendix D
distance weight scheme is employed. Specifically, in a context window of size 2n + 1, we calculate the token-token co-occurrence counts for positions [t − n, ..., t, ..., k + n] with the rule [clang/(n + 1), ..., clang/2, 0, clang/2, ..., clang/(n + 1)] over the shared vocabulary, which means we do not calculate the unigram counts or self-co-occurrence Xwtwt for the centric token wt at position t. Meanwhile, we are aware that the probability is not normalized and equivalent to token-token co-occurrence counts on the multilingual corpora. However, not all the languages have the same amount of samples in the corpora (e.g., low-resource vs. high-resource). Considering this, we use the language-wise constant clang = CEn/Clang, where CEn is the total number of tokens in English corpora, and Clang is the total number of tokens in the language lang, i.e., cooccurrence counts are normalized by clang.

heading: 3.3 Multilingual MLM-GC Pre-training
text: In multilingual pre-training, we have a combined objective of MLM and global co-occurrence modeling2, attempting to train the model to understand the masked tokens from bidirectional information and linguistic structures surrounding the masked tokens from global co-occurrence counts, and the result is our MLM-GC pre-training:
LMLM−GC = LMLM + λLGC . (4)
In the early experiment, we experiment with λ ∈ {0.1, 0.5, 1, 2}. We find λ = 1 is a general choice for experiments. On the other hand, we find warm_up (Vaswani et al., 2017) of lr, √ d, and 12n (Eq. 3) are significant. The model might collapse to LGC without warm_up, √ d, or 12n because LGC converges too fast and is unstable. In this situation, the model ignores the objective of MLM. Then, the model can only learn co-occurrence information and does not learn the language knowledge. The result is presented in Table 1.
Improved Contextualized Representation LGC considers the correspondence in the context [t− n, ..., t) ∪ (t, ..., t+ n] with an explicit objective. In this way, the model is encouraged to learn from usefully hard context instead of easier multitokens under the supervision from co-occurrence information, where easier multi-tokens are in a subset of the context and result in sub-optimization
2We discuss the scope and limitation in §Limitation.
(Levine et al., 2021), as discussed in §Related Work. Meanwhile, co-occurrence counts help the model disambiguate word representations (Ai and Fang, 2022) in language modeling by distinguishing relevant information from irrelevant information and discriminating between the two relevant information in the language.
Improved Cross-lingual Transfer With the objective of LGC , we aim at associating HT[M]tHwt±n with HT[M ]t̃Hw̃t̃±n of different languages if Xwtwt±n = Xw̃t̃w̃t̃±n , where compound words wtwt±n and w̃t̃w̃t̃±n are analogical in different languages. In this way, it underlies the basic assumption that analogical compound words across languages have similar co-occurrence counts (normalized by clang), i.e., wtwt±n and w̃t̃w̃t̃±n are analogical compound words =⇒ Xwtwt±n = Xw̃t̃w̃t̃±n . Although Zipf’s law supports this assumption (Ha et al., 2002; Søgaard, 2020) in linguistics, we are still interested in the questions: how it reflects on the multilingual corpora we use and whether analogical pair of wtwt±n and w̃t̃w̃t̃±n =⇒ Xwtwt±n = Xw̃t̃w̃t̃±n . To answer these questions, we extract all the pairs of parallel compound words in En and De from the open-source translation tables (OPUS, Wikipedia v1.0)⋄, e.g., "ist die" (De) and "is the" (En), and compute co-occurrence counts on {De,En} Wikipedia dumps (the same dataset we use in our experiment). For any pair, we compute the absolute difference |log(De) − log(En)|, the sum log(De) + log(En) (sorted into bins), and the ratio |log(De)−log(En)|/(log(De)+log(En)) for statistics in Figure 1. The figure tells us that the absolute difference avg and the ratio avg for all the pairs are relatively small and have narrow confidence (95%) intervals. Although the absolute difference avg is proportional to the sum, the ratio avg has no proportional relationship with the sum and is small throughout all the bins. Note that some pairs have low translation scores resulting
in large absolute differences. The absolute difference avg is not 0, i.e., an exact match for any pair. However, it still confirms that analogical compound words across languages have similar (but not identical) co-occurrence counts, which might give weak (not 0) but stable (relatively small with high confidence) self-supervision for cross-lingual transfer. Meanwhile, the model is encouraged to distinguish relevant information from irrelevant information and to discriminate between the two relevant information across languages from co-occurrence counts and refine contextualized representations accordingly, which is beneficial for cross-lingual transfer. For example, in our experiment (n = 2), given the translation pair "ist die" (De) and "is the" (En), the relevant pair "ist die" and "is a" (En), the irrelevant pair "ist die" and "locally known" (En), we find |log(ist die) − log(is the)| = 0.67 < |log(ist die) − log(is a)| = 1.73 < |log(ist die) − log(locally known)| = 5.45 < |log(ist die) − log(En avg)| = 5.58, where log(En avg) is the avg co-occurrence counts in En.
Efficiency 1) Computing the co-occurrence matrix is laborious on large corpora. However, it requires a single pass through the entire corpora to collect the statistics, which is a one-time up-front cost and is easy to obtain new information from new corpora for updating. 2) For memories, the co-occurrence matrix is huge, e.g., ≈ 11 G for a 60k BPE vocabulary with float 32. However, it is somewhat trivial because the memory is allocated to CPUs, not GPUs. This can be automatically finished by DL platforms like TensorFlow. Also, the matrix can be formatted to float 16 or even float 8 by pre-logging the co-occurrence counts, which will significantly reduce the memory. 3) Meanwhile, we save the token-token cooccurrence matrix as dictionaries {(wi, wj): tokentoken co-occurrence counts} so that querying the co-occurrence counts for Xwiwj is O(1).
Tokenization Sub-token-level vocabularies may impact the co-occurrence counts. In extreme cases, several connective tokens of co-occurrence may only come from one word. However, As discussed in §Improved Contextualized Representation, even in this scenario, the model can be improved from the co-occurrence counts in MLM-GC pre-training. See experiments in Appendix B.

heading: 4 Experiment
text: All the links of datasets, libraries, scripts, and tools marked with ⋄ are listed in Appendix E. A preview version of the code is submitted, and we will open the source code on GitHub.

heading: 4.1 MLM Instance, Configuration, Data Preprocessing and Pre-training
text: We use XLM (Lample and Conneau, 2019) and MASS (Song et al., 2019) as the MLM instances, where XLM is a token-based encoder model, and MASS is a span-based encoder-decoder model (see Appendix §A.1 for more details). The Transformer configuration is identical to XLM and MASS, where word embeddings, hidden states, and filter sizes are 1024, 1024, and 4096 respectively (default). To be fair, we reimplement all the baseline models with our configurations, using official XLM⋄, Tensor2Tensor⋄, and HuggingFace⋄ as references. We compare the results of our reimplementation with the reported results on the same test set to ensure the difference less than 2% in overall performance (Appendix C). For the context window size 2n+ 1 of the co-occurrence counts and Eq.4, we set n = 2 for all the experiments, which is decided by our dev experiment.
Data preprocessing is identical to XLM and Mass. Specifically, we employ fastBPE⋄ to learn BPE (Sennrich et al., 2016b) with a sampling criterion from Lample and Conneau (2019) for all the experiments. To tokenize {Zh, Th,Ne}, we
use Stanford Word Segmenter⋄, PyThaiNLP⋄, and Indic-NLP Library⋄, respectively. For the others, we use the Moses tokenizer⋄ with default rules.
Our code is implemented on Tensorflow 2.6 (Abadi et al., 2016). We use Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9,β2 = 0.999, ϵ = 1e − 8, warm_up = 10000 (Vaswani et al., 2017) and lr = 1e− 4. We set dropout regularization with a drop rate rate = 0.1. The mini-batch size is set to 8192 tokens.

heading: 4.2 Multilingual Task
text: Readers can refer to Appendix §A.2 or references for more introductions to these tasks.
Cross-lingual Embedding We attempt MUSE⋄ (Lample et al., 2018a) tasks that measure similarities between two paired words to generally evaluate the degree of the isomorphism of languages’ embedding spaces. As discussed in Lample and Conneau (2019); Wang et al. (2020) and our preliminary experiment, the performance of the isomorphism is potentially proportional to the performance of cross-lingual transfer. We treat this experiment as our dev experiment to search for n.
UNMT UNMT (unsupervised neural machine translation) (Lample and Conneau, 2019; Lample et al., 2018b; Song et al., 2019; Liu et al., 2020) tackles bilingual translation (Bahdanau et al., 2015; Vaswani et al., 2017) on non-parallel bilingual corpora without any cross-lingual signal.
Cross-lingual Classification We test XNLI⋄ (Conneau et al., 2020b) on 15 languages (including English) under the cross-lingual transfer setting. The model is pre-trained on multilingual corpora and fine-tuned on the English dataset, aiming at zero-shot classification for other languages.
Cross-lingual Question Answering MLQA⋄ (Lewis et al., 2020b) on 7 languages (including English) requires identifying the answer to a question as a span in the corresponding paragraph. We pre-train the model on multilingual corpora and fine-tune it on the English dataset, and then we attempt zero-shot prediction for other languages.

heading: 4.3 Secondary Monolingual Task
text: Recall that, as presented in Eq.2, H[M]t is the contextualized representation or the final hidden state. Therefore, MLM-GC pre-training is general and can work for other MLM instances such as BERT (Devlin et al., 2019), mBART (Liu et al., 2020), SpanBERT (Joshi et al., 2020), BART (Lewis et al., 2020a), and ALBERT (Lan et al., 2020). Meanwhile, MLM-GC pre-training is substantially better than MLM pre-training beyond multilingual tasks. We provide further experiments on monolingual tasks including SQuAD v1&v2 (Rajpurkar et al., 2016) in Appendix §B, using ALBERT as the MLM instance.

heading: 5 Result

heading: 5.1 Cross-lingual Embedding and Understanding Co-occurrence
text: Setup We configure an identical MLM instance to XLM with a 12-layer Transformer encoder. However, instead of 80K BPE and 15 languages in the original work, we learn 60K BPE and pretrain the model on Wikipedia dumps⋄ of the 2 languages. After 400K pre-training steps, we extract the embeddings required by the test set from the embedding space of the model. For words split into 2+ sub-tokens, we average all the sub-token
embeddings. See details in Appendix A.2.1. As mentioned early, this is our dev experiment.
Performance We follow the instruction to compute the cosine similarity for the MUSE task, reporting the result in Table 2 for En ↔ De test sets. MLM-GC pre-training outperforms the baseline model with different n. A large n does not consistently improve performance. We suspect that a large n may impact the capacity of the contextualized representation, which makes the model hard to be trained. Furthermore, n = 2 shows the best performance, and we may explain that in our comparison of co-occurrence counts (Figure 1), n = 2 has slightly smaller absolute difference avg and the ratio avg and narrower confidence (95%) intervals in large-count bins (> log(1e7)) contributing to over 45% co-occurrence counts on the multilingual corpora. Since we do not inject any cross-lingual supervision into the embedding space, this test can quantitatively report how MLM-GC refines the language spaces from co-occurrence counts for the isomorphic space and multilinguality.
Visualization and Multilingual Word Analogy We visualize all the words from the MUSE test sets. Since the task is originally designed for word translation including nouns, verbs, and other meaningful words, analogical words should be clustered and aligned in isomorphic spaces. As reported in Google’s NMT (Johnson et al., 2017), the t-SNE can visualize isomorphic spaces across languages. Then, we employ the t-SNE visualization 3 to observe the isomorphic space. Figure 2 shows that MLM-GC pre-training help the model learn to form a better isomorphic space than MLM pre-training. Another insight is from the classic analogy test: "English: King - Man + Woman = Queen and German: König-Mann+Frau = Königin", and we show results in Table 3. MLM-GC pre-training consistently improves the performance on monolingual tests (only English or German) and multilingual tests (mixing English with German). Then, we can further observe the effectiveness of our method in improving the quality of isomorphic spaces across languages.

heading: 5.2 UNMT
text: Setup&Training We consider two similar language pairs {De,Ro} ↔ En from WMT⋄ (Bojar et al., 2018) and a dissimilar language pair
3We reduce the dimension of embeddings to 3 by using PCA and then configure the t-SNE visualization.
En ↔ Ne (Nepali) from FLoRes⋄ (Guzmán et al., 2019). Transformer, configurations, corpora, and BLEU scripts are identical to XLM and MASS. We pre-train the model around 400K iterations on only monolingual corpora of the two languages. After MLM-GC pre-training, we follow XLM and MASS to train the model for translation from pre-trained weights. In the training phase, we use Adam optimizer with parameters β1 = 0.9, β2 = 0.997 and ϵ = 10−9, and a dynamic learning rate with warm_up = 8000 (learning_rate ∈ (0, 7e−4]) is employed. We set dropout with rate = 0.1 and label smoothing with gamma = 0.1. After around
400K iterations, we report results. See details in Appendix A.2.2.
Performance In Table 4, we report multiBLEU.perl⋄ to compare with XLM and MASS and sacreBleu⋄ to compare with mBART (Liu et al., 2020) so that the evaluation is based on the same BLEU script. MLG-GC pre-training consistently improves the performance of baseline models on all the similar language pairs by 3% ∼ 7% and on the dissimilar pair by 2.5 ∼ 5.0 BLEU. The performance on the dissimilar pair is competitive to SOTA: mBART and is better than mBART on similar language pairs. However, mBART uses CC25 (Wenzek et al., 2020) for pre-training and obtains benefits from more languages (25 languages) and samples. The global co-occurrence information across languages is general and abstract for isomorphic spaces, which allows for cross-lingual representations. It eventually helps the model understand translation knowledge. Meanwhile, we observe substantial gains on MASS + OURS (and ALBERT (Lan et al., 2020) in Appendix B), where MASS (ALBERT) is based on span masking. As discussed in §Related Work and Introduction, spanbased masking (also including Whole Word Masking (Devlin et al., 2019) and PMI-Masking (Levine et al., 2021) ) implicitly leverages co-occurrence information for improving context understanding. In addition to the empirical study in Figure 1, the gain further confirms that global co-occurrence information significantly injects some signals for crosslingual transfer beyond improving context understanding.

heading: 5.3 Cross-lingual Classification
text: Setup&Fine-tuning The model configuration, preprocessing, and corpora are identical to XLM4. For the classification objective, we deploy a linear classification layer on top of the encoder. Af-
4In the literature, this setup also refers to XLM-15.
ter pre-training, we deploy the randomly initialized linear classifier and fine-tune the encoder and the linear classifier on the En NLI dataset with mini-batch size 16. We use Adam optimizer with lr = 5 × 10−4 and linear decay of lr. After finetuning, we make zero-shot prediction for the other 14 languages. See details in Appendix A.2.3.
Performance We report the result in Table 5. Our method consistently improves baseline models by 3.5% (Avg). As discussed in previous models (Conneau et al., 2020b; Karthikeyan et al., 2020; Wu and Dredze, 2019; Pires et al., 2019; Dufter and Schütze, 2020), multilinguality is essential for this task. Then, we confirm the effectiveness of MLM-GC pre-training. Furthermore, our method outperforms XLM + PMI-Masking (span-based). Similar to the comparison in UNMT, MLM-GC pre-training uses co-occurrence information for better context understanding and cross-lingual transfer, whereas XLM + PMI-Masking leverages cooccurrence information for context understanding but performs worse for cross-lingual transfer because of the lack of a mechanism to help crosslingual transfer. We also include XLM + TLM (Lample and Conneau, 2019) for comparison. In this experiment, XLM + TLM using parallel sentences in pre-training slightly outperforms MLMGC, which indicates the knowledge gap between co-occurrence information and parallel sentences for cross-lingual supervision. Besides, when applying MLM-GC pre-training for XLM + TLM, we still observe gains. We attribute the additional gains to the contextualized representations that are further refined by co-occurrence information to represent similar abstractions for cross-lingual transfer.
Intuitively, the co-occurrence information gives extra cross-lingual supervision beyond a limited amount of parallel sentences.

heading: 5.4 Cross-lingual Question Answering
text: Setup&Fine-tuning The setup is similar to §Cross-lingual Classification. We follow the instruction of SQuAD from BERT, fine-tuning the model with a span extraction loss on the English dataset. We use Adam optimizer with lr = 5 × 10−5 and linear decay of lr. As suggested, we finetune the model on SQuAD v1.1 (Rajpurkar et al., 2016) and then make zero-shot prediction for the 7 languages of MLQA. See details in Appendix A.2.4.
Performance In Table 6, MLM-GC pre-training substantially improves the performance (Avg) in both F1 and EM metrics by 4.8 % and 5.0 % respectively. Meanwhile, MLM-GC pre-training yields more improvements for low-resource languages. We attribute all the improvements to the global cooccurrence objective the model learns in MLM-GC pre-training. Intuitively, spans (groups of words) of answers across languages are most likely to consist of nouns and terms and can be easily represented, clustered, and aligned in the improved isomorphic space because they are analogical and might have similar co-occurrence counts as discussed in the empirical study (Figure 1).

heading: 6 Conclusion
text: In this work, we leverage the global co-occurrence information from multilingual corpora. The result is MLM-GC pre-training with a combined objective of MLM and global co-occurrence modeling.
Our experiments show that MLM-GC pre-training can substantially improve the performance of naive MLM pre-training for 4 multilingual tasks, and additional experiments show that it can work for monolingual tasks. The isomorphic space across languages benefits from co-occurrence information, which allows for cross-lingual transfer. Meanwhile, the model is encouraged to distinguish relevant information from irrelevant information and to discriminate between the two relevant information across languages from co-occurrence counts (normalized) and refine contextualized representations accordingly. We believe that leveraging cooccurrence information for cross-lingual transfer is an interesting avenue in multilingual pre-training.

heading: 7 Limitation
text: Theoretically, our method might benefit from comparable corpora across languages, where words and compound words might have similar distribution because Zipf’s law might be satisfied only for similar domains. For instance, as presented in Figure 3, word distributions of De and En on Wikipedia are similar after applying BPE. In our experiments, we only confirm the effectiveness of our methods on Wikipedia corpora in different languages, which are comparable across languages. This might limit the scope of our method. However, multilingual models are commonly pre-trained on comparable corpora, e.g., Wikipedia and CC.
Another limitation is about the combined objective in Eq. 4. In our experiments, we try to
eliminate the MLM objective, only considering global regression modeling LGC . The result is not promising, and it seems that LGC can not work well without the help of the MLM objective. However, our experiment is very simple. This might be further confirmed or designed in future work.

heading: A Experiment
text: A.1 MLM Instance
We adapt our method to two MLM instances: XLM (Lample and Conneau, 2019) and MASS (Song et al., 2019). We follow the instructions of BERT (Devlin et al., 2019) that each selected token is replaced with the probabilities (p[unchanged], p[random], p[mask]) = (0.1, 0.1, 0.8).
XLM XLM is similar to BERT (Devlin et al., 2019) but uses text streams of an arbitrary number of sentences. Following the instruction, we randomly select 15% of the tokens from the input sentence for replacing.
MASS MASS is different from XLM and BERT but similar to SpanBERT (Joshi et al., 2020), using spans to replace consecutive tokens. Given an input sentence with length N , we randomly select consecutive tokens with length N/2 for replacing.
A.2 Multilingual Task
A.2.1 Cross-lingual Embedding We are interested in the isomorphism of languages’ embedding spaces. To investigate, we attempt MUSE⋄ tasks (Lample et al., 2018a) that measure similarities between two paired words. This test can generally evaluate the degree of the isomorphism of languages’ embedding spaces. Meanwhile, as discussed in Lample and Conneau (2019); Wang et al. (2020) and our preliminary experiment, the performance of the isomorphism is potentially proportional to the performance of cross-lingual transfer learning tasks. Therefore, we treat this experiment as our dev experiment to search for n.
Setup We configure a 12-layer Transformer encoder and use Moses tokenizer⋄ with default rules for tokenization, identical to XLM (Lample and Conneau, 2019). For fast dev experiment, we employ fastBPE⋄ to learn 60K BPE (Sennrich et al., 2016b) from concatenated corpora with a sampling criterion from (Lample and Conneau, 2019) and pre-train the model on 2 languages instead of 80K BPE and 15 languages in the reported work.
Training In the pre-training phase, we pre-train the model on Wikipedia dumps⋄ of the two languages for 400K steps. After pre-training, we extract the words’ embeddings required by the test set from the embedding space of the model. For
words split into 2+ sub-tokens, we average all the extracted embeddings of sub-tokens. We then evaluate paired embeddings in cosine similarity.
A.2.2 UNMT UNMT (unsupervised neural machine translation) (Lample and Conneau, 2019; Lample et al., 2018b; Song et al., 2019; Liu et al., 2020) tackles bilingual translation (Bahdanau et al., 2015; Vaswani et al., 2017) on non-parallel bilingual corpora without having access to any parallel sentence. In other words, there is no supervision for translation. The model requires pre-training to obtain some initial multilingual knowledge for decent performance.
Setup We configure an identical Transformer model to XLM (Lample and Conneau, 2019) and MASS (Song et al., 2019), which has 6 layers in both the encoder and decoder using default configurations. We consider multiple families of languages. Specifically, we consider similar language pairs {De,Ro} ↔ En, using the same dataset as previous works (Lample and Conneau, 2019). The dataset consists of monolingual corpora {De,En} from WMT 2018⋄ (Bojar et al., 2018) including all available NewsCrawl datasets from 2007 through 2017 and monolingual corpora Ro from WMT 2016⋄ (Bojar et al., 2016) including NewsCrawl 2016. We report the performance for {De,Ro} ↔ En on newstest2016. Meanwhile, we share the FLoRes⋄ (Guzmán et al., 2019) task to evaluate a dissimilar language pair Ne ↔ English (Nepali). For tokenization, we use the Moses tokenizer⋄ developed by Koehn et al. (2007) with default rules except for Ne that is tokenized by Indic-NLP Library⋄. We employ fastBPE⋄ to learn 60K BPE (Sennrich et al., 2016b) from concatenated corpora of paired languages, using the same sampling criteria in Lample and Conneau (2019). We use learnable language embeddings and position embeddings.
Training In MLM-GC pre-training, the model is pre-trained around 400K iterations on only monolingual corpora of different languages. In the training phase, we use Adam optimizer (Kingma and Ba, 2015) with parameters β1 = 0.9,β2 = 0.997 and ϵ = e − 9, and a dynamic learning rate with warm_up = 8000 and learning_rate ∈ (0, 7e− 4]) (Vaswani et al., 2017) is employed. We set dropout regularization with a drop rate rate = 0.1 and label smoothing with gamma = 0.1 (Mezzini, 2018). On-the-fly back-translation (Sennrich et al.,
2016a) (the inference mode of the model) performs to generate synthetic parallel sentences that can be used for training of translation as NMT (neural machine translation) is trained on genuine parallel sentences in a supervised manner. Meanwhile, UNMT learns an objective of denoising language modeling (Vincent, 2010) to maintain language knowledge in the training phase except for MASS. After around 400K iterations, we report BLEU computed by multi-BLEU.perl⋄ and scareBLEU⋄ with default rules, according to baseline models. In conclusion, in pre-training, we only have the objective of MLM-GC, and in training, we have the two objectives: 1) denoising language modeling for XLM or MASS itself and 2) translation (i.e., NMT), where the translation objective is finished by using synthetic pairs sentences from on-the-fly back-translation.
A.2.3 Cross-lingual Classification We experiment with XNLI⋄ (Conneau et al., 2020b), a general cross-lingual classification task on 15 languages (including English) under the cross-lingual transfer setting. The model takes in two input sentences and is required to classify into one of the three labels: entailment, contradiction, and neutral. The model is fine-tuned on the English dataset and then attempts zero-shot classification for other languages.
Setup Following the previous work5 (Lample and Conneau, 2019), we use raw sentences including 15 XNLI languages from Wikipedia dumps downloaded by WikiExtractor⋄. Sentences in different languages are sampled with the method of Lample and Conneau (2019). The model configuration and preprocessing are identical to XLM that we use a 12-layer transformer encoder and 80K BPE. For the classification objective, we deploy a linear classification layer on top of the encoder. To tokenize {zh, th}, we use Stanford Word Segmenter⋄ and PyThaiNLP⋄ respectively. For the others, we use the Moses tokenizer⋄ with default rules. Similar to the Cross-lingual Embedding experiment, we use fastBPE⋄ and the sampling strategy to learn BPE.
Fine-tuning After pre-training on the corpora, we deploy a randomly initialized linear classifier and fine-tune the encoder and the linear classifier on the En NLI dataset with mini-batch size 16. We
5In the literature, this setup also refers to XLM-15.
use Adam optimizer (Kingma and Ba, 2015) with lr = 5e − 4 and linear decay of lr. After finetuning, we make zero-shot prediction for the other 14 languages. We use categorical cross-entropy with three labels: entailment, contradiction, and neutral.
A.2.4 Cross-lingual Question Answering We consider the MLQA⋄ (Lewis et al., 2020b) dataset for a cross-lingual question answering task. Given a question and a passage containing the answers, the aim is to predict the answer text span in the passage. This task requires identifying the answer to a question as a span in the corresponding paragraph. The evaluation data for English and 6 other languages are obtained by automatically mining target language sentences that are parallel to sentences in English from Wikipedia, crowdsourcing annotations in English, and translating the question and aligning the answer spans in the target languages. Similar to the cross-lingual classification task, the model is fine-tuned on the English dataset and then attempts zero-shot prediction for other languages.
Setup The setup is similar to the experiment of cross-lingual classification.
Fine-tuning We follow the instruction of SQuAD from BERT (Devlin et al., 2019), finetuning the model with a span extraction loss on the English dataset. We use Adam optimizer (Kingma and Ba, 2015) with lr = 5e − 5 and linear decay of lr. Meanwhile, as suggested, we fine-tune the model on the SQuAD v1.1 (Rajpurkar et al., 2016) dataset and then make zero-shot prediction for the 7 languages of MLQA. Given a sequence T , we only have a start vector S ∈ Rhidden and an end vector E ∈ Rhidden during fine-tuning. The probability of word i being the start of the answer span is computed as a dot product Ti and S d by a softmax over all of the words in the sequence pi =
eSTi∑ k∈T e ETk . Similarly, we can compute the
end of the span. The score of a candidate span from position i to position j is defined as STi+ETj and the maximum scoring span where j ≥ i is used as a prediction.

heading: B Additional and Supportive Result
text: B.1 Pre-training for Monolingual Task Although we derive our method from the observation of multilingual models, MLM-GC pre-training
is substantially better than MLM pre-training. We provide further experiments on monolingual tasks including SQuAD v1&v2 (Rajpurkar et al., 2016).
setup For this monolingual task, our configuration is identical to 12-base-ALBERT (Lan et al., 2020). Specifically, We set the model dimension, word embedding dimension, and the maximum number of layers to 768, 128, and 12. As recommended, we generate a masked span for the MLM targets using the random strategy from Joshi et al. (2020), and we use LAMB optimizer⋄ with a learning rate of 0.00176 (You et al., 2020) instead of Adam optimizer. Following the instructions, we pre-train models on BooksCorpus⋄ (Zhu et al., 2015) and English Wikipedia⋄ (Devlin et al., 2019) for 140k steps.
Fine-tuning Similar to the cross-lingual question answering task, we fine-tune the pre-trained model on SQuAD(v1.1 and v2.0)⋄ (Rajpurkar et al., 2016, 2018).
Result Table 7 shows that MLM-GC pre-training is substantially better than MLM pre-training when pre-training 12-base-ALBERT for monolingual tasks. These observations confirm the effectiveness of MLM-GC pre-training on monolingual tasks.
B.2 Impact of Tokenization Method We are interested in how the tokenization method affects the performance because it potentially affects the token-token co-occurrence counts. For evaluation, we use all the configurations in UNMT and additionally configure a word-level vocabulary for the model. The word-level vocabulary has the same number of tokens as the BPE vocabulary. Table 8 shows that our method can work with different tokenization methods. Our method can generally improve the performance, regardless of the difference between the two baseline models in the same configuration.

heading: C Reimplementation
text: We compare our reimplementation with reported results in Table 9.

heading: D Alternative
text: In MLM pre-training, when wt at the position t is replaced by the artificial masking token [M]t, the output distribution for wt is obtained by applying a pre-softmax linear transformation O ∈ Rd×V from the final hidden state or the contextualized representation H[M]t to the output vocabulary size V , followed by a softmax operation which generates an output matrix normalized over its rows. Specifically, Q[M]twt = exp(HT [M]t Owt )∑V
k=1 exp(H T [M]t Owk ) is the model
for the probability of wt in the context of H[M]t , where Owt and Owk are vectors factorized from O, i.e., self-recognizing. In this way, the probability of wn in the context H[M]t is similar to Q[M]twt in the global regression model. Specifically, for wn, Q[M]twt could be extended to:
Q[M]twn = exp(HT[M]tOwn)∑V k=1 exp(H T [M]tOwk) . (5)
For all the neighboring tokens wt±n of the input sentence at position [t−n, ..., t)∪(t, ..., t+n], i.e., excluding position t, we have the model Q[M]twt±n . Then, we employ the new global log-bilinear regression model in MLM pre-training. Formally, given the factorized Owt±n and Xwtwt±n from O and X respectively, we have the model:
LGC = 1
2n
∑
n
f(Xwtwt±n)
( HT[M]tOwt±n√
d − logXwtwt±n)2,
(6)
where d is the model dimension.
In Table 10 and Table 11, we show the experimental results (also see our previous revision https://openreview.net/forum?id=DswOSXvLfuy). In conclusion, the presented method Eq. 3 slightly outperforms the alternative Eq. 6 on sentence-level tasks. We explain that the alternative involves neighboring embeddings in the objective, which directly improves the quality of cross-lingual embeddings. Compared to that, the presented method of the main paper considers contextualized representations of the masked tokens and their neighboring tokens, which is better for cross-lingual transfer.

heading: E Source
text: We list all the links of dataset, tools, and other sources in Table 12. Note that for multilingual tasks, datasets can be downloaded from the XTREME link except for UNMT and crossembeddings.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1 line 75
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.3 line 320 and section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.3 and Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This might be further confirmed or designed in future work