Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Year: 2023

Abstract:
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing stateof-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications. We show that going beyond Englishcentric bitexts, coupled with a novel sampling strategy aimed at reducing under-utilization of training data, substantially boosts performance across model sizes for both Electra and MLM pre-training objectives. We introduce XY-LENT: X-Y bitext enhanced Language ENcodings using Transformers which not only achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands, is also competitive across bands. Our XYLENTXL variant outperforms XLM-RXXL and exhibits competitive performance with mT5XXL while being 5x and 6x smaller respectively. We then show that our proposed method helps ameliorate the curse of multilinguality, with the XYLENTXL achieving 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band. We then analyze our models performance on extremely low resource languages and posit that scaling alone may not be sufficient for improving the performance in this scenario.

heading: 1 Introduction
text: Recent advancements in Natural Language Processing (NLP) have been a direct consequence of leveraging foundational models (Bommasani et al., 2021), pretrained on a large text corpora in a selfsupervised fashion. This has also been the case for multilingual NLP where pre-trained models like multilingual BERT (mBERT) (Devlin, 2018; Devlin et al., 2019), XLM (Conneau and Lample, 2019), XLM-Roberta (Conneau et al., 2020), XLMElectra (Chi et al., 2022) and mT5 (Xue et al., 2021)
∗Equal Contribution.
have all shown non-trivial performance gains, especially in the setup of zero-shot transfer, and have been the work-horse for a diverse number of multilingual tasks. Given their ubiquitous applicability in zero-shot downstream scenarios, improving the quality and enabling their usage in resourceconstrained applications is also an important vein of research which we explore in this paper.
A source of improvement for these models has been leveraging bitext data for better representation learning (Conneau and Lample, 2019; Chi et al., 2022). Most prior work, however, has focused on leveraging English-centric (EN-X) bitext data. Contemporaneously, the related area of Massively Multilingual Machine Translation (a single model for translating between different pairs of languages, eg: Aharoni et al. (2019); Zhang et al. (2020); Fan
15354
et al. (2021)) has shown tremendous progress, with Fan et al. (2021) showing that a crucial aspect of this improvement has been moving beyond EN-X parallel corpora and leveraging web-based mined X-Y bitexts spanning 1000s of translation directions (Schwenk et al., 2021a; El-Kishky et al., 2020; Schwenk et al., 2021b). This makes a compelling case to explore if leveraging X-Y bitexts can also improve multilingual representation learning.
In this work, we introduce XY-LENT (pronounced as "Excellent"): X-Y bitext enhanced Language ENcodings using Transformers. We first identify problems with using the commonly used sampling strategy proposed in Fan et al. (2021), showing that it induces sparse sampling distributions leading to under-utilization of data, and thus propose a novel strategy to mitigate this issue (§3.2). We then propose leveraging X-Y bitexts in conjunction with the improved sampling strategy, as well as a VoCAP (Zheng et al., 2021) style sentencepiece vocabulary re-construction for improving multilingual representation learning (§3.1). We show that our proposed method improves performance across all model size bands (§6). Furthermore, these performance gains hold for both Masked Language Models (MLM) and ELECTRA style models. Our approach results in an almost 12x speedup in training for MLM model training (§6.2). We systematically analyse the impact of model scaling with respect to the curse of multilinguality (Conneau et al., 2020) to observe that the gap between current English only SoTA models and multilingual models can be considerably reduced (§6.3). Our analysis reveals that XY-LENT improves performance across language families (§6.4) and helps reduce the cross-lingual transfer gap in multilingual tasks (§6.5). We then demonstrate that the training dynamics of such models can be used to better understand the underlying datasets and use it to find interesting defects in them (§6.6). Finally, we show some limitations of such multilingual representational models vis-à-vis extremely low resource languages, identifying potential shortcomings that are not addressed with scaling of such models, as well as issues around catastrophic forgetting in the way current models are used for domain adaptation.
In doing so, we establish state of the art on 5 multilingual downstream tasks (XNLI, PAWS-X, TYDIQA, XQuAD and MLQA) within a model size band, and achieve competitive performance across
size bands, thereby showing for the first time (to the best of our knowledge) an interesting notion of parameter efficiency: XY-LENTXL outperforms XLM-RXXL (Goyal et al., 2021) and performs competitively with mT5XXL (Xue et al., 2021), whilst being 5x and 6x smaller respectively (Figure 1). Furthermore, our proposed model reduces the gap for English specific tasks: XY-LENTXL achieves 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band.

heading: 2 Related Work
text: Large scale self-supervised learning has emerged as a prominent way of building cross-lingual language models that can be adapted for numerous multilingual downstream applications. Especially for building multilingual encoder transformer (Vaswani et al., 2017) models, two popular paradigms have been Masked language modeling (MLM; Devlin et al. (2019); Conneau et al. (2020)) and pre-training encoders as discriminators (ELECTRA; Clark et al. (2020b); Chi et al. (2022)), with the latter showing considerable compute efficiency. These approaches can further be improved by leveraging parallel corpora in different ways: Conneau and Lample (2019) propose a Translation Language Modeling task (TLM) wherein the model predicts masked tokens in concatenated translation pairs, Chi et al. (2022) propose a Translation Replaced Token Detection (TRTD) task, an analogous task for Electra-style models. Other approaches include using bitexts to construct code-switched sequences as inputs during pre-training (ALM; Yang et al. (2020)) and for contrastive learning (InfoXLM; Chi et al. (2021a)), or using token-level alignments in parallel data to improve cross-lingual modeling (Hu et al., 2021; Chi et al., 2021b, inter alia). However, all the aforementioned works rely on Englishcentric bitexts.
Fan et al. (2021) show that moving beyond EN-X bitexts for Massively Multilingual Machine Translation affords substantial improvements over approaches that rely solely on English-centric data (Aharoni et al., 2019; Zhang et al., 2020). The primary factor responsible for this improvement has been the curation of X-Y aligned bitext data, constructed by mining bitexts from publicly available web data (Schwenk et al., 2021a; El-Kishky et al., 2020; Schwenk et al., 2021b). The dataset construction either follows a local mining approach
(first aligning documents using heuristics, and then mining parallel bitexts from the aligned documents; used in CCAligned (El-Kishky et al., 2020)), or a global mining approach (all bitexts are embedded in a common vector space, and then aligned candidates are found by looking at the normalized nearest neighbors; used in CCMatrix (Schwenk et al., 2021b)). Fan et al. (2021) also propose a sampling strategy for leveraging the X-Y bitexts, wherein the marginals are constrained to be similar to what is used for En-X bitexts, and show their proposed method improves over uniform sampling. However, as we show in (§3.2), their proposed strategy has the undesirable artefact of inducing extremely sparse solutions, thereby resulting in data wastage.

heading: 3 Leveraging Many-to-Many Bitexts

heading: 3.1 Dataset
text: Prior representation learning works usually consider English-centric (EN-X) bitexts to improve model quality. Thus, given the emergence of mining based approaches for extracting parallel bitexts from large monolingual datasets that are approximate translations of each other and are multi-way aligned (the source and target languages are not restricted to be English only), in this work we explore leveraging these many-to-many (X-Y) bitext datasets for better representation learning. We consider two such publicly available datasets: CCMatrix and multiCCAligned.

heading: 3.2 Sampling Distribution
text: A common method used for balancing training data for the EN-X framework is using a temperature based exponential sampling approach (Aharoni et al., 2019), wherein the probability of sampling a language is chosen from a temperature smoothed distribution to downsample high resource languages, whilst upsampling low resource languages. This work was extended by Fan et al. (2021), wherein the authors propose Sinkhorn Temperature sampling: given a joint probability matrix Q across L×L language pairs (L being the number of unique languages), and the marginal distribution p of the L languages, the authors estimate a sampling distribution P∗ as:
max P
Tr(PQ) | P1L = p 1 T = P⊤1L (1)
where Tr is the trace operator. The primary advantage of using this is that P∗ can be efficiently
estimated with the Sinkhorn-Knopp algorithm and also allows us to set the marginal to be the temperature sampled based distribution which we know works well in practice. The authors found this to work better than uniform sampling.
However, in practice, we observed this to generate extremely sparse sampling distributions: Figure 2a show the sparsity induced by the naive application of Eq. 1.
We note that one potential way of overcoming the above issue is by modifying the optimization problem to also maximize the entropy of P. Consequently, we propose the following modified optimization objective :
P∗ = argminPTr (P (− logQ))−H (P ) | P1L = p 1 T = P⊤Q) | P1L = p 1 T = P⊤1L
(2) where H(P ) denotes the entropy of P and KL(P ||Q) denotes the Kullback-Leibler divergence between P and Q.
This can be solved by using the Sinkhorn-Knopp algorithm for the entropic regularized optimal transport problem (Cuturi, 2013), by setting the cost matrix to be − log(Q+ ϵ) (in practice, since Q can have zero entries, ϵ is used for smoothing). Since the cost of assigning a non-zero probability value to a zero entry is extremely high (− log (ϵ)), we never observe any entry of P∗ to be non-zero if it’s corresponding entry in Q was zero. In addition, since Eq. 2 also maximizes the entropy of P, it encourages its entries to be non-sparse, thereby avoiding the problem present in the solution of Eq. 1. In practice, we did not see this losing out on any data: if Q was non-zero, then P∗ was also non-zero (Figure 2b).

heading: 3.3 Vocabulary Construction
text: We construct our vocabulary using Sentence Piece Models (SPM) (Kudo and Richardson, 2018) which cater to language specific complexities (tokenization, accent removal, etc. ). We increase the vocabulary size to 500k tokens to better serve the varied scripts encountered while working in the multilingual setting. For this construction, we follow the VoCAP algorithm (Zheng et al., 2021) to quantify the vocabulary capacity for each language separately and account for varied corpora sizes across languages. Better capacity allocation leads to smaller representative sequences (especially for mid and low resource languages) which
in-turn improves the computational efficiency of the model. Increasing the size of the vocabulary, however, comes at the cost of inflating the model parameters which is particularly observed in the case of XY-LENTBase and XY-LENTLarge where the embeddings layer constitute 80.5% and 62.9% of the total parameters respectively.

heading: 4 Pretraining Details
text: We follow the XLM-E (Chi et al., 2022) pretraining approach and only introduce a few architectural changes to improve the overall performance of the model. We use the Transformer model (Vaswani et al., 2017) trained with ELECTRA (Clark et al., 2020b) style of replace token detection (RTD) on both monolingual (MRTD) and bitext (TRTD) data. In the current setup of training, we use two Transformer encoders in conjunction: a generator G and a discriminator D, where the generator G is trained with masked language modeling objective (MLM; Devlin et al. (2019)) and the discriminator is trained on replaced token detection objective (RTD; Clark et al. (2020b) on all the tokens passing through the generator.
In addition to using the Gated Relative Position Bias introduced in Chi et al. (2022), we do not mask the [CLS] token and flip bitext language order with probability p = 0.5 for the TRTD task.

heading: 5 Experiments
text: Baselines: We compare the cross-lingual performance of our proposed model against 3 popular
cross-lingual models: XLM-R, mT5 and XLM-E (across all model size variations). Note that Chi et al. (2022) use a 250k vocabulary size for XLMEBase and 500k vocabulary for their large and XL variants. As a follow-up, we re-train XLM-EBase with the same vocabulary as used by XY-LENT for a fair comparison. Thus all references to XLMEBase refer to the re-trained model variant with a 500k vocabulary size.1 For our downstream English evaluation (§6.3), we compare against the SoTA English model METRO-LM(Bajaj et al., 2022). Note that Bajaj et al. (2022) also train the models in an ELECTRA style framework, thereby allowing for a fair comparison.
Pretraining Data: For our monolingual data, we follow Chi et al. (2022) and use the CC-100 dataset2 (Conneau et al., 2020; Wenzek et al., 2020) which contains texts in 100 languages collected from Common Crawl. As mentioned in (§3.1), we explore the utility of the CCMatrix and the multiCCAligned X-Y aligned bitext data. CCMatrix consists of 1015 language pairs (97 unique languages) 3; while the multiCCAligned dataset consists of 2959 language pairs (92 unique languages) 4. We contrast this against only using EN-X bitexts (CCAligned, El-Kishky et al. (2020)).
1We also ablate out the impact of the vocabulary change, with Table 2 showing that this yields a 1.5 pt gain on XNLI.
2http://data.statmt.org/cc-100/ 3For some language pairs that are present in CCAligned and not in CCMatrix, we combine the data from those languages. Since de-duplication is expensive, we don’t merge language pairs common to both datasets.
4We filter out languages with less than 50k pairs
Model Size Bands: While our base and large models have more parameters when compared with XLM-R, most of the additional parameters come from the increased vocabulary size (§3.3). Concretely, our base model has 12 layers and 768 hidden states, while the large model has 24 layer and 1024 hidden states, which is identical to XLMRBase and XLM-RLarge respectively. However, even with the increased parameter count, the computation cost on a text classification task is roughly the same within a model size family (since mapping tokens to an embedding is a lookup operation). Finally, it is noteworthy that even with the increased vocabulary size, the number of parameters for XYLENTXL is less compared to the XL and XXL variants of both XLM-R and mT5.
Pretraining Setup: For the base model, we train for 125k steps with a batch size of 8192 for MRTD task and for the large model, we train the model for 500k steps with a batch size of 2048. Finally for the XL model, we train for 150k steps with a batch size of 8192. We use a dynamic batch size for TRTD task which is based on original length of the translated bi-text pair. Please refer Appendix A for additional details. We adopt the standard practice of using a linear warmup schedule for the learning rate and use the Adam (Kingma and Ba, 2014) optimizer for all the models. Following Meng et al. (2021), we do not apply any dropout to the generator.
Cross-lingual Downstream Evaluation: For evaluating the cross-lingual understanding of the model, we consider 5 multilingual evaluation benchmarks. We consider 2 classification tasks and 3 question answering tasks. For classification, we evaluate on the cross-lingual Natural Language Inference dataset (XNLI; Conneau et al. (2018)) and the cross-lingual paraphrase adversaries from word scrambling dataset (PAWS-X; Yang et al. (2019)). For cross-lingual question answering, we consider MLQA (Lewis et al., 2019), XQuAD (Artetxe et al., 2019) and TyDiQA-GoldP (Clark et al., 2020a). For all the aforementioned tasks, we perform the evaluation in zero-shot setting, i.e. only using the English data for fine-tuning. To further assess the model’s performance when translated data is available, we evaluate the model on the translate-train setup for the classification tasks.
English Downstream Evaluation: To further assess XY-LENT’s performance on English and see
how the curse of multilinguality impacts the model, we also assess the model’s performance on the commonly used GLUE benchmark (Wang et al., 2018), comprising of 8 tasks: MNLI (Williams et al., 2017), SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2018a), MRPC (Dolan and Brockett, 2005), CoLA (Warstadt et al., 2018), QQP , STS-B (Cer et al., 2017) and RTE. Additionally, we also evaluate the English performance of our model on a question answering task, using the SQuAD 2.0 dataset (Rajpurkar et al., 2018b).
Please refer to Appendix B for additional details on the datasets.

heading: 6 Results and Analysis

heading: 6.1 Main Results
text: Table 1 presents our proposed model’s performance across different model sizes for zero-shot transfer on sentence classification as well as question answering tasks (detailed results for all languages and all tasks can be found in Appendix D). We see that XY-LENT outperforms the baselines of XLM-E, XLM-R and mT5 across all model sizes, establishing (to the best of our knowledge) the state-of-theart (SoTA) for all the 5 considered multilingual datasets within the model size bands: with XYLENTBase outperforming XLM-EBase by 3.1 pts, XY-LENTLarge outperforming XLM-ELarge by 1.8 pts and XY-LENTXL outperforming XLM-EXL by 0.9 pts (averaged across all 5 datasets). Another interesting observation is that XY-LENT is competitive across model size families: the XYLENTBase model out-performs XLM-RLarge and mT5Large variants on 4 out of 5 datasets, similarly the XY-LENTLarge outperforms the mT5XL model on 4 out of 5 datasets. Furthermore, the XY-LENTXL model outperforms XLM-RXXL and is competitive with mT5XXL while being 5x and 6x smaller respectively. A practical implication of these better performing smaller models is their easy usage in downstream tasks.
This behaviour is also consistent in the TranslateTrain setting where the translated version of the training data is present across all languages for training. Table 1 presents XY-LENT’s performance on this setup for sentence classification tasks. We see that even in this setting, XY-LENT outperforms other models with the same size band, and is competitive across model size bands.

heading: 6.2 Ablations
text: Different Many-to-Many Datasets Table 2 shows the impact of moving from English-centric bitexts to X-Y bitext data. Using multiCCAligned dataset gives a +1.4 pt improvement on average XNLI performance over the baseline which uses only the CCAligned data, thereby showing that the utility of leveraging multi-way bitext data is not limited to CCMatrix dataset. However, we still see an additional improvement of 1.0 pt with usage of CCMatrix data and we hypothesize this gain to more diversity present in it which in-turn helps in improving the multilingual representations.
Different Pretraining Objectives While the gains are more substantial with ELECTRA training
objective, Table 2 shows that the benefits of having a better quality bitext data is not just restricted to the ELECTRA paradigm of training and can also be observed with the Masked LM objective. For the ablation experiment, we train a base model model with the MLM objective for 125k steps with a batch size of 8192. Comparing this with XLMRBase’s setup, which uses only monolingual data with MLM objective and trains for 1.5M steps (i.e. 12 times longer), finally achieving an XNLI (Avg) of 76.2, we observe that introduction of X-Y data not only brings performance gains but also significantly improves the training efficiency of these models.

heading: 6.3 On English Performance of Multi-Lingual Models
text: Given the strong performance of multilingual models on the English subset of XNLI, one interesting question that arises is how does model scaling impact the performance on English centric downstream tasks. In order to evaluate that, we measure the performance of XY-LENT on the commonly used GLUE benchmark (Wang et al., 2018) and the SQuAD 2.0 benchmark. To compare the multilingual model performance on English, we also consider English specific encoder models trained in an Electra pre-training paradigm. Specifically, we consider the Base, Large, XL and XXL models
presented in (Bajaj et al., 2022).
Table 3 shows the performance of our proposed method against the SoTA monolingual as well as other multilingual baselines. As observed in the results, with an increase in the number of parameters, we see that the gap in the performance of an English centric model and a multilingual model decreases, with the XL model being just 0.6 points behind on GLUE and 1.3 points on SQuAD 2.0. We hypothesize that an increase in model capacity alleviates the issues caused by the curse of multilinguality (Conneau et al., 2020); and when that is the case, English performance actually benefits from the presence of other languages in the training data.
It is noteworthy that the even for the English language performance, having an X-Y centric data is more beneficial compared to an EN-X data (XLM-E vs XY-LENT). Furthermore, our proposed method outperforms XLM-R on large and XL sizes.

heading: 6.4 Performance Across Language Families
text: Figure 3 shows the performance the delta of performance between XLM-E and XY-LENT across different language families. Following Hu et al. (2020), we use the number of Wikipedia articles as a proxy for a language family being high or low resource. As can be seen, leveraging X-Y bitexts helps improves performance consistently across language families.
N iger-Congo Kra-D ai IE:Indo-Aryan IE:G reek Turkic Koreanic Afro-Asiatic ideogram s Japonic Austro-Asiatic IE:Rom ance IE:G erm anic
0
1
2
3
4
5
6
XNLI PAWSX XQuAD MLQA TYDIQA
Language Families (Ordered By Number of Wikipedia Articles)
S c o r e Δ
b e t w
e e n X
Y - L E N
T a
n d X
L M
- E
Figure 3: Performance ∆ between XLM-E and XYLENT across language families
Model XQuAD MLQA TyDiQA XNLI PAWS-X MBERT 25.0 27.5 22.2 16.5 14.1 XLM-R 15.9 20.3 15.2 10.4 11.4 XLM-E 14.9 19.2 13.1 11.2 8.8 XY-LENT 15.3 19.9 8.6 7.8 6.8
Table 4: Crosslingual Transfer Gap scores on 5 multilingual benchmark tasks. A lower score indicates better cross-lingual transfer. For QnA datasets, this is computed using the EM scores.

heading: 6.5 Crosslingual Transfer Gap
text: In order to further evaluate the cross-lingual transferrability of our model, we follow Hu et al. (2020) and evaluate the cross-lingual transfer gap (the difference between the performance on the English test set and the average test set performance for
other languages) for XY-LENTBase. This score indicates how much end task knowledge is not transferred to other languages post fine-tuning, with a smaller gap indicating better transferrability. As seen in Table 4, XY-LENT achieves lower scores on 3 out of 5 tasks, thereby demonstrating strong transferrability.

heading: 6.6 Using Training Dynamics to Explore Dataset Quality
text: So far we have seen that leveraging X-Y aligned bitexts improves model quality. In this section, we consider the inverse direction: whether training dynamics of representation learning models can be used to identify dataset artifacts. Given these bitext datasets span over 1000 language pairs, a manual inspection of these datasets is extremely hard. Thus an automated method for spot-checking the dataset quality is quite valuable.
To do so, we first train a model in line with the methodology presented by Zhou et al. (2021) for Distributionally Robust Multilingual Machine Translation. Specifically, we train XY-LENT with the following modified objective:
min θD,θG sup P:χ2(P,Q)≤ρ
∑
i
pi(LD(x; θD)+
λLG(x; θG)) (3)
Here LG and LD refer to the generator and discriminator losses respectively (§4), P is the joint distribution over the bitext language pairs that we want to estimate (i.e P = pi | 1 ≤ i ≤ L× L; ∑ i pi = 1); and Q is the original training distribution (i.e the probability distribution over the bitexts when the training starts, equal to P∗ as estimated in §3.2). At a high level, the objective minimizes the training loss over a χ2 ball around the original training distribution, with the supremum up-weighing language pairs with higher loss values, and down-weighing languages with lower loss values 5. We train a model with the Distributional Robustness Optimization objective (DRO) using Iterated Best Response strategy, as proposed by Zhou et al. (2021) and resample 10 times throughout the training. We hypothesize that the two extremities (i.e language pairs that are highly upsampled as well as those that are downsampled) would be bitext datasets of interest for spot-checking.
5Table 11 in the Appendix shows that such an approach achieves reasonable performance on XNLI.
Figure 4 presents the top 10 upsampled and 10 downsampled languages between the initial and final language distributions. Manual inspection of these language pairs shows that our hypothesis indeed holds true: we observe that the translations for English and Xhosa (en - xh) are extremely noisy and aligned with non-sensical text, with multiple different English sentences being aligned to the same Xhosa sentence. This can potentially be a manifestation of the hubness issue for Nearest Neighbor lookups in high dimensional spaces (Radovanović et al., 2010; Dinu and Baroni, 2014). Bitexts for Catalan and Spanish (ca - es) and Czech and Slovak (cs - sk) are near duplicates, since the language pairs are very similar. Both of these issues can cause the TRD task to be trivial, explaining the downsampling. Similarly, looking at languages that are up-sampled, we observe a lot of translation quality noise in bitexts for Spanish and Tamil (es - ta), Turkish and Urdu (tr - ur) and Sinhala and Turkish (si - tr).

heading: 7 Conclusion
text: In this work, we introduced a family of models which achieve SoTA performance over 5 multilingual benchmarks compared to other models belonging to similar model size bands and are competitive across the bands. Our XY-LENTXL model outperforms XLM-RXXL and is competitive with mT5XXL being 5x and 6x smaller respectively. Furthermore, the XL model variant also achieves 99.3% and 98.5% of the current best performant models on GLUE and SQuAD 2.0 respectively, thereby aiding in reducing the curse of multilinguality. The
performance gains are consistent across language families.

heading: 8 Limitations
text: Even though XY-LENT paves the way towards better general-purpose multilingual representation foundation models, in this section, we highlight the limitations associated with this work. We first expound upon the limitations associated with selfsupervised learning on large web extracted corpora. Then we show that while XY-LENT achieves strong performance on multiple multilingual benchmarks, when the downstream task involves unseen (during pretraining) languages, the performance drops by a substantial margin. Finally, we show the potential limitation associated with a common methodology used for domain adaptation associated with leveraging these multilingual foundation models, illustrating how catastrophic forgetting exacerbates certail issues pertaining to low resource language performance.
Training Data
XY-LENT uses CC-100 which a static multilingual corpus extracted from Common Crawl for 100 languages. As noted by Wenzek et al. (2020), several data filtering strategies have been applied to remove duplicated documents, paragraphs with high ratio of punctuations, digits and profanities, the resultant data may still result in many potential biases requiring further analysis. Additionally, these issues might be aggravated for models that leverage bitext data, since the bitexts themselves are mined from web crawls, and thus potentially have all the associated biases, stereotypes and other associated harms. Furthermore, the raw data was compiled from static Common Crawl snapshots from January, 2020 to December, 2020 and hence may not include information about some of the recent events such as COVID-19.
Performance on Unseen Languages
Given the performance improvements observed with scaling, we investigate how it impacts extremely low resource languages which are not present in the pre-training data. In order to do so, we consider our model’s performance on the AmericasNLI dataset (Ebrahimi et al., 2022) which extends the XNLI dataset to 10 Indigenous languages of the Americas.
Table 5 presents the results on the AmericasNLI
dataset. As can be seen, XY-LENT does outperform XLM-R, indicating that better representation learning also benefits these extremely low resource languages. However, we do not see an increase in performance while scaling our models. Specifically, the performance of XY-LENTBase and XYLENTXL model is nearly the same, and substantially worse that the performance observed on the XNLI dataset. This indicates that, while parameter scaling can help improve performance on languages that the model has seen during pre-training, it does not automatically improve performance in the extremely low-resource regime 6. Thus, while model scaling allows for improvements across numerous dimensions, it is far from a panacea, especially if not done in conjunction with data scaling efforts. To be able to improve performance for unseen languages, an intervention would need to be made at the data collection efforts during pretraining, which we aim to assess in future works.
Continued Training for Domain Adaptation in Pre-Trained Encoders
In recent years, continued training on domain specific corpora has been considered a viable approach for domain adaptation of MLM style pre-trained models (Gururangan et al., 2020; Yao et al., 2021) where the core idea is to continue train the pretrained model on domain specific corpora with the goal of improving in-domain downstream evaluation.
We first show that this phenomenon can be extended to models pretrained with an ELECTRA style training objective. Concretely, we apply domain adaptation in the biomedical domain where we continue to train our XY-LENTBase as well as XY-LENTMLM + TLM model on the PubMed data presented in Yao et al. (2021), and evaluate it on the ChemProt task (which aims at extracting relations between chemicals and proteins) presented in Gururangan et al. (2020) as the in-domain downstream task.
We observe that the continued training approach presented in Gururangan et al. (2020) for the ELECTRA style models, using the same peak learning rate as used during pre-training, results in divergence. Interestingly, this neither happens for the generator of the ELECTRA model nor for the
6Note that since the tokenizer is a sentencepiece tokenzier, there are extremely few UNK words in the low-resource languages. Consequently, the poor performance is not explained by excessive occurrences of UNK tokens
MLM style pre-trained model. Thus, for an ELECTRA style continued training setup, we posit reducing the peak learning rate to be a crucial change. Table 7 shows the performance on the downstream task post the continued training approach and unsurprisingly it helps with improving in-domain performance.
However, given the multilingual nature of such models, we test the multilinguality of these models
before and after continued training; using crosslingual zero-shot XNLI as a proxy for multilingual model quality. Table 6 shows the drop in performance across all languages pre and post continued training. We first note that this drop in performance is present for both MLM and ELECTRA style of models, and thus is not an artifact of the pre-training objective. We observe that the drop in performance is not uniform across all languages and the drop is worse for MLM style models (with using the same peak learning rate suffering more from this issue; Table 7). While we expect the drop in English performance to be relatively less, we do see that the drop is substantially more for the mid and low resource languages (especially Hindi, Turkish, Urdu and Swahili; see Fig. 5). While this can potentially be ameliorated by using techniques like Adapters (Houlsby et al., 2019) etc., we would like to draw attention towards the fact that general purpose continued training does suffer from this issue.

heading: A Pre-Training and Model Hyperparameters
text: Table 8 shows the hyper-parameters of XYLENT across various model sizes. All the models are trained with a vocabulary size of 500K and we use batch size of 1M or 4M tokens based on model size as mentioned in Table 9. For multilingual replace token detection task we work with a fixed input sequence length of 512 and hence maintains a constant batch size. For translation replace token detection task, the input sequence length is dynamically set as the length of original translation pair and the max one is chosen across the batch. For the base and large models, we train on 128 Nvidia A100-40GB GPU cards, and for the XL model, we use 512 Nvidia A100-80GB GPU cards.

heading: B Downstream Performance
text: For evaluating cross lingual understanding, we consider five multilingual evaluation benchmarks. We use XNLI (Cross-Lingual Natural Language Inference) and PAWS-X for classification and XQuAD, MLQA and TyDiQA-GP for question answering. Additionally, we use GLUE benchmark and SQuAD2.0 to evaluate the English performance of our model.
XNLI The XNLI dataset (Conneau et al., 2018) comes with ground-truth dev and test sets in 15
languages, and a ground-truth English training set. The training set has been machine-translated to the remaining 14 languages, providing synthetic training data for these languages as well. We evaluate our model on cross-lingual transfer from English to other languages in two modes: (i)zero-shot: the model is fine-tuned only using the English training data and (ii) translate-train-all: the English training set is machine-translated to each language and we fine-tune a multilingual model on all training sets. For translations, we use the original XNLI data for consistency.
PAWS-X The PAWS (Paraphrase Adversaries from Word Scrambling) dataset (Zhang et al., 2019) requires to determine whether two sentences are paraphrases. We use the subset of the PAWS dev and test sets translated to six other languages by professional translators, dubbed as PAWS-X (Yang et al., 2019) for evaluation, while using the PAWS set for training.
XQuAD The English SQuAD v1.1(Rajpurkar et al., 2016) requires identifying the answer to a question as a span in the corresponding paragraph. In XQuAD(Artetxe et al., 2019), a subset of the English dev set was translated into ten other languages by professional translators which is then used for evaluation.
MLQA The Multilingual Question Answering(Lewis et al., 2019) dataset is another crosslingual question answering dataset. In this dataset, the evaluation data for English and six other languages was obtained by automatically mining target language sentences that are parallel to sentences in English from Wikipedia, crowd-sourcing annotations in English, and translating the question and aligning the answer spans in the target languages. We use the SQuAD v1.1(Rajpurkar et al., 2016) training data for training and evaluate on the test data of the corresponding task.
TyDiQA-GP We use the gold passage version of the Typologically Diverse Question Answering(Clark et al., 2020a) dataset, a benchmark for information-seeking question answering, which covers nine languages. The gold passage version is a simplified version of the primary task, which uses only the gold passage as context and excludes unanswerable questions. It is thus similar to XQuAD and MLQA, while being more challenging as questions have been written without seeing the answers,
leading to 3× and 2× less lexical overlap compared to XQuAD and MLQA respectively. We use the English training data for training and evaluate on the test sets of the target languages.
GLUE and SQuAD 2.0 We evaluate English performance of our model on the GLUE benchmark (Wang et al., 2018) which is a benchmark of eight diverse NLU tasks spanning over single-sentence tasks (CoLA, SST-2), similarity and paraphrase tasks (MRPC, STS-B, QQP) and inference tasks (RTE, MNLI, QNLI). The benchmark is also varied in terms of the training data sizes across tasks which makes it an effective benchmark for testing NLU capabilities of a pretrained model in a robust fashion. We also evaluate the English performance on SQuAD 2.0 (Rajpurkar et al., 2018b) task which is a collection of 100k crowdsourced question/answer pairs collected from Wikipedia where given a passage and a question, the task is to predict the answer span in the passage. The task also has the possibility that no answer exists, making the problem more grounded.

heading: C Sampling Sparsity Across All Language Pairs
text: Figure 6 shows the sampling distribution as induced by the M2M sampling method and by our proposed method for all language pair directions. Our proposed method induces a much less sparse distribution, resulting in less data wastage.

heading: D Detailed Performance on All Tasks and Languages
text: We present the detailed results associated with all tasks and languages in this section.
Model en ar de el es hi ru th tr vi zh Avg Base mT5 84.6/71.7 63.8/44.3 73.8/54.5 59.6/35.6 74.8/56.1 60.3/43.4 57.8/34.7 57.6/45.7 67.9/48.2 70.7/50.3 66.1/54.1 67.0/49.0 XLM-E 84.9/72.9 70.5/54.3 78.9/63.2 75.6/57.8 78.4/60.8 71.2/54.5 75.9/59.7 68.7/58.8 71.6/55.4 75.9/56.4 65.5/56.9 74.3/59.2 XY-LENT 87.2/76.0 72.9/56.0 80.0/64.5 79.6/63.5 81.2/63.1 75.3/59.7 77.7/61.5 70.9/59.5 74.0/58.7 77.4/59.2 69.0/61. 76.8/62.1 Large XLM-R 86.5/75.7 68.6/49.0 80.4/63.4 79.8/61.7 82.0/63.9 76.7/59.7 80.1/64.3 74.2/62.8 75.9/59.3 79.1/59.0 59.3/50.0 76.6/60.8 mT5 88.4/77.3 75.2/56.7 80.0/62.9 77.5/57.6 81.8/64.2 73.4/56.6 74.7/56.9 73.4/62.0 76.5/56.3 79.4/60.3 75.9/65.5 77.8/61.5 XLM-E 87.1/75.5 75.1/58.1 82.1/66.0 80.9/64.0 82.5/64.3 77.5/61.3 80.3/63.7 73.4/59.4 76.8/60.8 79.2/59.0 70.5/61.6 78.7/63.1 XY-LENT 88.1/77.4 76.3/59.6 82.6/67.1 82.5/65.1 83.9/66.6 77.9/61.3 80.2/63.6 74.3/63.8 78.5/62.9 80.6/61.6 71.4/64.6 79.7/64.9 XL mT5 88.8/78.1 77.4/60.8 80.4/63.5 80.4/61.2 82.7/64.5 76.1/60.3 76.2/58.8 74.2/62.5 77.7/58.4 80.5/60.8 80.5/71.0 79.5/63.6 XLM-R 89.5/79.0 78.4/61.6 81.3/64.1 82.3/63.9 84.6/66.2 78.8/63.2 81.5/65.0 76.0/65.5 73.9/57.9 81.7/61.8 72.3/66.1 80.0/64.9 XLM-E 89.1/79.0 78.5/62.0 82.4/66.9 81.8/65.5 84.3/67.1 79.3/63.4 82.2/66.9 75.4/65.1 78.3/62.5 81.5/62.9 71.6/65.1 80.4/66.0 XY-LENT 89.4/79.2 79.2/62.0 84.1/68.3 83.5/66.1 84.9/66.6 80.4/64.5 82.9/67.1 75.0/61.7 79.5/64.5 83.2/64.1 72.7/65.0 81.3/66.3 XXL XLM-R 89.3/79.4 80.1/63.7 82.7/65.8 83.4/65.5 83.8/66.0 80.7/65.4 82.4/65.4 76.6/65.6 76.8/61.7 82.2/63.0 74.1/67.4 81.1/66.3 mT5 90.9/80.1 80.3/62.6 83.1/65.5 83.3/65.5 85.1/68.1 81.7/65.9 79.3/63.6 77.8/66.1 80.2/60.9 83.1/63.6 83.1/73.4 82.5/66.8
Table 14: XQuAD results (F1/EM) for each language.

heading: E Hyperparameters for Fine-Tuning
text: In Table 15, we report the hyperparameters for fine-tuning XY-LENT on the downstream tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A and E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To be able to improve performance for unseen languages, an intervention would need to be made at the data collection efforts during pretraining, which we aim to assess in future works. Continued Training for Domain Adaptation in Pre-Trained Encoders
In recent years, continued training on domain specific corpora has been considered a viable approach for domain adaptation of MLM style pre-trained models (Gururangan et al. , 2020; Yao et al. , 2021) where the core idea is to continue train the pretrained model on domain specific corpora with the goal of improving in-domain downstream evaluation. We first show that this phenomenon can be extended to models pretrained with an ELECTRA style training objective. Concretely, we apply domain adaptation in the biomedical domain where we continue to train our XY-LENTBase as well as XY-LENTMLM + TLM model on the PubMed data presented in Yao et al. (2021), and evaluate it on the ChemProt task (which aims at extracting relations between chemicals and proteins) presented in Gururangan et al. (2020) as the in-domain downstream task. We observe that the continued training approach presented in Gururangan et al. (2020) for the ELECTRA style models, using the same peak learning rate as used during pre-training, results in divergence. Interestingly, this neither happens for the generator of the ELECTRA model nor for the
6Note that since the tokenizer is a sentencepiece tokenzier, there are extremely few UNK words in the low-resource languages. Consequently, the poor performance is not explained by excessive occurrences of UNK tokens
MLM style pre-trained model. Thus, for an ELECTRA style continued training setup, we posit reducing the peak learning rate to be a crucial change. Table 7 shows the performance on the downstream task post the continued training approach and unsurprisingly it helps with improving in-domain performance. However, given the multilingual nature of such models, we test the multilinguality of these models
before and after continued training; using crosslingual zero-shot XNLI as a proxy for multilingual model quality. Table 6 shows the drop in performance across all languages pre and post continued training. We first note that this drop in performance is present for both MLM and ELECTRA style of models, and thus is not an artifact of the pre-training objective. We observe that the drop in performance is not uniform across all languages and the drop is worse for MLM style models (with using the same peak learning rate suffering more from this issue; Table 7). While we expect the drop in English performance to be relatively less, we do see that the drop is substantially more for the mid and low resource languages (especially Hindi, Turkish, Urdu and Swahili; see Fig. 5). While this can potentially be ameliorated by using techniques like Adapters (Houlsby et al. , 2019) etc. , we would like to draw attention towards the fact that general purpose continued training does suffer from this issue