Soft Language Clustering for Multilingual Model Pre-training
Year: 2023

Abstract:
Multilingual pre-trained language models have demonstrated impressive (zero-shot) crosslingual transfer abilities, however, their performance is hindered when the target language has distant typology from source languages or when pre-training data is limited in size. In this paper, we propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods. On the tasks of XTREME including text classification, sequence labeling, question answering, and sentence retrieval, both baseand large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer1.

heading: 1 Introduction
text: Multilingual pre-trained language models (mPLMs) such as mBERT (Devlin et al., 2019), mBART (Liu et al., 2020), XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2021) have lately produced notable advancements in a number of downstream NLP tasks. In particular, the use of mPLMs significantly enhances few-shot fine-tuning and makes possible efficient zero-shot cross-lingual transfer (Hu et al., 2020). Essentially, an ideal mPLM should satisfy two properties: alignment between language pairs, which has been widely studied in the literature (Chi et al., 2022; Ouyang et al., 2021; Chi et al., 2021a); and a good trade-off between high-resource and low-resource
∗Corresponding author. 1Code and model are available at
https://github.com/lemon0830/XLMP.git
languages, which remains largely unexplored despite the success of mPLMs.
In this paper, we focus on the second property, specially the potential for model performance to suffer when a large number of languages are added. This can occur due to restricted model capacity or computational limitations, resulting in underrepresented languages being allocated less capacity (Conneau et al., 2020). Furthermore, the model’s coverage of world’s languages remains inadequate, limiting the range of language technology applications it can support (Ansell et al., 2021). A typical solution for the coverage-performance trade-off in multilingual learning is to assign additional model parameters to specific languages, such as language identity embeddings (Conneau and Lample, 2019), adaptors (Houlsby et al., 2019; Üstün et al., 2022; Ansell et al., 2021), and language-aware layers (Zhang et al., 2021). However, it is impractical for multilingual pre-training to maintain a separate component for each language, which can lead to more complicated and challenging optimization, especially for low-resource languages.
We propose to approach the above languageaware components from a different perspective. In linguistic typology, some patterns such as nominative-accusative alignment have broad global distributions, whereas others like morphology are more specific and detailed (Donohue and Wichmann, 2008). To take advantage of this, we introduce XLM-P, which uses a set of compact embeddings to represent soft clustering of the language patterns beyond language identity. We refer these embeddings as prompts, due to their similarity to prompt tuning (Lester et al., 2021). Concretely, we build a key-value prompt pool and use the attention mechanism to look up the prompts for each input. The retrieved prompts are then prepended to the input embeddings, and serve as categorization information to adapt the model weights conditionally. This allows for more efficient and effective mul-
7021
tilingual learning by leveraging the patterns and similarities across languages rather than maintaining separate components for each language.
We evaluate the proposed XLM-P on CrossLingual Natural Language Understanding tasks and Cross-Lingual Sentence Retrieval tasks of the XTREME benchmark, and the consistent improvement in performance demonstrates its effectiveness. In addition, we conduct empirical analyses to investigate the underlying reasons of the improvement of XLM-P. The advantages of XLM-P can be summed up as follows:
• The prompt pool and instance-wise prompt retrieval are lightweight and only result in 0.35% and 0.23% increase in parameters for the base and large models, respectively. When fine-tuning on downstream tasks, the prompt module can be easily added or removed as needed.
• Our XLM-P divides the prompts into general and specific ones without any explicit supervision. The dynamically retrieved instance-wise prompts tame the sentence encoding, thus enhancing the capability of multilingual pretrained models.
• The prompt module is model-agnostic and can be outfitted with the other frameworks (e.g., encoder-decoder style PLMs) and multilingual pre-training objectives (e.g., contrastive learning used in this paper).
Overall, XLM-P is a versatile and efficient approach for improving multilingual pre-training.

heading: 2 Related Work

heading: 2.1 Cross-lingual LM pre-training
text: Trained by the masked language modeling (MLM) loss with a shared multilingual vocabulary, multilingual BERT (Devlin et al., 2019) achieves promising results in cross-lingual natural language understanding tasks (Hu et al., 2020), which has attracted increasing attention to improve the cross-lingual transferability. XLM-R (Conneau et al., 2020) increase the model capacity and use large-scale monolingual training data. In addition to monolingual data, XLM (Conneau and Lample, 2019) performs MLM on bilingual parallel corpus, while ALM (Yang et al., 2020) constructs code-switched sequences. In respect of training objectives, a series of studies have explored various pre-training
tasks to enhance the models’ transferability (Huang et al., 2019; Ouyang et al., 2021; Chi et al., 2021b, 2022, 2021a; Luo et al., 2021).
Compared to the above methods, our XLM-P exploits a small number of compact prompt vectors to tame the sentence encoding. Moreover, the prompt module is model-agnostic, and can be combined with the above methods to achieve further improvement.

heading: 2.2 Language-aware Components
text: To alleviate the issue of the ‘curse of multilinguality’, various language-aware components have been proposed, which can allocate additional capacity for individual languages especially underrepresented languages. Conneau and Lample (2019) use language identity embeddings to explicitly guide the model. Ansell et al. (2021) present MAD-G, which contextually generates language adapters from language representations. Üstün et al. (2022) propose a single hypernet-work that unifies multi-task and multilingual learning with efficient adaptation. On multilingual neural machine translation, Philip et al. (2020) trained language-specific adapters. Zhang et al. (2021) use conditional routing to select shared and language-specific parameters. Stickland et al. (2021) use language-agnostic task adapters for fine-tuning BART and mBART to bilingual and multilingual MT.
We differ from them in that we do not use any language indicators, and regard the prompts as a bottleneck for storing a small number of discriminative features. The prompts are encoded by the large language model along with the input, which learns language-invariant and language-specific features via the deep modular interaction.

heading: 2.3 Prompt-based Tuning
text: Our work builds upon the recent results showing the effectiveness of adapting PLMs to downstream tasks conditioning on lightweight prompt vectors (Brown et al., 2020; Lester et al., 2021; Schick and Schütze, 2021; Sanh et al., 2021). Differentiable prompts (Li and Liang, 2021; Lester et al., 2021; Gu et al., 2022) show the power of adapting pretrained language model to multiple downstream tasks by simply prepending a few learnable parameters to the input. More recently, the effectiveness of prompting has been investigated in multilingual (or cross-lingual) tasks, which is largely unexplored despite the success of prompting in English (Zhao
and Schütze, 2021; Zhou et al., 2022; Huang et al., 2022a).
Instead of exploring prompting in fine-tuning, we propose to adopt dynamic retrieval of prompts, which jointly optimized with the mPLM, as a method of soft language clustering to enhance multilingual pre-training.

heading: 3 Method
text: Figure 1 depicts the addition of a Prompt Pool to the transformer-based language model. Before feeding an input to the model, we perform Instancewise Prompt Retrieval and convert the input to a prompt-wrapped one. Both the prompts and the model are jointly optimized on multilingual data using Masked Language Modeling, which trains the model to make predictions based on both context and clustering information. At the fine-tuning stage, we experiment with two strategies: standard fine-tuning and prompt-based fine-tuning.

heading: 3.1 Prompt Pool
text: In our proposed framework, we use a prompt pool to store fine-grained patterns sharing across languages as well as language-specific knowledge. Formally, the prompt pool is defined as:
P = {P1, P2, ..., PM}, (1)
where M is the total number of prompts in the prompt pool, and Pj ∈ RLp×D is a single prompt with Lp vectors, whose dimension is the same as the embedding size D of the mPLM. We associate each prompt as value to a learnable key: {(k1, P1), (k2, P2), ..., (kM , PM )}, where ki ∈ RDk . And we denote the set of all keys K={kj}Mj=1.

heading: 3.2 Instance-wise Prompt Retrieval
text: Ideally, we would like to let the input itself to decide which prompts to choose through querykey matching. Formally, given an input sentence X={x0, ..., xn}, we first map each token xi to a real-valued vector ei by an embedding layer. The sequence of embeddings is then concatenated with the embedding of “[CLS]” token E={ecls, e0, ..., en}. Next, we operate a pooling strategy2 on the embedding sequence to obtain the input representation:
r = Pool(E). (2) 2In this paper, we use max pooling.
Next, we obtain the prompts for the input using the attention mechanism:
P̂ = ∑ αjPj , (3)
αj = softmax((rW )k T j ), (4)
where W is a trainable parameter. Finally, we prepend the retrieved prompt P̂ to the input embeddings Ê=[P̂ , E], and feed the extended sequence Ê to the model to get the contextualized sentence representations H={hP0 , .., hPLp , hcls, h0, ..., hn}.
Notably, unlike language identity embeddings used by Conneau and Lample (2019), this soft manner allows the model to capture fine-grained language-invariant and language-specific features from massive monolingual data without explicitly assigning a cue to each input.

heading: 3.3 Prompt-based Masked Language Modeling for Pre-training
text: Our XLM-P is trained to predict the masked tokens using the prompt-augmented contextualized representations. Following Devlin et al. (2019), we randomly mask 15% of the tokens in a monolingual sentence. With probabilities of 0.8, 0.1, and 0.1,
we replace each masked token with a special token “[MASK]”, a random token, or the unchanged token, respectively. The monolingual MLM loss is defined as:
LMLM = − ∑
j∈Mx logp(xi|H,X\Mx) (5)
where X\Mx is the masked version of input X. Since the prompt module is agnostic to training objectives, we can enhance the model with additional objectives, as demonstrated in Section 4.2.

heading: 3.4 XLM-P for Downstream Applications
text: Benefiting from the plug-and-play property of the proposed prompt module, we can choose either plug in or plug out it on demand. We explore two fine-tuning strategies to use our XLM-P on downstream applications, i.e., Standard Fine-tuning and Prompt-based Fine-tuning, and the detail is presented as follows.
Standard Fine-tuning. In this setting, we unplug the prompt module from the pre-trained model. Concretely, we simply feed the embedding features E = {ecls, e0, ..., en} excluding the prompt embeddings to the model, which is identical to XLM-R.
Prompt-based Fine-tuning. We can keep the prompt module and conduct the retrieval during fine-tuning, as we do in training. For tokenlevel prediction tasks (e.g., structured prediction and question answering), we remove the hidden states of the prompt after encoding and take the rest H\P̂={hcls, h1, ..., hn} as the input of taskspecific classifiers. For sentence-level classification tasks, we apply a pooling operation on all of the hidden states H\X={hP0 , .., hPLp , hcls} to obtain the sentence-level representation used for classification.

heading: 4 Experiments

heading: 4.1 Settings
text: Pre-training. To train XLM-P, we extract a subset from CC-100 (Conneau et al., 2020) which involves monolingual data in 50 languages. We use XLM-R (Conneau et al., 2020) as the backbone. The XLM-Pbase model has 12 layers with 768 hidden units and 12 attention heads, and the XLMPlarge model has 24 layers with 1024 hidden units and 16 attention heads. We set M=256 and Lp=4 for both base model and large model, and XLM-P introduces 983,040 and 1,310,720 parameters to
the original pre-trained model, merely accounting for 0.35% and 0.23% of the total parameters, respectively. The detail of the pre-training settings can be found in Appendix A.
Evaluation. We evaluate our model on XTREME (Hu et al., 2020), which is designed to assess the the cross-lingual generalization capabilities of pre-trained language models, with a specific focus on Cross-lingual Natural Language Understanding and Cross-lingual Sentence Retrieval. There are seven tasks for cross-lingual natural language understanding, which can be grouped into three categories: 1) Structured prediction: part-of-speech tagging (POS) on the Universal Dependencies v2.5 (Daniel Zeman and et al., 2019), and named entity recognition (NER) on the WikiAnn (Pan et al., 2017) dataset; 2) Question answering: cross-lingual question answering on MLQA (Lewis et al., 2020) and XQuAD (Artetxe et al., 2020), and gold passage of typologically diverse question answering (TyDiQA-GoldP, Clark et al. (2020)); 3) Sentence classification: cross-lingual natural language inference (XNLI, Conneau et al. (2018)), and cross-lingual paraphrase adversaries from word scrambling (PAWS-X, Yang et al. (2019)). The aim of the cross-lingual sentence retrieval task is to retrieve relevant sentences across languages, and we use the Tatoeba (Artetxe and Schwenk, 2019) dataset.
Baselines. As baselines, we employ the multilingual pre-trained language models listed below: 1) MBERT (Devlin et al., 2019) is pre-trained with MLM and next sentence prediction on Wikipedia in 104 languages; 2) XLM (Conneau and Lample, 2019) is pre-trained with MLM on 100 languages and translation language modeling (TLM) on 14 language pairs; 3) MT5 (Xue et al., 2021) is the multilingual version of T5 pre-trained with text-totext tasks; 4) XLM-E (Chi et al., 2022) is trained with two pre-training tasks: namely multilingual replaced token detection (MRTD) and translation replaced token detection (TRTD). We report XLME (-TRTD) for fair comparison, which does not use parallel data during pre-training; and 5) VECO (Luo et al., 2021) is a unified cross-lingual language model for both NLU and NLG. 6) InfoXLM (Chi et al., 2021a) is jointly pre-trained with a crosslingual contrastive learning task.

heading: 4.2 Main Results
text: Cross-Lingual Natural Language Understanding. Following Hu et al. (2020), we adopt the zero-shot transfer setting for evaluation, in which the models are fine-tuned on English training data but evaluated on all the target languages. Rather than selecting a single model for each language, we use only one model for evaluation. The detail of the hyper-parameters used for fine-tuning can be found in Appendix B.
The results, which are averaged across all the target languages and five runs with different random seeds, are illustrated in Table 1. Compared to the XLM-R based models, the XLM-P based models achieve significantly better performances. Besides, our XLM-Pbase consistently outperforms the baselines MBERTbase, MT5base, and XLME (-TRTD)base, which are pre-trained without any parallel corpora. Moreover, XLM-Plarge brings notable improvements over all the baselines on most of the tasks. Concretely, XLM-P models perform better on the structure prediction and the question answering tasks, while preserving competitive results on the sentence classification tasks. The overall experimental results demonstrate that multilingual pre-training can benefit from the guidance of our proposed prompt module.
Surprisingly, there appears to be minimal differ-
ence in performance between standard fine-tuning and prompt-based fine-tuning. This can be explained by the fact that only English training data was used during fine-tuning, and the prompt embeddings were not specifically optimized for the task at hand. Furthermore, recent studies have highlighted the challenges of prompt tuning in cross-lingual natural language understanding and have shown that this area is gaining more attention (Qi et al., 2022; Huang et al., 2022b). We plan to explore this further in future work. Finally, we compare our model with XLM-R under the translate-train-all setting and the results are reported in Appendix C. These results further support the effectiveness of our model.
Cross-Lingual Sentence Retrieval. Following Chi et al. (2021a) and Hu et al. (2020), we use 14 and 36 languages of the parallel corpora for evaluation, respectively. For sentence representations, we take the average of hidden states in a specific layer, and we use the 10-th layer for XLMP. Then, we induce translation pairs using a nearest neighbor search with cosine similarity. As illustrated in Table 2, XLM-P achieves 64.0 and 61.7 accuracy scores on Tatoeba-14, and 63.8 and 61.0 accuracy scores on Tatoeba-36 in the directions of en → xx and xx → en, respectively, which outperforms XLM-R significantly. The improvement
possibly due to that the sentence representations obtained from XLM-P encoding extra shared features across languages learned by our prompts. Moreover, under the setting of pre-training on multilingual monolingual corpus, XLM-P performs greatly better than XLM-E (-TRTD) and competes with InfoXLM (-XLCO).
Notably, InfoXLM outperforms our XLM-P due to the benefit of the additional cross-lingual contrastive objective. Our prompt module is designed to be compatible with other multilingual pre-training techniques, and to validate this, we added a simple dropout-based InfoNCE objective (van den Oord et al., 2018; He et al., 2020) to our XLM-P model. More detailed is introduced in Appendix D. We post-train XLM-P on monolingual corpora in 50 languages with both MLM loss and InfoNCE loss. The resulting model, XLMP+, gives a significant improvement over XLM-P and performs better than XLM-E and InfoXLM on Tatoeba-36, but slightly worse on Tatoeba-14 against InfoXLM. This is due to the fact that we did not use the parallel training data utilized by InfoXLM and XLM-E3. In a nutshell, the results show that our proposed method can be effectively integrated with other pre-training objectives.
Finally, in Figure 2, we illustrate the effect of layer selection on sentence representations. The accuracy score is calculated by taking the average of all the 36 language pairs in xx → en directions. The figure shows that all the models exhibit a parabolic trend across layers. Different from XLM-R that achieves the highest accuracy of 56.5 at the 7-th
3There are promising training strategies for pre-training mPLM using parallel corpus with our XLM-P, such as the interaction between retrieved prompts of sentence pairs. This is an intriguing research question that merits further investigation in future studies.
layer, the curve of XLM-P rises more steadily until it peaks at the 10-th layer. It can be observed that XLM-P outperforms XLM-R on the top layers, and XLM-P+ achieves notably higher average scores than XLM-R at all layers.

heading: 5 Analysis
text: We carry out a number of analyses in order to comprehend the design of our proposed XLM-P better.

heading: 5.1 Effect of Prompt Capacity
text: In XLM-P, the length of a simple prompt Lp and the size of the prompt pool M are two critical hyperparameters that determine the total capacity of the learnable prompt embeddings. M determines the granularity of prompt selection and Lp decides the expressiveness of a single prompt. To investigate the effects of the two hyper-parameters, we train small-size XLM-R and XLM-P from scratch and keep the other setting unchanged (see Appendix A for more details). We evaluate the pre-trained
(a) Prompt selection histograms for MLQA in different languages (b) Prompt selection histograms for different tasks in English
Figure 4: Prompt selection histograms. We only display the top 16 selected prompts. The prompts retrieved can be grouped into language-shared and -specific prompts. Similar tasks tend to have similar patterns of prompt selection across different datasets, whereas tasks that are more irrelevant to each other prefer selecting more diverse prompts.
models on TydiQA and report the average of F1 and EM scores in Figure 3.
We can see that removing the prompt pool and only using a single prompt (i.e., pool size M = 1) results in a significant performance drop, suggesting that it is not sufficient to maintain a single shared prompt for all languages. Increasing the size of prompt pool shows a positive effect on performance, but excessive prompts degrade the performance. The result verifies our motivation, that the prompts are used to capture abstract clustering information. Too many prompts can dilute the effect, and thus negatively affect the generalization and transferability of representations.

heading: 5.2 Patterns of Retrieved Prompts
text: As depicted in the left portion of Figure 4, we present the prompt selection histograms for XLMP on test sets of MLQA in different languages. The prompts retrieved can be easily distinguished into two categories: language-shared prompts (e.g., prompt 207, 42, 23 and 2) and language-specific prompts (e.g., prompt 14, 66, 221, 120). In addition, we display the histograms on test sets of different tasks in English in the right portion of Figure 4. In particular, the similar tasks tend to have similar patterns of prompt selection across different datasets (e.g., xquad and mlqa, both of which are question answering tasks), whereas the tasks that are more irrelevant to each other (e.g., xquad and pawsx, pawsx is a sentence classification task) prefer selecting more diverse prompts. This phenomenon echos the effectiveness of our
XLM-P. Even without any explicit supervision, our model can learn to group the prompts into general and specific ones.

heading: 5.3 Visualization of Retrieved Prompts
text: Based on the observation in Section 5.2, we further plot t-SNE visualizations (van der Maaten and Hinton, 2008) of prompts retrieved by sentences in 15 languages on XNLI test set4 in Figure 5 to support our hypothesis that prompts can serve as a representation of soft clustering for language.
We can find that the prompt representations retrieved by the sentences in different languages are evidently distinct, while the prompt representations retrieved by the sentences in similar languages (e.g., ar and ur, bg and ru) are closer. It implies that the prompts are able to capture the shared patterns across languages as well as language-specific patterns during multilingual pre-training.

heading: 5.4 Trade-off on High and Low Resource Languages
text: In Section 4.2, we display the average performance across all of the languages, and we present more details about the performance in this section. First, we choose the PAWS-X task with 7 languages including English (i.e., the language of the training set), three European languages (i.e., German, Spanish, French), and three Asian languages (i.e., Japanese, Korea, and Chinese). As shown in Table 4, compared with XLM-R, XLM-P achieves better or more competitive performance on the test sets in
4We sampled 200 sentences from each language.
Model Train European Langs Asian Langs Avg. en de es fr ja ko zh
XLM-R 87.8 94.3 88.6 88.8 77.1 75.1 80.4 84.59 XLM-P 89.0 95.1 90.0 90.6 79.4 80.1 81.9 86.59
Table 4: Experimental results on PAWS-X in terms of accuracy (%). We adopt the zero-shot transfer setting for evaluation, in which the models are fine-tuned on English training data but evaluated on all the target languages.
European languages, while obtaining huge gains on the test sets in Asian languages, which are different from English.
Then, we illustrate the results on Tatoeba in both the xx → en and en → xx directions in Table 3. Due to the limited space, we only select the seven languages with the highest scores and the seven languages with the lowest scores based on the performance of XLM-R on Tatoeba-36 in xx → en direction, which can be grouped as High resource
language and Low resource language pairs, respectively. We compare our XLM-P+ with XLM-R and InfoXLM, and report the mean and the standard deviation of accuracy. Both XLM-P+ and InfoXLM provide advantages in both directions. Specifically, XLM-P+ yields similar benefits across different data scales, and greatly benefits low-resource languages (e.g., eu, te, and jv). By contrast, InfoXLM performs well on high-resource languages but has a marginal impact on low-resource languages.
We argue that mPLMs still suffer from insufficient modeling capacity, and adding more languages can result in a decline in representation quality. Our proposed XLM-P can indeed alleviate this issue, especially for languages with limited data or languages that differ from the training data. Intuitively, the dynamically retrieved instance-wise prompts in XLM-P make the sentence encoding specific to the soft clustering information, thereby enhancing the model capability of multilingual pretrained models.

heading: 6 Conclusion
text: This paper presents XLM-P, a new multilingual pre-trained language model equipped with contextually retrieved prompts. In particular, we prepend prompts-like learnable vectors to the input for modeling language interdependence and other potential sharing information. Compared with other language-aware components, the retrieved prompts are parameter-efficient and more flexible without the requirement of language detection. Experiments and analyses validate the effectiveness and robustness of XLM-P. In addition, our method is compatible with various existing multilingual pretraining objectives.
Limitations
In this paper, we simply prepend the retrieved prompt to the input embeddings before encoding. A well-designed method of combining prompts with the input embeddings, such as Prefix Tuning (Li and Liang, 2021), may result in additional enhancements. Finally, as observed in Section 4.2, prompt-based fine-tuning does not present obvious superiority over standard fine-tuning. Exploring the prompt tuning on cross-lingual natural language understanding is a challenging task that has recently gained attention (Qi et al., 2022; Huang et al., 2022b), and we leave it as future work.

heading: Acknowledgements
text: We would like to thank all of the anonymous reviewers for the helpful comments.

heading: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational
text: Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 3440–3453. Association for Computational Linguistics.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. 2022. Hyper-x: A unified hypernetwork for multi-task multilingual transfer. CoRR, abs/2205.12148.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9386–9393. AAAI Press.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547–8555, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Meng Zhou, Xin Li, Yue Jiang, and Lidong Bing. 2022. Enhancing cross-lingual prompting with mask token augmentation. arXiv preprint arXiv:2202.07255.

heading: A Pre-training Details
text: For monolingual data, following XLM-R (Conneau et al., 2020) and Veco (Luo et al., 2021), we build a clean CommonCrawl Corpus using an open-source tool CCNet (Wenzek et al., 2020). We use monolingual data in 50 languages for base-sized and large-sized XLM-P and monolingual data in 15 languages for small-sized XLM-R and XLM-P. Table 5 reports the language codes for pre-training. Please ref (Luo et al., 2021) for the detailed data statistic of the monolingual pre-training corpus. Following Chi et al. (2021a) and Luo et al. (2021), we initialize the parameters of XLM-P with XLMR (Conneau et al., 2020). We use the Adam optimizer (Kingma and Ba, 2015) with the learning rate 3e-4 for the base model and 1e-4 for the large model, respectively. The full set of pre-training hyperparameters for small-sized, base-sized and large-sized XLM-P are listed in Table 6. We conduct the pre-training experiments using 64 Nvidia A100-40GB GPUs with 8,192 batch size for base and large XLM-P.

heading: B Hyperparameters for Fine-tuning
text: In Table 7, we present the hyperparameters for finetuning baselines and our XLM-P on the XTREME end tasks. For each task, the hyperparameters are searched on the joint validation set of all languages.

heading: C Translate-Train-All Setting
text: In this section, we investigate another fine-tuning setting, Translate-Train-All, in which we fine tune a PLM on a mixed corpus consisting of golden training data in English and translated training data in other languages.
Table 8 presents the results on NER, TyDiQA, and PAWS-X. XLM-Plarge outperforms XLMRlarge across all the tasks, confirming our model’s effective capacity for cross-lingual transfer.

heading: D Detailed of Dropout-based InfoNCE
text: Specifically, we construct the “positive pairs” by passing the same sentence to the model twice, and take other sentences in the same mini-batch as “nagatives”. We use the average of prompt hidden states and “CLS” hidden states of the latest layer as the sentence representation. The model is trained to predict the positive one among the samples as follows:
li = − log esim(vi,v
+ i )/τ
∑N k=1 e sim(vi,v + k )/τ
(6)
We post-train our XLM-P in monolingual corpora in 50 languages with the MLM loss and InfoNCE loss. The learning rate is 5e-5, the total number of training step is 100k, and the warmup steps is 10k.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
7 A2. Did you discuss any potential risks of your work? We use the open data to pre-training our model.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract & 1. Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments?

heading: 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.1 Settings, Appendix A, Appendix B, Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4.1 Settings, Appendix A, Appendix B, Appendix C

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.2 Main results
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: , 2022b), and we leave it as future work