MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in China (2024)

Abstract: Current large language models demonstrate deficiencies in understanding low-resource languages, particularly the minority languages in China. This limitation stems from the scarcity of available pre-training data. To address this accessibility challenge, we present MC2, a Multilingual Corpus of Minority Languages in China, which is the largest open-source corpus of its kind so far. MC2 includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian. Notably, we focus on the less common writing systems of Kazakh and Mongolian, i.e., Kazakh Arabic script and traditional Mongolian script, respectively, which have been long neglected in previous corpus construction efforts. Recognizing the prevalence of language contamination within existing corpora, we adopt a quality-centric solution for collecting MC2, prioritizing accuracy while enhancing diversity. Furthermore, we underscore the importance of attending to the multiplicity of writing systems, which is closely related to the cultural awareness of the resulting models. The MC2 corpus and related models are made public to the community1.

heading: 1 Introduction
text: Recently, the rapid development of large language models (LLMs) has been fueled by the availability of high-quality pre-training data. However, only a handful of high-resource languages, such as English and Chinese, have benefitted from the advantages of this progress in LLMs. Despite having a substantial user base, many languages remain excluded from the benefits of these advancements due to a lack of suitable corpora. In this paper, we focus on such underrepresented languages, particularly addressing the minority languages in China, including Tibetan, Uyghur, Kazakh, and Mongolian.
*Equal contribution. †Corresponding author. 1https://github.com/luciusssss/mc2_corpus
As illustrated in Figure 1, the four languages, although spoken by tens of millions, face a critical deficiency in linguistic resources, posing obstacles to both academic research and the development of practical AI applications. Notably, there is no open-source corpus available for Kazakh in Arabic script and Mongolian in traditional Mongolian script – two language variants in China adopting underrepresented writing systems. In the case of Tibetan and Uyghur, although several multilingual datasets such as OSCAR (Abadji et al., 2022) and CulturaX (Nguyen et al., 2023) cover these languages, their quality falls short. Our preliminary study reveals a critical issue of language misidentification. Up to 34% of data in the Uyghur split of CulturaX is actually Kazakh or Arabic texts. This raises concerns for future research in lowresource NLP, as conducting experiments on these significantly contaminated monolingual corpora may yield misleading conclusions.
8832
To facilitate more transparent and reproducible NLP research on the minority languages in China, we present MC2, a Multilingual Corpus of Minority Languages in China, which is the largest open-source corpus for these languages so far. During data collection, we adhere to a quality-centric principle. We carefully design strategies for the selection of web pages to crawl, ensuring the language purity of the crawled texts. Afterwards, we conduct a thorough data cleaning process, eliminating extraneous elements such as web page headers and retaining only meaningful content. Our newly collected MC2 dataset surpasses existing datasets by containing more than twice the amount of Tibetan and Uyghur data, while maintaining superior data quality.
MC2 particularly emphasize underrepresented writing systems, recognizing their significance both technically and culturally. In languages with multiple writing systems, such as Kazakh and Mongolian, previous research has predominantly focused on the more common scripts (Abadji et al., 2022; Nguyen et al., 2023). We illustrate that transliterating texts from high-resource scripts into lowerresource ones may not be effective for training, especially when the script conversion is flawed. Moreover, the texts collected in different scripts inherently embody cultural nuances of various language communities. Through probing, we demonstrate that writing systems play a crucial role in developing culturally-aware NLP systems.
In addition, we continually train language models with MC2. They achieve comparable perfor-
mance to their counterparts trained with closedsource data. This further validates the value of our data in facilitating transparent NLP research.
To summarize, we make the following contributions: (1) We present MC2, the largest open-source corpus to date for four underrepresented languages in China. (2) We highlight quality issues, especially language misidentification, prevalent in previous corpora, which may threaten trustworthy research on low-resource languages. (3) We reveal the cultural complexities arising from the multiplicity of writing systems within languages.

heading: 2 Related Works
text: Multilingual Corpus To facilitate the development of multilingual LLMs, a series of large-scale multilingual corpora has been released, including CC100 (Conneau et al., 2020), mC4 (Raffel et al., 2020), ROOTS (Laurençon et al., 2022), OSCAR (Abadji et al., 2022), CulturaX (Nguyen et al., 2023), and Madlad-400 (Kudugunta et al., 2024). These corpora cover both high-resource languages and a handful of low-resource ones. Recently, there are also efforts in constructing opensource corpora for low-resource languages in certain regions (Cahyawijaya et al., 2021; Teodorescu et al., 2022; Doddapaneni et al., 2023). However, the minority languages in China are still underrepresented in existing datasets, which calls for more attention to constructing open-source corpus for these languages.
NLP for Minority Languages in China In terms of the minority languages in China, previous works try to improve the accessibility of these languages, by collecting a handful of annotated datasets for specific NLP tasks, each covering one or some languages investigated in our study. They mainly focus on three types of tasks: text classification (Qun et al., 2017; Yang et al., 2022; Shi et al., 2022b), question answering (Sun et al., 2021), and machine translation (Costa-jussà et al., 2022; Zhang et al., 2024). The models trained specifically for the minority languages in China include CINO (Yang et al., 2022), MiLMo (Shi et al., 2022b), and CMPT (Li et al., 2022). However, none of these works releases their pre-training corpus. In contrast, we release a large-scale high-quality corpus for four minority languages in China, which, we hope, will help relieve the data scarcity problem and improve transparency in research on lowresource languages.
Culturally-Aware NLP Cultural considerations are increasingly recognized as crucial in NLP research and applications (Hershcovich et al., 2022). An important aspect is the connection between cultural factors and linguistic forms within languages. Previous works on intra-language variations have primarily focused on dialects (Zampieri et al., 2020; Ziems et al., 2022; Liu et al., 2023) and sociolects (McCormack et al., 2011; Zhang et al., 2021). However, the dimension of writing systems within languages has received little attention. We demonstrate that texts in underrepresented writing systems may encode unique cultural nuances that differ significantly from those found in communities using more common scripts.

heading: 3 Data Collection and Analysis
text: Here we describe the procedure of creating MC2, the largest open-source corpus so far for four minority languages in China.
Before collecting MC2, we first conduct a preliminary audit in the low-resource language splits of previous multilingual corpora and find severe quality issues such as language contamination. Therefore, we propose a quality-centric solution for data collection of low-resource languages, which aims to ensure accuracy while improving the comprehensiveness and coverage of the data.
The collection procedure of MC2 mainly consists of three steps: (1) gathering web crawls, (2) incorporating existing datasets, and (3) deduplicat-
ing and filtering. Throughout the collection process, we adhere to the protocol of quality-centric collection. We hope to establish a reliable groundwork for subsequent language model training or linguistic research. Notably, our framework for data collection is generally language-agnostic and can be easily applied to collecting web corpora for other low-resource languages.

heading: 3.1 Quality Issues in Previous Corpora
text: When auditing previous multilingual web corpora for low-resource languages (Abadji et al., 2022; Nguyen et al., 2023), we find two critical quality issues: language misidentification and insufficient data cleaning. These defects pose a significant threat to effective model training and might undermine the credibility of research findings.
Language Misidentification An important step in collecting web corpora for low-resource languages is to identify the language of a web page. However, current language identification tools such as fastText (Joulin et al., 2016) are prone to error (Kreutzer et al., 2022). This issue is more severe in languages with similar writing systems.
Specifically, we audit the Uyghur splits in CulturaX and OSCAR. Since the documents in these two datasets are paired with their respective URLs, we can efficiently label their languages according to the website of their URLs. We manually check the 654 websites comprising the Uyghur corpus in CulturaX. Our examination reveals that 16% of the data is actually in Kazakh or Arabic, languages utilizing scripts akin to that of Uyghur. Similarly, 33% of the Uyghur corpus in OSCAR are Kazakh or Arabic. See the example of misidentification in Appendix A.1.
This issue presents a significant concern for future research in low-resource NLP, as experiments conducted on heavily contaminated corpora can produce misleading results. For instance, this contamination can adversely impact research on language transfer. Researchers may aim to evaluate whether a model trained on a monolingual corpus in Language A can transfer its capabilities to Language B. However, if the corpus already contains a considerable amount of Language B, the target language, the conclusions from such experiments are unlikely to be reliable.
Insufficient Data Cleaning The web crawls in previous corpora often contain unwanted texts such
as sidebars, headers, and footers, as shown the example in Appendix A.1. We sample 100 pages from the Tibetan corpus of CulturaX and our manual annotation shows that 42% contain headers or footers. These undesired texts affect the coherence of the document texts, which might hinder models from learning the linguistic patterns of low-resource languages.

heading: 3.2 Step 1: Web Crawling
text: Our corpus is mainly made up of web crawls. We combine both human labor and AI assistance to prevent the flaws in the previous corpora.
Language Identification Different from previous efforts for curating massively-multilingual web crawls (Nguyen et al., 2023), the number of websites in each language is limited for the four minority languages in our study. We manually maintain a list of high-quality websites for each language of our study, to avoid language contamination resulting from mislabeling by identification tools. These websites are collected from a variety of sources, including URLs in existing corpora, search engines, web portals, website recommendations on social media, and hyperlinks on these websites. For each website in the list, we start from the homepage and crawl all the pages through a breadth-first search.
Text Extraction The web crawls in previous corpora often contain unwanted texts such as sidebars, headers, and footers. As the number of websites in our corpus is manageable and the web pages in the same website usually share the same structure, we automatically design text extraction rules for each website. These rules can precisely extract the title and main content of a web page, discarding other distracting texts. Specifically, for each website, we ask Github Copilot2 to analyze the HTML structure of a sampled web page and write a Python code to extract its title and main content. In this way, we can extract the wanted texts in the raw web crawls with high accuracy and efficiency.
In total, we collected 2.0G Tibetan, 1.1G Uyghur, 1.3G Kazakh, and 1.7G Mongolian texts in our new web crawls.

heading: 3.3 Step 2: Incorporation of Existing Datasets
text: Thanks to the existing efforts in the community, we incorporate open-source resources into our corpus, including CulturaX (Nguyen et al., 2023),
2https://github.com/features/copilot
Wikipedia3, and NLGIW 2023 Shared Task4. We remove the data with language contamination when merging these datasets.
CulturaX is a cleaned collection of mC4 and OSCAR. Its Tibetan and Uyghur splits can be used to supplement our corpus. However, there are errors in language identification in CulturaX, especially the confusion between Uyghur texts and Kazakh texts in Arabic scripts. We thus conduct language re-identification on the Uyghur split of CulturaX, using the website-centric strategy explained in Section 3.1. From CulturaX, we obtained 718M Tibetan, 338M Uyghur, and 65M Kazakh texts.
Wikipedia is a high-quality resource for lowresource languages. We collected 127M Tibetan and 41M Uyghur texts from Wikipedia. NLGIW 2023 Shared Task provides an open-source corpus of Tibetan news, where we add 342M texts.

heading: 3.4 Step 3: Deduplication and Filtering
text: Previous works claim that properly filtered and deduplicated web data is crucial to effective pretraining (Lee et al., 2022; Penedo et al., 2023; Ranathunga et al., 2024). Therefore, we take a series of measures for deduplication and filtering to ensure the high quality of our corpus.
Deduplication We find two sources of duplication in our corpus. One is that the websites in minority languages often repost from each other. The other is the duplication between the web crawls in Step 1 and the existing corpus merged in Step 2. We thus use a combination of deduplication methods, including URL-based, exact, and fuzzy deduplication. See the implementation details in Appendix A.2
Filtering We design the filtering rules from several aspects, including repetition, document lengths, and unexpected characters. See details in Appendix A.2.
We additionally make efforts to remove privacy information from the corpus. We use heuristics to identify emails, telephone numbers, and Resident Identity Card numbers. We replace them with special tokens, i.e., [email], [phone], and [idcard].
After deduplication and filtering, we obtain 2.2G Tibetan, 736M Uyghur, 937M Kazakh, and 970M Mongolian texts, which compose the current version of MC2.
3https://huggingface.co/datasets/graelo/ wikipedia
4http://nlg.cipsc.org.cn/evaluation_12.html

heading: 3.5 Statistical Analysis of MC2
text: We provide an in-depth analysis from the perspective of domain diversity and document length.
Domain Diversity For the pretraining corpus, both quality and diversity are crucial. When curating the website list for crawling, we comprehensively collect high-quality websites from diverse domains to enhance the dataset’s diversity while ensuring quality. Specifically, our data sources fall into five categories: News, Culture, Government, Wikipedia, and Others. Figure 2 shows the proportion of different categories of websites in MC2. Our corpus covers formal official documents, and miscellaneous news articles about politics, important events, daily life, and entertainment, as well as cultural content related to literature, art, and religion. It is noteworthy that news articles constitute the majority of our datasets. News articles are usually grammatically correct, which could potentially facilitate learning language modeling under a low-resource setting.
Document Length Constrained by the absence of high-quality long-form texts, prior efforts like CINO (Yang et al., 2022) have to train their models with a collection of individual sentences, which might restrict their abilities on longer texts. In contrast, our work introduces a document-level corpus. We report the distribution of document length for each language in Figure 3. The average document length of MC2 far exceeds the maximum input length, i.e., 512, of the previous BERT-size model for minority languages in China.
We observe that CultureX also includes long documents. However, as mentioned in Section 3.2, its data cleaning might not be adequate, introducing
сайн уу
сайн уу
undesired texts like sidebars, headers, and footers. These issues could disrupt the semantic coherence of the entire document, making it challenging for models to learn correct dependencies within long texts. We expect that our high-quality, long-form corpus will assist future efforts in training models with better capacities for understanding and generating longer texts in minority languages.

heading: 4 Discussion on Underrepresented Writing Systems
text: Many languages adopt distinct writing systems across various regions. For instance, in China, minority languages such as Kazakh and Mongolian employ scripts that differ from the Cyrillic scripts used in Kazakhstan and Mongolia, as illustrated in Table 3. Unfortunately, existing datasets predominantly concentrate on the more prevalent writing systems, neglecting the less common ones. In re-
sponse to this issue, MC2 is the first effort to collect native corpora for the two underrepresented writing systems, i.e., the Kazakh Arabic script and the traditional Mongolian script.
To delve into the multiplicity of writing systems, we conduct empirical studies from both the technical and the cultural perspectives. We find that imperfect transliteration between writing systems harms model training. Furthermore, corpora in different writing systems of the same language encapsulate unique cultural context specific to their respective language communities. These findings underscore the importance of gathering resources for minority writing systems.

heading: 4.1 Flawed Training with Imperfect Transliteration
text: A common approach to addressing different writing systems is transliteration (Nakov and Ng, 2009; Chakravarthi et al., 2019; Muller et al., 2021; Sun et al., 2022). To obtain a model for low-resource scripts, it is intuitive to transliterate the corpus in the high-resource scripts into low-resource ones for training. However, there are no one-to-one conversion rules between scripts for languages such
as Mongolian (Bao et al., 2014). The transliteration between traditional and Cyrillic Mongolian is context-dependent and current open-source tools are far from perfect. We investigate how the noisy transliteration results affect the training of models.
Experimental Setup We prepare two versions of training data in traditional Mongolian. One is 900M transliterated traditional Mongolian data, which is converted from the Cyrillic Mongolian data in CulturaX, using the only open-source transliteration tool available so far5. The transliterated results often contain errors, as shown by the examples in Appendix C. The other version of data is an equivalent volume of authentic traditional Mongolian data in MC2. We then train XLM-RoBERTa-large (Conneau et al., 2019) with these two different versions of data respectively and compare the convergence speed and performance of the models.
Results The training processes are illustrated in Figure 4. The training loss of transliterated data is significantly higher than that of authentic data. This discrepancy arises due to the substantial noise in transliterated data, making it challenging for the model to capture underlying linguistic patterns. We further measure the perplexity of both models on a hold-out validation set of traditional Mongolian text. The perplexity of the model trained on authentic data steadily decreases with the training steps, maintaining a similar magnitude to the training loss. Conversely, the model trained on transliterated data exhibits a sharp increase in perplexity when evaluated on authentic data, indicating that it fails to learn the language modeling of true traditional Mongolian.
Lessons Learned Using noisy data transliterated from high-resource writing systems will greatly hinder the learning of low-resource writing systems. This highlights the necessity of collecting an authentic corpus for underrepresented scripts in the absence of mature transliteration tools.

heading: 4.2 Cultural Differences Behind Writing Systems
text: For some languages such as Kazakh, we can achieve perfect transliteration between different writing systems using pre-defined rules. Nevertheless, there exist disparities in the cultural backgrounds between the language variants using dif-
5https://github.com/tugstugi/mongolian-nlp
ferent scripts. With the technique of probing (Jiang et al., 2020; Zhang et al., 2021), we investigate whether the training data collected from different writing systems will lead to distinct cultural knowledge in the resulting models.
Probing Questions We take the Kazakh language as our research target. The Kazakh community in China uses the Arabic script while the Cyrillic script is adopted in Kazakhstan. We design probing questions that reflect the cultural differences between the two Kazakh communities, such as those in Table 4. Each question offers two options: Option A aligns with the cultural context of the Kazakh community in China, while Option B corresponds to the cultural context of Kazakhstan. These questions are manually formulated by comparing Wikipedia articles that describe the two communities. We compile 25 pairs of questions covering topics such as geography, politics, and daily life. Originally written in English, these questions are translated into Cyrillic Kazakh using Google Translate, and then transliterated into Arabic Kazakh.
Experimental Setup We train two distinct Kazakh language models based on XLMRoBERTa-large, each tailored to one of the writing systems. One is trained with 900M authentic Cyrillic Kazakh texts from CulturaX. And the other is trained with an equivalent volume of Arabic Kazakh texts from our MC2 corpus. We subject the two models trained on different scripts to the cultural probing questions. We query the Arabic Kazakh model with questions written in the Arabic script. Similarly, for the Cyrillic Kazakh model, we use questions written in the Cyrillic script.
Results The Arabic Kazakh model selects Option A for 84% of the probing questions, indicating a strong alignment with the cultural characteristics of Kazakh communities in China. Conversely, the Cyrillic Kazakh model chooses Option B for 56% of the questions, reflecting the cultural characteristics of Kazakh communities in Kazakhstan.
The cultural distinction is probably inherent in the corpus collected from websites using different writing systems. Subsequently, it is captured by the models trained on data in different scripts.
In Table 4, we present three examples of probing questions. For Query 1, the two models yield divergent responses concerning the holiday celebrated on May 1st. This discrepancy arises from the fact that the official holidays on this date differ
between China and Kazakhstan. Similarly, Query 2 and Query 3 reflect the cultural differences in terms of economy and geography. See their explanations in Appendix B.
Lessons Learned The corpora in different writing systems contain unique cultural context specific to their respective communities. Hence, it is sub-optimal to directly transliterate high-resource scripts into low-resource ones for training, which may undermine the cultural nuances underlying the low-resource writing system. The inclusiveness of writing systems is crucial to the construction of culturally-aware models. We encourage future research on properly leveraging the data in the highresource writing systems while preserving the cultural uniqueness in the underrepresented scripts.

heading: 5 Continual Pretraining with MC2
text: To demonstrate the practical value of our corpus, we train models with MC2 and compare their performance with competitive counterparts. Instead of training from scratch, we conduct continual pretraining on existing models with MC2, which is an effective technique to adapt models to new languages (Ebrahimi and Kann, 2021; Muller et al., 2021; Yong et al., 2023). We continually pretrain XLM-RoBerta-large (Conneau et al., 2019) with MC2, obtaining MC2XLMR-large. Thanks to the high quality of MC2, it performs comparably to CINO, a counterpart trained with closedsource data. We also attempt to adapt larger models such as Llama (Touvron et al., 2023) to the four languages, obtaining MC2Llama-13B. The newly trained model shows the superior ability of in-context learning in these languages.

heading: 5.1 Model Training
text: Two models with different architectures and sizes are used for continual pretraining. One is XLMRoBERTa-large (Conneau et al., 2019), a 560M encoder-only multilingual model. The other is Llama2-ZH-13B (Huang et al., 2023), a bilingual model supporting both English and Chinese.
The vocabularies of XLM-RoBERTa-large and Llama2-ZH-13B hardly contain tokens for the four languages in our study. Thus, we add 3K new tokens for each language, obtained by BPE (Sennrich et al., 2016).
To avoid the catastrophic forgetting of learned languages, we add 0.25B tokens of Chinese data from Wanjuan (He et al., 2023) and 0.25B tokens of English data from C4 (Raffel et al., 2020) for training. See more training details in Apppendix D.1.

heading: 5.2 Evaluation Setup
text: Datasets For the four minority languages, we can only find limited datasets for evaluation. WCMv2 (Yang et al., 2022) is a 10-category text classification dataset. TibetanQA (Sun et al., 2021) is a Tibetan machine reading comprehension (MRC) dataset. Flores-200 (Costa-jussà et al., 2022) contains machine translation (MT) tasks for Tibetan and Uyghur. We additionally construct an MRC dataset, XQuAD-MT, by translating XQuAD (Artetxe et al., 2020) into Uyghur and Tibetan with Google Translate6. All the evaluated datasets only consist of testing sets. See data statistics in Appendix D.2.
Settings For smaller encoder-only models, we adopt zero-shot transfer, i.e., finetuning a model with instances in a high-resource language and directly testing it on the low-resource language instances (Artetxe et al., 2020). For WCMv2, we use its Chinese instances for training. For TibetanQA and XQuAD-MT, we use CMRC 2018 (Cui et al., 2019), a Chinese MRC dataset for
6https://translate.google.com/
training. For larger generative models, we adopt in-context learning, providing exemplars in the prompt (Brown et al., 2020). See the prompts in Appendix D.3.
Compared Models We compare our models with a wide range of multilingual models, including CINO (Yang et al., 2022), mBERT (Devlin et al., 2019), XLM-RoBERTa (Conneau et al., 2019), BLOOM (BigScience et al., 2022), mT5 (Xue et al., 2021) and ByT5 (Xue et al., 2022). These models (potentially) process one or more languages in our study. XLM-RoBERTa is a multilingual pretrained language model (PLM), whose vocabulary contains 3 Tibetan tokens, 5 Mongolian tokens, and more than 14K tokens in the Arabic script. However, it has only been pretrained in Uyghur. CINO is the largest PLM so far focusing on the minority languages in China, but its pretraining corpora are closed source. BLOOM, mT5, and ByT5 are multilingual LLMs, which can represent tens of languages. Although they are not trained in the four minority languages, their tokenizers can encode unseen languages instead of treating them as unknown tokens. See more details of the supported languages and numbers of parameters in Appendix D.3.

heading: 5.3 Results
text: We show the experiment results of zero-shot transfer in Table 5, with XLM-RoBERTa-large as the baseline. We find both MC2XLMR-large and CINO can outperform vanilla XLM-RoBERTalarge on the classification and MRC tasks. It indicates that pretraining on minority language corpora can effectively enhance the model’s ability to represent these languages. When comparing the two models for Chinese minority languages, we find our MC2XLMR-large can exhibit comparable performance to CINO, which is trained on a closed-source corpus three times larger than MC2. This proves the high quality of MC2, which can contribute to more transparent and reproducible
research on minority languages in China. Table 6 illustrates the results of LLMs under the in-context learning setting. The performance of evaluated multilingual LLMs on WCM-v2 resembles random guessing, with an accuracy of around 10% in the ten-class classification tasks. These models struggle with the Tibetan MRC task. In contrast, the MC2Llama-13B model, pretrained on the MC2 corpus, outperforms others by more than +26% accuracy and F1 on WCM-v2. It also achieves 31.5% F1 on TibetanQA. These results show that MC2Llama-13B has a superior ability to handle tasks in the four minority languages of China. Besides smaller models, MC2 can also be leveraged for the pretraining of LLMs, thereby effectively enhancing the models’ ability to represent these languages.
The results on Flores-200 and XQuAD-MT are reported in Appendix E. The models trained on MC2 also demonstrate superior performance on these tasks.

heading: 6 Conclusion
text: We present MC2, an open-source corpus for four minority languages in China. It is also the first corpus focusing on two underrepresented writing systems, i.e., the Kazakh Arabic script and the traditional Mongolian script. To address the severe quality issues in previous low-resource datasets, we adopt a quality-centric strategy to collect MC2. We also emphasize the cultural significance of including low-resource writing systems through empirical studies. Our data and models are openly accessible to the community to facilitate research and applications on low-resource languages.
Limitations
Data Sources MC2 is mainly composed of web crawls. We have done our best to enrich the domains covered by the MC corpus. However, it is
not feasible to collect data from all websites worldwide. In this work, due to copyright constraints, we are also unable to collect and publicly release a corpus sourced from books in these minority languages. We note that the texts from books can also be used to pretrain language models.
News articles are one of our primary data sources, which are crawled from public websites. The data could potentially contain the biases of these media. Therefore, the data should be used with caution to avoid potential biases and misrepresentations.
Probing Questions It is challenging to collect the kind of probing question in our study, which requires mining the differences between the two communities and formulating them as multi-choice questions. We made great efforts and collected 25 verified probing questions covering a diverse range of topics. As a preliminary study, the size of probing questions is small. We plan to design efficient methods for cultural question sourcing and expand the data scale.

heading: Acknowledgments
text: This work is supported in part by NSFC (62161160339) and Beijing Science and Technology Program (Z231100007423011). We thank the anonymous reviewers for their valuable suggestions. We thank native speakers of minority languages who contribute to this work. For any correspondence, please contact Yansong Feng.

heading: A Corpus Details
text: A.1 Quality Issues in Previous Corpora We provide several cases of quality issues in previous corpora.
Language Misidentification In the Uyghur split of CulturaX, the website kazakh.people.com.cn ranks third in terms of document count, comprising 9% of the total documents in the dataset. However, this website is a Kazakh-language news website. These documents are mislabeled as Uyghur because the website uses the Kazakh Arabic script, which is similar to the Uyghur Arabic script.
Insufficient Data Cleaning In Figure 5, we show an example of insufficient data cleaning from the Tibetan corpus in CulturaX. The highlighted parts are irrelevant texts on a web page, such as headers and footers.
A.2 Deduplication and Filtering We describe in detail the deuplication and filtering process of MC2.
Deduplication We first discard the web pages from different sources with the same URLs. We then adopt both exact and fuzzy deduplication. For exact deduplication, we use the SHA-256 hash function to calculate the hash value of each web page’s content and keep only one page for the pages with the same hash value. For fuzzy deduplication, we remove similar web pages by applying MinHash (Broder, 1997): for each page, we compute the minhash values and measure its approximate
similarity with other pages, removing pairs whose minhash values are the same in at least one bucket. We use the same parameters as Lee et al. (2022): n = 5 (5-grams), b = 450, and r = 20.
Filtering We use the following rules:
• Repetition: We remove the documents that have a high ratio of paragraph repetition or n-gram repetition.
• Document Length: We remove the documents whose lengths are below the threshold.
• Unexpected Character: We remove the document whose ratios of the characters in the target language are below the threshold.
Note that the hyperparameters in the heuristic for filtering English corpora are not necessarily applicable to low-resource languages due to the difference in encodings and linguistic characteristics. One needs to decide the hyperparameters for each language respectively.
A.3 Code-Switching in MC2
Code-switching is an important and interesting phenomenon in multilingual NLP. Here we share some analyses and findings of the code-switching phenomenon present in our corpus. We mainly focus on the code-switching between Chinese and the four minority languages in China.
First, we calculate the proportion of documents containing Chinese characters for the four languages in MC2. The results are shown in Table 7. 3.3% of the documents in MC2 contain Chinese characters, indicating the commonness of the codeswitching phenomenon in our corpus. Uyghur exhibits the highest rate of code-switching. We find that the Uyghur documents in our corpus are more likely to attach Chinese translations after entity mentions.
Second, we analyze the types of code-switching present in our corpus. Inspired by previous works (Doğruöz et al., 2021; Winata et al., 2023),
we classify documents containing Chinese characters into three types:
• Intra-sentential code-switching: switching that occurs in a sentence where a word was switched to another language.
• Inter-sentential code-switching: switching that occurs in two different sentences.
• Attached translations of entities: In many cases, authors of articles often append Chinese translations after entity mentions. Strictly speaking, this doesn’t fall under codeswitching, but this situation is quite common.
We sample 50 documents containing Chinese characters and manually classify them into three types. The proportion of each type is shown in Table 8. The most common phenomenon is attaching Chinese translations to the entities mentioned in the documents. The common entities under these circumstances include scientific and cultural terms. Regarding intra-sentential code-switching, the switched phrases are often entities of Chinese origin, such as Han Chinese names and literary works written in Chinese. These Chinese terms currently lack official translations in these languages, so they are directly borrowed in their original form. Many instances of inter-sentential code-switching are sentences in different languages, where the sentences before and after each other are translations of one another. Based on this property, we can mine parallel sentences from the corpus, which we leave as future work.

heading: B Cultural Probing
text: We provide a detailed explanation of the probing cases in Table 4. For Query 1, May 1st is a public holiday in both China and Kazakhstan. However, the people in the two countries celebrate different festivals. In China, people celebrate International Workers’ Day on May 1st, to commemorate the struggles of laborers and the working class for the eight-hour workday. However, according to
the Kazakhstan law signed on October 18, 1995, May 1st was renamed People’s Unity Day, and the Soviet-era Labor Day was formally canceled.
For Query 2, in mainland China, the official currency is Renminbi. Meanwhile, in Kazakhstan, the official currency is Kazakhstani Tenge.
For Query 3, Sayram Lake is an endorheic freshwater lake in the northern Tianshan Mountains, near to Ili Kazakh Autonomous Prefecture in China. Its name, Sayram, is believed to originally derive from Kazakh, meaning blessing. Sayram Lake is famous in China as the last drop of the Atlantic tears. Lake Balkhash is also famous in Kazakhstan since it is the largest lake in Kazakhstan. Around 20% of Kazakhstani people are living in the drainage basin of Lake Balkhash, including the residents of Almaty, the largest city in Kazakhstan.

heading: C Mongolian Transliteration
text: We use an open-source tool to transliterate Cyrillic Mongolian into traditional Mongolian7. It is trained on 80K sentence pairs crawled from a closed-source transliteration system. When transliterating a Cyrillic Mongolian corpus, we find that the transliteration quality deteriorates with the increase in the input length. So we split each document into sentences and do the transliteration at a sentence level to improve the quality. However, we find that the transliteration results are imperfect, especially in the case of long sentences. In Figure 6, the transliteration tool produces a bad result for a long input sentence, repeating similar words at the end of the output.

heading: D Experiment Details
text: D.1 Model Training Hyperparameters We list in Table 9 the hyperparameters for the continual pretraining of MC2XLMR-large and MC2Llama-13B.
Training Frameworks For MC2XLMR-large, we use Transformers (Wolf et al., 2020) for training. For MC2Llama-13B, we use Megatron (Shoeybi et al., 2019) for training.
Computing Infrastructure We continually pretrain MC2XLMR-large on one A100 GPU, and an epoch takes 17 hours. We continually pretrain MC2Llama-13B on eight A100 GPUs, and an epoch takes 20 hours.
7https://github.com/tugstugi/mongolian-nlp/ tree/master/bichig2cyrillic
Input:
Даваа гарагт Шведийн академи дахь Нобелийн уран зохиолын
хорооны байнгын нарийн бичгийн дарга Сара Даниус Швед дэх
Сверижес радиогийн нэвтрүүлгийн үеэр 2016 оны уран зохиолын
D.2 Data Statistics
WCM-v2 Wiki-Chinese-Minority-v2(WCMv2) is a classification dataset. It covers Mongolian, Tibetan, Uyghur, Cantonese, Korean, Kazakh and Chinese. However, the training set is only available in Chinese. The data statistics is shown in Table 10
TibetanQA Tibetan Machine Reading Comprehension Dataset (TibetanQA) is a Tibetan reading comprehension dataset, which contains 20,000 question-answer pairs and 1,513 articles. However, the released dataset contains only 2007 questionanswer pairs. The average lengths (in characters) of contexts, questions, and answers are 688, 58, and 93 respectively.
Flores-200 Flore-200 is a machine translation dataset for low-resource languages, which covers two languages of our study, i.e., Tibetan and Uyghur. We sample 200 testing instances for
Tibetan-to-Chinese and Uyghur-to-Chinese translations.
XQuAD-MT Since there are no existing QA datasets for the languages of interest except Tibetan, we translate the Chinese split of XQuAD into Uyghur and Kazakh through Google Translate8. We translate an instance’s passage, question, and answer separately. Consequently, the translated answer may not exactly appear in the translated passage. We then extract an EXACT subset from the translated dataset where the translated answer exactly appears in the translated passage. We obtain 1,190 QA pairs for Uyghur, 257 of which belong to the EXACT subset, and 1,190 QA pairs for Kazakh, 417 of which belong to the EXACT subset.
D.3 Evaluation Setup Models We list the numbers of parameters and supported languages of all models we use in Table 11. We only consider four minority languages used in our experiments. Other supported languages can be found in these models’ original papers.
Prompts Regarding in-context learning, we follow the previous work (Shi et al., 2022a) to use native exemplars with a high-resource language (Chinese) for prompt description. We use 3 exemplars for WCM-v2, 2 exemplars for TibetanQA, and 5 exemplars for Flores-200. In Table 12, we list the format of our prompts along with English translation in italics.
Result Reporting We report the results of single runs. In the experiments of in-context learning, the
8Google Translate does not support traditional Mongolian currently. As the Kazakh translation results by Google Translate are in the Cyrillic script, we transliterate them into the Kazakh Arabic script.
models sometimes fail to output a label in the label list. In these cases, we assign random labels to these samples.

heading: E Additional Results
text: E.1 WCM-v2 In Table 13, we report the performance in different languages on WCM-v2.
E.2 Flores-200 Considering that encoder-only models cannot perform the MT task, we use generative models for experiments. Since the models have limited abilities to generate texts in low-resource languages, we only conduct experiments to translate the lowresource languages into Chinese.
The results are shown in Table 14. As they are not finetuned for the translation task, the four models generally perform poorly on this task merely through in-context learning. However, our model achieves non-zero scores, demonstrating its preliminary ability to understand these languages, thanks to the training on MC2. We leave the supervised training setting of MT for future work.
E.3 XQuAD-MT In Table 15 we report the performance on the full set and EXACT subset of XQuAD-MT.

heading: Future Work
text: We leave the supervised training setting of MT for future work. E. 3 XQuAD-MT In Table 15 we report the performance on the full set and EXACT subset of XQuAD-MT