Enhancing Few-shot Cross-lingual Transfer with Target Language Peculiar Examples
Year: 2023

Abstract:
Few-shot cross-lingual transfer, fine-tuning Multilingual Masked Language Model (MMLM) with source language labeled data and a small amount of target language labeled data, provides excellent performance in the target language. However, if no labeled data in the target language are available, they need to be created through human annotations. In this study, we devise a metric to select annotation candidates from an unlabeled data pool that efficiently enhance accuracy for few-shot cross-lingual transfer. It is known that training a model with hard examples is important to improve the model’s performance. Therefore, we first identify examples that MMLM cannot solve in a zero-shot cross-lingual transfer setting and demonstrate that it is hard to predict peculiar examples in the target language, i.e., the examples distant from the source language examples in cross-lingual semantic space of the MMLM. We then choose high peculiarity examples as annotation candidates and perform few-shot cross-lingual transfer. In comprehensive experiments with 20 languages and 6 tasks, we demonstrate that the high peculiarity examples improve the target language accuracy compared to other candidate selection methods proposed in previous studies. The code used in our experiments is available at https://github.com/hwichan0720/ fewshot_transfer_with_peculiarity.

heading: 1 Introduction
text: Sufficient labeled data is essential to train an accurate model. However, few languages have abundant language resources for both labeled and unlabeled data like English (Joshi et al., 2020). In addition, constructing a large amount of labeled data through human annotators is costly and time-consuming. The use of Multilingual Mask Language Models (MMLMs) is one way to overcome this problem
*Now at Hitotsubashi University
as they show good zero-shot cross-lingual performance in target languages by fine-tuning with only task-specific labeled data of source language, such as English. While this zero-shot cross-lingual transfer ability is promising for the target languages with no or limited task-specific resources, there is a divergence in accuracy between the source and target languages, meaning the zero-shot cross-lingual transfer ability is imperfect.
To analyze the characteristics of hard examples for predictions in zero-shot cross-lingual transfer, we visualize representations of the source and target languages’ examples and highlight the target ones with accuracy in the zero-shot setting, as shown in Figure 1. We can observe that the accuracies of target language examples distant from the source language examples are low compared to others. We refer to these as peculiar examples of the target language. We should address peculiar examples to further enhance performance in the target language.
Few-shot cross-lingual transfer, adapting
747
MMLM with a small number (0.1k–1k) of task-specific labeled examples in the target language, is a promising approach to enhance performance for target languages. Specifically, Lauscher et al. (2020) showed that a small number of examples randomly selected from labeled dataset significantly improves the accuracy for the target languages. However, if there are no labeled examples at all, we should create them by selecting annotation candidates from an unlabeled data pool. From this perspective, Kumar et al. (2022) evaluated candidate selection methods proposed in active learning research for selecting the annotation candidates for few-shot cross-lingual transfer. They split the target languages based on their zero-shot cross-lingual transfer performance into “good” and “poor” (or “good”, “fair”, and “poor”) language groups1 and demonstrated that effective methods vary for each language group. Our preliminary analysis (Figure 1) suggests that adapting MMLM for peculiar examples is crucial to improve performance for the target language and recommends selecting peculiar examples as annotation candidates.
Therefore, in this study, we first propose a metric to measure peculiarity of the target language examples. Note that peculiarity is defined without labels of the downstream tasks. Then, we select high peculiarity examples as annotation candidates and conduct few-shot cross-lingual transfer using languages with “good” and “poor” zero-shot crosslingual performance. Our experiments show that the proposed metric peculiarity is very simple yet effective in selecting candidates for few-shot crosslingual transfer. Our contributions in this study are threefold:
1. We propose a simple metric to measure peculiarity and show that the prediction accuracy of high peculiarity examples is low compared to others in the zero-shot setting.
2. We conduct few-shot cross-lingual transfer using high peculiarity examples and demonstrate that these examples can improve accuracy compared to other candidate selection methods regardless of language groups in few (2–3) label classification tasks. In addition, our analysis shows that peculiarity is robust
1They referred the language groups as “C1” and “C2” (or “C1”, “C2”, and “C3”), respectively.
for hyperparameters and brings consistent performance in few-shot cross-lingual transfer.
3. However, high peculiarity examples do not work well for the “poor” language group in many-label (5) classification and sequencetagging tasks. Our analysis indicates that these examples are redundant to fine-tuning MMLM. Therefore, we design a method that combines existing methods to select diverse examples. Our experiments demonstrate that our method enhances accuracy across target languages.

heading: 2 Notation and Task Setting
text: In this section, we define notations and explain our task setting. We denote the source and target language as S and T , respectively. For the source language S, we assume that labeled data exist for downstream tasks DS = (XS , YS), where XS = {x1S , ..., xiS} are monolingual data and YS = {y1S , ..., yiS} are corresponding labels. For the target language T , we only have monolingual data XT = {x1T , ..., xjT }. We denote an MMLM as M and one fine-tuned by DS as M
S. We use M
S to the target language inputs in the zero-shot cross-lingual transfer setting.
In this study, we conduct few-shot cross-lingual transfer. In this setting, we select annotation candidates X ′T ⊂ XT and limit ∣X ′T ∣ to n. Then, human
annotators annotate labels Y ′T for X ′ T 2. We additionally fine-tune MS using D′T = (X ′T , Y ′T ) and denote the model as MS→T. To support understanding of zero-shot and few-shot cross-lingual transfer, we provide an overview of zero-shot and few-shot cross-lingual transfer in Figure 2. The objective of our task is to select the candidates X ′T that lead to better performance of MS→T in the target language T .

heading: 3 Related Works
text: Zero-shot cross-lingual transfer. mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) are the representative MMLMs. mBERT is an extension of BERT that is pre-trained on Wikipedia data in various languages. XLM-R is trained on 2.5T data of more than 100 languages extracted from Common Crawl (Wenzek et al., 2020). The most interesting property of these MMLMs is that they show a strong zero-shot cross-lingual transfer performance even though they did not explicitly use bilingual data.
Several studies have analyzed their zero-shot cross-lingual transfer ability and indicated that source-target languages’ similarity is important for the transfer. Pires et al. (2019) analyzed the ability of mBERT in NER and POS tagging tasks and demonstrated that more overlap in WALS (Dryer and Haspelmath, 2013) features between the source and target languages, better transfer. Lauscher et al. (2020) quantitatively measured the similarity between the source and target languages using LANG2VEC (Littell et al., 2017) and showed that it has a strong correlation with the zero-shot crosslingual transfer performance.
Yang et al. (2022) analyzed relations of alignment quality between the source and target languages and the transfer performance. Specifically, they measured the alignment quality between languages in cross-lingual semantic space of an MMLM using CKA (Kornblith et al., 2019) and showed a strong correlation between the CKA scores and the zero-shot cross-lingual transfer performances. They also proposed a method to improve the alignment quality using pseudo-bilingual data and demonstrated that it enhances the zeroshot cross-lingual transfer performance. In addition, there have been several attempts to enhance the zero-shot cross-lingual transfer ability using ad-
2In our experiments, since we employ previously annotated labels, we do not annotate the candidates ourselves.
ditional bilingual resources (Lample and Conneau, 2019; Cao et al., 2020; Chi et al., 2021; Dou and Neubig, 2021; Yang et al., 2021).
Few-shot cross-lingual transfer. Few-shot crosslingual transfer is another approach to improve the performance for target languages. Lauscher et al. (2020) randomly selected annotation candidates X ′ T from XT and conducted few-shot cross-lingual transfer. They demonstrated that these candidates surprisingly boost performance compared to the zero-shot cross-lingual transfer setting (e.g., gains are 27.3 points in POS tagging task).
The most relevant previous work is Kumar et al. (2022) who conducted few-shot cross-lingual transfer and evaluated representative candidate selection methods used in active learning research, such as entropy and BADGE (Ash et al., 2020). BADGE is a method to select diverse and uncertain examples. It first calculates gradient embeddings (GEs) for each example xT ∈ XT , which are vectors of hidden states multiplied by MS’s confidences about each example, and then selects the most typical examples in the GEs space. However, BADGE is expensive for sequence-tagging tasks takes because it has to calculate the GE for each token. Therefore, Kumar et al. (2022) simplified the GEs to loss embeddings (LEs), which are vectors consisting of cross-entropy loss for each token, by taking M
S’s prediction as actual labels. Their experiments showed that the method using LEs obtains consistent gains over other methods for sequencetagging tasks.
The methods using embeddings, such as GEs and LEs, can select diverse candidates from XT . However, we consider that these methods also select the candidates well aligned with the source language examples XS , which are predicted accurately in the zero-shot cross-lingual transfer setting (Figure 1). Therefore, in this study, we propose a metric, peculiarity, which isolates examples that cannot be covered by source language data XS , and select the candidates based on our metric.

heading: 4 How to Measure Language Peculiarity
text: Sorscher et al. (2022) revealed that training with hard examples for a neural model can improve the model performance exponentially beyond power law scaling, both in theory and practice. In terms of few-shot cross-lingual transfer, we need to finetune MS using target language examples that will not be predicted correctly. Therefore, we study
how to extract these examples as the annotation candidates from XT without knowing their labels.
Yang et al. (2022) showed that there is a correlation between cross-lingual transfer performance and language-level alignment quality. Based on the preliminary analysis (Figure 1), we infer that there are correlations not only at the language-level but also at the example-level, which means that the accuracy of the target language examples depends on whether they are aligned with the source language examples. This analysis indicates that we can detect hard examples for MS from XT if we can measure the example-level alignment quality.
Therefore, we devise a simple but effective metric to isolate the examples based on their alignment quality without additional language resources. Intuitively, the examples that have no source language examples in the neighborhood in cross-lingual semantic space are not well aligned3. In addition, Figure 1 shows that accuracies of these examples are low compared to the others. We use this aspect instead of the example-level alignment quality as a proxy to detect the hard examples for MS. We refer to this aspect of whether there are source language examples in the neighborhood as peculiarity and define peculiarity of xjT ∈ XT as:
Peculiarity(xjT ;MS) = 1
k ∑ xiS∈k-NN(xjT ) distance(xjT ,xiS) (1)
where distance serves as a metric for measuring the distance between input hidden states, encompassing options such as Cosine and Euclidean distance. xiS ∈ XS is a source language example of k nearest neighbor based on distance and k is a hyperparameter. xiT and x j S are hidden states of BOS tokens from the final layer of MS. Peculiarity is higher when there are no source language examples in the neighborhood of the input target language example. In experiments, we first confirm whether the peculiarity can isolate the examples with low or high prediction accuracy in the zero-shot setting. Then, we select high peculiarity examples as annotation candidates and conduct few-shot crosslingual transfer.
3To strictly measure the example-level alignment quality for each xT ∈ XT , we should translate xT into the source language and calculate similarities between their representations. However, this method is costly because it requires a human translator or a high-quality machine translation system.

heading: 5 Experimental Settings
text: We explain experimental settings, which are largely similar to those used in previous works such as Lauscher et al. (2020) and Kumar et al. (2022), including the dataset and training hyperparameters.

heading: 5.1 Tasks and datasets
text: We experiment with XNLI (Conneau et al., 2018) for a classification task, and NER, POS-tagging for sequence tagging tasks. We use WikiANN (Rahimi et al., 2019) and Universal Dependency tree banks (UD, Nivre et al. (2016)) for NER and POS-tagging, respectively. In addition, we use PAWS-X (Yang et al., 2019) and Multilingual Amazon Review Classification (MARC, Keung et al. (2020)) as datasets for classification tasks. There are two- and fivelabel classification settings in MARC, and we refer to them as MARC-2 and MARC-5, respectively.
WikiANN, UD, and XNLI include several languages. We use the same languages used in Kumar et al. (2022)’s experiments. For PAWS-X and MARC, we use all languages. The previous studies (Lauscher et al., 2020; Kumar et al., 2022) indicated that the effectiveness of candidate selection methods in each language depends on the accuracy of downstream tasks in the zero-shot setting. Therefore, we evaluate each method by splitting the target languages into “good’ and “poor” groups, where the target languages achieve the median or higher accuracy and lower than the median accuracy, respectively. We detail the languages of each group and the datasets in Appendix B.

heading: 5.2 Model and training
text: We use XLM-R Base4 as the MMLM in all experiments5. Following Devlin et al. (2019); Pires et al. (2019), we attach token-level and sentencelevel classifiers to the last layer of XLM-R to train sequence-tagging and classification models, respectively.
We use English as the source language and finetune XLM-R with English dataset of each task. We limit the sequence length to 128 subword tokens and set the batch size as 32. For the sequencetagging tasks, we fix the number of training epochs to 20 and the learning rate as 2 ⋅ 10−5. For the classification tasks, we set the training epochs to 3 and learning rate as 3 ⋅ 10−5.
For few-shot cross-lingual transfer, we conduct additional fine-tuning with combined English data and sampled target language examples. We change the training epochs to 1 for the classification tasks and use the same hyperparameters mentioned above.
4 https://huggingface.co/xlm-roberta-base
5In the previous studies (Lauscher et al., 2020; Kumar et al., 2022), mBERT Base cased and XLM-R Base were used in the experiments, with XLM-R achieving better crosslingual performances. In addition, they showed these MMLMs performing with the same trend.

heading: 5.3 Candidate selection methods
text: We select n annotation candidates, and set n to 100, 500, and 1,000. We compare our method with the methods (Random, Entorpy, GE-KM, and LEKM) used in Kumar et al. (2022)6 and an additional method (KM).
Random. We select the candidates randomly.
Entropy. We select the candidates with the highest entropy. We average the entropy per each token for the sequence-tagging tasks.
KM. We cluster BOS hidden states of the last layer using k-means++ (k is the same number of n) and select medoids, most typical candidates, from each cluster following Chang et al. (2021); Hacohen et al. (2022). We use scikit-learn7 for performing k-means++.
GE-KM or LE-KM. We use the gradient embeddings (GEs) for the classification tasks and the loss embeddings (LEs) for the sequence tagging tasks instead of the hidden states and apply the same
6They also employed a method named Data Cross-Entropy (DCE). However, their experiments showed DCE is not effective for few-shot cross-lingual transfer compared to the other methods. Therefore, we exclude DCE in our experiments.
7 https://scikit-learn.org/stable/
steps with KM8. We use Yuan et al. (2020)’s implementation for GE-KM and our re-implementation for LE-KM.
Peculiarity (ours). We select the candidates with the top-n peculiarity according to Equation 1. We use Faiss (Johnson et al., 2019) library9 to search the k-NN source language examples. We set 20 and Euclidean distance as k and distance, respectively, in all experiments.

heading: 6 Experimental Results
text: In this section, we confirm the effectiveness of peculiarity. We first show that peculiarity isolates the examples based on their accuracy in the zeroshot cross-lingual transfer setting (Subsection 6.1), and then demonstrate that the examples of high peculiarity are useful for few-shot cross-lingual transfer (Subsection 6.2).
6.1 Zero-shot transfer for peculiar examples First, we examine whether peculiarity (Equation 1) can isolate the examples predicted correctly and incorrectly in the zero-shot setting. We construct two subsets by extracting 10% examples of bottom and top peculiarity from each language’s validation data and measure their accuracy. We show the
8GE-KM and LE-KM correspond to the methods proposed by Ash et al. (2020) and Kumar et al. (2022), respectively.
9 https://github.com/facebookresearch/faiss
average accuracy of all languages for each subset in Table 1 and more details about each language in Appendix B. This table shows that the examples with high peculiarity cannot be predicted accurately in the zero-shot setting. Specifically, the average score is 85.1% in “Lowest”, but is 59.2% in “Highest”. This result indicates that peculiarity can extract the examples of low accuracy in the zero-shot setting without their labels.
6.2 Enhancing few-shot transfer with high peculiarity candidates
We conduct few-shot cross-lingual transfer using candidates extracted by peculiarity. We show the experimental results for few-label (2–3) classification tasks (XNLI, PAWS-X, and MARC-2) in Table 2 and many-label (5) classification and sequencetagging tasks (MARC-5, NER, and POS) in Table 310. We report delta scores in accuracies between 0-shot and n-shot models following Kumar et al. (2022). These scores are the average across languages in each group.
Table 2 shows that the methods proposed in the previous studies enhance accuracy for each language group but do not outperform random baseline consistently. However, peculiarity consistently achieves the highest scores regardless of the language groups. Therefore, we conclude that peculiarity is useful for selecting annotation candidates
10In Appendices B and C, we provide more detailed results including statistical significance tests.
that could improve the performance for these fewlabel classification tasks.
On the contrary, for MARC-5, NER, and POS the best methods differ in each language group. In the “good” group, peculiarity consistently achieves the highest improvements in the all tasks and nshots. However, in the “poor” group, peculiarity does not work as well as in the “good” and is worse than random in some settings. In addition, the scores show that the methods considering the diversity of candidates (KM, GE-KM, and LE-KM) are the best choice for the “poor” group11.

heading: 7 Analysis
text: 7.1 Redundancy of high peculiarity examples
As mentioned above, peculiarity outperforms the other methods in the few-label classification tasks
11We confirm that GE-KM and LE-KM achieve the highest scores in 500-shot settings. The results for 100- and 500-shot experiments are in Appendix C.
(XNLI, PAWS-X, and MARC-2), but the KMbased methods perform better in the “poor” group for sequence-tagging and many-label classification tasks (NER, POS, and MARC-5). The KM-based methods select candidates by considering their diversity, but peculiarity does not.
Therefore, we assume that peculiarity selects more redundant candidates than KM-based methods. To confirm our assumption, we measure the token type ratio (TTR) of the candidates extracted by KM and peculiarity. Table 4 shows the results and that the TTRs of KM and peculiarity are almost the same in the few-label classification tasks. By contrast, the TTRs of peculiarity are lower than that of KM in the sequence-tagging and many-label classification tasks, which means that peculiarity selects redundant candidates in these tasks. Intuitively, training a model with only similar examples harms generalization performance because the model is optimized only for limited data. We consider that the redundancy is one of the causes of peculiarity not working well across languages.
Then, we verify why peculiarity works well in the “good” group but not the “poor” group. To do this, we chose an NER task and Finish and Arabic from the “good” and “poor” groups, respectively, and visualize the hidden states of the candidates extracted by KM and peculiarity (n = 1, 000). We compress the hidden states to two dimensions using UMAP (McInnes et al., 2018) and show the results in Figure 3. This figure shows that peculiarity selects local candidates compared to KM in the both languages. For Finnish, these candidates are enough to cover the Finish examples that are not covered by English (source language). However,
for Arabic, the candidates selected by peculiarity cannot complement all of the Arabic examples because almost all of the examples are not covered by English examples. Several studies (Yuan et al., 2020; Hacohen et al., 2022) have suggested that training a model with representative and diverse examples, in other words, examples that approximate the entire data pool, is beneficial to enhance the model’s performance when the training examples are limited. In the case of few-shot cross-lingual transfer, we should fine-tune MS using source and target language examples that approximate an unlabeled data pool in the target language XT . Figure 3 indicates that peculiarity selects ideal candidates in Finish, but not in Arabic. Therefore, it is recommended to select examples by considering diversity for the “poor” group, such as Arabic.
Motivated by the previous analyses, we design a new candidate selection method that considers peculiarity and diversity of candidates. Specifically, first, we extract 50% of candidates X ′′T ⊂ XT using peculiarity. Then, we select candidates X ′T ⊂ X ′′ T by adapting KM to X ′′T . We denote this method as peculiarity-KM and provide evaluation results on NER, POS, and MARC-5 tasks in Table 5. Peculiarity-KM achieves the highest scores in the both “good” and “poor” groups for the NER and POS tasks, which means that we can mitigate the weakness of peculiarity by considering the diversity. However, the scores drop 0.24 points compared to peculiarity in the “good” group of MARC-5. Therefore, there are still challenges to address in the candidate selection method that can efficiently enhance few-shot cross-lingual performance across languages and tasks.
7.2 Robustness of peculiarity
Peculiarity (Equation 1) has two hyperparameters k and distance. Finally, we analyze the robustness of the hyperparameters of peculiarity.
We measure peculiarity using various k (1, 5, 10, 20, 40, 80, and 160) and extract 1,000 candidates based on each peculiarity. Figure 4 shows overlap rates between each subset of extracted candidates in MARC-2 and NER tasks averaged across all languages. We indicate the overlap rates for the other tasks in Appendix C. The overlap rates gradually decrease as the k value changes. However, even the lowest overlap rates (between 1 and 160) are 0.72 and 0.74 in MARC-2 and NER, respectively. Therefore, this figure reveals that peculiarity selects almost same examples regardless of k. We also conduct few-shot cross-lingual transfer with these candidates. In this experiment, we chose German (De) and Chinese (Zh) from the “good” and “poor” groups and use MARC-2 and NER tasks. Figure 5 shows the evaluation results. The figure demonstrates that the accuracy are almost consistent between each k value. This result reveals that peculiarity can lead to robust performance regardless of k.
In previous experiments, we adopted Euclidean distance as distance; however, Cosine distance is another option. We extract 1,000 candidates based on Euclidean and Cosine distances and measure the overlap rates between each subset of extracted candidates. The overlap rates averaged across all languages are 0.93, 0.99, 0.98, 0.97, 0.96, and 0.93 in XNLI, PAWS-X, MARC-2, MARC-5, NER, and POS tasks, respectively. This indicates that peculiarity extract consistent candidates regardless of distance metric.

heading: 8 Conclusion
text: In this study, we proposed a simple metric called peculiarity, which measures whether source language examples exist in the neighborhood of target language examples. We showed that high peculiarity examples are not predicted correctly in the zeroshot setting and demonstrated that these examples can enhance accuracy for few-shot cross-lingual transfer regardless of language groups.
In addition, we showed that peculiarity-KM, the candidate selection method that considers peculiarity and diversity, further boosts few-shot crosslingual transfer performance compared to peculiarity or KM alone. However, there are some configurations in which peculiarity-KM does not work well, such as the “good” group in MARC-5. Therefore, we would like to attempt to analyze the causes of this result and develop a new candidate selection method that enhances the few-shot cross-lingual performance across languages and tasks.
Limitations
Although we demonstrated that the proposed metric peculiarity is useful for selecting candidates for few-shot cross-lingual transfer, our current work has the following limitations.
Lack of evaluations to argue the usefulness of peculiarity. We demonstrated that peculiarity selects candidates to efficiently enhance few-shot cross-lingual performance in several tasks and languages. In addition, peculiarity is robust for hyperparameter k. However, further verification is required to evaluate the usefulness of peculiarity.
In this study, we only used XLM-R as the MMLM in the experiments, because previous works (Lauscher et al., 2020; Kumar et al., 2022) have demonstrated that mBERT and XLM-R show the same trend and XLM-R achieves better zeroshot and few-shot cross-lingual performance. However, it is not obvious that peculiarity will work well in mBERT. In addition, recently, Lin et al. (2022) proposed XGLM, a pre-trained multilingual causal language model, that demonstrates strong multilingual capabilities. We would like to experiment using these pre-trained multilingual models to show the usefulness of peculiarity regardless of models.
We fine-tuned the MMLM using a standard training objective, predicting true labels or tags for inputs. On the contrary, Zhao and Schütze (2021)
revealed that fine-tuning in a prompting format encourages better zero-shot and few-shot crosslingual transfer than the standard fine-tuning. It is worthwhile to examine few-shot cross-lingual transfer performance when fine-tuning the MMLM with high peculiarity examples in a prompting format because it may be possible to achieve higher accuracy in the target languages with a smaller amount of examples.
We experimented using English as the source language. However, if possible, it is better to use a language that is linguistically close to the target language as the source language (Pires et al., 2019; Lauscher et al., 2020; Chai et al., 2022). In our experiments, we did not show that peculiarity works well regardless of source languages. Therefore, verifying this aspect is also a remaining challenge.
Definition of annotation cost. In this study, we defined annotation cost in terms of the number of candidates following previous studies (Pires et al., 2019; Lauscher et al., 2020; Chai et al., 2022). However, a small number of candidates does not necessarily mean less work for annotators. If a candidate (sentence) length is long or hard, it is considered to take longer to understand. On the other hand, if the candidate length is short or easy, annotation time per candidate will be shorter, and the annotators can annotate more candidates in the same time. Therefore, we should evaluate candidate selection methods based on total time required for annotation.
In addition, aligning the cross-lingual representations between source and target languages using bilingual data is one approach to enhance accuracy for the target languages (Lample and Conneau, 2019; Cao et al., 2020; Chi et al., 2021; Dou and Neubig, 2021; Yang et al., 2021). To align the representations, we should create bilingual data through a human or automatic translator. Verification whether labeling or translating is less labor intensive and further boosting performance is one of the future goals.
Developing a better peculiarity-based candidate selection method. In this study, we used the BOS hidden states to measure peculiarity; in other words, it measures example-level peculiarity. In classification tasks, using example-level peculiarity to select candidates is intuitive because we predict labels based on the BOS hidden states. On the other hand, in the sequence-tagging tasks, we
predict token tags based on hidden states of each token. In addition, we consider that it is necessary to fine-tune MS with peculiar tokens, tokens that are not covered by the source language, to ensure that the model predicts tags of these tokens correctly. Therefore, we will attempt to select candidates that contain peculiar tokens by using token-level peculiarity and conduct few-shot cross-lingual transfer in the sequence-tagging tasks.
We observed that peculiarity selects more redundant candidates compared to the KM-based methods and argued that this aspect is the reason that peculiarity does not work in the “poor” group. We consider the possibility of other reasons for this behavior. Several studies (Swayamdipta et al., 2020; Sorscher et al., 2022; Hacohen et al., 2022) have suggested that if only a small amount of examples can be used for training, it is important to use not only hard (atypical) examples but also some easy (typical) examples for training in order to improve model performance. In terms of few-shot crosslingual transfer using peculiarity, we should finetune MS with the both highest and lowest peculiarity examples. In addition, using typical examples selected by KM instead of the lowest peculiarity examples is one of the approaches. For future work, we would like to verify the effectiveness of these methods for few-shot cross-lingual transfer.
Ethics Statement
Impact of our work. Thanks to the efforts of various researchers, pre-trained models have been proposed that can solve NLP tasks with high accuracy. However, labeled data is essential for fine-tuning these models, and few languages have abundant language resources like English. In addition, construction of labeled data is not easy. Therefore, attempts to train high-quality models with little effort, as in our study, are very important for low-resource languages. Although our method and study have the limitations mentioned above, our experiments provided useful insights into selecting annotation candidates for few-shot cross-lingual transfer. In addition, we will publish the code used in our experiments, which will facilitate the reproduction of our experiments and contribute to further research.
Potential risks for bias. In recent years, bias in data has become an issue. Training a model on such data can lead to unwarranted predictions or generate negative sentences for a particular person or group. When the training data is small and
contains biases, such problems may be more pronounced because the model is optimized only for the provided data. In this study, we did not take into account this issue, and our proposed method is not designed to select bias-less candidates. Therefore, when using the proposed method, sufficient attention should be paid to the problem of bias.

heading: Acknowledgments
text: This work was supported by TMU research fund for young scientists and JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2139.

heading: A Experimental Settings
text: We experiment with WikiANN12, UD13, XNLI14, PAWS-X15, and MARC (MARC-2 and MARC5)16 datasets. Example numbers of train, dev, and test data are the same across all languages in the XNLI (392,702, 2,418, and 5,010), PAWSX (49,401, 2,000, and 2,000), MARC-2 (160,000, 4,000, and 4,000), and MARC-5 (200,000, 5,000, and 5,000) datasets. We show example numbers for the other datasets in Table 6.

heading: B Zero-shot Cross-lingual Transfer
text: We show the accuracy of the test and validation datasets for each target language in zero-shot crosslingual transfer in Tables 7, 8, 9, 10, and 11. Low and High are subsets of extracted 10% examples of the lowest and highest peculiarity from the validation dataset. These scores are the average accuracy of three models. The accuracies of High are obviously lower than others in all languages and tasks. Therefore, these tables indicate that peculiarity can isolate the languages depending on their zero-shot cross-lingual performance. We also indicate the median (Mdn.) and macro-average (Avg.) scores
12 https://huggingface.co/datasets/wikiann 13 https://huggingface.co/datasets/universal_
dependencies 14 https://github.com/facebookresearch/XNLI
15 https://huggingface.co/datasets/paws-x 16 https://huggingface.co/datasets/amazon_
reviews_multi
(excluding the English score). We underline the scores higher than the median score in the validation dataset, and we use those languages as “good” group and the others as “poor” group.

heading: C Few-shot Cross-lingual Transfer
text: We show the accuracy of the test datasets for each target language in few-shot cross-lingual transfer in Tables 12, 13, 14, 15, and 16. We perform pairwise t-Test to measure statistical significance. The † (or ‡) indicates the statistical significance (p < 0.1)17 between peculiarity (or peculiarity-KM) and the underlined method that achieves the highest average score except for the peculiarity-based methods. In addition, the ⋆ indicates the statistical significance between peculiarity and peculiarity-KM.
In PAWS-X, MARC-2, and MARC-5 (Tables 15 and 16) tasks, the peculiarity-based methods consistently achieves the highest accuracy. In NER, POS, and XNLI tasks (Tables 12, 13, and 14), the peculiarity-based methods bring the best performances in almost all languages. We also show differences in accuracy between 0-shot and each nshot model averaged across all languages in Table 17.
Finally, we show overlap rates of examples extracted by peculiarity measured at each k and observe that they are higher than 0.70. Therefore, peculiarity is robust for hyperparameter k.
17We set p value following Kumar et al. (2022).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section named Limitations
3 A2. Did you discuss any potential risks of your work? Section named Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sections 1 and 8
3 A4. Have you used AI writing assistants when working on this paper? We use DeepL and Grammarly in all sections.
B 3 Did you use or create scientific artifacts? Sections 5, 6, and 7
3 B1. Did you cite the creators of artifacts you used? Section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendices A and B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Sections 6 and 7
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6 and 7
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Verification whether labeling or translating is less labor intensive and further boosting performance is one of the future goals. Developing a better peculiarity-based candidate selection method. In this study, we used the BOS hidden states to measure peculiarity; in other words, it measures example-level peculiarity. In classification tasks, using example-level peculiarity to select candidates is intuitive because we predict labels based on the BOS hidden states. On the other hand, in the sequence-tagging tasks, we
predict token tags based on hidden states of each token. In addition, we consider that it is necessary to fine-tune MS with peculiar tokens, tokens that are not covered by the source language, to ensure that the model predicts tags of these tokens correctly. Therefore, we will attempt to select candidates that contain peculiar tokens by using token-level peculiarity and conduct few-shot cross-lingual transfer in the sequence-tagging tasks. We observed that peculiarity selects more redundant candidates compared to the KM-based methods and argued that this aspect is the reason that peculiarity does not work in the “poor” group. We consider the possibility of other reasons for this behavior. Several studies (Swayamdipta et al. , 2020; Sorscher et al. , 2022; Hacohen et al. , 2022) have suggested that if only a small amount of examples can be used for training, it is important to use not only hard (atypical) examples but also some easy (typical) examples for training in order to improve model performance. In terms of few-shot crosslingual transfer using peculiarity, we should finetune MS with the both highest and lowest peculiarity examples. In addition, using typical examples selected by KM instead of the lowest peculiarity examples is one of the approaches. For future work, we would like to verify the effectiveness of these methods for few-shot cross-lingual transfer