SlowGuess/NAACL_2022_OCR huggingface dataset:
Shanu Kumar<sup>1</sup> Sandipan Dandapat<sup>1</sup> Monjit Choudhury<sup>2</sup>
<sup>1</sup> Microsoft R&D, Hyderabad, India
2 Microsoft Research, India
{shankum,sadandap,monojitc}@microsoft.com
Few-shot transfer often shows substantial gain over zero-shot transfer (Lauscher et al., 2020), which is a practically useful trade-off between fully supervised and unsupervised learning approaches for multilingual pretrained model-based systems. This paper explores various strategies for selecting data for annotation that can result in a better few-shot transfer. The proposed approaches rely on multiple measures such as data entropy using  $n$ -gram language model, predictive entropy, and gradient embedding. We propose a loss embedding method for sequence labeling tasks, which induces diversity and uncertainty sampling similar to gradient embedding. The proposed data selection strategies are evaluated and compared for POS tagging, NER, and NLI tasks for up to 20 languages. Our experiments show that the gradient and loss embedding-based strategies consistently outperform random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. Furthermore, the proposed method shows similar trends in improvement even when the model is fine-tuned using a lower proportion of the original task-specific labeled training data for zero-shot transfer.
Language resource distribution, for both labeled and unlabeled data, across the world's languages is extremely skewed, with more than  $95\%$  of the languages having hardly any task-specific labeled data (Joshi et al., 2020). Therefore, cross-lingual zero-shot transfer using pretrained deep multilingual language models has received significant attention from the NLP community. During cross-lingual zero-shot transfer, first a multilingual model (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Xue et al., 2020; Ouyang et al., 2020) is created using only unlabeled data from a large number of languages
(typically in the range of 100) with some self-supervised learning objectives. These pretrained models are then fine-tuned with task-specific labeled data from one or more languages (we refer to these as the pivot languages) and tested on all the other languages (here referred to as the target languages) for which no annotated data was used during fine-tuning.
Many recent work (Pires et al., 2019; Karthikeyan et al., 2019; Wu and Dredze, 2019; Artetxe et al., 2020; Lauscher et al., 2020) have studied the efficacy of zero-shot cross-lingual transfer across languages and factors influencing it. Other work have shown that a few-shot transfer, where very little labeled data in the target language is also used during fine-tuning, can result in substantial gains over the zero-shot transfer. For instance, Lauscher et al. (2020) show that zero-shot transfer does not hold much promise for transfer across typologically different languages or when there is not enough unlabeled data in the target language during model pretraining. In such cases, the gap in the cross-lingual transfer can be effectively reduced by fine-tuning it on a little annotated data in the target language. However, very few languages have readily available annotated resources for different NLP tasks, and collecting annotated data for a large set of target languages can be expensive and time-consuming (Dandapat et al., 2009; Sabou et al., 2012; Fort, 2016). Therefore, it is essential to carefully select and annotate target language data for a few-shot transfer, reducing the transfer gap effectively.
Training data selection has been investigated for several NLP tasks, especially for domain adaptation (Blitzer et al., 2007; Søgaard, 2011; Liu et al., 2019). The majority of these approaches use different techniques to rank the entire data and use top  $n$  data points to train the system (Moore and Lewis, 2010). In addition, active learning (Fu et al., 2013; Settles and Craven, 2008) has been widely used to
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1042 - 1055 July 10-15, 2022 ©2022 Association for Computational Linguistics
improve annotation efficiently by using model predictions to select informative data. Active learning is generally used in an iterative setting, in which a model is learned at each iteration, and samples are selected for labeling to improve performance. However, in this paper, we are trying to select a few samples. Hence we are limiting the training to one iteration. In the past, Chaudhary et al. (2019) have used active learning to annotate only uncertain entity spans for Dutch and Hindi languages. However, to the best of our knowledge, none of these approaches have been studied for a large set of languages in a cross-lingual few-shot transfer setting.
The central goal of this work is to propose specific strategies for data selection (and subsequent annotation) for few-shot learning so that the performance in a target language is maximized, given a data budget. The main contributions of this work are: [1] We propose different data selection strategies based on the notions of cross-entropy, predictive entropy, gradient embedding and loss embedding, and perform various reliability analyses of these strategies. [2] We conduct experiments on a set of 20 typologically diverse languages including some syntactically divergent from the pivot language - English & Chinese. [3] We propose a loss embedding-based method for sequence labeling tasks which incorporates both diversity and uncertainty sampling. [4] Through experiments on three NLP tasks, we show that embedding-based strategies perform consistently better than random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. We also observe several language and data-size dependent trends in the performance across different data selection strategies. [5] Finally, we provide a concrete set of recommendations for data selection based on features such as zero-shot performance and the amount of unlabeled data available for a target-language.
The rest of the paper is organized as follows. The next section introduces the novel data sampling strategies. Section 3, 4, and 5 present the experimental setup, results and related research in the area, respectively. Concluding remarks are made in Section 6.
2 Data Sampling Strategies
Assuming we have a pre-trained multilingual language model and enough labelled data in a particu
lar language such as English (EN) for fine-tuning on a task. We can measure the zero-shot performance on a set of target languages. We observe that zero-shot performances are not uniform and often vary with the tYpological similarity between the target and pivot language as stated by (Pires et al., 2019; Lauscher et al., 2020). Nevertheless, all the target languages show a drop in zero-shot performance compared to the performance achieved in the pivot language. Hence, there is a cross-lingual transfer gap for all the target languages. This gap can be attributed to the inherent linguistic property of the target languages however, Lauscher et al. (2020) have shown that the cross-lingual transfer gap can be reduced by fine-tuning on a little annotated data in a target language.
Consequently, few-shot performance can reduce the transfer gap for all the languages. Given a fixed budget, let say  $k$  examples, we want to maximize the few-shot performance in a target language by carefully choosing the effective  $k$  examples. To this end, we are proposing several data selection strategies in this section. We compare them with random sampling where  $k$  target language examples are randomly selected from task-specific fine-tuning data collection. Note that the sampling strategies are oblivious to the actual labels of the data points, as annotation would follow the data selection step in practice.
2.1 Data Cross-Entropy (DCE)
Cross-entropy (Moore and Lewis, 2010; Axelrod et al., 2011; Dara et al., 2014) has been widely used for domain adaptation by selecting in-domain data from a large non-domain-specific (contains both in- and out-domain data) corpus. In our scenario, like Dara et al. (2014), the target language labeled data acts as the large non-domain-specific corpus and using cross-entropy, novel and diverse data is selected from it. Assuming that there is little overlap between the tokens of the pivot and target language during the zero-shot cross-lingual transfer, we presume that no in-domain labeled data for a particular target language is available initially. We further assume that we have access to a non-domain-specific collection of data points, the entire target language unlabeled corpus (may or may not be distinct from the pretraining corpus).
First, two N-gram language models  $\mathbf{M_I}$  and  $\mathbf{M_O}$  are trained on the sentences selected  $L_{I}$  (initially an empty set) and sentences left in the target lan
guage corpus  $L_O$  (staring with the entire corpus), respectively. We use SRILM $^1$  (Stolcke, 2002) to build the N-gram (for  $\mathbf{N} = 3$ ) language models. We do not want to select sentences which are similar to already picked  $L_I$ ; hence we measure data cross entropy (DCE) and select sentences from  $L_O$  that have high entropy with respect to  $L_I$  and low entropy with respect to  $L_O$ .
$$
H _ {I} (x) = H \left(\mathbf {M} _ {\mathbf {I}} (x)\right) \tag {1}
$$
$$
H _ {O} (x) = H (\mathbf {M} _ {\mathbf {O}} (x)) \tag {2}
$$
$$
\mathrm {D C E} (x) = \frac {H _ {O} (x)}{\sum_ {s \in L _ {O}} H _ {O} (s)} - \frac {H _ {I} (x)}{\sum_ {s \in L _ {O}} H _ {I} (s)} \tag {3}
$$
where  $H(x)$  is the measure entropy of a sentence  $x$  using a N-gram language model. The size of  $L_{O}$  and  $L_{I}$  will vary across the iterations, therefore we appropriately normalize the entropy  $H_{I}$  and  $H_{O}$  for calculating cross-entropy.
Algorithm 1 describes the data selection method using data cross-entropy, where  $g$  is the number of data points to be selected in one iteration, and  $k$  is the total number of sentences to be selected. The overall time complexity of this method is  $\mathcal{O}(nk / g)$ , where  $n = |D^t|$ . For reducing the computation time, we can increase  $g$ , which we set to 10 in our experiments.
2.2 Predictive Entropy (PE)
We employ predictive entropy to measure the task-specific knowledge of a fine-tuned model. For a sequence labelling task, we define the predictive entropy  $E(x_{i})$  of a token  $x_{i}$  of a sentence  $x$  given a fine-tuned model  $\mathbf{M}$  as follows:
$$
p \left(y _ {i} \mid x _ {i}\right) = \mathbf {M} \left(x _ {i}\right) \tag {4}
$$
$$
E \left(x _ {i}\right) = - \sum_ {j = 1} ^ {C} p \left(y _ {i} = c _ {j} \mid x _ {i}\right) * \log p \left(y _ {i} = c _ {j} \mid x _ {i}\right) \tag {5}
$$
where  $c_{1}, c_{2}, \ldots, c_{C}$  are the class labels.
We define the predictive entropy of the sentence using the equation (6):
$$
\mathrm {P E} (x) = \frac {1}{N _ {x}} \sum_ {i = 1} ^ {N _ {x}} E \left(x _ {i}\right) \tag {6}
$$
where  $N_{x}$  is the number of tokens in sentence  $x$ . For classification tasks,  $N_{x}$  will be 1.
To define the scoring function for data selection using predictive entropy that can generalize to the corpus with different domain-shift, we use the statistics of the predictive entropy from the entire target language corpus. We use  $\mu_{PE}$  mean and  $\sigma_{PE}$  standard deviation of the predictive entropy of all the sentences in the corpus. Selecting sentences with very low predictive entropy will not help improve the performance as they have less novel information to enhance the knowledge of the model. Furthermore, picking sentences with very high predictive entropy can be harmful to training. It can be high due to either noise or out-of-domain data. As we want to select very few data instances for few-shot learning and improve further upon the zero-shot performance, we consider selection around  $\mu_{PE}$ , the mean of the predictive entropy. But if the zero-shot performance is excellent, then  $\mu_{PE}$  will be very low, and selecting data closer to mean may not improve over the zero-shot performance. Therefore, we add  $\sigma_{PE}$ . We formally define the scoring function in equation (7).
$$
\operatorname {s c o r e} _ {P E} (x) = | \mathrm {P E} (x) - (\mu_ {P E} + \lambda * \sigma_ {P E}) | \tag {7}
$$
Here,  $\lambda$  controls the distance of the preferred selection zone from  $\mu_{PE}$ .
2.3 Gradient Embedding (GE)
Most of the data selection strategies use either representative sampling such as DCE or uncertainty
<sup>1</sup>http://www.speech.sri.com/projects/srilm/
Task	Model	EN	AR △	BG △	DE △	EL △	ES △	EU △	FI △	FR △	HE △	HI △	JA △	KO △	RU △	SV △	SW △	TH △	TR △	UR △	VI △	ZH △
POS	B	96.4	-55.4	-11.4	-11.7	-18.9	-13.27	-37.4	-19.8	-	-46.9	-35.2	-53.0	-49.7	-12.2	-7.3	-	-	-26.3	-43.4	-	-38.5
X	97.2	-43.6	-9.7	-9.9	-14.6	-13.1	-27.9	-14.7	-	-44.4	-27.6	-74.3	-46.7	-10.2	-6.3	-	-	-20.4	-37.2	-	-63.9
NER	B	84.2	-45.3	-7.4	-6.7	-12.8	-11.9	-24.3	-7.7	-5.7	-28.6	-20.2	-54.9	-25.2	-21.3	-9.9	-	-83.5	-12.4	-47.9	-11.3	-40.7
X	82.5	-39.6	-5.7	-9.2	-9.8	-8.8	-23.9	-8.7	-5.6	-32.5	-15.3	-60.5	-36.6	-18.7	-13.4	-	-78.1	-8.6	-34.5	-16.4	-53.5
XNLI	B	81.9	-16.7	-13.2	-11.1	-14.8	-7.1	-	-	-7.8	-	-22.1	-	-	-12.9	-	-32.2	-28.8	-20.8	-24.3	-11.7	-13.2
X	84.1	-12.7	-6.3	-8.3	-8.8	-5.7	-	-	-6.5	-	-15.0	-	-	-9.0	-	-20.4	-12.7	-12.0	-18.6	-9.9	-10.6
Table 1: We report the zero-shot cross-lingual transfer performance drops relative to EN for all the languages on POS, NER, and XNLI tasks with mBERT (B) and XLM-R (X). The results are medians over three RAND initialization (seeds). The darkness of the cell indicates the drops in zero-shot performance.
sampling such as PE. Recently, Ash et al. (2019) proposed BADGE that combines both diversity and uncertainty sampling. BADGE uses gradient embedding to capture uncertainty from the model, assuming the norm of the gradients will be smaller if the model is highly certain about its predictions and vice versa. As we don't have access to the ground truth labels, the gradient embedding  $g_{x_i} \in \mathbb{R}^d$  is computed for a input sentence  $x_i$  by taking model's  $(M)$  prediction as the true label  $\hat{y}_i$ .
$$
\hat {y} _ {i} = \arg \max  \mathbf {M} \left(x _ {i}\right) \tag {8}
$$
$$
g _ {x _ {i}} = \frac {\partial}{\partial \theta_ {o u t}} l _ {C E} (\mathbf {M} (x _ {i}), \hat {y} _ {i}) \tag {9}
$$
where  $l_{CE}$  is the cross-entropy loss function,  $\theta_{out} \in \mathbb{R}^d$  refers to the parameters of last layer and  $d$  is the number of parameters. We have used hidden states of the [CLS] token from last layer classification tasks, hence we have computed the gradients with  $\theta_{out}$  as the last layer of the pre-trained models.
BADGE selects samples by applying  $k$ -MEANS++ (Arthur and Vassilvitskii, 2006) clustering on the gradient embedding. The selection is made on the assumption that examples with gradient embedding of small magnitude will tend to cluster together and not be selected repeatedly.  $k$ -MEANS++ tend to select samples that are diverse and highly uncertain. For simplicity, we will call BADGE method as GE.
As we want to select very few data instances for few-shot learning and improve further upon the zero-shot performance, we consider applying GE selection on examples satisfying the following criteria:
$$
\operatorname {G E} (\lambda) = \operatorname {G E} \left(\left\{x: g _ {x} > \mu_ {g} + \lambda * \sigma_ {g} \right\}\right) \tag {10}
$$
where  $\mu_{g}$  and  $\sigma_{g}$  are the mean and standard deviation of magnitude of the gradient embedding of all the examples in the corpus.  $\lambda$  controls the final value of the selection criteria.
We noticed that in certain cases selecting samples sharing similar context but having different
true labels may be more helpful for few-shot learning. To incorporate this, we propose  $\mathbf{GE}(\gamma)$ , which adds  $\gamma$  similar examples for each  $k$  sample selected using the GE method. As gradient embedding loses information about the sentence, we use Multilingual Sentence XLM-R (Reimers and Gurevych, 2020) for calculating similarity based on sentences. We do not apply any constraints to ensure similar examples have different true labels but the gradient embedding can be used for ensuring it.
2.4 Loss Embedding (LE)
Sequence labelling tasks require prediction over all the tokens of a sentence, and therefore we have to calculate the gradient embedding for each token classification. Considering the maximum number of allowed tokens in a sentence to be  $m$ , the resulting gradient embedding  $g_{x_i}$  will of dimension  $d \times m$ . Due to its high dimensionality, applying  $k$ -MEANS++ will be expensive. We solve this dimensionality issue by proposing the Loss Embedding method, which has a dimension of  $m$ , considering  $lm$  is usually less than  $d$ .
Instead of calculating gradient, we consider only using classification loss at each token. For a sentence  $x_{i}$ , we compute loss embedding  $l_{x_i} \in \mathbb{R}^m$  by computing cross-entropy loss for each token by taking the model's prediction as actual labels. As the norm of loss embedding will be smaller if the model is highly certain about its predictions and vice-versa, it satisfies the primary assumption of BADGE method. Another property preferable for sequential tasks is that the sentences with similar syntax will have a similar structure in the loss embedding as it depends upon the position of tokens in a sentence. Therefore applying  $k$ -MEANS++ clustering on the loss embedding will induce both diversity and uncertainty sampling.
Similar to GE, we also experiment with selection of examples satisfying the following criteria:
$$
\operatorname {L E} (\lambda) = \operatorname {L E} \left(\left\{x: l _ {x} > \mu_ {l} + \lambda * \sigma_ {l} \right\}\right) \tag {11}
$$
where  $\mu_{l}$  and  $\sigma_{l}$  are the mean and standard deviation of magnitude of the loss embedding of all the examples in the corpus.
We conduct various experiments to evaluate effectiveness of our proposed data sampling techniques in a few-shot transfer setting with up to 20 languages from various language families on two different sequential tasks and one classification task.
We evaluate our methods on three benchmarks datasets on POS-tagging, NER, and NLI. The complete statistics of training and test data available in each language is provided in Appendix A.
Named Entity Recognition (NER). We perform NER experiments using NER Wikiann dataset (Rahimi et al., 2019) on 20 languages. We also remove duplicates data points from the training corpus as these will hinder data selection.
Part-of-speech Tagging (POS). We perform POS experiments using Universal Dependency treebanks (Nivre et al., 2016) on the same set of languages of NER except French (FR), Thai (TH), and Vietnamese (VI) due to unavailability of substantial amount of training data after removing duplicates.
Cross-lingual Natural Language Inference (XNLI). The XNLI dataset (Conneau et al., 2018) consists of translated train, dev and test sets in 14 languages of English hypothesis-premise pairs.
We conduct all our experiment using the 12 layer multilingual mBERT Base cased (Devlin et al., 2019) and XLM-R (Conneau et al., 2020). We use the standard fine-tuning technique as described in (Devlin et al., 2019; Pires et al., 2019) for all the experiments. We limit the sentence length to 128 subword tokens and set the batch size as 32. Following (Lauscher et al., 2020), we fix the number of training epochs to 20 and the learning rate as  $2.10^{-5}$  for NER and POS. For XNLI, we set the training epochs to 3 for zero-shot and 1 for few-shot training, and learning rate as  $3.10^{-5}$ . We report  $F_{1}$ -score for NER and POS, and accuracy for XNLI. All the reported results are medians over three random initializations (seeds).
3.3 Zero-Shot Transfer
Throughout our experiments, we assume EN as the pivot language. We report the zero-shot cross-lingual transfer results in Table 1. We observe similar trends in zero-shot performance as reported in (Lauscher et al., 2020), where there are significant drops in performance for TH, JA, AR, ZH, UR, KO, VI. In TH, we observe the highest transfer gap with nearly  $0\mathrm{F}_{1}$ -score, which indicates no cross-lingual transfer has happened.
3.4 Few-Shot Transfer
We add  $k$  additional examples from a target language and report the improvement of few-shot performance over the zero-shot performance reported in Section 3.3, where  $k$  examples are chosen according to the proposed strategies in Section 3, namely random sampling (RAND), DCE, PE, GE, and LE. We use similar training and evaluation setups for the few-shot transfer experiments as we used in the zero-shot setting and repeat the experiments with three random seeds. We consider three RAND baselines and report the average for all the data selection experiments.
We calculated the difference between the  $\mathrm{F_1}$  scores of few-shot and zero-shot setups,  $deltas(\triangle)$ , for each language separately, but we observed different sampling strategies to work better depending upon the cross-lingual transfer gap. Therefore, we present the experimental results after categorizing languages by the transfer gap as indicated by the zero-shot performance, shown in Table 1. We categorize the languages in three groups:  $C_1$ ,  $C_2$  and  $C_3$ , and are coloured as light grey, dark grey and very dark grey respectively in Table 1. For NER task, groups are defined as  $C_1 \in \{\mathrm{BG}, \mathrm{DE}, \mathrm{EL}, \mathrm{ES}, \mathrm{EU}, \mathrm{FI}, \mathrm{FR}, \mathrm{HI}, \mathrm{RU}, \mathrm{SV}, \mathrm{TR}, \mathrm{VI}\}$ ,  $C_2 \in \{\mathrm{AR}, \mathrm{HE}, \mathrm{JA}, \mathrm{KO}, \mathrm{UR}, \mathrm{ZH}\}$ , and  $C_3 \in \{\mathrm{TH}\}$ . For POS, groups are defined as  $C_1 \in \{\mathrm{BG}, \mathrm{DE}, \mathrm{ES}, \mathrm{EL}, \mathrm{FI}, \mathrm{RU}, \mathrm{SV}, \mathrm{TR}\}$ , and  $C_2 \in \{\mathrm{AR}, \mathrm{EU}, \mathrm{HE}, \mathrm{HI}, \mathrm{JA}, \mathrm{KO}, \mathrm{JA}, \mathrm{UR}, \mathrm{ZH}\}$ . For XNLI, the groups are different for XLM-R and mBERT, hence we have mentioned them in the Appendix.
We report the deltas, for NER and POS tasks in Table 2 and 3, respectively. The reported deltas are averaged across all the target languages for each language group. All the reported values are positive, which means in all cases, performance for the few-shot is higher than that for the zero-shot. The proposed methods require two parameters  $\lambda$
Method	k=10	k=50	k=100	k=500	k=1000
△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3
mBERT	RAND	2.9	9.9	0.5	6.4	15.8	1.3	7.7	17.4	1.3	12.1	26.9	18.6	14.0	30.4	31.2
DCE	1.8	8.4	4.0	5.1	12.8	2.8	5.2	11.8	3.3	9.9	23.0	18.8	12.3	28.5	29.2
PE (λ=1)	3.1	10.0	5.7	6.8	14.5	4.7	7.5	16.1	3.9	12.4	24.5	19.8	14.2	27.4	27.4
LE	5.5	11.3	2.5	7.4	18.4	1.1	8.9	18.9	0.7	13.0	27.6	18.9	14.9	30.6	31.0
LE (λ=0)	5.6	11.0	0.7	8.4	18.3	0.1	8.7	18.4	-0.0	12.9	26.9	15.1	14.8	30.0	29.8
XLM-R	RAND	1.4	8.0	0.3	6.7	15.3	0.4	7.8	16.8	1.5	12.8	26.1	20.7	14.6	29.4	27.7
DCE	-3.8	0.5	2.4	3.4	10.0	0.9	4.3	10.5	-0.2	10.5	23.8	19.2	13.2	27.8	26.3
PE (λ=1)	4.4	8.6	3.6	7.0	16.5	1.2	7.9	17.8	0.5	12.3	26.1	19.0	14.2	29.9	28.4
LE	3.0	8.1	5.7	7.9	15.7	3.4	9.0	16.7	2.4	13.0	26.1	18.2	14.5	29.1	23.4
LE (λ=0)	2.4	8.2	1.4	7.4	16.0	5.0	8.5	16.9	4.0	13.0	26.1	16.5	14.5	29.3	23.2
Table 2: Few-shot cross-lingual transfer performance on NER tasks with varying number of target language examples  $k$  using EN as the pivot language. We have reported the  $\bigtriangleup$  delta between few-shot and zero-shot performance averaged across the languages in each category  ${C}_{1},{C}_{2}$  ,and  ${C}_{3}$  .
Method	k=10	k=50	k=100
△C1	△C2	△C1	△C2	△C1	△C2
mBERT	Rand	4.1	22.6	6.7	27.5	7.3	28.0
DCE	2.3	18.7	5.2	24.3	6.0	25.9
PE (λ=1)	4.4	23.4	7.0	27.8	7.4	28.1
LE	3.9	20.1	6.3	26.3	7.1	26.9
LE (λ=0)	4.5	21.9	6.8	27.3	7.5	27.9
LE (λ=0.5)	4.1	23.3	6.8	28.2	7.5	28.6
XLM-R	RAND	3.1	24.6	5.2	28.5	5.6	28.8
DCE	1.8	22.1	3.7	26.0	4.5	27.2
PE (λ=1)	3.4	24.7	5.6	29.1	6.1	29.2
LE	2.9	22.1	5.5	27.6	6.2	28.5
LE (λ=0)	3.1	24.9	5.6	28.6	6.1	28.8
LE (λ=0.5)	3.5	25.2	5.8	29.2	6.4	29.2