SlowGuess/NAACL_2022_OCR huggingface dataset:
Haoran Xu, Kenton Murray
Johns Hopkins University {hxu64, kenton} @ jhu.edu
The current state-of-the-art for few-shot cross-lingual transfer learning first trains on abundant labeled data in the source language and then fine-tunes with a few examples on the target language, termed target-adapting. Though this has been demonstrated to work on a variety of tasks, in this paper we show some deficiencies of this approach and propose a one-step mixed training method that trains on both source and target data with stochastic gradient surgery, a novel gradient-level optimization. Unlike the previous studies that focus on one language at a time when target-adapting, we use one model to handle all target languages simultaneously to avoid excessively language-specific models. Moreover, we discuss the unreality of utilizing large target development sets for model selection in previous literature. We further show that our method is both development-free for target languages, and is also able to escape from overfitting issues. We conduct a large-scale experiment on 4 diverse NLP tasks across up to 48 languages. Our proposed method achieves state-of-the-art performance on all tasks and outperforms target-adapting by a large margin<sup>1</sup>, especially for languages that are linguistically distant from the source language, e.g.,  $7.36\%$  F1 absolute gain on average for the NER task, up to  $17.60\%$  on Punjabi.
The cost of linguistic data annotation and a plethora of differences across language resources and structures of natural language processing (NLP) tasks result in the problem that sufficient labeled data is only available for a handful of high-resource languages (Bender, 2011). The lack of data for low-resource languages leads to the need for effective cross-lingual transfer learning, which aims to leverage abundant labeled high-resource languages to improve model performance on low-resource
ones. The majority of methods for cross-lingual transfer are mainly based on multilingual language models (LMs) (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021) which are pre-trained on massive multilingual data. Zero-shot cross-lingual transfer is widely explored where a multilingual LM is trained on a large amount of labeled data in the source language without any target data, and then is directly evaluated on the target test set, frequently achieving surprisingly good performance (Wu and Dredze, 2019; Pires et al., 2019; Conneau et al., 2020). Recently, Lauscher et al. (2020) emphasize the effective mechanism of few-shot cross-lingual transfer for improving target-language performance, where only a few (such as 10) extra target examples can obtain substantial improvements. The current state-of-the-art methods for few-shot cross-lingual transfer learning (Lauscher et al., 2020; Hedderich et al., 2020; Maurya et al., 2021; Zhao et al., 2021) utilize the source-trained model (the same model training on the source data in zero-shot learning) to fine-tune on small target examples, which is termed target-adapting.
In this paper, we dissect the potential weaknesses of the ubiquitous target-adapting method and propose a one-step mixed training method that trains on both source and target data with a novel gradient-level optimization, stochastic gradient surgery. Specifically, we highlight 6 benefits (contributions) of our method in this paper as follows:
(1) State-of-The-Art Performance: Our proposed method achieves significant improvements compared to target-adapting on 4 diverse NLP tasks across up to 48 languages. For instance, averaged over all target languages, we demonstrate an absolute F1 improvement of  $7.36\%$  on NER using 5-shot learning, with our best performance gains on Punjabi where the gap is  $17.60\%$  (Section 4).
(2) One Model for All Languages: The target-adapting step generally focuses on only one target language. With the proposed method, we do not
1Code is available at: https://github.com/felixxu/Mixed-Gradient-Few-Shot.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2043 - 2059 July 10-15, 2022 ©2022 Association for Computational Linguistics
need to fine-tune specialized models for every target language, which is of particular interest when scaling to dozens or even hundreds of languages. We discuss the benefits of mixed training one model on all target languages, even when their number of shots is extremely small (Section 3.2).
(3) Efficient Gradient De-Conflicting and Information De-Dilution: Two issues arise when mixed training uses data from all target languages in addition to the source language — conflicting gradients among languages and target information dilution. Stochastic gradient surgery efficiently de-conflicts gradients and de-dilutes the target information (Section 3.4 and 3.5).
(4) Single Language Friendly: Though our proposed method normally uses information from multiple target languages, in the simplest setting, where we only have a single target language, stochastic gradient surgery trained on source and target still substantially outperforms standard target-adapting. The improvement is especially pronounced for languages linguistically distant from the source language (Section 5.3).
(5) The Same Script Helps: For a specific language, the model is able to use information learned from other languages. In Section 5.4, we show that this gain is most pronounced in languages that use the same script.
(6) Development-Free for Target Languages: Target development (dev) set used by previous studies (Hsu et al., 2019; Zhao et al., 2021) significantly outnumber training examples in few-shot cross-lingual learning, which is not realistic in the true low-resource settings. However, target-adapting can be prone to overfit on small examples without target dev sets. In comparison, our proposed method is development-free for target languages and able to escape overfitting issues (Section 5.5).
2.1 Cross-Lingual Transfer Learning
Cross-lingual transfer learning enables systems to co-learn the meaning of words across languages and facilitates model transfer between languages, particularly from high-resource to low-resource languages (Ruder et al., 2019), even for languages that are linguistically distant (Xu et al., 2021; Yarmohammadi et al., 2021). Language transfer is based on finding a shared cross-lingual space for source and target languages. One of the most common methods is to align the source and target embed
ding spaces, termed cross-lingual word embeddings (CLWEs) (Mikolov et al., 2013; Artetxe et al., 2016; Conneau et al., 2018a; Vulic et al., 2019; Xu and Koehn, 2021a). Recently, multilingual pre-trained encoders have shown stronger effectiveness over CLWEs for cross-lingual transfer in various tasks (Artetxe and Schwenk, 2019; Wu and Dredze, 2019). While some studies utilize static pre-trained encoders for transfer learning (Wang et al., 2019; Xu and Koehn, 2021b), the majority of studies continuously train encoders for cross-lingual transfer (Conneau et al., 2020; Luo et al., 2021; Xue et al., 2021) based on the finding that source and target representations are still aligned after only fine-tuning on the source data (Hsu et al., 2019).
Few-shot learning was firstly investigated in computer vision (Fei-Fei et al., 2006). Currently, the majority of studies for NLP tasks are designed for one single language (usually English), e.g., model agnostic meta-learning (Finn et al., 2017) and prototypical networks (Snell et al., 2017). However, limited few-shot studies are explored in cross-lingual settings. Recent works mainly focus on zero-shot cross-lingual transfer to evaluate the cross-lingual generalization capabilities of multilingual representations, e.g., XTREME (Hu et al., 2020; Ruder et al., 2021) and XGLUE (Liang et al., 2020). Lauscher et al. (2020) further emphasize that additional fine-tuning on a few inexpensive labeled target-language instances is surprisingly effective across broad NLP tasks. Zhao et al. (2021) highlight the sensitivity of the selection of the few examples (shots) and suggest using the same shots for fair comparisons. State-of-the-art methods for few-shot cross-lingual learning follow the source-training + target-adapting paradigm. In this paper, we investigate deficiencies of this approach and propose more effective methods which significantly improve the transfer performance compared to target-adapting.
2.3 Gradient Surgery
Previous works on gradient optimization (Chen et al., 2018; Sener and Koltun, 2018; Yu et al., 2020) have successfully utilized gradient-level techniques to improve the performance of multi-task models. In fact, mixed training multilingual data can be categorized into multi-task learning (Zhang and Yang, 2018) but in a monolithic manner by
using a single language-agnostic objective on the concatenated data from all languages. Recently, multilingual machine translation utilizes gradient-level regularization to improve the translation performance (Wang et al., 2020; Yang et al., 2021; Wang et al., 2021b). In this paper, our experiments mainly focus on training multiple target languages, so we propose stochastic gradient surgery (Section 3.5) which improves upon the original gradient surgery method (Yu et al., 2020) to improve the overall performance.
3.1 Ordinary Few-Shot Learning
The current state-of-the-art few-shot cross-lingual transfer learning method (Lauscher et al., 2020; Hedderich et al., 2020; Zhao et al., 2021) includes two stages, source-training and target-adapting. In the source-training stage, a pre-trained LM such as mBERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020) is fine-tuned with sufficient labeled data in the source language (which is usually English). In the target-adapting stage, the source-trained model is then fine-tuned only with a few examples in the target language. We abbreviate the name of this method to ord-FS.
3.2 Mixed Fine-Tuning on All Target Languages
The ord-FS method brings up a question: is it necessary to fine-tune a language-specific model for each target language? Can we use one model to handle all target languages to avoid excessively language-specific models? One straightforward method to have such a model is fine-tuning the source-trained model on concatenated examples of all target languages, instead of only one target. Here, we are interested in whether more few examples of other target languages will improve/degrade the overall performance. We abbreviate the name of this method to mix-FT.
3.3 Mixed Training on Source and Target Languages
Ord-FS and mix-FT follow the transductive transfer $^2$  learning method that first trains on the source domain and then fine-tunes on the target domain (Pan and Yang, 2009). However, recently, Xu et al.
(2021) show that abruptly shifting the source domain to the target domain is not an optimized solution due to catastrophic forgetting (McCloskey and Cohen, 1989). Thus, we should be careful about the language domain gaps between the source and target languages, especially for distant languages. One naive but effective approach to preserve the source knowledge and escape catastrophic forgetting is simply training both the source and target data<sup>3</sup> (all target languages), where we simplify source-training and target-adapting into only one mixed training step. We abbreviate the name of this method to naive-mix-train.
3.4 Gradient Surgery in Mixed Training
One issue of naive-mix-train is conflicting gradients (Yu et al., 2020) among languages, which makes training more difficult because gradients point away from one another. We define that two gradients are conflicting if they have a negative cosine similarity. Another issue is that the information of the target domain will be diluted due to the overwhelming source data. Specifically, the gradient of source data is much larger in magnitude than the other languages in one batch training due to the small or even no target training instances in this batch. Hence, the source gradients will dominate the average gradient and result in information dilution of the target data and underestimation of the target language performance.
The main idea of using gradient surgery (Yu et al., 2020) to mitigate the two issues above is, in each backpropagation step, projecting the dominant gradient to the normal plane of a target gradient to de-conflict their gradients and 'remind' the model of target instances. Specifically, we denote  $g_{s}$  as the gradient for the source language and  $g_{t}$  as the gradient for the target language. We first compute the cosine similarity between  $g_{s}$  and  $g_{t}$  and judge  $g_{s}$  and  $g_{t}$  are conflicting gradients if their similarity is negative. Next, we project  $g_{s}$  into the normal plane of  $g_{t}$  only if they are conflicting:
$$
g _ {s} ^ {\prime} = g _ {s} - \frac {g _ {s} \cdot g _ {t}}{\| g _ {t} \| ^ {2}} g _ {t} \tag {1}
$$
The modified  $g_{s}^{\prime}$  replace the original dominant source gradient to update the model parameters.
3.5 Stochastic Gradient Surgery
However, target data is usually not guaranteed to exist in the batch due to the small training size.
${}^{3}$  Target data is randomly interpolated in the source data.
2The pre-training (source-training) and the fine-tuning (target-adapting) are the same task.
Even though we assume that we have target data for all target languages in each batch training, we should detect conflicting gradients not just between source and target languages, but also between every target language. However, this is extremely computationally expensive, especially when it comes to large-scale languages for training. Based on this, we propose stochastic gradient surgery approach, composed of two parts, oracle dataset creation and stochastic training.
Oracle Dataset Creation In the case of  $K$ -shot learning, the oracle dataset comprises  $K$  training instances for each target language. To not use any external information, the oracle datasets of target languages are exactly the  $K$  target instances used in mixed training. Similar to Wang et al. (2020, 2021a); Yang et al. (2021), we create an oracle dataset to ensure that we can pair any one of the target languages with the source language to operate gradient surgery.
Stochastic Training In each batch training, we randomly pick oracle data of a random target language in a uniform distribution to conduct gradient surgery with the source batch data. Moreover, in order to avoid that small number of target examples constrain the source gradients into a sub-optimal place (especially for tasks which need higher-level semantic understanding), we also have a pre-set threshold  $\alpha$  to control the probability of gradient surgery in each training step. The gradient surgery is conducted only if a sampled value  $p \sim \mathrm{uniform}[0,1]$  is smaller than  $\alpha$ .
The advantages of this method are that 1) we only focus on gradient de-conflicting between the source and one of the target languages, which only computes the gradient one additional time to avoid expensive computation, 2) and more importantly, the source language could be a pivot language which also helps gradients of target languages deconflict between each other (more discussion in Section 5.2). The detailed workflow is shown in Algorithm 1. We abbreviate the name of this method to gradient-mix-train.
4.1 Development-Free Training
Importantly, Zhao et al. (2021) notice that few-shot learning easily tends to overfit quickly at a small
Algorithm 1: Stochastic Gradient Surgery
Input: Language Set L; Pre-Trained Model
θ; Mixed Training Data Dtrain;
Oracle Data Dloracle, l ∈ L; Pre-Set Threshold α.
1 Initialize θ0 = θ, step t = 0
2 while not converged do
▷ Iterate batches Btrain from data Dtrain
3 for Btrain in Dtrain do
4 gtrain = ∇θtL(θt, Btrain)
5 Sample a language l from set L
6 goracle = ∇θtL(θt, Dloracle)
7 Sample a value p ~ uniform[0, 1]
▷ Gradient surgery
8 if goracle · gtrain < 0 and p < α then
9 | gtrain = gtrain - ∑goracle||goracle||2goracle
10 end
11 Update t← t+1
12 Update θt with gradient gtrain
13 end
14 end
number of shots, where the model performs best on the dev set at the beginning of training. One good solution to avoid overfitting is using target dev set for early stopping. Previous studies (Hedderich et al., 2020; Zhao et al., 2021) utilize a large amount of dev sets for each target language for model selection, e.g., even around 10K dev examples for Arabic in the NER task. However, it is unlikely that such a dev set would be available in reality, especially for the extreme low-resource training such as 1-shot and 5-shot learning, since it would be more effective to use it for training instead (Kann et al., 2019). The true standard setup of zero-shot cross-lingual learning only uses the source dev set (Zhao et al., 2021), and few-shot learning should also follow this setup, particularly at a small value of shots. Thus, we suggest only using the source dev set for model selection. However, for target-adapting, it does not make sense to use the source dev for model selection due to the different languages in the training and dev steps. Hence, the two-step methods, ord-FS and mix-FT, use the last checkpoint for evaluation. Since naive-mix-train and gradient-mix-train train on both source and target data, they are suitable for using the source dev set for target model selection. We show that our methods substantially outperform target-adapting whatever it uses unrealistic dev sets or
$^{4}$ XNLI use  $K$  examples from every class followed by the "N-way K-shot" discussion in Section 4.3.
not in Section 4.4.
We consider all introduced methods in the experiment, including two-step methods (ord-FS, mix-FT), and one-step methods (naive-mix-train, gradient-mix-train). Moreover, in order to investigate the difference between using and not using dev sets, we add another baseline, ord-FS+dev, whis is ord-FS with unrealistically large dev sets for model selection as Zhao et al. (2021) conduct.
4.2 Tasks and Datasets
We consider two lower-level (structured prediction) tasks, Wikiann Named-Entity Recognition (NER) task (Pan et al., 2017) and Part-of-Speech Tagging (POS) (Nivre et al., 2018) and two different types of higher-level tasks, Typologically Diverse Question Answering-Gold Passage $^6$  (TyDiQA-GoldP) (Clark et al., 2020) and Cross-lingual Natural Language Inference (XNLI) (Conneau et al., 2018b). We download datasets from the XTREME-R benchmark (Hu et al., 2020; Ruder et al., 2021). NER and POS cover 48 and 38 languages, respectively. Our experiments use 35 languages on POS because the remaining three languages, Thai(th), Tagalog(tl) and Yoruba(yo), do not have target training data in XTREME-R. TydiQA and XNLI cover 9 and 15 languages, respectively. We conduct aforementioned methods on all tasks for all languages. English is the source language and the others are targets. Statistics about languages are listed in Appendix B.
Two-step training methods, ord-FS(+dev) and mix-FT, have two different settings for source-training and target-adapting. For one-step methods, naive-mix-train and gradient-mix-train, their settings are the same as source-training in the two-step methods. We run 10 epochs for NER and POS, 60 for TyDiQA, and 10 for XNLI in both source-training and target-adapting. The batch size of all tasks is 32 for source-training and  $K$  for target-adapting with a 2e-5 learning rate. Pre-set threshold  $\alpha$  is 1 for NER and POS and 0.1 for TyDiQA and XNLI unless otherwise noted. The values of  $\alpha$  are empirically selected, which might not be optimal but strongly effective. The model architecture of NER and POS is based on pre-trained XLM- $\mathbf{R}_{\text {large }}$  attached with
a feed-forward token-level classifier. For TydiQA, the representations of all subwords in XLM-Rbase are input to a span classification head — a linear layer computing the start and the end of the answer. For XNLI, the model architecture is XLM-Rbase with a simple softmax classifier on the vector of the start token. The number of examples we consider is  $K \in \{1, 5, 10\}$ . The sampling method is simply extracting random  $K$  shots. The only exception is XNLI, where we adopt the sampling method of conventional few-shot classification learning — "N-way  $K$ -shot" (Fei-Fei et al., 2006) — we sample  $K$  examples for  $N$  classes. Here,  $N$  is the total number of classes in XNLI. We repeat every experiment 5 times with 5 different random seeds<sup>7</sup> suggested by Lauscher et al. (2020). All methods use the same  $K$  shots for a fair comparison. We finally report the average accuracy (XNLI) or F1 scores (other tasks) and their standard deviation.
The main results on each task, conditioned on the number of examples  $K$  and averaged across all languages, are presented in Table 1. The full results with each target language are shown in Appendix C. For all values of  $K$  and all tasks, gradient-mix-train performs the best among all introduced few-shot learning methods.
The zero-shot cross-lingual transfer results  $(K = 0)$  deliver similar results comparable to Ruder et al. (2021). Similar to the findings in Lauscher et al. (2020); Zhao et al. (2021), we notice substantial improvements with ord-FS(+dev) on lower-level tasks (NER and POS) and modest improvement on XNLI over zero-shot performance.
However, ord-FS significantly degrades the zero-shot performance on TyDiQA because it suffers from a tendency of overfitting on target training instances (more discussion in Section 5.5). On the other hand, with the help of dev sets in model selection, ord-FS+dev achieves higher performance than ord-FS on all tasks and particularly solve the overfitting issue.
Compared to ord-FS, NER and TyDiQA benefit most from mix-FT, e.g., from  $65.91\%$  to  $70.60\%$  with  $K = 5$  in NER. However, it still suffers from the overfitting issue, but the impact decreases with more target examples. Training source sentences with target data (naive-mix-train) is a better solution. It consistently outperforms mix-FT on all
<sup>5</sup>Detail information of dev sets are shown in Appendix A. <sup>6</sup>We try to not use translated data such as XQuAD (Artetxe et al., 2020) to avoid unrealistic artifacts such as preserving source words (Clark et al., 2020).
7 Shots are different with different seeds.
K	Methods	NER	POS	TyDiQA	XNLI
Avg. F1 (%)	sd.	Avg. F1 (%)	sd.	Avg. F1 (%)	sd.	Avg. Acc. (%)	sd.
K=0	Zero-Shot	64.56	-	77.32	-	55.80	-	73.55	-
K=1	ord-FS+dev (Zhao et al., 2021)	65.92	0.84	80.37	0.16	55.81	1.01	73.95	0.19
ord-FS (Zhao et al., 2021)	64.11	0.98	80.24	0.19	47.44	1.47	73.70	0.17
mix-FT (Ours)	65.71	0.90	79.37	0.12	48.73	2.15	73.54	0.61
naive-mix-train (Ours)	67.31	0.58	80.04	0.23	57.03	0.56	73.29	0.43
gradient-mix-train (Ours)	69.58	0.99	81.14	0.27	57.64	1.02	74.09	0.54
K=5	ord-FS+dev (Zhao et al., 2021)	68.22	0.69	83.15	0.23	55.60	1.07	74.08	0.36
ord-FS (Zhao et al., 2021)	65.91	0.91	82.95	0.20	51.19	1.29	73.73	0.60
mix-FT (Ours)	70.60	0.85	81.95	0.16	54.49	1.76	73.13	0.74
naive-mix-train (Ours)	72.06	0.68	82.79	0.19	58.59	1.45	73.69	0.80
gradient-mix-train (Ours)	73.27	0.60	83.48	0.24	59.34	1.04	74.41	0.26
K=10	ord-FS+dev (Zhao et al., 2021)	69.85	0.60	84.92	0.07	55.59	1.62	74.19	0.39
ord-FS (Zhao et al., 2021)	68.75	0.67	84.66	0.08	53.17	1.56	74.03	0.38
mix-FT (Ours)	73.89	0.56	83.54	0.07	55.54	1.05	73.62	0.98
naive-mix-train (Ours)	74.13	0.45	84.52	0.17	58.88	1.37	74.23	0.37
gradient-mix-train (Ours)	75.92	0.61	85.03	0.16	59.47	1.73	74.44	0.38