Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages
Authors: Linjuan Wu
Zongyi Guo
Baoliang Cui
Haihong Tang
Weiming Lu
Venue: EMNLP 2023 Findings (2023)
Keywords: Few-shot Cross-lingual Transfer Learning
Low-resource Languages
Model-agnostic Meta-learning

Abstract: Model-agnostic meta-learning has garnered attention as a promising technique for enhancing few-shot cross-lingual transfer learning in low-resource scenarios. However, little attention was paid to the impact of data selection strategies on this cross-lingual meta-transfer method, particularly the sampling of cross-lingual meta-training data (i.e. meta-tasks) at the syntactic level to reduce language gaps. In this paper, we propose a Meta-Task Collector-based Cross-lingual Meta-Transfer framework (MeTaCo-XMT) to adapt different data selection strategies to construct meta-tasks for meta-transfer learning. Syntactic differences have an effect on transfer performance, so we consider a syntactic similarity sampling strategy and propose a syntactic distance metric model consisting of a syntactic encoder block based on the pre-trained model and a distance metric block using Word Move's Distance (WMD). Additionally, we conduct experiments with three different data selection strategies to instantiate our framework and analyze their performance impact. Experimental results on two multilingual NLP datasets, Wikiann and TydiQA, demonstrate the significant superiority of our approach compared to existing strong baselines.

Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages
Linjuan Wu', Zongyi Guo Baoliang Cui Haihong Tang 2.3 Weiming Lub-
'College of Computer Science and Technology, Zhejiang University
A Alibaba International Digital Commerce Group, China
Alibaba Group, China
[wwllinjun525, luwm)@zzj. edu. cn
2,3 (zongyi.gzy, gzy, moqing.com cbl, piaoxue]@al wwweeaalaaabinnnn.oe
Abstract
Model-agnostic meta-learning has garnered at- tention promising technique for enhancing few-shot cross-lingual transfer learning in low- resource scenarios. However, little attention was paid to the impact of data selection strate- gies on this cross-lingual meta-transfer method, particularly the sampling of cross- tross-lingual meta- training data (i.e, meta-tasks) at the syntactic level to reduce language gaps. In this paper, we propose Meta-Task Collector-based Cross- lingual Seta-Transfer framework (MeTaCo- XMT) to adapt different data selection strate- gies to construct meta-tasks for meta-trans learning. Syntactic differences have an effect transfer performance, so we consider syn- tactic similarity sampling strategy and propose a syntactic distance metric model consisting of syntactic encoder block based on the pre- trained model and a distance metric block using Word Move's Distance (WMD). Additionally, we conduct experiments with three different data selection strategies to instantiate our frame- work and analyze their performance impact. Experimental results two multilingual NLP datasets, WikiAnn and TydiQA, demonstrate the significant superiority of approach pared to existing strong baselines!.
65.0
59.1
57.9
X-METRA
58.1
68.1
X-METRA,,
X-METRA
X: X-ME:TX,,,
bm
41.8
42.5
X-METRAwaEH
42.2
53.3
56.8
54.9
Figure I: EM score of 4 languages on the TydiQA dataset based on the X-METRA model. refers results from M''hadd et al., 2021). The subscript imp means the result from our implementation. The seed indicates the random seed number for random sampling.
tasks (Nooralahzadeh et al., 2020; Ponti al., 2021; Liu et al., 2021; M' hamdi et al., 2021). Our focus in this paper is primarily on cross-lingual meta- transfer methods for low-resource languages (Liu et al., 2021; M Thandi al., 2021), utilizing sup- port sets from high-resource languages to establish an effective initialization for training in the low- resource target language.
1 Introduction
Few-shot cross-lingual transfer surpasses zero-shot transfer (Lauscher et al., 2020; Hu et al., 2020; Zhao et al., 2021) using multilingual pre-trained language models (PLMs) (Devlin et al., 2019; Pires et al., 2019; Conneau and Lample, 2019; Conneau al., 2020; Chi al., 2021). It significantly im- proves model performance in the target language with minimal annotation costs. Recent studies have highlighted the benefits of Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) for few-shot cross-lingual transfer learning in NLP
The selection strategies for meta-tasks in cross- lingual meta-transfer learning typically involve ran- dom sampling (Nooralahzadeh et al., 2020; Ponti et al., 2021; Liu et al., 2021) or semantic similar- ity (Wu et al., 2020a; M'hamdi et al., 2021). We compare these two strategies using the X-METRA model (M'hamdi et al., 2021) on four languages from the TydiQA dataset (Clark et al., 2020). Fig- illustrates that models trained with randomly sampled meta-tasks (X-METRAsea)) in different seed setting generally perform a big variance of results compared with X-METRAjmp. Different random seed means the different selection of in-
"Corresponding authors.
'The code is available at httpps//ggithh com/
wul wulinjuan/MeTaCo. XMT

---

stances construct meta-task, and better selection can generate better results. It emphasize the piv- otal role of meta-task construction in enhancing learning.
syntactic level for syntactic similarity sam- pling.
We investigate three different data selection strategies and experiment two cross-lingual datasets (WikiAnn and TydiQA) demon- strate that our framework equipped with syn- tactic similarity sampling strategy signifi- cantly outperforms existing strong baselines.
cross-lingual
Motivated by the above finding, we propose a Meta-Tass Collector- based Cross-lingual Meta- Transfer framework (MeTaCo-XMT)) As shown in Figure 2(a), the meta-task collector includes data encoder and data selector with a distance metric block, which can be designed according different data selection strategies. For instance, multilingual semantic representation and cosine similarity met- rics can realize semantic similarity sampling strat- egy. Specifically, the data encoder encodes all data to semantic space, and the data selector selects top- v cosine-similar support instances for each query data as candidates. Finally, the meta-task can be constructed according the setting,
Related Work
We focus on two threads of related work: (1) meta- learning for cross-lingual transfer and (2) train- ing data selection. Sherborne and Lapata (2023) and Wu et al. (2020b) use meta-learning for cross- lingual NER and Semantic Parsing with slight enhancement in minimal resources. X-MAML (Nooralahzadeh et al., 2020) combines the MAML and cross-lingual transfer method based on PLM and demonstrates improvement in zero-shot and few-shot settings. X-MAML samples the support and query data from the same language, which lim- its the ability of the model for cross-lingual trans- fer. XLA-MAML (Liu et al., 2021) performs direct cross-lingual adaptation in the meta-learning stage by sampling the meta-tasks from two or more lan- guages. X-METRA-ADA (M' MM'hamdi et al., 2021) follows the setting of cross gross-lingual meta-transfer training and adds meta-adaptation stage for fur- ther improvement. While Wu et al. (2020b) and M'hamdi al. (2021) use semantic similarity se- lect the meta-tasks, explore more data selection strategies to get better cross-lingual meta-transfer performance.
However, the semantic similarity sampling is limited for different tasks. For example, in the ma- chine reading comprehension (MRC) task, most semantically similar samples are subject-related, ignoring the relationship between question and paragraph. In practice, different questions with a common paragraph may choose the same sup- port instance. For structural tasks like named entity recognition (NER), syntactic similarity sampling is more effective due srgguage-speciffe syntac- tic differences. Therefore, we propose a Syntactic Distance Metric Model (SDMM) based on multilin- gual PLM and the Word Mover's Distance (WMD) (Kusner et al., 2015). As shown in Figure 2(b), the SDMM incorporates syntactic linear layer for syntactic tree learning and employs triplet loss to distinguish WMD between close and distant languages from pivot languages (c.g., English).
Training data selection has been extensively stud- ied for several tasks, such as domain adaptation (Liu et al., 2019; Ivison et al., 2022) and cross- lingual transfer (Maurya and Desarkar, 2022; Ku- mar et al., 2022). In the cross-lingual transfer set- ting, the selection of training data can reduce the performance gaps across languages. Kumar et al. (2022) proposed approaches of data selection rely on multiple measures such as data entropy using an n-gram language model, predictive entropy, and gradient embedding. Maurya and Desarkar (2022) use meta-learning techniques for cross-lingual gen- cration and choose centroid languages to meta- training the model and improve other languages. In this paper, we propose a cross-lingual meta-transfer framework based on meta-task collector and ex- plore the performance of different data sampling strategies.
In order to further explore the impact of data se- lection strategy on cross-lingual meta-transfer per- formance, we compare three sampling strategies to build meta-tasks including semantic similarity sampling, task sisk-iimilarity sampling, and syntactic similarity sampling. We conduct experiments on 13 typologically diverse target languages of two cross-lingual tasks: NRE and MRC. Our main con- tributions are listed below:
We propose meta-task collector- based cross- lingual meta-transfer framework (McTaCo- XMT) to accommodate different data select strategies reducing the gap of languages.
We propose a syntactic distance metric model to calculate the distance of text pairs at the

---

Target
Prodictoo
English labelled
Task: Task-Speciff Specificc Base PLM
WMD
WMD
Distance
a Layer
Support
Sanoder
S
Syntactic Linear Layer
Target
Dissance Natti Srock
s
labelled
Layer
Multilingua PLM
Query
Syntactic Encoder Block
Cross Coos--inguall Meta-
Collector
(a)
(b)
Figure 2: Diagram of (a) the proposed meta-task collector-base based cross-lingual meta-transfer framework and (b) the syntactic distance metric model trained with triplet loss Luis and syntax tree loss LSTI. The superscript d and e in (h) represent the distant and close language from pivot language (without superscript) respectively.
3 Preliminaries
Methodology
Model Agnostic Meta-Iearning (MAML) (Finn al.. 2017) as an optimization-baser meta- learning method is compatible with any model fo that updates parameters through gradient descent. Formally, given a task тæ from meta-tasks set r with loss function L the optimal target of MAML is:
Figure 2 shows the architecture of our meta-task collector-based cross-lingual meta-transfer frame- work and the proposed syntactic distance metric model. Our framework consists of three essential components: task-specifi base PLM, cross-lingual meta-transfer learner, and meta-task collector. this section, we first introduce the training proce- (1) dure of McTaCo-XMT ($ 4.1) and detailed de- scription of the meta-task collector ($ 4.2). Then syntactic distance metric model ($ 4.3) is proposed to instantiate the meta-task collector and two other data selection strategies described in 4.4.
arg min Σ (fa) TEC
8, = avoLT,
(2)
where the T; (,; Qi). consisting of support set S; and query set Q,. MAML optimizes the parameters toward the optimal target in Equation via an inner step and a meta step. The support set S is used for the inner step (as Equation 2) to update the A to 8' with learning rate a. The meta step uses the updated model for and optimizes 0 with learning rate 3 across query set Q,. as follow:
4.1 Training Procedure of MeTaCo-XMT
As shown in Figure 2(a), for the task-specific base PLM, we first initialize our model fo with multilin- gual PLM such mBERT (Pires et al., 2019) XLM-R (Conneau et al., 2020) and fine-tune it on English monolingual labeled data. This step allows the PLM to take benefit of the high resource data and serve as a baseline model.
0 0
(3)
The optimized-basse meta-learning algorithm can perform the model fast adaptation to a new task during the adaptation training phase.
Before the cross-lingual meta-transfer stage, we sample batch of meta tasks T (S, Q) from dataset ID (IDeres Bgge)) by meta-task collector, which uses high-resource source language (typi- cally English) data support sets and low-resource target language data in the query sets. For every task т; = [S,, Q.), we update f over r steps us- ing support instances in S, as Equation 2. At the end of inner loop, we compute the gradients with respect to the loss of 0f on Qi After each batch training,
Cross-lingual Meta-transfed Learning mainly fo- cuses on the meta-training phase of MAML. In the inner step, the model fo learns good initializa- tion of parameter by repeatedly simulating the learning process source language support set S Dive. The meta step is fine-tuning the initial- ized model in a target language query set Q Digt- These two steps are iterated repeatedly to optimize the parameters 0. We skip the adaptation phase and directly evaluate the target language test dataset.
over all pre-computed gradients and update 0, thus completing one outer loop. At test stage, we directly evaluate the optimized

---

model fe on the target language test dataset.
propose Syntactic Distance Metric Model (SDMM) to select syntactic-similar source in- stances for meta-learning. which consists of syn- tactic encoder block and distance metric block.
4.2 Meta-Tase Collector
The large variations illustrated in Figure demon- strate that meta-task selection has profound im- pact on the performance of cross-lingual meta- transfer learning. So we equip the cross-lingual meta-transfer framework with meta-task collector to select the training data strategically. As shown in Figure 2(a), our meta-task collector is to construct meta-task with q query instances and support instances, described in the following.
As shown in Figure 2(b), We first encode the multilingual text into a universal syntactic space and use Word Mover's Distance (WMD) (Kusner et al., 2015) to measure the syntactic distance of text pairs. To train the model, we optimize triplet loss by using three-way parallel texts (x, x" .r"). The three languages include the pivot language (English), a distant language 1á, and a close lan- guage I' compared to English. The distance be- tween languages is calculated by the lang2vec (Lit- tell et al., 2017), a tool that extracts features of different languages by querying the URIEL typo- logical database². Based on the relationship be- tween language distance and transfer performance in (Ahmad al., 2019), the threshold for language classification is set as 0.52. A language with a dis- tance of more than 0.52 from English a distant language, otherwise, is a close language. Next, we introduce the two blocks and losses.
In this few-shot eross- cross-lingens lingual transfer setting, we have a rich-resource source language (i.e. English) dataset sre = (Erse: Usrc)} } and a low-resource target language dataset Dage = [ Mege)) ) where (ar, v) is a pair of text and ground truth labels y. We form the meta-tasks T (S, Q)
with the following process:
1. Sample num target data as query instances (8)
set Q = (1ige J J E Dige- The entire source dataset is used as candidate support instances S
(i) ) E Dare- (<
4.3.1 Syntactic Encoder Block
2. Encode text æ in query and support instances set by encoder block to obtain vector
Many studies have found that PLMs can encode syntactic structures of sentences (Hewitt and Man- ning (2019); Chi al. (2020)). For learning syn- tactic representation,
3. Calculate the distance dy between iig and Sare by distance metric block (DMB):
design a syntactic encoder block consisting of the multilingual PLM layer and the syntactic linear layer. Specifically, for an in- put text æ = {wi]g the output representations h(x) E Raxb from the frozen multilingual PLM are fed into the syntactic lincar layer (a matrix B E Rlxe). Then h(x) is transformed into univer- sal syntactic space g(x) Bh(x), so the syntactic vectors of ith word uæ can be defined as:
(5)
dy = DMB(8 Egtt (
For each query instance 4 (aager Vegt). choose top-v closest instances as its candidate support subset Sy = (( (1g) ) S.
4. Finally, we draw task 7; = {S, 0.) T by first randomly choosing y query instances. forming Q, For each query instance 9j in 2,, we draw the k/g most closest candidate support instance from Syj thus forming S,, The number of meta-tasks is also hyper- parameter.
8; Bh (w;)
(5)
Inspired by Hewitt and Manning (2019), we adopt the syntactic labels from Universal Depen- dencies (UD²) to learn syntactic embedding with two tasks: depth prediction of word and distance prediction of two words in the parse tree T. The losses of these two subtasks are defined as:
How to adapt the meta-task collector to syntactic similarity sampling is described in detail below.
Ldepth
Σ (|wy) - 1 1xx
(6)
4.3 Syntactic Distance Metric Model
Linguistic disparities affect the performance of cross-lingual transfer (Pires et al., 2019; K al.,
Ldiistonee
Σ dr(w,, wy) - ds (s,, s;)l
(7)
2020; Wu et al., 2022), such as the difference in word order or other syntactic differences. So
hhtp://www.csccmm edu/ dmontens/ppojjccts/7 project httpps//mnivvraadddpeeeeeeeeerrr

---

where |tca| is the parse depth of word defined the number of edges from the root of the parse tree to wi, and |s ll is the tree depth L2 norm of the syntactic vector. dr(w,, wj) is the number of edges in the path between the ith and jth word in
where the represents margin between distant language and a close language from English. Finally, the loss the Syntactic Distance Metric Model (SDMM) is:
- LSTL + LISTI + LiSTI. + Ldis,
(14)
the parse tree. As for d 8y), it can be defined
the squared L2 distance:
where (LSTL, LITT LSTL) are syntactic tree losses of three-way parallel texts.
( 85) (8: F -8))
(8)
4.4 Other Data Selection Strategies
To induce parse trees, we minimize the summa- tion of the above two losses deppth and Idiistacce and define the syntactic tree loss (STL) as:
In this section, we introduce other two data selec- tion strategies to instantiate the meta-task collector. Semantic Similarity Sampling was used in recent works (Wu et al., 2020a; M I'hamdi et al., 2021) to construct meta-tasks. In the experiment, we fol- low (M' (M'hamdi al., 2021) to use the cross-lingual extension to SBERT's pre-trained model (Reimers and Gurevych, 2019, 2020) as encoder block and use cosine similarity algorithm in distance metric block. For NER and MRC, the input is only the text of the task and the concatenation of the question and paragraph, respectively.
LSTL= deppth
Liistance
4.3.2 Distance Metric Block
Syntactic information is at the word level, so introduce Word Mover's Distance (WMD) (Kusner al., 2015) to calculate text syntactic distance incorporating the dissimilarity between word pairs. WMD is the cost of transporting a set of word vectors to the other in an embedding space.
Formally, the three-way parallel texts (z,x", x²) can be encoded into a syntactic space by our syn- tactic encoder to obtain the vectors (s, ,,, s*). The inputs of WMD are probability weight and trans- portation cost function. As shown in the upper of Figure 2(b), the transportation cost function is:
Task-level Similarity Sampling is proposed for tasks with long input text or multiple input texts because semantic similarity sampling ignores task information between texts. We use a pre-traince model fine-tuned on the English dataset as an en- coder, called task-specific pre-trained model. The distance metric block uses the cosine similarity al- gorithm. For MRC or NER, the input is the form of task fine-tuning followed by Devlin et al. (2019).
cd cos(s, s²), of cos(s, (ssssss)) 8")
(10)
where -))(:.) is cosine similarity function. Fol- lowing (Yokoi et al., 2020), we use norm a word vector as the probability weight, and the norm of äth word vector 8; is:
5 Experiments
5.1 Languages and Datasets
a &ill
(11)
We evaluate the performance of our framework on NER and MRC benchmarks from XTREME (Hu et al., 2020) in 13 target languages, including Afrikaans (af), Arabic (ar), Bengali (bn), Finnish (fi), Javanese (jv), Indonesian (id), Korean (ko). Russian (ru). Swahili (sw), Telugu (te). Tagalog (tI),
Therefore, the syntactic distance is defined as:
N Tige
N
dy
diy
C
(12)
Yoruba (yo), and Chinese (zh). Among them, two languages (bn and te) are considered low-resource, while five languages (sw, af, tl, jv, and yo) are classified as extremely low-resource according to the classification method described in Bang et al. (2023). Additional details on language classifica- tion and statisties can be found in Appendix A. For MRC, we use the gold passage version of the TydiQA dataset (TydiQA-GoldP) (Clark et al., 2020). more challenging than XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al.,
where T is a transportation matrix, which is the solution for WMD get the minimum cumulative cost of moving s to target. The transport values Túj or T5 is subjected to the probability weight Oj or aj, respectively. In the experiment, we use EMD (Yu and Herman, 2005) class of the python package evá to solve the T. The triplet loss is:
Lais = max (0, m dy + dly
(13)

---

TydiQA- (EM)
Model
6
xd
kn
bn
te
PRE et al., 2020)
38.8
42.8
45.3
45.8
50.0
32.7
38.4
37,9
41.5
PRT:
39.4
44.8
43.2
48.5
46.6
35.2
38.4
39.9
42.0
X-METRA M'handi et al., 2021)
48.910.4
63.3:088
59. 19.1111
63.2100.5
39.011.
49.7
61.410.4
$5.2
FT
52.0+0.5
62. 1..4
19
65
48.8: 1.0
49.74:09
61.1#3.2
57.6400.
$7.0
IBER
FT wayassemple
51.740.7
62.6:1.11
59.840.4
64.11005
49.5#1.2
4944116
62.9204
60.1+1.5
57.5
52.8+0.9
66.4+144
51.343.7
49.9+1.
62.5+0.9
MeTaCo-XMT.co MeTaCa-XMTioo
51.740.3
62.7:00.3
61.54099
65.71-0.8
50.9
48.1+2244
62.710.3
64.3
58.4
52.74-0.6
63.5-002
61.240.7
66.0+0.9
51.3++13
52.8+112
64.0+0.3
63.4+-..
59.4
McTaCo-XMT..m
$3.0+0.5
63.4:10.3
61.640.7
66.810.4
51.9=0.4
54.3+10.9
64.5=0022
64.3
60.0
PRE (Hu et al., 2020)
42.1
40.4
53.2
61.9
10.9
47.8
43.6
48.1
45.0
PRE
41.9
56.5
64.2
41.4
50.1
54.7
52.7
XLM-RRegg
54.240.9
63.2440.7
64.74-111
60.94288
FT wissn_ssmple
53.610.5
62.9=110
66.011111
70.7110
$2.4
60.422.7
65.3=0.8
66.7:11.2
62.3
XMT
55.14-088
6.65+116
65.2440.88
72.6+00.5
55.8++0.7
66.74330
67.4+008
69.940.22
64.7
Ours
McTaCo-XMT.... Me
55.7+10.9
64,8: 1.4
65. 65.3-:0.6
71.0#0.4
53,9=0.5
66.11:0.6
66,5=0.44
71.7=0022
64.4
56.0:11.0
6.,6:11.2
65.6:10.9
73.0
55.1
67.310.9
67.4=0.9
71.7=113
65.1
MeTaCo-XMTTp
56.0+-03
6..3++1.
66.3+-0.4
72.9+0.4
56.9-0.5
68.400.7
67.5+0.1
71.6+0.4
65.6
Table 1: EM score and standard deviation of target languages and average on the TydiQA-Gold dataset.
2020) as questions have been written without see- ing the answers. The dataset is segmented follow- ing M' d'maadi et al. (2021), with English training data as Train and 10% of training data from other languages as Dev for few-shot or meta-transfer. The provided test sets are used for evaluation. For the NER task, we employ the multilingual WikiAnn dataset (Pan et al., 2017), reserving num 100 instances from the training data of other languages Dev. Appendix B.1 provides detailed dataset statistics.
question, answer) triples. For generality, we only concatenate the paragraph and question.
For WikiAnn, we added the competitive and challenging zero-shot baselines with pseudo- labeled data, including CROP(Yang et al., 2022) and SL LEU(Xu et al., 2021). They leverage trans- lation or self-training methods to obtain pseudo- labeled data for target languages. Further details on these baselines be found in Appendix C.
5.3 Implementation Details
MeTaCo-XMI initialized by PRE following the hyper-parameters settings in XTREME (Hu et al., 2020). The meta-task collector employs English Train data as support sets and target lan- guage Dev data as query sets. We implemented the MAML using the learn2learn library. MeTaCo- XMT syns MeTaCCoXMMT,n and MeTaCo-XMI sem respectively represent the framework of meta-tasks constructed by syntactic similarity sampling, task- similarity sampling, and semantic similarity sam- pling. For each model (except that PRE uses a fixed seed 42), we run 3 random initialization and report the average and standard deviation.
5.2 Baselines
mBERT (Pires et al., 2019) and XLM-Rlarge (Con- neau et al., 2020) are used as the base PLM. We compared our model with the following baselines:
PRE: An initial task-specific base PLM base- line is fine-tuned on the English Train and evaluated other languages Test.
FT: A standard few-shot transfer baseline to fine-tune the PRE on target language Dev.
FT w/syn_sample: We fine-tune the PRE model on Dev split of the target languages and the selected English support dataset by syntactic sampling method in Section 4.3.
For SDMM, we collected a syntax-labeled cor- pus of 7k instances from the UD 2.7 Treebank (Ze- man et al., 2020), covering 7 distant languages and languages close to English (detailed information can be found in Appendix B.2). We utilize Uni- versal HEAD tags in UD 2.7 for optimizing the syntactic tree loss. Further hyper-parameter details can be found in Appendix D.
XMT The framework is similar to Figure 2(a), except the data selection is im- plemented by random sampling.
For TydiQA-GoldP, focused on the baseline X-METRA (M' (M'hamdi et al., 2021) which shares similar setting with our framework, utilizing se- mantic similarity sampling based on (paragraph,
hhtnp///www.aaptttooo..m//pllllp

---

WILIAam
Modd
1
87
16
50
(fterll 2000)
64.0
42.7
41.1
17.3
53.5
59.6
4K.5
78.9
67.5
13.3
33.6
99.4
MRF
61.9
45.3
46.2
76.5
56.5
99.9
67.6
49.2
75.5
56.8
68.5
68.4
51.1
60.3
CROP Vanger et al. 2023)
69.7
56.4
48.0
79.1
46.4
62.6
74.9
61.6
81.0
57.7
65.3
75.5
52.6
64.0
SE.UEU (No al. 2021)
79.9
54.8
70.0
86.2
51.4
71.8
K36
60.9
81.5
65.3
2044
61.3
43.5
76.1
2581106
166.18
724103
81010.1
534.32
667106
705112
07910.4
909104
8333003
831116
7833107
318110
FT whyn cample
75.741.0
735-43
0
819-09
720119
65210.1
824105
855104
61.3-0.
93
75.3205
7433208
120303
700308
74221111
738107
M1.1003
504209
357700
76.3212
91ME111
Oura
MeTOCo NMT
7522300
000
7511200
$12:01
742222
7518206
S14E11
782200
92.5303
75.5
MrrCCoXMTeo XMT
75610.3
74.8:06
R222103
$61.001
70.110.1
732.110
751106
918:002
85. 91111
17.1110
918102
75.6
77,60.2
61.9:04
74.7
63.2-0.1
846-003
71.7+00
762:0044
746+08
84.5+04
98.7-0.2
88.0+0.3
$1,8006
938H004
80.4
2020)
0931
13.1
000
78.9
70.5
PRG
71.6
26.5
57.6
81.5
55.2
63.2
78.0
59.6
71.3
65.0
08.1
70.1
40.2
62.5
XLMHlery
F7
M022119
48.3:7.5
77.0g2.4
662
780:556
72.2333
77.3499
70.1662
K3.0104
$5.4:111
86.222.4
786213
84.262.4
77.4
PT whyn sample
794310.3
89
199150
76.1.22
K2.7102
86.2 2.2.00
706124
77.2220
714422
N2.410.5
64.9.122
M6.119.
76.9 76.9.00
89.01.2.2
77.9
XMT...sbe
80.720.5
$3.9g16
71.7=0.7
83.8505
429924.
72.4606
7988009
77.56066
84.060.11
50:26533
96.860.6
50.5:0077
66.9644.4
79.0
Ours
51
K3233.1
154
91.8213
McToCo XMTi
79.71166
33.3
77:11:01
73.7113
2
75.2.05
K2.9108
11:2:09
87.12001
7.0:4044
89112008
McBCCAMMT...
80.900.3
54.3=002
781=001
83.5002
87.8=02
749
62.6
75.7002
83.860.11
88.:00.3
87.26005
77.0=0.3
92.1.022
Table 2: Fl score and standard deviation of 13 languages and average on the WikiAnn dataset.
Languages
zh
37
ko
10
y0
352
T50004GGshiPP (EM)
0BBEXT
McDato XM
53.0
634::0.
6162207
6682004
$19:04
54.3009
6452002
64.3
60.0
63,006
63.3
61.0115
63.4004
$12212
53.7332
64110.8
62.94088
WMD 100s.
53
63.020.3
60820.3
05,209
518205
516629
642205
62.1.10.6
Witiboon (FI)
McT.Co-MMT..
77.600
61.9204
74.700.1
82.2-0.1
84.6203
71,7500
7622004
74620.8
84.5004
93.70022
880:0033
8188006
93M#04
SOL
13.12:11
39.9104
739204
7111100
7EU:03
8811106
9211204
884ELL
82331.3
WMDD-ooe
73
80.01 0.
738.003
82.1004
80.8.0. 0.2
698.00
756.0088
75910.
840105
91.7104
855110
821114
866.1.11
78.8
Table 3: Ablation results for MRC and NER task based on MBERT.
5.4 Results
tic similarity sampling for structural tasks like NER. Overall, the results from both tasks demonstrate the efficacy of our MeTaCo-XMT framework.
Table shows the results of the Exact Match (EM) score on TydiQA-GoldB while F1 scores are re- ported in Table 9 of Appendix E. Our method based on SDMM (i.e. MeTaCo-XNN XMT sym) is supe- rior to the baselines in terms of EM for all 8 target languages. Notably, McTaCo-XMT syn achieves a significant EM improvement of 4.4%, 1.6%, and 2.2% over the mBERT baseline for the low- resource languages Bengali (bn), Telugu (te), and the cxtremely-low resource language Swahili (sw), respectively. MeTaCo- XMT sym based on mBERT and XLM-RRarge demonstrates average EM im- provements of 1.4% and 0.9% compared to the strong baseline XMTrondom. Moreover, among the models with reported standard deviations, MeTaCo- XMT sym exhibits the highest stability, indicated by a lower average standard deviation.
The results of three data selection strategies and random sampling strategy show that MeTaCO- XMT syn obtains best performance and with lower standard deviation. Semantic similarity sampling is less effective than random sampling may due to limited diversity in the support samples caused by high semantic similarity among query samples. The task- sisk-iiillarity sampling strategy is better than the semantic similarity sampling, demonstrating that task information is significant for meta-task selection in MRC and NER.
Ablation study was conducted based on the MeTaCo-XMT syn model with different structures of SDMM. We study the effectiveness of the syn- tactic tree loss (STL) and WMD metric, as shown in Table 3. The results show the importance of all proposed components, as removing syntactic tree loss (STL) or using cosine similarity instead of Word Mover's Distance (WMD) leads to slight decrease in average effectiveness across both tasks.
The results of WikiAnn in 13 languages are shown in Table 2. The MeTaCo XMCCCoXXMT.... syn model achieves an average improvement of 2.8% and 1.3% over the strong baseline XMTranden mBERT and XLM-Riarge> respectively. Our mBERRTbased MeTaCo-XMT outperforms other models on all extremely-low resource languages and exhibits the lowest average standard deviation. These findings validate the effectiveness of syntac-
6 Analysis
Our analytical experiments consider the perfor- mance of the four cross-lingual meta-transfer mod- els with random, semantic-similar task-similar,

---

15
id: ditalis olch [Manhew Rellany) PER en: This song was written by [Manthew Bellamy Ip P
random
[The History England from the Accession James lllo
11
Belll p/ composer
9
task
The artwook was credited to [Amold Podh PER
All tracks written [Jsonn p
Table 4: A example of different data select strategies in WikiAnn.
-1
3
random
task
syn
14
20 800
o8 bn
sem
random
Figure 3: The effect gain ó relative to the FT model in language WikiAnn. distances between language and English increase from left right. Lan- guages starting with Arabic (ar) are distant languages.
70.0
and syntactic-similar samples named 'random', 'sem', 'task', and 'syn' for short, respectively.
61.0
mBERT baseline
Syntactic Distance Analysis Our McTaCo-XMMT framework significantly improved the average performance on NER task, especially MeTaCo- XMT sym Therefore, we explored the influence of different sampling strategies on languages with different syntactic distances from English (calcu- lated by lang2vec (Littell et al., 2017)). As shown in Figure the y-axis is the effect gain percent 8 relative to FT model based mBERT:
58.0
Figure 4: Avarage Fl in five extremely-low resource languages of NER task with different nam.
transfer, analyzed in Appendix E.2.
Query Data Size Analysis In the few-shot cross- lingual transfer scenario, target language has a certain number of annotated data. Low resource languages are challenging to collect annotation data, so we explore (i) whether small amount of data can achieve an improvement and (ii) the robust performance of different data selection strategies with different sizes num of query instances.
8 F1 FlMeTTCo XMT FIFT (15)
FIFT
The results show that MeTaCo-XMT syn always obtains the positive gain in 13 languages, and the syntactic similarity sampling schema is more ad- vantageous in distant languages from English than other strategies.
With different num settings, we reported the av- erage FI in five extremely-low resource languages of the NER task (the entire results can be found in Appendix E.4). As shown in Figure 4, even with only 5 query samples, the method MeTaCo- XMT sy based on syntactic similarity sampling can achieve the best results with great improve- ment than strong baseline F7 and XMT random- For MeTaCo-XMTros and MeTaCo-XM'T sem. they are not suitable for only 5 query instances; and when the query examples are less than 30, the ef- feet is lower than XMT strategies are suitable for scenarios with certain amount of target language data. However, even when query data is scarce, XMTyandom always has an advantage. This may be because randomly sam- pled instances preserve the diversity of the samples.
Case Study for Meta-task In order observe the sampling effect of different data selection strate- gies, an example of English (eno-ennnnnnian (id) match pairs in WikiAmn are listed in Table 4 (more cases shown in Appendix E.5). In example #1, original Indonesian and syn English sentences have similar syntax and even semantics, and the task sen- tence has a similar syntax to the original sentence. random and sem sentences are no obvious connec- tion the Indonesian query sentence. Combining the results of four strategies on NER, the struc- turally similar examples might benefit cross-lingual meta-transfer learning. Furthermore, the struc- turally similar examples can more effectively stim- ulate the ability learning in cross-lingual
So task and sem

---

According to the better performance of McTaCo- XMT agn and the cases study, the sample selection with syntactic similar conforms to the syntactic distribution of the target language and ensures the diversity of samples.
to be done on syntactic differences and syntactic distance metries.
References
Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Ed- uard H. Hovy. Kai- Kai-ee Chang, and Nanyun Peng. 2019. On difficulties of cross- mons-lingual transfer with or- der differences: case study on dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 2440-2452.
7 Conclusion
In this paper, we have presented a novel meta-task collector-based cross-lingual meta-transfer frame- work, which can adapt different meta-task selec- tion strategies to construct meta-transse transfer training data, reducing the cross-lingual performance gap between languages due to language differences. To close the syntactic distance between languages, we propose the syntactic distance metric model that encodes text pairs to syntactic space and se- lects meta-task by WMD for meta-transfer learn- ing. Two other data selection strategies are ex- plored: semantic similarity sampling and task- similarity sampling. We demonstrate the valid- ity of our framework on both the NER and MRC tasks, especially the syntactic similar sampling- based method reaches new offte-ofthhe-att for most languages. Further analyses suggest that the proposed MeTaCo-XMT with syntactic similar sampling can effectively improve the cross-lingunt transfer performance with only a small amount of data, especially for low-resource languages.
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 4623-4637.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji. Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. multitask, multilin- gual, multimodal evaluation of chatgpt on reason- ing, hallucination, and interactivity. arXiv preprint XiV:22302.04023.
Ethan A. Chi, John Hewitt, and Christopher D. Man- ning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, ACL 2020, pages 5564-5577.
Zewen Chi, Li Dong. Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang. Xia Song. Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. Infoxlm: An informatiionthhoretee framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 3576-3588.
Acknowledgement
This work is supported by the Key Research and Development Program of Zhejiang Province (No. 2021C01013), the Fundamental Research Funds for the Central Universities (No. 226-2023-00060), National Key Research and Development Project of China (No. 2018AAA010190M Alibaba-Zhejiang University Joint Research Institute of Frontier Tech- nologies, and MOE Engineering Research Center of Digital Library.
Jonathan H. Clark. Jennimaria Palomaki, Vitaly Niko- lacv, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. 2020. Tydi QA: A bench- mark for information-seeking question answering in typologically diverse languages. Trans. Assoc. Com- pul. Linguistics, 8:4544470
8 Limitations
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACI. 2020, pages 8440-8451.
Due to the computation constraints, we were not able to experiment with our framework on more NLP tasks with more languages, which will be sup- plemented in the future. The syntactic difference is key factor affecting cross-lingual, and syntac- tic distance can help to understand or quantify the transfer differences. Our framework with SDMM is a simple attempt to use syntactic distance metrics to construct meta-task and reduce the language gap in cross-lingual transfer learning. More work needs
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Proceedings of the 2019 Annual Conference on Neural Informa- tion Processing Systems, NeucIPS 2019, pages 7057- 7067.

---

Jacob Devlin, Ming-Wo Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, NAACL-HILT 2019, pages 4171-4188
Patrick S. H. Lewis, Barlas Oguz, Ruty Rinott, Sebas- tian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answer- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 7315-7330.
Patrick Littell, David R. Mortensen, Ke Lin, Kather- ine Kairis, Carlisle Turner, and Lori S. Levin. 2017. URIEL and lang2vec: Representing languages as ty- pological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, EACL 2017, pages 8-14.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agmostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML, 2017, volume 70 of Proceedings of Machine Learning Re- search, pages 1135.
John Hewitt and Christopher D. Manning. 2019. A
Miaofeng Liu, Yan Song, Hongbin Zou, and Tong Zhang, 2019. Reinforced training data selection for domain adaptation. In Proceedings of the 57th Con- ference of the Association for Computational Linguis- tics, ACL 2019. pages 1957-1968.
structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, pages 4129-4138.
Qianying Liu, Fei Cheng, and Sadao Kurohashi. 2021. Cross-lingual adaption model-agnostic meta-learning for natural language understanding. arXiv preprint arXiv:2111.05800
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, pages 4411-4421.
Kaushal Kumar Maurya and Maunendra Sankar De- sarkar. 2022. Mega-Xplg. A meta-learning approach based on language clustering for zero-shot cross- lingual transfer and generation. In Findings of the As- sociation for Computational Linguistics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 269-284.
Hamish Ivison, Noah A. Smith, Hannameh Hajishirzi, and Pradeep Dasigi. 2022. Data-efficient finetuning using cross-task nearest neighbors. arXiv preprint arXiv:2212.00116.
Meryem M' hamdi, Doo Soon Kim, Franck Demoncourt, Trung Bui, Xiang Ren, and Jonathan May. 2021. X- METRA-ADA: cross-lingual meta- transfer learning adaptation to natural language understanding and question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2021, pages 3617-3632.
Karthikeyan K, Zähan Wang. Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In 8th International Con- ference Learning Representutions, ICLR 2020.
Shanu Kumar, Sandipan Dandapat, and Monojit Choud- hury. 2022. "diversity and uncertainty in moderation" are the key to data selection for multilingual few-shot transfer. In Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 1052-1055.
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kil- ian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd In- ternational Conference on Machine Learning, ICML 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 957-966.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2017, pages 1946-195 1958.
Vict Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english to- wards a comprehensive evaluation of large language models in multilingual learning labeled sequence translation. arXiv preprini arXivv2304.05613
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual multilingual bert? In Proceed- ings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 4996-
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2020, pages 4483-4499.
5001.
Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivas- tava, Siva Reddy, and Anders Søgaard. 2021. Mini- max and neyman-pearson meta-learning for outlier

---

languages, In Findings of the Association for Com- putational Linguistics: ACL/JJCNP 2021, Findings of ACI,, pages 1245-1260.
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word rotator's s distance. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 2944-2960.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLP 2019, pages 3980-3990.
Zhenghua Yu and Gunawan Herman. 2005. On the earth mover's distance as a histogram similarity metric for image retrieval. In Proceedings of the 2005 IEEE International Conference on Multimedia and Expo, ICME 2005, July 6-9, 2005, Amsterdam, The Nether- lands, pages 686-689, IEEE Computer Society.
Nils Reimers and Tryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, pages 4512-4525.
Daniel Zeman, Joakim Nivre, Mitchell Abrams. et al. 2020. Universal dependencies 2.7. LINDAT/CLARIAA-CZ. digital library at the Insti- tute of Formal and Applied Linguisties (ÚFALL, Fac- ulty Mathematics and Physics, Charles University.
Tom Sherborne and Mirella Lapata. 2023. Meta- learning a cross-lingual manifold for semantic pars- ing. Trans. Assoc. Comput. Linguistics, 11:49677.
Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulic, Roi Reichart, Anna Korhonen, and Hinrich Schlitze. 2021. A closer look at few-shot crosslingual transfer: The choice of shots matters. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACL/JCNLP 2021, pages 5751-5767.
Linjuan Wu, Shaojuan Wu, Xiaowang Zhang, Deyi Xiong. Shizhan Chen, Zhiqiang Zhuang, and Zhiy- ong Feng. 2022. Learning disentangled semantic representations for zero-shot cross- cross-lingual transfer in multilingual machine reading comprehension. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), ACI. 2022, pages 9-1-00
Qianhui Wu, Zijia Lin, Guoxin Wang. Hui Chen, Börje F. Karlsson, Biqing Huang, and Chin-Yew Lin. 2020a. Enhanced meta- veea-learning for cross-lingual named entity recognition with minimal resources. In The Thirty-Fourth AAAI Conference Artificial In- telligence, AAAI 2020, pages 9274-9281.
Appendices
A Classify the resource levels for Languages
Following Bang et al. (2023) and Lai et al. (2023), the 13 languages in our study are grouped into categories based on their data ratios in the Com- momCrawl (i.pus the main data to pre-train multilingual language models). In particular, a lan- guage will be considered as High Resource (H), Medium Resource (M), Low Resource (L), and Extremely-Low Resource (X), if its data ratio is greater than 1 1% (> 1%), between 0. % and 1% (> 0. %), between 0.01% % and 0.1% (> 0.01 0.01%) and smaller than 0.01% % (< 0.01 %) respectively. Ta- ble presents information and categories for the languages considered in our work.
Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Börje F. Karlsson, Biqing Huang, and Chin- Yew Lin. 2020b. Enhanced meta-learning for cross -oossll named entity recognition with minimal resources. The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Chirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, pages 9274-9281.
Liyan Xu, Xuchao Zhang. Xujiang Zhao, Haifeng Chen, Feng Chen, and Jinho D. Choi. 2021. Boosting cross- lingual transfer via self-learning with uncertainty es- timation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, pages 67 6716-6733.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards token-free future with trained byte-to-byte models. Trans. Assoc. Comput. Linguistics, 10:291-306.
B Dataset Statistics
B.1 Benchmark Dataset
Tables 6 and 7 show the statistics of WikiAnn and TydiQA-GoldP respectively per language and split in our study.
Jian Yang, Shaohan Huang. Shuming Ma. Yuwei Yin. Li Dong, Dongdong Zhang, Hongcheng Guo, Zhou- jun Li, and Furu Wei. 2022. CROP: zero-shot cross- cross-lingual named entity recognition with multi- lingual labeled sequence translation. arXiv preprint arXiv:2210.07022.
"htpp://commoncrawl. www.hi.tghhhhhhc......... htt.bucoo/c-rrawl--- staistisspons/linguugss..m

---

Lang
Script
Family
Dist. to English
Cat.
English
Latin
Indo European
Pop.
CC size
Spunish
Latin
Indo-Furopean
04
Language
Code
(M)
(%)
Cat.
German
Latin
Indo: European
0.42
Clos.
Latin
Indo-European
leelandic
Latin
Indo-Puropean
0.47
Clos.
English
en
1,452
43.8846
Latin
Indo-Furopean
Russian ru
258
9.2012
H
Russian
Cyrillic
Indo European
0.49
Clos.
Chinese
zh
1.118
5.1984
H
Ttalian
Latin
Indo-Eure Europpan
0.51
Clos
Indonesian
199
0.7399
Chinese
Han (Trddiional)
Tibe-an
0.57
Dist
Arabic
274
0.6688
Anbiic
Anabic
Korean
81
0.5944
Hindi
Devanagari
Indo- Europeun
0.39
Dist.
Finnish
4.9
0,3535
M
Korean
Hungui
Kareumi
0.02
Japanese
Japanese
Japonic
0.66
Dist
Bengali
bn
272.7
0.0454
L.
Turkish
Latin
Torkic
0177
Dist.
Telugu
95.7
0.017
Swahili
sw
71
0.0074
Table 8: List of Languages, languages script, language family, and the distance English. A distance larger than 0.53 is the distant language (Dist.) to English, and conversely the close language (Clos.) English.
Afrikaams
6.2
0,0072
X
Tagalog
72
0.0068
X
Javanese
60
0.0012
X
Yoruba
yo
42
0.0004
X
Table 5: List of languages, language codes, numbers of first and second speakers, data ratios the Common- Crawl corpus, and language categories.
B.2 Dataset for SDMM
The training data of SDMM is from UD 2.7 Tree- bank. In particular, we select 15 languages (includ- ing English) from Parallel Universal Dependen- cies (PUD) treebanks', each language containing 1000 sentences. According to the distance between each language and English, we construct a three- way parallel corpus with distant languages and 7 close languages for SDMM, including 6500 data for training and 500 data for development. The distance information between each language and English is shown in Table 8.
Lang
ISO
Train
Dev
Test
English en 20,000
Afrikuams af 5,000
100
1,000
Arabic ar 20,000
100
10,000
Bengali bn 10,000
100
1,000
Finnish
20,000
100
10,000
Javanese
100
100
100
Indonesian id
20,000
100
10,000
Korean ko
20,000
100
10,000
Russian
20,000
100
10,000
C Baselines for NER
Swahili
1,000
100
1,000
Telugu
1,000
100
1,000
For WikiAnn, we compared our method with two high-performance zero-shot baselines:
Tagalog
10,000
100
1,000
Yoruba
yo
100
100
100
Chinese
20,000
100
10,000
CROP (Yang al., 2022): A Cross-lingual Entity Projection framework (CROP) with a multilingual labeled sequence translation model. obtains the labels on the English NER model by translating the target language raw corpus (more than 100k instances) into English and then utilizes the multilingual la- beled sequence translation model to obtain the labels of the target language corpus. The whole pipeline is integrated into an end-to-end NER model by way of self-training.
Table 6: 13 languages statistics of WikiAnn dataset in our study.
Lang
ISO
Train
Dev
Test
English en
3,326
Arabic ar 13,324
1,481
921
Bengali bn 2,151
239
113
Finnish
fi
6,169
686
782
Indonesian
id
5,131
571
565
Russian ko
Korcan
1462
163
276
5,841
649
812
SL-LEU (Xu et al., 2021): A self-learning framework (SL) that further utilizes unlabeled model
Swahili sw
2,479
499
Telugu
te 5,006
557
669
data of target languages based fine-tuned on English training data, combined with uncertainty estimation in the process
Table 7: Statistics of TydiQA-GoldP dataset per lan-
guage and split.

---

select high-quality silver labels. The best per- formance of SL for the NER task is achieved by adopting Language Heteroscedastic Uncer- tainty (LEU) as the uncertainty estimation. uses all of the dev set of target languages on task data as the source of unlabeled data.
MoTaCO-MW
syn
They leverage translation or self-training methods to obtain pseudo-labeled data for target languages.
85
Hyperparameters
For the meta-task collector, we set q k: = 6 or q k: 8 to construct meta-task. The number of meta-tasks is [300, 100, 500, 600). The learning rate (Ir) a and are 3e-5 on the NER task. For Ty- diQA, a is 3e-5 and / is le-5. The step size r for the inner loop is 4 or 3. The batch size of the outer loop is 4 or 5. The three seeds for NER and MRC tasks are [111, 22, 3333]. We experimented us- ing [3, 4, 5, 6, 7, epoch number for meta-transfer training, referring the setting in M' M'hamdi et al. (2021); Liu et al. (2021); Nooralahzadeh et al. (2020). However, the 5 8-epoch setting led to the best results in our experiments.
Ko bn SW
(d) madeon
Figure 5: The results of MeTaCo-XMT and FT w/sample in NER task (except rich-resource languages) with four data selection methods.
The results of our MeTaCo- MMTT framework are also shown in Figure 5 to compare the performance of different learning methods (meta-learning and fine-tuning) using these selected data.
Overall, meta-learning methods outperform fine- tuning methods in most languages, especially medium-rcsource languages. From the compari- son different data selection strategies, MeTaCo- XMT syn always outperforms FT w/syn_sampp Me MeTaCo-XMTT... can also surpass the fine-tuning method in most languages. However, MeTaCo- XMT sem and XMT random difficult outper- form the FT w/sample in extremely-low resource languages. The examples based on syntactically similar selection can more effectively stimulate the ability of meta-learning cross- tross-lingual transfer learning.
For typer-parameten size of 16 and 20 for the model based on mBERT and XLM-R large baselines respectively, Adam with the Ir of 1e-3, and the epoch is 20. The mar- gins were 0.6 or 0.5, and the syntactic vector dimensions were 32 and 64 for SDMM based mBERT and XLM-R large baselines respectively.
SDMM, we use a batch
E
More Result
E.1 F1 Score of TydiQA-GoldP
Tables 9 show the F1 score of TydiQA-GoldP in 8 target languages. MeTaCo-XMT syn outper- forms the strong baselines based on mBERT and XLM-R by 1.4% and 0.6% average, respec- tively. And its standard deviation is also sig- nificantly lower than the random sampling-bes method XMT random- For extremely-lov resource language, our MeTaCo-XMT framework can al- ways outperform XMT yandom with a lower stan- dard deviation.
E.3 Comparison with Translation-tral Baselines
The method fine-tuned on translation-tras data is strong baseline, such as translate- baseline in XTREME (Hu et al., 2020) and translate-train setting in ByT5 (Xue et al., 2022). They fine-tuned on one or all target languages data translated from English data, which is significantly more than the DEV data (10% of training data in one target lan- guage) we used. Compared with ByT5 based on the m'T5 base model (580M parameter) in the translate- train setting, we can have improvement by 2.7 and 3.9 on the avarage of Fl and EM metric in TydiQA- GoldP. Furthermore, the results reported in the XTREME benchmark for the Translate- train base-
E.2
Data Selection in Meta-learning and Fine-tuning
We extended the baseline FT s/syy_samppl ex- periment on four data selection methods (called FT w/sample), and the results the NER task (except rich-resource languages) are shown in Figure 5.

---

TydiQA HolQA-GoldP&oo (FI)
Model
fi
id
ko
bn
te
PRE (RE al., 2020)
60.0
62.2
59.7
64,8
58.8
49.3
49.6
57.5
57.7
PRE
59,4
64.3
59.0
64.8
57.1
51.8
47.8
59.8
58.0
X- X-METRA (M' h'mdddi et al., 2021)
66.111011
78.4:0.6
12.740.4
77.71:022
53.240.5
66.6
717400.2
69.5
FT
68.340077
77.5=0.4
12.02100
77.9110
59.0=0.7
63.42100
76,522.6
69 09.240..5
70.5
m
FT wwwnnssmmpp sumple
68.64:0.6
78.2+00.5
72.240.6
77.2+004
58.6
63.9+199
77.3+0.3
71.0+1.7
70.9
68.74:0.7
77.5=0.44
73.3::0.9
77.811.
60.423.2
62.14222
76.6=0.9
71.61111
McTaCo-XMT... McTaCo-XMTiunl
67.410.2
77.8:113
72.740.7
78.21:0.8
59.622.4
60.5::55
76.420.6
73.5+0.7
70.8
68.3:10.4
78.9=0.55
73.3:0.77
77.0=:0.
60.3#2.3
65.6:22.3
78.2
73.24:..1
71.8
McTaCo-XMI..
69.1+007
79.0=00,2
72.9:0.6
78.020.4
61.2=0.7
67.0::0.8
78.7=003
73.9
72.4
(RE et ul., 2020)
67.0
67.6
70.5
77.4
31.9
64.0
70.1
66.1
64.3
PRE
67.6
75.6
73.3
79.6
59.8
66.5
74.2
71.4
71.0
FT
72.8
83.3
75.5
XLM-Rarg
FT saspl
75.0::0.8
80.410.9
83.62:08
76.24:.0
80.820.6
77.020.7
XMToo
73.520..9
80,5=00.9
78.411.2
83.710.4
66.2=114 1.4
79.242.44
82.1
78.42:0.3
Ours
McTaCo-XMI
73.420.1
80.7=110
78.620.3
83,420.7
64.620.8
76.81114 4
81.5=0.2
79.74:0.3
77.3
MeTaCo-XMM XMT
74.1+008
81.1-0.6
78.8+1.2
83,3+-0.2
64.9-0.7
79.9+-2.8
82,5 --33
79.8++0.2
78.1
McTaCo-XMTT...
73.8
81.1=0.5
79.240.9
83.710.11
66.0:2088
80,412.1
82.4:20.5
80.2:10.3
78.3
Table 9: Fl score and standard deviation of 8 target languages and average on the TydiQA-GoldP dataset.
WikiAnn (FI)
Model
af
jv
yo
avg
mBert baseline
75.5
56.8
68.6
68.4
51.1
64.1
FT
7.14-E1.1
68.0:10.6
71.5
73.4
74.4
71.9
random
80.0=F0.4
76.1
83.2
75.8+00.4
73.2
77.7
num 30
sem 78.5+1.2
7..41
83.1
74.7
75.3
77.4
task 78.8++0.5
76.2+-0.1
83.0+00.4
74.7+=111
1..3+
77.6
syn 82.0
75.7+10.4
83.6
77.0
75.8
78.8
FT
77.5=10.2
62.5
71.8
74.1
75.4
72.3
random 79.5+
72.3+0077
8..
76.5
61.3
74.4
num 20
sem 77.1
72.811.9
82.0
74.6#00.9
64.0
74.1
task 77.110.2
72.71188
82.0
74.71..0
61.911..1
73.7
syn
81.2+0033
74.0
83.6
76.4
71.8
77.4
FT
78.9
63.4::2.8
69.2::0.4
63.0+:..2
57.1
66.3
random 77.2-1..2
61.9110.3
79.4
6..711.2
60.6
68.7
10
73.8+0..4
61.9+:0.3
79.5
61.5+
19.6
67.2
73.810..4
62.110.5
79.5
61.5
59.9
67.4
syn
80.3++0.2
74.0
83.6
74.0
60.2
74.4
FT 1.2
62.8+
68.1
58.4+00.
59.8
63.0
random 74.7+1..3
57.0
67.2
61.410.6
59.7+1..
64.0
num = 5
sem 68.2+10.3
57.1
6..5
54.5+
59.1
60.3
task 68.5+10.2
56.910044
62.5
54.3
59.0
60.2
syn 79.3
65.9-L0.4
66.0
75.0
63.9
70.0
Table 10: Fl score and standard deviation of extremely-low resource languages on the NER task with different num setting.

---

line are also lower than our model in the TydiQA- GoldP task. Performance increasement demon- strates the advantages of our model in the few-shot scenario.
E.4
The Results with Different Query Data Size
Table 10 shows the Fl score and standard deviation of 5 extremely-low resource languages on the NER task with different num settings.
E,5 Case Study
We report more data select cases in Table 11.

---

fi: Hän syntyi (Stralsundissal..oc Loc ja opiskeli [Leipzigin konservatoriossalooc.
#2
en: He was born [Stralsund] LoC and studied at the [Leipzig Conservatorylooe.
random
1997: [Pust Mirom Pravit Lyubovlonc
sem
The project went from [New Haven, Indiana to Toledo, Ohiolnoo-
task
He moved to [Memphis. Tennessce]aoc with his family at the age of twelve.
syn
He was horn in [Teleška]xoo,, [Zapadna BačkaJong, [Serbia]n.oc.
id: Saat ini ia hermain untuk [PSIS Semaranglone. He currently plays for [PSIS Semaranglozo.
#3
random
[Crystal Tovar AragónJren
sem
He plays for (Thailand Premier Leaguelorg clubside [Samut Songkhram FCloro.
task
[Regional District of Cheall-Ch
syn
He currently plays for [Sivassporloxe in the [Super Ligloxc.
zh: + [f
#4
en: [Jiang Zhongzheng ZhongzhenglreeaaPeeiee (reseresid of the Republic of China, President of the Chinese Kuomintang)
random
His reign was also marked by the highly controversial exccution his son, [Prince SadolpER. in 1762.
sem
[Terengganu] "*
task
[Governor of KentuckylegR : [William Owsleylega ( [Whig loxa) until September ) [John J.
Crittenden)ees ([hiiglorc) ( starting September 6)
syn
[Bao ZhenglPeR (L)
Table 1I: The examples of different data select strategies in the WikiAnn dataset of three target languages (Finnish(fi) Indonesian(id), and Chinese (zh)).


Reviews:
[{"review_id": "PdhpdwQNeV", "reviewer": "Reviewer_cvjj", "summary": "", "questions": "- Although the framework is proposed as a data selector, multilingual NLP has other challenges such as data availability and quality, as pointed out in Yu et al. (2022) https://arxiv.org/abs/2211.15649. Can the proposed framework tackle these challenges? Why would we use this framework rather than collected suggestions?\n- What is $x$, $x^d$ and $x^c$? Is it the same as in the WMD paper? There are a lot of different notations so maybe explain before using them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt thanks to you for your valuable insights and thoughtful assessment of our work. Below is my response.\n\n- **R1: About more in-depth analysis regarding the framework on other tasks.**\n\nNER and MRC represent token-level and passage-level comprehension tasks, respectively. We employ these two tasks of different levels to validate the effectiveness of our method. However, the NER task aligns more closely with our syntactic similarity selection strategy, as it is the base for syntactic analysis tasks. Therefore, due to space limitations, we have chosen to conduct a detailed analysis specifically on the NER task. \n\nWe are also conducting analytical experiments on the MRC data, which will be included in the appendix of the new version. The following are some experimental results completed so far, mainly about the results of different $NUM$ settings on low- (bn) and extremely low-resource languages (sw, te) in the TydiQA-GP dataset. It can be seen from the experimental results that when the target language only has 20 annotation instances, the semantic similarity selection is not even as good as the random selection method. The selection based on task similarity still has improvement over the random scheme. Our proposed MeTaCo-XMT model based on the syntactic distance measure can always maintain the advantage in the few-shot learning setting.\n\n|     | **model**      | **bn**                  | **sw**                  | **te**                  | **avg**         |\n| --- | -------------- | ----------------------- | ----------------------- | ----------------------- | --------------- |\n| NUM | mBert baseline | 35.2 / 51.8             | 38.4 / 47.8             | 39.9 / 59.8             | 37.8 / 53.1     |\n| 30  | FT             | 36.2±3.9 / 49.8±3.0     | 41.5±1.0 / 58.4±1.4     | 47.0±1.7 / 58.2±2.2     | 41.6 / 55.5     |\n|     | random         | 39.6±1.4 / 54.0±1.3     | 38.3±1.4 / 57.0±0.6     | 50.5±1.7 / 62.8±1.5     | 42.8 / 57.9     |\n|     | sem            | 37.8±1.1 / 52.5±0.5     | 38.7±0.1 / 57.2±0.1     | 50.2±1.0 / 62.7±0.9     | 42.2 / 57.5     |\n|     | task           | 38.9±0.6 / 56.2±0.7     | 43.6±0.6 / 56.7±0.7     | 51.2±0.2 / 63.4±0.4     | 44.6 / 58.8     |\n|     | syn            | **41.2±1.1 / 56.1±0.8** | **47.6±0.5 / 60.4±0.8** | **53.2±0.4 / 65.3±0.3** | **47.6 / 60.6** |\n| 20  | FT             | 34.9±2.7 / 47.6±1.9     | 39.3±2.1 /58.6±0.6      | 42.1±3.0 / 53.9±3.8     | 38.8 / 53.4     |\n|     | random         | 33.8±2.5 / 48.1±2.4     | 39.0±1.8 /57.6±2.2      | 46.5±1.0 / 60.2±1.2     | 39.8 / 55.3     |\n|     | sem            | 34.2±2.3 / 47.8±1.5     | 38.2±1.3 / 56.2±1.0     | 45.8±1.8 / 60.3±0.5     | 39.4 / 54.8     |\n|     | task           | 36.3±1.5 / 48.8±1.2     | 40.4±0.7 / 58.8±0.6     | 46.8±0.9 / 58.7±0.4     | 41.2 / 55.4     |\n|     | syn            | **37.8±0.7 / 54.6±0.8** | **42.8±0.6 / 59.4±0.5** | **48.8±0.8 / 61.3±0.3** | **43.1 / 58.4** |\n\n- **R2R3: About presentation and more analysis.**\n\nWe truly value your insights, which are vital for enhancing our paper's quality. Our comprehensive method description is intended to aid readers in understanding our motivations and the experimental setup. We have also included additional experiments and analysis in Appendix E to further showcase our method's performance.\n\nWe understand your concern that an excessive amount of method and implementation details might divert readers' attention and affect the comprehensibility of the experimental analysis. In the new version, we will simplify the description of our methods and implementation, placing more space on experiments and analysis. This adjustment aims to ensure that readers can more easily comprehend our work and be persuaded by our experimental results.\n\nAdditionally, we appreciate you pointing out the errors in our reference formatting, and we will address these issues one by one.\n\n- **Q1: About challenges of data that our approach can address in multilingual NLP**\n\nWhile our proposed framework primarily serves as a data selector, it indirectly contributes to addressing these challenges of data by:\n\n(1)Efficient Resource Utilization: Our framework can maximize the utility of available multilingual data, as shown in Table 10, we can use only 5 target labeled data to obtain great improvement in extremely-low resource languages NER task, thereby we can mitigate data scarcity issues to some extent.\n\n(2)Enhanced Data Quality: By selecting data with similar syntactic characteristics, our framework can potentially improve data quality, as it is more likely to contain relevant and high-quality examples.\n\nWhile our framework offers a solution, further research and improvements are ongoing to enhance its effectiveness in tackling challenges related to multilingual data availability and quality, such as using translated English Datasets. Although it is difficult to obtain high-quality translation models for low-resource languages, combining our method may also be a new way to solve this difficulty. Hope for your further discussion.\n\n- **Q2: About the explanation of notations**\n\nI apologize for any confusion caused by the organization of our text. In lines 281-282, we introduce these symbols, which represent three parallel sentences: the source language sentence, the distant language sentence, and the close language sentence. Explanations for these superscripts are also provided in Figure 2. It's possible that the explanations for identical characters were too distant in the text, and we will make appropriate adjustments in the new version. We will also check and correct the descriptions of other notations one by one.\n\nThank you again and look forward to further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting framework and metric! The metric focuses on syntactical distance rather than semantic.", "reasons_to_reject": "- While the experiment is conducted on both NER and MRC, Section 6 almost only mentioned NER task, why is that? I would love to see more in-depth analysis regarding the framework on other tasks in Section 6.\n- Presentation should be improved. The authors use many space to describe the method and implementation, yet the analysis is not enough to convince me that the proposed framework is useful. \n- Many bibliography reference style errors. ", "questions_to_authors": "- Although the framework is proposed as a data selector, multilingual NLP has other challenges such as data availability and quality, as pointed out in Yu et al. (2022) https://arxiv.org/abs/2211.15649. Can the proposed framework tackle these challenges? Why would we use this framework rather than collected suggestions?\n- What is $x$, $x^d$ and $x^c$? Is it the same as in the WMD paper? There are a lot of different notations so maybe explain before using them?", "paper_topic_and_main_contributions": "This paper proposes MeTaCo-XMT, a meta-task collector-based cross-lingual meta-transfer framework for data selection in meta-transfer learning. The paper also propose a syntactic distance metric model consisting of a syntactic encoder block based on pretrained models and a distance metric block using word move’s distance. They conduct experiments with different data selection strategies to validate the framework on WikiAnn and TydiQA. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Bibliography style is incorrect and inconsistent, such as line 135, 138, 152 and 156. If it is the beginning of the sentence, you should use Kumar et al. (2022) instead of (Kumar et al. 2022). "}, "extra_text": {}}, {"review_id": "nnRJeYWQMA", "reviewer": "Reviewer_CVtE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough reading of our paper and your professional assessment. We will revise our expression based on your detailed suggestions to make it more reader-friendly. Below is our response to the points you raised:\n\n- **R1: Regarding the code.**\n\nWe've already provided the code as supplementary materials with the submission and will publicly release the code to facilitate reproducing our results.\n\n- **R2: About replacing PLM with a seq2seq model like mT5 and the strong translation-based baselines.**\n\nOur method is adaptable to various types of pre-trained models, including seq2seq models like mT5, with only minor code modifications required. The models mentioned in the reference you provided fine-tuned on all target languages data or translation data, which is significantly more than the DEV data (10\\% of training data in one target language) we used. The result 69.4 or 79.3 is obtained by a mT5\\_XXL model with 13 billion parameters, much larger than our model with 550 million parameters based on XLM-R\\_large. \n\nIf we compared with ByT5 based on the mT5\\_base model (580M parameter) in the translate-train setting, we can have improvement by 2.7 and 3.9 on the F1 and EM metric of TydiQA-GoldP. Furthermore, the results reported in the XTREME benchmark for the Translate-train baseline are also lower than our model in the TydiQA-GoldP task. Regarding the strong translation-based baselines, intuitively, they have access to more target language training data than we do. We will put the results of this baseline in the appendix to supplementally demonstrate the advantages of our model in the few-shot scenario.\n\nMoreover, your suggestion has prompted us to consider expanding our application to include NLG tasks to further validate the effectiveness of our approach.\n\n- **About Typos Grammar Style And Presentation Improvements:**\n\nThank you for your careful reading and patient revisions. We will also make thorough revisions to our new draft.\n\n(1) Thank you for your suggested corrections to Line 58, which have made the sentence clearer and more on-topic.\n\n(2) In Figure 1, except for the 'ar' result, which is the correct source, all others were incorrectly excerpted as results of the FT method. We have corrected this in the new version.\n\n(3) Line 183: MAML consists of two stages, meta-learning, and adaptation. So, what Line 183 intended to convey is that after the optimization by meta-learning step, the model can adapt more quickly to new meta-learning tasks.\n\n(4) **The reason for choosing the CROP and SL\\_LEU models** is that they share the same pre-trained model baseline with us, and they use only monolingual data from a single target language to generate pseudo-labels. As for scenarios like ByT5 where all target language training data is used, SL\\_LEU also provides comparisons, and the results surpass this setting.\nAlthough ensuring fairness is challenging compared to our use of a small amount of labeled data, these models do provide a strong baseline. \n\n(5) $\\delta$ represents the gain of our method relative to the FT method, which falls into the category of relative gains. In the new version, we will formalize $\\delta=\\frac{MeTaCo-XMT\\ -\\ FT}{FT}$.\n\nThank you again for your recognition of our work and for pointing out our deficiencies. Looking forward to further communication.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work is well situated within previous work, with improvement using syntactic distance based rather than semantic one\n* Experiments compare two models (mBERT & XLMR), two datasets and multiple baselines and data selection strategies", "reasons_to_reject": "* The method is quite elaborate and there is no mention of releasing the paper code which will make replication difficult\n* Authors compare two encoder models mBERT and XLMR, perhaps more interesting would have been replacing mBERT eg with a seq2seq model like mT5, [1] shows good results with mT5 (79.5 with full training and 69.4 with translate train approach, compare to best reported results by authors 65.6 which uses 10% of other language as DEV)\n\n[1] ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. https://arxiv.org/pdf/2105.13626v3.pdf", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work explores model agnostic meta learning (MAML) for cross-lingual learning, focusing on data selection strategies that construct the meta tasks for MAML. Data selection strategies include semantic similarity (previous work) and a newly proposed syntactic similarity metric model. Experiments on TydiQA (QA) and WikiANN (NER) show improved results for the proposed approach.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* line 58: \"models trained with randomly 056 sampled meta-tasks (X-METRAseed ) generally per- 057 form either better or worse than models trained 058 with meta-tasks sampled by semantic similarity\" is quite confusing, perhaps rephrase into your main point that there is a big variation of results according to the selection and hence better selection can generate better results.\n\n* 94: Figure(b) -> Figure 1(b)\n\nFigure 1: X-METRA1 results are taken from Mhamdi (Table 8, X-METRA block), though results don't seem to match: bn 39 vs 38 (Mhandi vs this paper), fi 59.1 vs 56.6, id 65.2 vs 63.8 (ar matches)\n\n* 108: \"to adopt data select strategy\" -> as a data selection strategy\n\n* 138: \"In recent X-METRA-ADA M’hamdi 138 et al. (2021), it follows the setting of cross-lingual\" sentence needs improvement\n\n* 178: \"for the inner step as Equation 2\" -> \"for the inner step in Equation 2\"\n\n* 183: \"The optimized-based meta-learning algorithm can perform the model fast adaptation to a new task.\" ?\n\n* 261: \"randomly choose q query\" -> choosing\n\n* 263: \"draw the k/q\" k is not defined?\n\nTable 1: TydiQA-GP -> TydiQA-GoldP to align with previous work notation\n\n* 440: \"For WikiAnn, we added the competitive and challenging zero-shot baselines with pseudo- labeled data, including CROP(Yang et al., 2022) and SL_LEU(Xu et al., 2021)\" it is not clear why these baselines are added? Perhaps also add the more competitive translate-train baseline from ByT5 (which is better than the reported results in the paper)\n\n* Figure 3: \"The effect gain\", do you mean the absolute gain?\n\n* 616: \"There are more work will need to be done on the syntactic differences and syntactic distance metric.\" rephrase"}}, {"review_id": "03SsZN6G1N", "reviewer": "Reviewer_5wKC", "summary": "", "questions": "1.\tL229-L230, why did you evaluate the optimized model on the target language test dataset directly? If my understanding is correct, the model should be optimized again after the meta-learning process to make the model better for the target task.\n\n2.\tThe task selection process seems like a language-agnostic process. Language similarity can be easily replaced by other similarity metrics. Is there any specific reason that you apply the method to the cross-lingual scenario? Can the method be applied to other monolingual few-shot knowledge transfer tasks?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for dedicating your time to reviewing our paper and providing your professional evaluation. Here are my responses to the questions you raised:\n\n- **Q1: About the reason to evaluate the optimized model on the target language test dataset directly.**\n\nYou understand correctly. MAML indeed comprises two stages: the first stage involves meta-learning training, and the second stage involves adaptation learning on new meta-tasks. We did not perform the adaptation learning stage because our primary focus was on evaluating the few-shot learning capability after data selection. This capability can be assessed after training in the first stage. As seen from the results in X-METRA-ADA, the complete two-stage learning process leads to better performance. However, this was not the primary focus of our study.\n\n- **Q2: About the particularity of data selection strategy design in cross-language scenarios.**\n\nSemantic and task-based similarity data selection strategies are language-agnostic and find application in various scenarios, as mentioned in our related work. Therefore, the main focus of this paper lies in choosing more suitable data based on syntactic distance measurement. Lines 81 to 98 elaborate on why we opt for syntactic similarity sampling in cross-lingual scenarios. For short, due to the existence of syntactic differences between languages, we believe that selecting samples with more similar syntactic features will be more beneficial for cross-lingual transfer. Perhaps this idea can also be generalized to knowledge transfer tasks that share structural differences as a distinguishing feature.\n\nI hope I have effectively answered your questions. If you have any other specific suggestions or requests, we would be more than happy to consider and address them appropriately. Once again, thank you for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe method is intuitive and effective. Meanwhile, the paper writing is clear.\n2.\tExperiments are comprehensive and support the claims of the paper.\n", "reasons_to_reject": "1.  I didn’t see any obvious weakness in this paper. \n\n", "questions_to_authors": "1.\tL229-L230, why did you evaluate the optimized model on the target language test dataset directly? If my understanding is correct, the model should be optimized again after the meta-learning process to make the model better for the target task.\n\n2.\tThe task selection process seems like a language-agnostic process. Language similarity can be easily replaced by other similarity metrics. Is there any specific reason that you apply the method to the cross-lingual scenario? Can the method be applied to other monolingual few-shot knowledge transfer tasks?\n", "paper_topic_and_main_contributions": "This paper proposes to use a task collector to select better support tasks for the cross-lingual MAML algorithm. The collector is built based on a syntactic distance metric model, which is used to calculate the Word Mover’s Distance between the target task samples and the source task instances. The syntactic distance metric model contains a syntactic encoder block based on the pre-trained model and a distance metric block. Experiments show the superiority of the method over several baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes MeTaCo-XMT, a framework based on model agnostic meta-learning (MAML) for cross-lingual learning, focusing on data selection strategies that construct the meta tasks for MAML. Overall, the empirical results in the paper show that MeTaCo-XMT has better results than comparable frameworks, and some interesting takeaways are that using syntactic distance is important rather than semantic one. However, some questions on the limitations of experimenting only on mBERT and XLMR, and the data availability for each language on different tasks.  Based on the reviews, the AC acknowledges that the analysis and results across multiple tasks are interesting, but recommends improvement in presenting the results, especially in resolving the extremely small fonts used in multiple results tables.
Decision: Accept-Findings