ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters
Authors: Vipul Kumar Rathore
Rajdeep Dhingra
Parag Singla
Mausam .
Venue: EMNLP 2023 Main (2023)
Keywords: Multilinguality
Low Resource Languages
Parameter-Efficient Fine-Tuning (PEFT)
TL;DR: We propose a strong method that ensembles Language Adapters at train and test-time for Zero-shot transfer to Low Resource Languages

Abstract: We tackle the problem of zero-shot cross-lingual transfer in  NLP tasks via the use of language adapters (LAs). Most of the earlier works have explored training with adapter of a single source (often English), and testing either using the target LA or LA of another related language. Training target LA requires unlabeled data, which may not be readily available for low resource *unseen* languages: those that are neither seen by the underlying multilingual language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data for them. 

We posit that for more effective cross-lingual transfer, instead of just one source LA, we need to leverage LAs of multiple (linguistically or geographically related) source languages, both at train and test-time - which we investigate via our novel neural architecture, ZGUL.
Extensive experimentation across four language groups, covering 15 unseen target languages, demonstrates improvements of up to 3.2 average F1 points over standard fine-tuning and other strong baselines on POS tagging and NER tasks. We also extend ZGUL to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language. We find that ZGUL continues to outperform baselines in these settings too.

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters
Vipul Rathore
Rajdeep Dhingra Parag Singla
Mausam
Indian Institute of Technology
New Delhi, India
rathorevvipul28@gail com, ra jadeep. wew. www.iitdegmaill........o...... paragsêcse iitd. ac. in, mausam@ www.:ieeeeee..eee........mee awcidddac..... ac.in in
Abstract
of a PLM on task-specific training data in source language(s), and (2) Language Adapters (LAs) small trainable modules inserted within a PLM transformer, and trained on target language's 's un- labeled data using Masked Language Modeling (MLM). At test time, in SFT the fine-tuned PLM is applied directly to target language inputs, whereas for the latter, the LA of source language is replaced with that of target language for better zero-shot performance. Unfortunately, while in the former case, one would expect presence of unlabeled data for a target language to pre-train the PLM for good performance, the latter also requires the same unla- beled data for training target adapter.
We tackle the problem zero-shot cross- lingual transfer in NLP tasks via the use of language adapters (LAs). Most of the earlier works have explored training with adapter of single source (often English), and testing either using the target LA or LA of another related language. Training target LA requires unla- beled data, which may not be readily available for low resource unseen languages: those that are neither seen by the underlying multilingual language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data for them.
We posit that for more effective cross-lingual transfer, instead of just one source LA,, we need to leverage LAs of multiple (linguistically or geographically related) source languages, both train and test-time- which investigate via our novel neural architecture, ZGUL. Ex- tensive experimentation across four language groups, covering 15 unseen target languages, demonstrates improvements of up to 3.2 av- erage Fl points over standard fine-tuning and other strong baselines on POS tagging and NER tasks. We also extend ZGUL to settings where either (I) some unlabeled data (2) few-shot training examples are available for the target language. We find that 7GUL. continues to out- perform baselines in these settings too.
For the large number of low resource languages, curating a decent-sized unlabeled corpus is chal- lenge. For instance, there are only 291 languages that have Wikipedias with over 1,000 articles. Con- sequently, existing works use English data for train- ing on the task and use the English LA (Pfeiffer et al., 2020b; He et al., 2021) or an ensemble of re- lated language LAs (Wang et al., 2021) at inference time. We posit that this is sub-optimal; for better performance, we should leverage multiple source languages (ideally, related target language) and their LAs, both at train and test-time.
To this end, we propose ZGUL, Zero-shot Generalization to Unseen Languages, which ex- plores this hypothesis. nents. First, it fuses LAs from source languages at train time by leveraging AdapterFusion (Pfeif- fer et al., 2021a), which was originally developed for fusing multiple task adapters. This allows ZGUL to locally decide the relevance of each LA for each token in each layer. Second, ZGUL leverages the typological properties of languages (encoded in existing language vectors) as addi-
1 Introduction
has three main compo-
Massive multilingual pretrained language models (PLMs) such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) support 100+ lan- guages. We are motivated by the vision of extend- ing NLP to the thousands (Muller et al., 2021) of unseen languages, i.e., those not present in PLMs, and for which unlabeled corpus is also not readily available. A natural approach is zero-shot cross- lingual transfer train the model on one or more source languages and test on the target language, zero-shot. Two common approaches are (1) Stan- dard Fine Tuning (SFT)- fine-tune all parameters
FFoloowig previous work (Wang et al., 2021), we assume the target language has the same script as one or more lan- guages in the PLM, so that the model does not have to deal with sequence of [UNK] tokens at test time. We present the script statistics for our target languages in App. Table 14.

---

tional information for computing global LA atten- tion scores. Finally, ZGUL also implements the Entropy-Minimization (EM)-based test-time tun- ing of LA attention weights (Wang et al., 2021).
2 Related Work
Single-source Adapter Tuning: We build on MAD-X (Pfeiffer et al., 2020b), which introduces two phases of adapter training. Pretraining Lan- guage adapter (LA) for each language Lj: inserting an LA in each layer of transformer model M (de- noted by L. and training on unlabeled data for language & using the MLM objective. 2. Training TA for a task Ty: stacking LA for source language Live with TA for task Т3 (denoted by T, MMM in which Ty and the task-specific prediction head are the only trainable parameters, During inference, Lere is replaced with Ligt, i.e. Tj o Ligt o M is used. The MAD-X paradigm uses only one LA for a given input sentence. Also, it assumes the avail- ability of Ligt- If not available, English adapter (He et al., 2021; Pfeiffer et al., 2020b) or related language's adapter (Wang et al., 2021) is used at test-time.
We denote a language group as a set of phylo- genetically or demographically close languages, similar to Wang et al. (2021). We experiment on 15 unseen languages from four language groups: Slavic, Germanic, African and Indo-Aryan, on POS tagging and NER tasks. In each group, we train on multiple (3 to 4) source languages (includ- ing English), for which task-specific training data and LAs are available. ZGUL obtains substantial improvements on unseen languages compared to strong baselines like SFT and CPG (Conditional Pa- rameter Generation (Üstün et al., 2020)), in a purely zero-shot setting. Detailed ablations show the im- portance of each component in ZGUL. We perform attention analysis to assess if learned weights to source languages' LAs are consistent with their relatedness to the target language.
Adapter Combination: Pfeiffer al. (2021a) in- troduce AdapterFusion, a technique that combines multiple pretrained TAs .I....Tp to solve a new target task Tz+1- It learns the attention weights of TI,...T while being fine-tuned on the data for Ta+1- Vu et al. (2022) adapt this technique for fusing domains and testing on out-of-domain data. This technique has not been applied in the context of LAs so far. The recent release of 50 LAs on AdapterHub² enables studying this for LAs. Recently, Wang et al. (2021) propose EMEA (Entropy Minimized Ensembling of Adapters) for efficiently combining multiple LAs at inference time. EMEA calculates the entropy of the predic- tion during test time and adjusts the LA attention scores (initialized uniformly) using Gradient De- scent, aiming to give higher importance the LA that increases the confidence score of the predic- tion. However, the training is still conducted using English as a single source.
Further, we study two additional scenarios, where (1) some unlabeled data, and (2) some task- specific training data are available for the target language. We extend ZGUL in these settings and find that our extensions continue to outperform our competitive baselines, including ones that use unlabeled data for target language either (1) pre-train mBERT or (2) train target's LA.
Our contributions can be summarized as: (1) We propose a strong method (ZGUL) to combine the pretrained language adapters during training itself. To the best of our knowledge, we are the first to sys- tematically attempt this in context of LAs. ZGUL further incorporates test-time tuning of LA weights using Entropy Minimization (EM). (2) ZGUL out- performs the competitive multi-souree baselines for zero-shot transfer on languages unseen in mBERT. (3) ZGUL exhibits a strong correlation between learned attention scores to adapters and the linguis- tic relatedness between source and target languages. (4) ZGUL achieves competitive results in a few- shot setting, where limited amount of labeled (target language) training data is available. (5) When target language unlabeled data is available, modification of ZGUL outperforms baselines for nine out of twelve languages. To encourage repro- ducibility, we publicly release our code and trained models.?
Generation of LA using Shared Parameters: Üstün et al. (2020) employ the Conditional Pa- rameter Generation (CPG) technique (Platanios et al., 2018) for training on multiple source lan- guages. They utilize a CPG module, referred to CPGAdapter, which takes a typological language vector as input and generates a Language Adapter (LA). The CPGAdapter is shared across all source languages and trained from scratch for a specific task. Since an LA is determined by the input lan-
hhtpss//aaappcchhhhm///clceeeee Jang/

---

Combining the two ensembling modules: We pass the input sentence through both networks, and for fth token receive the outputs of and of) corre- sponding the FUSION and LANG2VEC networks, respectively. These two vectors are concatenated and passed through a fully connected layer. The output of this linear layer, denoted as "LA serves guage's vector, this approach can directly gener- alize to unseen languages. However, it is worth noting that CPG data-intensive as learns the parameters of the CPGAdapter from scratch.
of = Softmaa (W (0) (
(9)
The FUSION output for t token is given by: (1) =a of @ (W,V(9)
(1)
Here, Wg. W,,W, are projection matrices which are different for every layer I.
We note that CPG comes under broader cate- gory of hypernetworks that generate weights for larger main network (Ha et al., 2016), which have been recently explored successfully for task mixing (Karimi Mahabadi et al., 2021). In our experiments, we include a comparison with the CPG method.
Language-vector-ben based Attention (LANG2VEC): This sub-network computes the global attention weights for source LAs using the input language vector (of the token) as the query, the language vec- tors of the source languages as the keys, and the outputs through individual LAs as the values. Math- ematically, the attention weights over the LAs are obtained through projected dot- dot-product attention between the input language vector lapp as query vector and the source language vectors stacked as key matrix Lorc-
3 Model for Ensembling of Adapters
Our goal is to combine a set of source LAs for op- timal zero-shot cross-lingus transfer to unseen lan- guages, which are neither in mBERT, nor have read- ily available labeled unlabeled data. Similar to previous works (Pfeiffer al., 2020b; Wang et al., 2021), we focus on languages whose scripts are seen in mBERT. Handling unseen languages with unseen scripts is a more challenging task, which we leave for future research.
Here the language vectors are derived by pass- ing the language features 1// (each feature being binary) through a single-layer trainable MLP: linp = MLP(I
The LANG2VEC attention scores for th token are given by:
Our approach can be described using two high- level components: (I) train-time ensembling and (2) test-time ensembling, explained below for every layer I (for notational simplicity, we skip using in notations but clarify wherever required).
at = -Softmaz
W' (W((Lrr))
Here W,. is a projection matrix associated differ- ently with each layer l. lang(t) [t] denotes language vector of 8th token in the input
The output of tth token is given by:
3.1 Train-time Ensembling
dt) - a. C (W,)(9)
During training, make of attention mech- anism inspired by the combination of task adapters explored in Pfeiffer et al. (2021a). While they fo- cus on creating an ensemble of task adapters, our focus is on combining language adapters. Addi- tionally, we identify valuable information available in typological language vectors (Littell et al., 2017), that we can leverage. To achieve this, we design two sub-components in our architecture, which later combine for each layer (refer Figure 1).
Since. lang[(] is same across all tokens in an exam- ple and also across all examples in a language, we refer LANG2VEC attention as global. On the other hand, FUSION computes local attention scores that depend purely on the token-level outputs of the feed forward layer and hence are local to cach token in any given input sentence.
Tosen-based Attention (FUSION): This sub- network computes the local attention weights over source LAs for each token using the output of the feed forward layer as its query, and the individual language adapters' outputs as both the key and the value. Mathematically, for 4thh token, the embed- ding obtained after passing through feed forward layer becomes the query vector (t)) The individual LAs' outputs following this become key (and value) matrices (t) (and V(ti). The attention weights of source LAs for y token are computed using the fot-product attention between projected query (") and projected key matrix W2 (!));
(t)
as input to the task adapter (TA) to obtain the final (f)
output finat.
(4) Linear Layer (t)
of
Final ((()
The above process is repeated for each layer / in
'Following the implementation of Üstün et al. (2020).
we use 103 "syntax' hased features available at http://gituh..co//attmisss/////t

---

Inpur to leyer 1+1
forward pass with the modified attention weights. This process is repeated for T iterations, where T and learning rate Ir are the hyperparameters, which are tuned on dev set of linguistically most related (based on distributed similarity, shown in figure 3) source language for each target (grid search de- tails in Appendix A). Detailed EMEA algorithm is presented in Algo. 1 (Appendix).
Concat
Linear Layer
Task Adapter
Fusión autpl of(1]
Lang2Vec output
(1)
Tokers-hased Attertion
Fusion Soores
Langg/Ves-bssed Attention
Langzves Scores
Softmax
Liinp
Experiments
Value
Key
Query
Value
LV1
LV2
LV3
We aim to address the following questions. (1) How does ZGUL perform in zero-shot setting compared to the other baselines on unseen lan- guages? What is the incremental contribution of ZGUL's is components the performance on LRLs? (2) Are LA attention weights learnt by ZGUL in- terpretable, i.e., whether seneticallysyynaactiaall more similar source languages get higher atten- tion scores? (3) How does ZGUL's performance change after incorporating unlabelled target lan- guage data? (4) How does ZGUL's performance vary in seoo setting, where a few training ex- amples of the target language provided the model for fine-tuning?
LA2
Feed Forward
Transto layer 1
Input [T]
Figure I: FUSION Network (left) and LANG2VEC Net- work (right) outputs are concatenated and sent Lin- car layer followed by TA in every transformer layer I
the transformer architecture. We note that the LAs are kept frozen throughout the training process, while only the TA and other parameters described in FUSION and LANG2VEC modules being train- able. The training objective is word-level cross- entropy loss for all models. For model selection, evaluate ZGUL and other baseline models the combined dev set of source languages' data. We do not use any dev set of target languages, as doing so would violate the zero-shot assumption on target language data.
4.1 Datasets, Tasks and Baselines
Datasets and Tasks: We experiment with di- verse language groups: Germanic, Slavic, African and Indo-Aryan. Following previous works (Wang et al., 2021; Pfeiffer et al., 2020b), we choose named entity recognition (NER) and part-of-speech (POS) tagging tasks. We select target languages which are unseen in mBERT subject to their avail- ability of test sets for each task. This leads total of 15 target languages spanning Germanic and Slavic for POS, and African and Indo-Ary Aryan for NER. For African and Indo-Aryan NER exper- iments, we use the MasakhaNER (Adelani et al., 2021) and WikiAnn (Pan et al., 2017) datasets re- spectively. For POS experiments, we use Universal Treebank 2.5 (Nivre et al., 2020). We pick training languages from each group that have pre-trained adapters available. The details of training and test languages, as well as corresponding task for each group are presented in Table For detailed statis- tics, please refer to tables 13, 14, 16.
3.2 Test-time Ensembling
Wang et al. (2021) introduced EMEA, an inference- time Entropy Minimization (EM))based algorithm to adjust the LA attention scores, initializing them from uniform (as mentioned in sec. 2). In our case, since we learn the attention scores during training itself, we seek further leverage the EM algorithm by initializing them with our learnt net- works' weights. First, we compute the entropy of ZGUL's predicted labels, averaged over all words in the input sentence, using an initial forward pass of our trained model. Since ZGUL has two dif- ferent attention-based networks FUSION and LANG2VEC, it's trainable parameters are the at- tention weights for both these networks. We back- propagate the computed entropy and update both these attention weights using SGD optimization. In the next iteration, entropy is computed again using
Baselines: We experiment with two sets of base- lines. In the first set, the baselines use only English as single source language during training:
SFT-En: Standard mBERT fine-tuning on En- glish

---

Language Group
TTrinn set"
Test
Task
Germanic
German( De), Islandic(Is)
Faroese(Fo), Gothic(Gott. Swiss German(
POS
Slavie
Rus)ian(Ru), Czech(Cs)
Poma)(Qpm), Upper Sorhian(Hsh)
POS
Old East Slavic(Orv), Old Church Slavonic(Cu)
African
Amharic(Amh)
Hausa(Hau), (go)) Kinyarwanda(
NER
Swahili(Swa), Wolof(Wol)
Luganda(Lug), Luo(Luo), Nigerian Pidgin(Pem)
Indo-Aryan
Hindi(Hii, Bengali(Bn), Urdu(Ur)
Assamese(As), Bhojpuri(Bh)
NER
Table Language groups or sets related source and target languages, along with tasks. *English (En) is added to train set in each case.
MADX-(S): Training with En LA (with train- able TA stacked on top) and using strategy S during inference, where S can be one of: (1) En: En LA inference, (2) Rel: Using linguistically most related (based on similar- ity in Table 3) LA from source languages and (3) EMEA?: Ensembling all source LAs and perform EM (initialized with uniform)
4.2 Results: Zero-Shot Transfer
Tables 2 and 3 present experimental findings for Germanic and Slavic POS, well as African and Indo-Aryan NER, respectively. ZGUL outper- forms other baselines for 10 out of 15 unseen test languages. In terms of POS, ZGUL achieves a respectable gain of 1.8 average F1 points for Ger- mamic and a marginal improvement of 0.4 points for Slavic compared to its closest baseline CPG the gains being particularly impressive for Gothic, Swiss German and Pomak languages. For NER, ZGUL achieves decent gains of 3.2 points and 0.9 points for the Indo-Aryan and the African groups respectively over the closest baseline i.c. SFT-M the gains being upto 4 Fl points for Luo. Moreover. baselines trained on a single En source perform sig- nificantly worse (upto 24 points gap in Indo-Aryan), highlighting the importance of multi-source train- ing for effective cross- linosll transfer. We note that CPG outperfoms SFT-M for POS tagging, but der switches for NER. This is to be expected due to huge number of parameters in CPG (details in Sec. A) and the smaller sizes of NER datasets compared to POS (details in App. 13).
In the second set, we compare against models trained upon multiple source languages (belong- ing to a group), in addition to English:
SFT-M: Standard fine-tuning data from all
source languages.
Conditional Parameter Generation (Üstün et al., 2020), see Section 2.
CPG:
MADX"alK-(ST -(S) We naturally extend MADX-En to the multi-source scenario by dynamically switching on the LA correspond- ing to the input sentence's language during training. S refers to inference strategy that can be one of the following (as described in English baselines above): En, Rel, Uniform Ensembling EMEA ensembling.
We also observe substantial performance gap between MADX multi _ and ZGUL the former performing upto 7.5 points average F1 worse than ZGUL for the Indo-Aryan group. This demon- strates that relying solely on the most related LA is sub-optimal compared to ZGUL, which lever- ages aggregated information from multiple LAs. Additionally, MADX -inttu--nn-nnifm,.....m which does naive averaging of LAs, performs even worse overall. Though MADXmulls LDDXxuiii_EEEEEA -EMEA shows some improvement over it, yet remains below ZGUL's performance by an average of about 4 points over all languages. This finding highlights the effective- ness ZGUL-style training, as the EM algorithm benefits from an informed initialization of weights, rather than a naive uniform initialization strategy.
It is important to note that the EM algorithm is not applicable to SFT and CPG basclines because SFT does not use an adapter, while CPG has only single (shared) adapter. Consequently, there are ensemble weights that can be tuned during in- ference for these methods. The EM algorithm is distinctive feature of the ensemble baaeeble-based methods like ZGUL, which allows for further optimization and performance improvement.
Evaluation Metrie: We report micro-F1 evaluated on each token using seqeval toolkit (Nakayama, 2018). For all experiments, we report the average F1 from three training runs of the models with three different random seeds. The standard deviation is reported in Appendix G.
'e do same grid search for EMEA as ZGUL, Ref. to A

---

Germanic
Slavic
Fo
Got
Gsw
Avg
Qpm
Hsb
Orv
Cu
Avg
SFT-Fn
72.4
13.2
55.1
46.9
40
64.8
56.5
28
47.3
MADX- En
71.6
15.6
57.1
48.1
41.8
63.2
54.3
29.5
47.2
MADX-Rel
72.8
15.2
57
48.3
39.3
60.2
57
32.2
47.2
MADX-ENO -IDX-EMEA
74
14.8
55,3
48
42.1
63.1
56.4
32
48.4
MADX m
Rel
75.8
14.7
60.4
50.3
47.2
74.8
35.8
55.2
MADX -nifform
75.6
8.3
56.4
46.8
46.4
74.6
62.1
34.3
54.4
MADX -EMFA
76.2
12.5
62.1
50.3
48
75
63.2
34.7
55.2
SFT-M
77.3
16.8
61.8
52.0
47.6
75.4
63.7
34.7
55.4
CPG
77
16
63.6
52.2
46.9
76.7
64.1
35.6
55.8
ZGUL
76.9
20.2
64.8
54"
50.7
76.8
63
34.4
56.2
ZGUL EM
76.8
14.9
51.6
49.7
76
62.6
33.7
55.5
ZGUL w/o
76.9
17.8
61.3
52
49.8
76
62.8
33.3
55.5
ZGUL w/o
76.9
18.7
62.4
52.7
49.8
76.5
33.7
55.8
Table 2: F1 of POS Tagging Results for Germanic and Slavic language groups, denotes p-value < 0.005 for McNemar's test aggregated results over all test languages in a group
African
Indo-Aryan
Hau
Tho
Kin
Lug
Luo
Pem
Avg
As
Bh
Avg
SFT-En
43.2
47.1
49.1
47.5
29
64.4
46.7
32.1
35.7
33.9
MADX-FI
41.3
51.2
46
49.8
29.2
64.5
47
33
44.8
38.9
MADX-RI Rel
39.1
49.1
46.3
48.7
29.7
65.1
46.3
42.4
46.9
44.7
MADX-EMEA
42
52.9
50.2
50.6
30.7
65.5
48.7
41.6
47.6
44.6
MADX HEUTEE Rel
47.8
49.6
49.6
46.8
31.8
62.4
48
61.7
61.9
61.8
MADX -Unfform
47
51.1
50.8
49.9
31.7
62.7
48.7
60
59
59.5
MADX -EMEA -EMEA
47.4
54.4
55
51.1
32.8
63.9
50.8
62.4
59.7
61.1
SFT-M
51.3
55.7
57.9
56.0
36.0
65.0
53.7
70.8
61.4
66.1
CPG
49
51.1
55.1
54
34.2
65.7
51.5
62
63.3
62.7
ZGUL
53.6
56.8
56.2
54.2
40.2
66.5
54.6
74.4
64.1
69.3"
ZGUL W/ EM
52.4
55.6
57.3
55.1
38.6
65.5
54.1
71.1
58
64.6
ZGUL w/ L
54.7
57.3
53.5
35.5
65.5
52.6
61.5
64.2
62.9
ZGUI. w/o F
53.1
54.1
57.7
55.7
36.6
65.6
53.8
67.2
61.7
64.5
Table Fl of NER Results for African and Indo- Aryan language groups test aggregated results over all test languages in group
denotes p-value < 0.005 for McNemar's
More analysis on this follows in Sec. 4.3.
this, we computed the correlation 6 between the (final) attention scores computed by ZGUL at in- ference time, and the language relatedness with the source, for each of the test languages. Since ZGUL has two different networks for computing the attention scores, i.e., the Fusion network and LANG2VEC, we correspondingly compute the cor- relation with respect to the average attention scores in both these networks. For both networks, we compute the average of scores across all tokens, layers as well as examples in a target language. To compute the language relatedness, we use the dis- tributed similarity metric obtained as the average of the syntactic and genetic similarities (we refer to Appendix C for details). Table 4 presents the results. Clearly, we observe a high correlation between the attention scores computed by ZGUL
Ablation results last three rows in Tables 2, examine the impact of each of the 3 components, FUSION, LANG2VEC and Entropy Minimization (EM), on ZGUL's performance. We observe a posi- tive impact of each component for each language group, in terms of average Fl scores. For indi- vidual languages as well, we see an improvement in Fl due to each component, exceptions being Kinyarwanda and Luganda, where EM marginally hurts the performance. This could occur when wrong predictions are confident ones, and further performing EM over those predictions might hurt the overall performance
4.3 Interpretability w.r.t. Attention Scores
We wanted to examine if the attention weights com- puted by ZGUL are interpretable. In order to do
Peeassn Product-Moment Correlation

---

Group
LANG2VEC
FUSION
M
Eng
Isl
Ger
Eng
Hin
Ben
Urd
Germanic
0.57
0.45
0.23
Fao
0.26
0.38
0.36
Ass
0.21
0.28
0.3
0.21
Slavic
0.93
0.85
0.14
Bho
0.18
0.32
0.27
0.23
Got
0.25
0.4
0.35
African 0.61
0.2
0.18
Indo-Aryan
Indo-Aryam 0.81
0.67
-0.01
Gsw
0.27
0.35
0.38
Germanic
En
Amh
Swa
Wol
Table Correlation between ZGUL's attention scores and syntactic-genett (averaged) similarity of source- target pairs for both LANG2VEC and FUSION networks and for the EMEA (multi-source) baseline
Hau
0.22
0.2
0.31
0.27
Eng
Rus
lbo
0.22
0.19
0.33
0.26
Qpm
0.25
0.41
0.34
Kin
0.2
0.2
0.35
0.25
Hsb
0.23
0.46
0.31
Lug
0.18
0.19
0.38
0.25
Orv
0.23
0.47
0.3
Luo
0.21
0.21
0.33
0.25
and language relatedness, especially for Slavic and Indo-Aryan groups, for both the attention networks. This means that the model is assigning higher atten- tion score to languages which are more related in linguistic sense, in other words, language relat- edness can be thought of reasonable proxy, for deciding how much (relative) importance give to LA from each of the source languages for build- ing an efficient predictive model (for our tasks). Further, we note that among the two networks, the scores are particularly higher for LANG2VEC, which we attribute to the fact that the network ex- plicitly uses language features as its input, and therefore is in better position directly capture language relatedness, compared to the Fusion net- work, which has to rely on this information implic- itly through the tokens in each language.
Chu
0.24
0.46
0.3
Pcm
0.24
0.2
0.33
0.23
Slavic
African
Figure LANG2VEC attention scores for each of the test languages (clustered group-wise).
Eng
Isl
Ger
Eng
Hin
Ben
Urd
Fao
0.34
0.64
0.42
Ass
0.14
0.35
0.56
0.29
Bho
0.21
0.47
0.41
0.43
Got
0.26
0.35
0.34
Indo-Aryan
Gsw
0.38
0.38
0.51
Germanic
Eng
Amh
Swa
Wol
Hau
0.27
0.24
0.22
0.27
Eng
Rus
Cze
Ibo
0.19
0.12
0.41
0.3
Qpm
0.33
0.48
0.44
Kin
0.11
0.1
0.48
0.21
Hsb
0.26
0.51
0.46
Lug
0.11
0.57
0.25
Orv
0.26
0.51
0.38
Luo
0.18
0.17
0.23
0.25
Chu
0.26
0.51
0.38
Pcm
0.66
0.21
0.2
0.19
Slavic
African
For the sake of comparison, we also include the correlations for MADX -EMEA model (re- ferred to as MD-EM for brevity), which does LA ensembling purely at inference time, contrast the impact of learning ensemble weights via training in ZGUL on correlation with language relatedness. Clearly, the scores are significantly lower in this case, pointing to the fact that EMEA alone is not able to capture good notion of language related- ness, which possibly explains its weaker perfor- mance as well (as observed in Tables 2 & 3).
Figure 3:
Language relatedness between target and source languages in each group, computed as average of syntactic and genetic similarity metrics.
swer this, use Wikipedia dumps, which are available for 12 out of our 15 target languages. For each language L (1) we train its Language Adapter LA and (2) pre-train mBERT model using MLM, denoted as F. We also ex- tend ZGUL to ZGUL++ as follows: we initialize ZGUL's encoder weights with mBBRTT and fine-tune it with the additional adapter LAtst, in- serted along with other source LAs. This is trained only on source languages' task-specific data tar- get language training
For completeness, Figures 2 & 3 present the at- tention scores for the LANG2VEC network, and also the language relatedness the source lan- guages, for each test language, grouped by corre- sponding language group. The similarity in the two heat maps (depicted via color coding) again corroborates the high correlation between the at- tention scores computed by ZGUL with language relatedness.
available). We compare ZGUL++ with (1) MADXmalti-Te which trains MADX in multi-source fashion and at inference, use the L and (2) SFT++, which initializes SFT's s encoder weights with E.FRRTg and fine- tunes on the source languages' data.
4.4 Leveraging Unlabeled Target Data
Table shows Z.GUL++'s average gains of 2 Fl points over our competitive baseline SFT++ ZGUL+ achieves SOTA performance for 9
Is ZGUL useful case some amount of unlabeled data is available for the target language? To an-

---

Fo
Gsw
Hsb
Hau
lbo
Kin
Lug
Pem
Avg
MADX mADXmui Tgt
81.8
15.5
76.1
87.6
32.8
69.7
64
53.1
54.8
61.8
57.7
64.4
59.9
SFT+
81.9
16.5
76.1
91.5
34.6
75.8
71.8
68
66.9
68.1
65.9
65.1
+
82.3
17.9
82.7
91.5
35.4
77.7
71.4
69.2
64.4
67.6
73.5
70.5
Z.++ w/o Tgt LA
82.1
17.8
79.7
90.5
35.1
77.1
71.7
69.7
63.7
67.2
74.1
67.9
66.4
7.++ w/o MBERT
78.5
15.8
69.5
87.1
34.2
64.1
60.4
60.5
56.2
65.8
64.6
66.7
60.3
Unlabeled datasize
160k
2k
100k
150k
8k
350k
230k
30k
35k
8k
250k
45k
Table Fl scores after incorporating target unlabeled data in various models. Unlaheled datasize in # sentences.
12 languages while it's ablated variant (not using LA does so for two more languages. The gains for Swiss German (6.6 Fl points) are particularly impressive. Ablations for ZGUL++ show that though incorporating the LAt is beneficial with average gain of about 0.6 F1 point over all lan- guages, the crucial component is initializing with mBERTT which leads to around 7 avg. Fl point gains. Hence, the additional target pre-training step is crucial, in conjunction with utilizing the target LA, for effectively exploiting the unlabeled data, We investigate how the performance scales for each model from zero-shot setting (no unlabeled data) utilizing 100% of the Wikipedia target data. We sample 2 bins, containing 25% and 50% of the full target data. then plot the average scores over all 12 languages for each bin fig. 4.
available (12 out 15). We sample training bins
of sizes 10, 30, 70 and 100 samples. We observe
70
ZGUL
65
SFT-M
CPG
25
75
75
65-
801
55-
25
50
75
100
25
30
75
100
67.5
Figure 5: Few-shot averaged over languages in group for various few-shot bins. Top row: Germanic, Slavic. Bottom row: African, Indo-Aryan.
65.0
62.5
in Fig. 5 that ZGUL scales smoothly for lan- guage groups, maintaining its dominance over the baselines in each case, except for Slavic, where its performance is similar to CPG baseline. The relative ordering of baselines, i.e. CPG outperform- ing SFT-M for Slavic and Germanic (POS), and SFT-M outperforming CPG for African and Indo- Aryan (NER), is also maintained across the regime of few-shot samples, similar to zero-shot setting. The learning plateau is not reached the curves for either of the language groups, showing that adding more target examples would likely result in further improvement of all the models, albeit at smaller pace. We present the detailed language- wise few- shot curves in Appendix H.
60.0
57.5
ZGUL+
SFT++
55.0
MADX-Tgt
25
50
75
100
Figure 4: Average F1 scores over target languages w.r.t.
percentage of Wikipedia data used. denotes zero-shot.
We observe ZGUL++ is effective across the regime. Compared to SFT++, the gains are higher on the 100% regime, while compared to MADX- Tgt baseline, a steep gain is observed upon just using 25% data.
4.5 Few-Shot Performance
4.6 EM tuning using Target Dev set In the purely zero-shot setting, we tuned EM parameters for ensemble-based methods, i.e. MADX:ulli_EMEEs and ZGUL, on the closest source language's dev set. However, we assume
In this experiment, take the trained ZGUL and other multi-source models, i.e., CPG, SFT-M, and fine-tune them further for a few labeled examples from the train set of each target language. We do this for those test languages, whose training set is

---

Germanic
Slavic
Fo
Got
Gsw
Avg
Qpm
Hsb
Orv
Cu
Avg
Closest baseline (CPG)
77
16
63.6
52.2
46.9
76.7
64.1
35.6
55.8
MADX -EMEA
77.1
14
62.4
51.2
49.7
75.4
63.4
35.7
56.1
ZGUL
77.1
22.6
65.1
54.9
51.3
77.4
63.2
34.7
56.7
Table 6: POS Tagging Results for Germanic and Slavic groups when utilizing target dev set for EM tuning
African
Indo-Aryan
Hau
Iho
Kin
Lug
Luo
Pcm
Avg
As
Bh
Avg
Closest baseline (SFT-M)
51.3
55.7
57.9
56.0
36.0
65.0
53.7
70.8
61.4
66.1
MADX EEMAA
55.2
55
52.4
34.2
52.1
63.6
63.2
2GUL
57.5
56.9
54.8
40.9
55.2
76.9
64.5
70.7
Table 7: NER Results for African and Indo-Aryan groups when utilizing target dev set for EM tuning
the target dev set availability, which indeed holds for our target languages, can leverage for EM hyperparameter tuning. We present the results for the same tables 6 and 7. The gains of ZGUL become more pronounced, obtaining up to 1.4 avg. Fl points in the Indo-Aryan group.
those with scripts unseen in mBERT (Pfeiffer al., 2021b). Extending approach to code-switched data, in which each input token can potentially be- long to different language, is another interesting future direction.
Acknowledgements
5 Conclusion and Future Work
Mausam is supported by grants from Google, Huawei, and Jai Gupta Chair Professorship from ITT Delhi. Parag was supported by the DARPA Explainable Artificial Intelligence (XAI) Program ##N66001-17-2-4033) Mausam and Parag are also supported by 1BM SUR awards. Vipul was sup- ported by Prime Minister's Research Fellowship (PMRF). We thank IIT Delhi HPC facility for com- pute resources. We thank Keshav Kolluru for help- ful comments on earlier drafts of the paper. Any opinions, findings, conclusions or recommenda- tions expressed here are those of the authors and do not necessarily reflect the views or official poli- cies, either expressed or implied, of the funding agencies.
We present ZGUL?, novel neural model for sembling the pre-trained language adapters (LAs) for multi-source training. This performed by fus- ing the LAs at train-time to compute local token- level attention scores, along with typological lan- guage vectors to compute a second global attention score, which are combined for effective training. Entropy Minimization (EM) is carried out at test- time further refine those attention scores. Our model obtains strong performance for languages unseen by mBERT but with the seen scripts. We present various analyses including that the learnt attention weights have significant correlation with linguistic similarity between source and target, and demonstrating scalability of our model in the un- labeled data and few-shot labeled data settings as well.
Limitations
Our method incurs high inference-time overhead for each forward pass owing to Adapters being inserted in each layer. Further, the entropy mini- mization typically needs 5 or 10 forward passes for effective performance, which leads to further mul- tiplying factor with each forward pass time. These trade-offss between performance and cost in- herited from (Wang et al., 2021) itself. Language Adapters have been trained on the Wikipedia dump of source/tareet languages. This might potentially impose restrictions to extending our technique's
In the future, our approach, being task-agnostic, can be applied to more non-trivial tasks, such as generation (Kolluru et al., 2022, 2021), semantic parsing (Awasthi et al., 2023), relation extraction (Rathore et al., 2022; Bhartiya et al., 2022), and knowledge graph completion (Chakrabarti et al., 2022; Mittal et al., 2023). Our technique may com- plement other approaches for morphologically rich languages (Nzeyimana and Rubungo, 2022) and for

---

efficacy to domain-specific tasks not having suffi- cient publicly available data in that language and domain as to train a strong Adapter (E.g. Medical domain for African languages). Presently, our tech- nique cannot be tested directly on unseen scripts because our tokenizatiion/embedding layer is same that of mBERT and may become a bottleneck for Adapters to directly perform well. Our approach is not currently tested on deep semantic tasks and interation-baseee tasks owing to the lack of suitable large-scale datasets for evaluation.
North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
David Ha, Andrew Dai, and Quoe Le. 2016. Hyper- networks. arXiv preprint 1rXiv
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Anmual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 2208- 2222.
References
David Iffoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D' souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel Happy Buzaaba, Shruti Rijh- wani, Sebastian Ruder, et al. 2021. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Compitational Linguistics, 9:1 11311
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hy hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Pupers), pages 565-576, Online, Association for Computational Linguistics.
Abhijeet Awasthi, Nitish Gupta, Bidisha Samanta, Shachi Dave, Sunita Sarawagi, and Partha Talukdar. 2023. Bootstrapping multilingual semantic parsers using large language models. Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics. pages 2455- 2467, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Keshav Kolluru, Muqeeth Mohammed, Shubham Mittal, Soumen Chakrabarti, and Mausam. 2022. Aligament-augmentter consistent translation for mul- tilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2502-2517, Dublin, Ireland, Association for Computational Linguisties.
Abhyuday Bhartiya, Kartikeya Badola, and Mausam 2022. DiS- RIS-ReX:AA A multilingual dataset for distantly supervised relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics ( (Volume 2: Short Papers). pages 849-863, Dublin, Ireland. Association for Computa- tional Linguistics.
Keshav Kolluru, Martin Rezk, Pat Verga, William W Cohen, and Partha Talukdar. 2021. Multilingual fact linking. In 3rd Conference on Automated Knowledge Base Construction.
Soumen Chakrabarti, Harkanwar Singh, Shubham Lo- hiya, Prachi Jain, and Mausam. 2022. Joint comple- tion and alignment of multilingual knowledge graphs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, Decem- ber 7-11, 2022, pages 11922-1 1938.. Association for Computational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers. pages 8-14, Valencia. Spain. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Shubham Mittal, Keshav Kolluru, Soumen Chakrabarti, and Mausam. 2023. mokb6: A multilingual open Proceed-
knowledge base completion benchmark. ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 201-214. Association for Computational Linguistics.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When heing un- seen from mbeet is just the beginning: Handling languages with multilingual language models. In Proceedings of the 2021 Conference of the North
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 448-462.
Vipul Rathore, Kartikeya Badola, Parag Singla, and Mausam 2022. PARE: A simple and strong base- line for monolingual and multilingual distantly super- vised relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 340-354, Dublin, Ireland. Association for Computational Lin- guistics.
Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://githhh. com/chakki- worksssekk....kssseeeell..
Joakim Nivre, Catherie Marneffe. Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster. Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034 4043.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gert- jan Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Association for Computational Linguistics.
Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. Kinyabert: a morphology-avee -ware kinyarwanda language model. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 5347-5363.
Thuy-Trang Vu, Shahram Khadivi, Dinh Phung, and Gholamreza Haffari. 2022. Domain generalisation of NMT: Fusing adapters with leave-one -aae--on-domain-ouu training. In Findings of the Association for Computa- rional Linguistics: ACL 2022, pages 582-588.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 9946-1958
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Gra- ham Neubig. 2021. Efficient test time adapter en- sembling for low-resource language varieties. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 730-737.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021a. la. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Tryna Gurevych. 2020a. AdapterHub: framework for adapting transformers. In Proceedings the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulié, Tryna Gurevych, and Se- bastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Association for Computa- tional Linguistics.
pages
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebas- tian Ruder. 2021b. Unks everywhere: Adapting mul- tilingual language models to new scripts. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186-10203.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Gra- ham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 425 435, Brussels, Belgium. Association for Computational Linguistics.

---

Hyperparameter A Hyperparameter and Implementation Details
We use a Tesla V100 GPU (32 GB) for training all models. We use AdapterHub (Pfeiffer et al.. 2020a) for all our experiments and analysis. We mention the hyperparameter grid for SFT and all adapter-basee models (including ZGUL) in table 8. For all experiments, we report the average of 3 training runs of the models (with 3 different random seeds). We tune the Adapter Reduction Factor (RF) in the range of (3,4) but generally find to be the best for all adapter-basedd methods on all datasets. The EM algorithm used in both ZGUL and EMEA uses grid search in the range {1, 5, 10) for iterations T and (0.05, 0.1, 0.5, 1.0) for learning rate Ir.
SFT
Adapter Based
Learning Rate
[2c-5, 3c-5, 5c-5)
[5c-5, Ic-4]
Max. epochs
for NER, 5 for POS
10 for NER, for POS
Reduction Factor
NII.
[3.4]
Batch Size
{16, 32)
[16, 32)
EM Steps
NIL
(1.5. 10)
EM LR
NIL
(0.05, 0.1, 0.5, 1.0)
Few-shot LR
(le-5, le-4)
(le-5, Se-5, le-4)
Few-shot epochs
(1, 5, 10)
{1,5, 10)
Few-shot Batch Size
(1, 4, 8)
(1,4.8)
Table 8: Hyperparameter grids for ours models
A.1 Trainable Parameters Training time
Per-Epoch Training time (in mins)
Model
Params
Indo-Aryan
African
Germanic
Slavic
Avg
SFT-M
177M
36
30
19.5
CPG
253M
13
115
76.5
ZGUL
41M
36
29
Table 9: Trainable Parameters & per-epoch training time of all models
A.2 Detailed EM Algorithm used in ZGUL
Algorithm EM ulgorithm during Inference
Input: Our model's scores a (afaal) test data learning rate ir, update steps T' Output: Prediction
1: function EMEA++0
2: for <- 0 to do
3:
8r + Softmar((')
Normalize the LA scores
4:
H(z, a) Entropy(TAAA (f, 8¹) M)
D Compute Entropy Compute Token Attention gradient &. Update Token Attention weights Compute I LangVee Attention gradient P Update Lang Vec Attention weights
5:
0r VaH(s,a
ap + Update(arr
7:
gi. = Har o
8:
at
(ippatefaa,ooo gi)
9:
++1 2 1+1)
10:
end for
11:
a² + Softmaz(("))
D. Final Normalize Compute Prediction
12:
- Predict(T A loooglh, a²) M))
13:
end function
We note that we made the following amendments the originaly proposed EMEA (Wang al., 2021) for our setting (1) we made each token-level attention weights in each layer trainable, while the original EMEA had tied it layer-wise. This gives the EM method more degree of freedom in our framework compared to EMEA. (2) we have attention networks, each initialized with its respective trained attention
"hunss//githuu..com/dddppaat

---

Hau
Ibo
Kin
Lug
Luo
Pem
As
Bh
Got
Gsw
Qpm
Hsb
Orv
Cu
Avg.
Layerwise Tied
(original EMEA)
53.2
56.9
55.2
54.1
39.8
65.8
74.1
64.1
76.4
18.6
61.9
49.5
76.7
62.3
33.4
56,1
ZGUL
53.6
56.8
56.2
54.2
40.2
66.5
74.4
64.1
76.9
20.2
64.8
50.1
77.2
63
34.1
56.8
Table 10: Ablation showing why keeping token-wise trainable weights in EMEA is marginally better than tying it layer-wise
scores, while the original technique had only single network, initialized naively with uniform weights. We found amendment (1) to be marginally better than the one when we tie the token-level weights layer-wise (as EMEA), as shown in table 10. Also, amendment (2) naturally follows from the fact that our architecture has 2 components for LA ensembling.
B Robustness with XLM-Roberta
We wanted to examine whether our findings related to ZGUL's superior performance over the existing baselines carry over to other language models. Specifically, we trained ZGUL and other multi moure baselines on XLM-R Base model available on Adapterhub (Pfeiffer et al., 2020a) for Germanic and Indo-Aryan language groups We had all the adapters for languages in these two groups, except for Bengali (Indo-Aryan) which we eliminated during training. Table 11 presents our findings. ZGUL beats both the baselines on both the language groups, with gain of 1.6 pts on the Indo-Aryan group, and gain of 0.9 pts the Germanic group, compared to its closest competitor. Ablation analysis conforms the trend observed with mBERT with EM being the most important component followed by LANG2VEC and FUSION being the least important. This clearly confirms the finding that ZGUL's 's gains are not particularly restricted to a specific choice of PLM.
Indo-Aryan
Germanic
Model
As
Bh
Avg
Fo
Got
Gsw
Avg
SFT-M
60.4
60.5
60.5
78
15.4
$4.9
49,4
CPG
55.7
60.3
58
78.1
18.3
57.6
51.3
ZGUL
59.3
64.9
62.1
77.8
21.7
57.1
52.2
ZG EM
58.1
61.5
59.8
77.6
15.3
$4.9
49.3
7G-F
60.3
62.1
61.2
77.6
21
54.7
51.1
ZG L
59.7
58.7
59.2
77.8
21.1
$4.2
51
Table NER Results for Indo-Aryan (trained on En,Hi, and POS Is, De) with XLM-R-Base Adapters. means without
Germanic models (trained on En,
C Quantifying similarity between source and target languages
We use the phlogenetic and syntactic distances between source and target languages for reference." 9
C.1 Target language assignment
We justify assigning the unseen target language to a group using nearest neighbour based on phylogenetic similarity (shown in fig. 6). E.g. Hausa is genetically most similar to Amharic, so it's been assigned to the African group. On the other hand, Luo has equal genetic similarity with all source languages, so we refer to the syntactic similarity (fig. 7), for tie-break, in which it's most similar to Wolof (also English, in this case, which common in every group), and hence it's been assigned the African group
C.2 Consistency with the learnt attention scores
We also make use of these similarity metrics to validate the attention scores being learnt the Lang2 modules (explained in detail in section 4.3). E.g. for Bhojpuri, highest attention score goes to Hindi

---

Figure Heatmap Genetic similarity
Target/S Eng
Amh Swa
Wol
Rus Cze
Isl
Ger
Ara
Hin
Ben
Bho
01
02
0.14
0.14
0.14
0.37
0.37
Hau
lbo
0.5
0.17
0
0
0
0
0
Kin
0.62
0.2
0
Lug
0.62
0.2
0
0
Pcm
Ass
0.1
0.2
0.14
0.14
0.14
0.37
0.75
Qpm
0.14
0.12
Hsb
0.6
0.57
0.14
0.14
0.12
0.12
0.1
0.6
0.43
0.14
0.14
0.12
0.12
Chu
Fao
0.3
0
0.2
0.14
0.86
0.43
0.12
0.12
Got
Gsw
0.4
0.2
0.14
0.43
0.57
0.12
0.12
Figure 7: Hcatmap Syntactic similarity
Target'S Eng
Amh Swa
Wol
Rus
Cze
Isi
Ger
Urd
Hin
Ben
0.32
0.37
0.18
0.22
0.39
0.37
0.32
0.38
0.58
0.45
Hau
0.55
0.35
0.44
0.55
0.41
0.28
0.46
0.22
0.38
0.22
lbo
0.39
0.24
0.33
0.44
0.28
0.29
0.39
0.32
0.18
0.17
0.18
Kin
Lug
0.22
0.22
0.52
0.3
0.22
0.33
0.32
0.23
0.09
0.13
0.2
0.37
0.34
0.48
0.51
0.31
0.33
0.41
0.34
0.23
0.28
0.17
Pcm
0.43
0.4
0.38
0.4
0.42
Ass
0.19
0.28
0.2
0.13
0.24
0.31
0.26
0.3
0.33
0.33
0.38
Qpm
Hsb
0.43
0.43
0.4
0.38
0.43
0.34
0.4
0.42
0.31
0.42
0.33
0.43
0.43
0.4
0.38
0.43
0.34
0.4
0.42
0.31
0.42
Chu
0.43
0.4
0.38
0.43
0.4
0.31
Fao
0.39
0.3
0.26
0.38
0.36
0.46
0.43
0.41
0.26
0.25
0.31
Got
Gsw
0.37
0.34
0.24
0.36
0.37
0.46
0.33
0.44
0.4
0.35
0.37
Figure 8: Various similarity metries between source and target languages (higher the more similar). This is used to validate the assignment of the target languages to corresponding groups as well as for depicting correlation with the LA attention scores learnt by LANG2VEC component.

---

LA while for Assamese, does for Bengali. Indeed, we observe that Bhojpuri closest Hindi while Assamese being closest Bengali, based upon the average of both genetic and syntactic similarities (fig.
3). Detailed correlation between the attention scores and similarity have been discussed in section 4.3
D Datasets' details & Stats
Code
Language
Code
Language
Amh
Amharic
Hsb
Upper Sorbian
Ar
Arabic
lbo
Igbo
As
Assamese
Is
Icelandic
Be
Belorussian
Kin
Kinyarwanda
Bg
Bulgarian
Lug
Ganda
Bh
Bhojpuri
Luo
Luo
Bn
Bengali
Mr
Marathi
Cs
Czech
No
Norwegian
Cu
Old Church Slavonic
Orv
Old East Slavik
Da
Danish
Qpm
Pomak
De
German
Ru
Russian
En
English
Swa
Swahili
Fo
Faroese
Ta
Tamil
Got
Gothic
Uk
Ukrainian
Gsw
Swiss German
Ur
Urdu
Hau
Hausa
Wol
Wolof
Hi
Hindi
Table 12: Languages and their codes
Group
Task
Train set
Train set size (combined)
Dev set size
Indo-Aryan
NER
(En,Hi,Bn,Ur
55026
13003
Germanic
POS
Den
222792
28394
Slavic
POS
[En,Ru,Cs]
179043
23479
African
NER
(En,AAm,,Sw,, Wol]
19788
4073
Table 13: Training & Dev size (# Sentences)
Group
Task
Train set
LA set
Test set
Indo-Aryan
NER
Bn,
[En,Hi,Bn,Ur Ur} Den Cun, (En.Amh.Swa. Wol.
[As,Bh]
Germanic
POS
De) Cs), (Bn,Al
(Fo,Got, }
Slavic
POS
Qpm, Hsb,Orv,C.. (ing Pcm)
African
NER
Table 15: Language groups and their corresponding train, LA and test sets

---

Test Language
Script
Size
Overlap( %
Fo
Latin
1208
61.7
Got
Latin
1031
21.6
Gsw
Latin
100
54.3
Qpm
Cyrillic
635
42.9
Hsb
Latin
626
43.1
Orv
Cyrilic
4204
45
Cu
Cyrillic
1 1141
19.2
As
Bengali
100
44.1
Bh
Devanagri
102
64
Hau
Latin
570
37.1
lbo
Latin
642
32
Kin
Latin
611
32.6
Lug
Latin 419
28.3
Luo
Latin 189
35.8
Pem
Latin
600
82.3
Table 14: Test languages with their scripts, size (# Sentences) and token-level overlap (in % IOU) with source training data
Group
Task
Tag Set
Test Lang Set
Indo-Aryan
NER
(PER.LOC.ORG]
[As.Bh}
Germanic
POS
TagPOS TagPOS
Goo, Gsw
Slavic
POS
[Qpm,,sb,Orv. Ory,.
African
NER
[PER,LOC,ORG.DATTE.
Table
Language
groups and
their
corresponding tag
sets.
TagPOS
PAA..CONAADLLADPADDLVEEEEEDDDDDD NOUN, PRON, PROPN, NUM, PUNCT, AUX, SYM.
X
E
Qualitative Analysis
We also did some qualitative analysis understand from where does the actual gain of ZGUL come from. To do this, for both POS and NER tasks, we examined label-wise performance of our model, vis-a-vis the basclines. For POS, we see that ZUGL does quite well on 'NOUN' which has a huge support, resulting overall better performance for ZUGL. ZUGL does somewhat worse on labels such as 'CONJ'. Similarly, for NER, ZUGL is able to do well on labels such as 'LOC' and 'PER', resulting in overall improved performance over its competitors. We refer to the Appendix E, for further details. Carefully examining the reasons behind improved performance on certain labels (while worse performance others) is direction for future work.
E.1 Examples
Model
Labels
Sentence
Daas
Buech
laufft
besser
als
jede
vo
sine
Krimi
Gold Labels
DET
NOUN
VERB
ADV
CONJ
PRON
ADP
DET
NOUN
PUNCT
ZGUL
DET
NOUN
VERB
ADV
CONJ
PRON
ADP
DET
NOUN
PUNCT
CPG
DET
PROPN
VERB
ADV
ADP
DET
ADP
DET
NOUN
PUNCT
Table 17: Example from the gsw language

---

Model
Labels
Sentence
весь
даеть
праведпыи
Gold Labels
DET
NOUN
VERB
CONJ
ADP
NOUN
VERB
ADJ
ZGUL
DET
PUNCT
VERB
PART
ADP
NOUN
VERB
ADJ
CPG
DET
PUNCT
VERB
CONJ
ADP
NOUN
VERB
ADJ
Table 15: Example from the orv language
Figure 9: Orv language
Model
Labels
Sentence
Jami an
tsaron
Lebanon
Jakadancin
Amurka
da
ke
birnin
Beirut
Gold Lables
0
O
LOC
O
LOC
O
0
0
LOC
ZGUL
0
0
LOC
O
LOC
0
0
0
LOC
SFT-M
0
O
LOC
PER
PER
0
0
LOC
Table 18: Example from the hau langauge
Model
Labels
Sentence
Doho
pachoka
David
Maraga
chiwo
jii
11
matiyo
Gold Labels
O
PER
PER
0
0
0
O
ZGUL
O
O
PER
PER
0
O
DATE
DATE
SFT-M
O
O
PER
PER
0
0
0
0
Table 19; Example from the luo language
F
Class-wise F1 scores
Table 20 and Table 21 show the classwise scores for each of the tasks. The scores are averaged over all languages in each task.
We have used the seqeval 10 framework for evaluating all the models, which is consistent with the previous works and used by XTREME" Seqeval removes the 'B' and prefixes the labels, hence the Table has only 4 classes (F.g. 'B-PER' and 'I-PER' are mapped to same label 'PER')

---

Class
7GUL
CPG
SFT-M
Support
PART
29.2
26.3
25.7
831
CONJ
63.8
64.6
64.1
9000
ADJ
44.3
44.1
43.6
8768
ADP
78.8
77.2
75.1
9960
ADV
27.7
27.6
27.8
5830
VERB
51.2
51.1
50.4
14385
DET
40.4
40.8
37.2
3027
INTJ
10.4
3.3
13.8
211
NOUN
58.9
57.3
58
18957
PRON
41.2
41
40.4
7068
PROPN
54.8
52.8
53.2
4506
NUM
58.4
57.2
57.3
1508
PUNCT
62.3
61
62.1
7670
AUX
42
41.2
40.3
2800
SYM
11.8
13.3
12.2
33
X
3.8
3.8
2.4
228
Micro-F1
55.9
54.4
54.1
94782
Table 20: Classwise Fi-scores for POS task. Averaged over all languages
Class
ZGUL
SFT-M
CPG
Support
DATE
21.2
22.1
21.9
623
LOC
55.2
53.4
50.3
1643
ORG
53
53
48.4
1034
PER
66.7
64.2
64.1
1483
Micro-FI
57.1
55.6
53.4
4783
Table 21: Classwise Fi-scores for NER task. Averaged all languages
G Standard Deviation Details
Germanic
Slavic
Fo
Got
Gsw
Avg
Qpm
Hsb
Orv
Cu
Avg
MADXmult-EEEEA
0.5
0.8
0.4
0.5
0.3
0.2
0.2
0.7
0.4
SFT-M
0.3
1.2
0.6
0.7
0.4
0.3
0.2
0.4
0.1
CPG
0.1
0.5
0.2
0.3
0.3
0.2
0.0
0.2
0.0
ZGUL
0.0
0.5
0.4
0.1
0.5
0.5
0.1
0.3
0.1
Table 22: F1 Std. Dev. (rounded decimal) of POS Tagging for Germanic and Slavic language groups. Note:- Avg column denotes std. dev. of average Fl and not the average of std dev.

---

African
Indo-Aryan
Hau
Ibo
Kin
Lug
Luo
Pem
Avg
As
Bh
Avg
MADX AEMEAA
1.4
1.2
0.9
0.4
0.5
1.4
1.1
1.2
1.3
0.5
SFT-M
1.3
0.8
0.7
0.7
0.6
0.5
0.7
1.4
1
0.9
CPG
1.3
0.9
1.4
0.9
0.9
0.8
1.1
1.5
0.8
ZGUL
1.1
0.9
0.7
0.4
0.5
0.3
0.4
0.7
1.1
0.5
Table 23: Fl Std. Dev. (rounded to decimal) of NER for African and Indo-Aryan language groups. Note:- Avg column denotes std. average not the average of dev.
Language-wise Few Shot Performance
SFTM
C 20.L
SPMM
GPG
23U.
CRR
767
763
60
702
05
00
25
22
15
72
133
Heb
CPO
200. S7T
D-4
A3.1
S- S-IM
CFG
7011
SFT4
CPS
70.1
SFT
12
12
72
13
75
130
11
CJ
Bh
o0o ZOU,
V
SFTM
ZGJL.
15
132
lbo
Lug
Figure 10: Language-wise few-shot performance results


Reviews:
[{"review_id": "3IKdN16F0n", "reviewer": "Reviewer_itj7", "summary": "", "questions": "What’s the training objective of the train-time Ensemble? It seems it’s not clarified on section 3.1", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello reviewer, \\\nWe would like to duly address your concerns as follows- \n1. **Lack of Novelty** - \n* _AdapterFusion_ - To our knowledge, we are the first to apply AdapterFusion (Pfeiffer et al., 2021a) in context of Language Adapters (LAs) and find that this standalone technique in itself is not sufficient (as can be verified from ablation results) and hence, explored other ideas such as use of language vectors to aid the model.\n\n* _Lang2Vec_ Attention - Littell et al., 2017 just provided the typological language vectors for 1000+ languages with no insight whatsoever on how to use them. We are the first, to our knowledge, to introduce the concept of global attention over LAs as a function of those language vectors. So, we claim this to be our novel component in the model.\n\n* _EM_ at inference - Wang et al., 2021 proposed EMEA - a pure inference-time algorithm. We modified it with the following amendments for our setting - (1) We made each token-level attention weight in each layer trainable, while the original EMEA had tied it layer-wise. This gives the EM method more degree of freedom in our framework compared to EMEA. (2) We have 2 attention networks, each initialized with its respective trained attention scores, while the original technique had only a single network, initialized naively with uniform weights. \\\nTo show why amendment (1) is useful (though marginally), we tie the token-level weights layer-wise (as in EMEA) and compare with our EM approach as follows - \n| Model | Hau | Ibo | Kin | Lug | Luo | Pcm | As | Bh | Fo | Got | Gsw | Qpm | Hsb | Orv | Cu | Average |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| Layer-wise Tied as in original EMEA | 53.7 | 57.5 | 55.6 | 54.5 | 40.5 | 66 | 74.1 | 65.4 | 76.6 | 19.2 | 62.7 |50.1 |76.9 | 62.9 | 34.2 | 56.7 |\n| ZGUL | 54.1 | 57.4 | 56.6 | 54.6 | 40.9 | 66.7 | 74.4 | 65.4 | 77.1 | 20.8 | 65.6 | 50.7 | 77.4 | 63.6 | 34.9 | 57.3 | \n\nAlso, amendment (2) naturally follows from the components of our architecture itself. \n\nOverall, each of the previous works have been adapted with suitable modifications for our setting and then combined novelly to yield a final strong working solution.\n\n\n2. **Weak motivation for Lang2Vec-based attention -**\n* Our problem is to improve the current multilingual models for the low-resource languages. The AdapterFusion technique alone doesn't create much impact which motivates us to look for ways to improve the model. \n* Our core hypothesis behind using Lang2Vec-based attention is that an effective combination of source languages’ LAs should be grounded in their linguistic similarity with the target language. We indeed justify our usage of Lang2Vec module in section 4.3 in terms of the correlation scores’ comparison w.r.t. other ensemble-based approaches. This analysis provides evidence for our hypothesis that linguistic similarity should be a crucial factor to be considered for cross-lingual transfer settings. \n* This hypothesis is also reflected in our experimental results (in the form ablations - Tables 2 and 3), where not utilizing language-vector-based attention results in a drop in overall performance.\n\n3. **Wrong division of African and Indic languages -** \nWe clustered the African languages together following the MasakhaNER paper published by Adelani et al 2021. While we will clarify the language families better (as per your comments) in the camera-ready, we note that all of this does not detract from the key point of the paper: that using multiple language data as source, our method obtains better performance on target low-resource language. We test this using high-resource language data (as per line 331), which (we will clarify that) sometimes will include geographically related but linguistically not very related source languages also. \\\nThe term Indic languages can be used to refer to Indo-Aryan family. See here: https://en.wikipedia.org/wiki/Indic_languages. Nevertheless, we will clarify this also.\n\n* **Questions-** \n\n1. What’s the training objective of the train-time Ensemble? \\\n**Ans**: It’s the word-level cross-entropy loss which is the standard loss function for sequence labeling tasks. We would include this detail in our camera-ready version.\n\n* **Typos Grammar Style And Presentation Improvements:**\n\n1. \"optimal\" in Line 13 - We acknowledge that the term “optimal” in line 13 shouldn’t have been used. We will replace it with “more effective” in the camera-ready version. \n\n2. \"From the last statement is not possible to conclude that there leverage multiple LAs is needed\"  in Line 13 - This line follows from last two sentences. In line 6 we say that existing works using target LA or single related LA. In line 8 we talk about absence of target LAs for low resource languages. In line 13, we go back to the other issue (single related LA), and posit that for more effective transfer, instead of using one, we should use multiple related LAs. We will clarify this subtle lack of fluency by adding the phrase, “instead of just one source LA”.\n\n3. Line 39 - We should have concretely mentioned that the acronym SFT represents “fine-tune **all parameters** of a PLM …..”. We are fine changing this acronym to FFT (Full-model Fine-Tuning), if it aids readability.\n\n4. Line 50 - Yes, your reading is accurate. To better clarify, we will add edit line 53 to say “the latter **also** requires the **same** unlabeled data…” \n\n5. Line 63 - We do show in our experiments why multiple LAs’ combination is useful. In fact, that is at the core of our work’s thesis. E.g. note that using simply one Adapter, that is most related to a given target language (assuming target LA is not available), has weaker performance (Check out the numbers of MADX-multi-{Rel} vs ZGUL in tables 2 and 3) compared to our method. \\\nWe further use only a single English Adapter to train our model on multiple languages and use the English adapter at inference. The performance of this approach is significantly lower than ZGUL which utilizes multiple Adapters, while training on the same multilingual data. We present the numbers below:\n\n| Model | Hau | Ibo | Kin | Lug | Luo | Pcm | As | Bh | Fo | Got | Gsw | Qpm | Hsb | Orv | Cu | Average |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| ZGUL w. En LA only | 52.8 | 54.8 | 54.2 | 51.9 | 35.2 | 64.7 | 62 | 60.5\t| 74.8 | 14.9 | 60.8 | 48.9\t| 74.4 | 62.9 | 34.3 | 53.8 |\n| ZGUL | 54.1 | 57.4 | 56.6 | 54.6 | 40.9 | 66.7 | 74.4 | 65.4 | 77.1 | 20.8 | 65.6 | 50.7 | 77.4 | 63.6 | 34.9 | 57.3 |\n\n\nWe clearly observe average gains of using multiple LAs by 3.5 points compared to using only En LA. \n\n6. Table 1 - We have addressed the \"African\" and \"Indic\" issues above. Regarding Swiss German, we don’t understand your concern. Swiss German has been included in the Germanic family only. Also, English is kept a common source language in every family (including \"African\" that has Nigerian Pidgin as a test language).\n\n7. Line 342 - Yes, MADX is same as Pfeiffer et al., (2020b). We have cited this method in Line 130 though we didn’t mention the term “MADX” in that paragraph. We would do this edit in our camera-ready version.\n\nLooking at your final scores, we can’t help but wonder that our paper is being judged too harshly and unfairly. While excitement can be more subjective (though scaling NLP technology to low-resource languages is an important area of inquiry today), a soundness score of 1 feels excessive and unreasonable. \\\nIn particular, almost all issues highlighted by you are minor clarifications; we have answered them in detail, and minor edits in the next version will help clarify them. Finally, we have given a detailed response to your question of novelty, exposing aspects of our contributions that may have been missed in the first version. We hope that in light of this response, you will reassess the value our paper brings to low-resource multilingual NLP and revise your opinion positively. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work address a realistic scenario (zero shot on extremely low resource languages)\n\nThe approach is reasonable, it has been built on top of established and proven techniques such as AdapterFusion.\n\nResults, specifically on Tables 2 and 3 show an improvement in performance compared to the baselines. Also, proves that all the components are needed to clearly surpass all the baselines.", "reasons_to_reject": "The proposed approach lacks enough novelty. It proposes an ensemble of adapters, fused with an already known technique (AdapterFusion, Pfeiffer et al., 2021a), global attention brought by Language-vector-based Attention (Lang2Vec, Littell et al., 2017) and an already established algorithm (Entropy-Minimization from Wang et al., 2021).\n\nThe approach is weak in motivation. The amendment of the language-vector attention is reasonable but is nowhere mentioned as a previous problem that needed to be solved. \n\nThe division of the languages on non existing \"African\" or \"Indian\" is wrong (see comments on Table 1 below). This division is technically wrong, and unfortunately, prevents the paper to be accepted as it is.", "questions_to_authors": "What’s the training objective of the train-time Ensemble? It seems it’s not clarified on section 3.1", "paper_topic_and_main_contributions": "This research piece deals with the problem of the support of language models for unseen low-resource languages, i.e. languages that are not part of the training data, nor have a corpus available.\nTo that end the authors propose an ensemble of language-specific adapters that have some degree of relatedness with the unseen languages. During train time, the model learns to combine language adapters of previously seen languages, and uses a language specific attention to bring additional information.\nAt test time, the model uses Entropy-Minimization from Wang et al., 2021).\nThe model is evaluated in terms of two NLP tasks, part-of-speech (POS) tagging and named\nentity recognition (NER) and compared to baselines as mBERT and other variations of MADX with different training strategies, and changing the amount of languages as part of the finetuning.\nTo finish, the work include a few ablation studies, as an analysis interpretability of the attention scores and the results if there is a minimal amount of data from the low resource language to use.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 13: This sentence is confusing. Authors are dealing with the problem of zero-shot cross-lingual transfer using adapters. Given the lack of corpora from low resource languages, authors *propose* to tackle the problem by leveraging multiple linguistically related languages. From the last statement is not possible to conclude that there leverage multiple LAs is *needed*. It is proposed, and it works as shown in the rest of the paper. Finally, authors never address the optimality of their cross-lingual transfer, only show that the approach provides better performance compared to previous ones.\n\nLine 39: The training techniques are not disjoint. SFT is a training objective “fine-tune PLMs on task-specific training data in source languages”. Language adapters also fine tune on “ task-specific training data in source languages” but only a new portion of the model is updated (the adapters) while keeping the rest of the model frozen. Authors might like to change (1) definition to full-model fine-tuning instead.\n\nLine 50 This sentence is not clear. In both cases the models are trained with data from a target language. The only difference is that on the latter, the only part of the model that is trained is the adapter. \n\nLine 63 See comment on line 13. Authors propose to use multiple LAs at training time to improve cross-lingual transfer, but nothing implies that people *should* use the approach.  \n\nTable 1 The “language families” presented in this table are not completely right. There is not an “African” family. There is an Afro-Asiatic language family, but not even this category clusters all the languages in this list. Languages from this list belong to different language families: Amharic is semitic (Afro-Asiatic), Wolof is Niger-Congo and Swahili belongs to the Bantu family. \nHauza is a Chadic language, belonging to Afro-Asiatic family as well. Igbo is a Niger-Congo language, as Wolof. Kinyarwanda and Luganda are Bantu as Swahili, Luo is a Nilo-Saharan language, from a language family not included in the training.\nThere is no “Indic” family. All languages listed in this table below to different branches of the Indo-Aryan family.\nFinally, Swiss German is a dialect from German, as well as Nigerian Pidgin, which could be considered an English dialect.\n\nLine 342 MADX-{X}, seems to be based on the MAD-X approach from Pfeiffer et al., (2020b) but it's never stated in the paper. Please clarify."}}, {"review_id": "F5jDNDiUgY", "reviewer": "Reviewer_RmJP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \\\nWe express our gratitude for your valuable time to provide us with your feedback. Also, many thanks for citing these many reasons for our work to be published. \n\nWe like to address your concerns as follows - \\\n**Rewriting Limitations section:** \\\nAs far as the limitations section is concerned, yes we would move some items from future work to there in the camera-ready version. The final text for this section would be as follows -  \\\n“Our method incurs high inference-time overhead for each forward pass owing to Adapters being inserted in each layer. Further, the entropy minimization typically needs 5 or 10 forward passes for effective performance, which leads to further multiplying factor with each forward pass time. These trade-offs between performance and cost are inherited from Wang et al, 2021 itself. Language Adapters have been trained on the Wikipedia dump of source/target languages. This might potentially impose restrictions to extending our technique's efficacy to domain-specific tasks not having sufficient publicly available data in that language and domain as to train a strong Adapter (E.g. Medical domain for African languages). Presently, our technique cannot be tested directly on unseen scripts because our tokenization/embedding layer is same as that of mBERT and may become a bottleneck for Adapters to directly perform well. Our approach is not currently tested on deep semantic tasks and generation-based tasks owing to the lack of suitable large-scale datasets for evaluation.” \n\n**Missing citations and typos**: \\\nWe really appreciate you taking your time to figure out missing citations and other typos. However, it seems that these typos are not consistent w.r.t. our latest version, but are valid for our previously submitted version. It appears that you mistakenly downloaded that PDF, instead of our main submission while checking for these typos. \\\nWe humbly point you to our main submission PDF (link - https://openreview.net/pdf?id=aIp5EZeO3f ). Kindly note that most of these typos have already been fixed in the main version. A few remaining ones would be incorporated in the camera-ready version. \\\nMoreover, an important section (Section 4.4) is completely new, and was added on the basis of the comments made by the previous set of reviewers. You might enjoy reading this as well.\n\nThank you once again for your meticulous review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new approach to extend multilingual language models to newer unseen languages. \n- Relatively inexpensive approach given that the script of the language is already represented in the existing multilingual model\n- The paper performs complete ablations of all the proposed components and show the utility of the proposed approach\n- They perform qualitative analysis of the outputs and also show the interpretable nature of the features used for training. Solidifying the results obtained.\n- Authors release the code and the trained models\n", "reasons_to_reject": "- This is not a criticism of the main work, but the limitations section is non-informing. The section needs to provide crucial cases where the approach fails or show potential failure conditions. And possibly shed some light on future directions. For example, (i) the approach not tested or may not be useful for scripts unseen is a possible limitation of the work (ii) the approach not tested for language generation, etc. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The goal of the paper is to propose a new method to extend already trained multilingual models to languages unseen during the pretraining phase. In particular they make use of language adaptors from a set of related languages to extend the capabilities to newer languages. It is important to note that the scripts for unseen languages are to be already present in the existing model. The methodology is described as follows, (i) language adapters from related languages fused together with adapter fusion, (ii) additional meta data is provided in the form of syntactic and phonological properties, and (iii) entropy maximization of adapter weights for inference. The experiments are run on 15 languages across 4 language families and 2 tasks. The proposed methodology leads to improved gains across most of the languages. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Line 206: missing citation Bahdanau attention\n- Line 285 - CItation required for UDPOS\n```\n@article{nivre2018universal,\n  title={Universal Dependencies 2.2},\n  author={Nivre, Joakim and Abrams, Mitchell and Agi{\\'c}, {\\v{Z}}eljko and Ahrenberg, Lars and Antonsen, Lene and Aranzabe, Maria Jesus and Arutie, Gashaw and Asahara, Masayuki and Ateyah, Luma and Attia, Mohammed and others},\n  year={2018}\n}\n```\n\n- Line 287 - Citation required for PAN-X\n```\n@inproceedings{Pan2017,\nauthor = {Pan, Xiaoman and Zhang, Boliang and May, Jonathan and Nothman, Joel and Knight, Kevin and Ji, Heng},\nbooktitle = {Proceedings of ACL 2017},\npages = {1946--1958},\ntitle = {{Cross-lingual name tagging and linking for 282 languages}},\nyear = {2017}\n}\n```\n- Line 202, 457 - missing citation mBERT\n- Line 442, 456 - Citation missing for XLMR\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- I would suggest improving figure 1, by increasing the font size of the text, arrows, providing a legend with the acronyms and operations symbols, export higher quality figures. Preferably pdf\n- Line 082 - first usage of acronym. Use the full form here than line 144\n- Line 251 - tained -> trained\n- Line 256 - Algo 11 -> Algorithm 1 (use \\Cref)\n- Line 290 - Remove additional full stop\n- Line 292 - Add full stop. “adapters available.”\n- Table 1 - Does “Is” imply Icelandic? \n- Consider changing language codes to “each language with a BCP 47 tag sequence using a three-letter ISO 639-3 code as the base subtag and complement with ISO 15924 script subtags” for example - Hindi - “hin_Deva”. As done in https://arxiv.org/abs/2207.04672\n- Line 319 - xtreme -> XTREME\n- Line 336 - move citation to the comma, for better readability\n- Line 339 - What is POR?\n- Line 339 - Mention Appendix section/number\n- Line 351-353 - ill formed sentence\n- Table 2 & 3 - have the same degree of precision for all entries. Example - Use 70.0 instead of 70\n- Table 2 & 3 - write explanatory captions with brief descriptions of the 3 sections of the table\n- Line 371 - Remove additional full stop\n- Line 371 - Table -> Figure\n- Line 401 - indic -> Indic \n- Line 457 - m-BERT -> mBERT, maintain consistency\n- Section 4.6 - multiple instances of ZUGL -> ZGUL\n"}}, {"review_id": "sz4XiKWZ1p", "reviewer": "Reviewer_m43w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \\\nWe are grateful to you for your valuable time to provide us with your feedback. Also, many thanks for appreciating our work. \\\nWe are happy to address your concerns as follows - \n\n**Lack of Novelty** - \n* _AdapterFusion_ - To our knowledge, we are the first to apply AdapterFusion (Pfeiffer et al., 2021a) in context of Language Adapters (LAs) and find that this standalone technique in itself is not sufficient (as can be verified from ablation results) and hence, explored other ideas such as use of language vectors to aid the model.\n* _Lang2Vec_ Attention - To our knowledge, we are the first to introduce the concept of global attention over LAs as a function of  language vectors (Lang2Vec). In contrast, the CPG method used these language vectors to generate the entire parameters of a (single) shared Adapter. So, we claim this to be our novel component in the model.\n* _EM_ at inference - Wang et al., 2021 proposed EMEA - a pure inference-time algorithm. We modified it with the following amendments for our setting - (1) We made each token-level attention weight in each layer trainable, while the original EMEA had tied it layer-wise. This gives the EM method more degree of freedom in our framework compared to EMEA. (2) We have 2 attention networks, each initialized with its respective trained attention scores, while the original technique had only a single network, initialized naively with uniform weights. \\\nTo show why amendment (1) is useful (though marginally), we tie the token-level weights layer-wise (as in EMEA) and compare with our EM approach as follows - \n| Model | Hau | Ibo | Kin | Lug | Luo | Pcm | As | Bh | Fo | Got | Gsw | Qpm | Hsb | Orv | Cu | Average |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| Layer-wise Tied as in original EMEA | 53.7 | 57.5 | 55.6 | 54.5 | 40.5 | 66 | 74.1 | 65.4 | 76.6 | 19.2 | 62.7 |50.1 |76.9 | 62.9 | 34.2 | 56.7 |\n| ZGUL | 54.1 | 57.4 | 56.6 | 54.6 | 40.9 | 66.7 | 74.4 | 65.4 | 77.1 | 20.8 | 65.6 | 50.7 | 77.4 | 63.6 | 34.9 | 57.3 | \n\nAlso, amendment (2) naturally follows from the components of our architecture itself. \n\nMoreover, our understanding of local and global attention mechanisms for LA ensembling is also novel as far as cross-lingual transfer literature is concerned. \n\nOverall, each of the previous works have been adapted with suitable modifications for our setting and then combined novelly to yield a final strong working solution.\n\n**Complexity of the proposed method**: \\\nAs far as overall complexity of ZGUL is concerned, we can infer from Appendix A.1 that ZGUL has around 23% and 16 % of the trainable parameters of SFT and CPG, respectively. This parameter efficiency is decent, given the degree of gains achieved by ZGUL in various settings compared to these methods. \n\nAs promised, we release our code & models in order to enable future researchers to build further on top of our proposed methodology. We have mentioned the scope for future work as well as current limitations of our work (Sections 5 and 6, respectively) that are open challenges to be addressed by the community. We also believe that some complexity is acceptable as long as it significantly improves the results, which we show it does, in this case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, I found this paper solid. It is well-aware of previous work and combines it together in a reasonable way.\n\nThe experimentation is also relatively thorough. I appreciate all of it, but particularly appreciate section 4.4, where unlabeled target language data is incorporated into training, which is a practical setting.\n\nI also appreciate that the code will be released, which will greatly improve reproducibility.", "reasons_to_reject": "The main disadvantage of this work is that it is somewhat incremental methodologically. It is mainly based on a combination of several existing methods. Overall complexity can be warranted when it improves results (as it seems to do in this paper), but it also can be detrimental to further progress compared to simpler methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method ZGUL for improving zero-shot transfer between languages using ensembles of adaptors.\nThe method is a combination of several methods:\n1. Train-time ensembling through token-based attention\n2. Train-time ensembling through language vectors\n3. Test-time ensembling through EMEA", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the usage of adapters  to support of language models for unseen low-resource languages during training time. This is done by a proposed architecture called ZGUL that is an ensemble of  ensembles of adapters. This method also claim to use linguistical related source languages to perform this. Overall the method is sound, and tackles a high impact problem. The authors also include a strong ablation study as well as a quantitative analysis. On the reject side, there are some issues that the authors need to tackle (using the correct linguistic family classifications). There are also room for future work as, how to include new scripts, and testing it on other tasks, like generative tasks.
Decision: Accept-Main