Venue: NAACL 2022

# Bad-x: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer

 Marinela Parovic\({}^{1}\)  Goran Glavas\({}^{2}\)   Ivan Vulic\({}^{1}\)   Anna Korhonen\({}^{1}\)

\({}^{1}\)Language Technology Lab, TAL, University of Cambridge

\({}^{2}\)CAIDAS, University of Wurzburg

{mp939,iv250,alk23}@cam.ac.uk goran.glavas@uni-wuerzburg.de

###### Abstract

Adapter modules enable modular and efficient zero-shot cross-lingual transfer, where current state-of-the-art adapter-based approaches learn specialized _language adapters_ (LAS) for individual languages. In this work, we show that it is more effective to learn _bilingual language pair adapters_ (BAs) when the goal is to optimize performance for a _particular source-target transfer direction_. Our novel bad-x adapter framework trades off some modularity of dedicated LAs for improved transfer performance: we demonstrate consistent gains in three standard downstream tasks, and for the majority of evaluated low-resource languages.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1791 - 1799

## 1 Introduction

Massively multilingual Transformers (MMTs) such as mBERT Devlin et al. (2019), XLM-R Conneau et al. (2020), and mT5 Xue et al. (2021) have dominated research in multilingual NLP and cross-lingual transfer recently. Pretrained on large amounts of unlabelled data in 100+ languages, they have been shown to achieve impressive performance for a wide range of languages and tasks, and in zero-shot cross-lingual transfer in particular Wu and Dredze (2019); K et al. (2019). However, their representational capacity is known to be limited by _the curse of multilinguality_: a trade-off between the language coverage and model capacity Conneau et al. (2020), which typically favors high-resource languages. Their limitations are thus especially pronounced in low-resource scenarios, in transfer between distant languages and towards resource-poor target languages Hu et al. (2020); Lauscher et al. (2020); Ansell et al. (2021), _inter alia_).

A standard approach to zero-shot cross-lingual transfer with MMTs (i) fine-tunes the full MMT on task-specific data in the source language and then (ii) applies it directly to make predictions in the target language Hu et al. (2020). On top of the expensive fine-tuning of the entire large model, this standard procedure also does not '_prepare_' the MMT to excel at _a particular target language_ or for a _particular source-target transfer direction_.

This has been alleviated through modular parameter-efficient adaptations of the MMTs Bapna and Firat (2019); Philip et al. (2020); He et al. (2021) which bypass full fine-tuning, most prominently through lightweight _adapters_Rebuffi et al. (2017); Houlsby et al. (2019): additional trainable parameters inserted into the MMT's layers. They have recently been used for language and task specialization of the MMTTs Pfeiffer et al. (2020), offering improved and more efficient zero-shot cross-lingual transfer.

Previous work Pfeiffer et al. (2020); Ustin et al. (2020, 2021); Vidoni et al. (2020); Ansell et al. (2021), _inter alia_) focused on creating: **1)** dedicated language adapters (LAs) for each individual language, and **2)** individual task adapters (TAs). Creating single-language LAS enables a very modular approach to cross-lingual transfer, where a source language LA (used in training) can be directly swapped with any target language LA at inference. Yet, this procedure still does not prepare nor adapt the MMT for a _particular source-target transfer direction_. Put simply, if one's incentive is to optimize the performance of a particular target language \(L_{t}\) given annotated data in a particular source language \(L_{s}\), especially under low-data regimes, one might try to capture the interplay between the two languages instead of learning separate LAs.

To address this gap, in this work we introduce the bad-x framework: bilingual adapters (Bs) for zero-shot cross-lingual transfer (see Figure 1), designed towards improving transfer performance for a particular transfer direction, with a focus on low-resource target languages. The goal of bad-x is to specialize the MMT for a particular language pair, while preserving all its existing knowledge encoded into the MMT's parameters.

We experiment with three standard tasks in cross

lingual transfer (Lauscher et al., 2020; Ansell et al., 2021): part-of-speech tagging (POS), dependency parsing (DP) and natural language inference (NLI), and with a total of 20 low-resource target languages. Our results demonstrate that trading off modularity of single-language LAs for less modular BAs (tailored for language pairs) indeed yields improved transfer performance over the current state-of-the-art (SotA) adapter-based transfer framework madx Pfeiffer et al. (2020), in all three tasks and for the large majority of target languages. We also show that, under the fixed fine-tuning budget and resources, further task performance gains can be achieved by varying the ratio of \(L_{s}\)-vs-\(L_{t}\) unannotated data when learning BAs. Finally, aiming to delve deeper into the trade-off between modularity and training efficiency, we experiment with multilingual adapters that are trained on the source language and all target languages under consideration at once. We show that such adapters, despite being more efficient to train, are unable to match the performance of their more specialized counterparts across a diverse set of target languages.

We share our code and pretrained BAs online at: [https://github.com/parovicm/BADX](https://github.com/parovicm/BADX).

## 2 Bad-x: Methodology

**Motivation and Overview.** The main idea can be summarized into the following: instead of adapting the MMT to languages \(L_{s}\) and \(L_{t}\) separately as done in the SotA adapter-based mad-x framework Pfeiffer et al. (2020), cross-lingual transfer might be more effective by adapting the MMT directly to the language pair \((L_{s},L_{t})\). This means that we learn a bilingual language-pair adapter instead of two separate monolingual LAs. We then learn a task adapter directly on top of the BA: since we focus on the zero-shot setting, this means using task-annotated examples only from \(L_{s}\) to fine-tune the TA. This procedure is summarized in Figure 1.1

Footnote 1: Inspiration for bad-x originates from neural machine translation (MMT), where bilingual adapters have been trained on parallel corpora of two languages to recover performance of a massively multilingual NMT model for high-resource languages Bapna and Firat (2019). bald-x, however, proposes bilingual adapters (i) without the use of any parallel data, (ii) with the goal to support the downstream cross-lingual transfer, and (iii) it targets low-resource target languages.

**bad-x Adapters.** bad-x adapts the mad-x adapter framework, where BAs are learnt instead of single-language LAs. The architecture of the adapter in each layer \(l\) consists of a down- and up-projection with a residual connection. More specifically, let the down-projection be a matrix \(\mathbf{D}_{l}\in\mathbb{R}^{h\times d}\) and the up-projection be a matrix \(\mathbf{U}_{l}\in\mathbb{R}^{d\times h}\) where \(h\) is a hidden size of the MMT and \(d\) is the hidden size of the adapter. Let us denote MMT's hidden state and the residual at layer \(l\) as \(\mathbf{h}_{l}\) and \(\mathbf{r}_{l}\), respectively. The adapter computation of layer \(l\) is then given by:

\[A_{l}(\mathbf{h}_{l},\mathbf{r}_{l})=\mathbf{U}_{l}(\text{ReLU}(\mathbf{D}_{ l}(\mathbf{h}_{l})))+\mathbf{r}_{l}, \tag{1}\]

with ReLU as the activation function. This formulation subsumes LAs and TAs in mad-x, as well as BAs and TAs in bad-x, where LAs/BAs receive the input from the (frozen) Transformer layer, while TAs receive the input from the (frozen) LA/BA put on top of the frozen Transformer layer (Figure 1).2

Footnote 2: mad-x also relies on so-called _invertible adapters_ for slightly improved performance, see Pfeiffer et al. (2020) for further details; they have a similar effect on bad-x, but we omit them to boost simplicity and clarity of the design and the experimental setup.

Mad-x LAs are trained via masked-language modeling (MLM) objective on the Wikipedia of the corresponding language, while TAs are trained on annotated task data. Once LA for \(L_{s}\) is available, TA is trained by stacking it on top of the fixed source LA. Transfer is done by replacing the \(L_{s}\) LA with the \(L_{t}\) LA. Unlike mad-x, bad-x trains a single bilingual adapter via MLM, alternating between the unlabelled (Wikipedia) data from both \(L_{s}\) and \(L_{t}\). The 'data alternations' are done according to a predefined _ratio_: e.g., the ratio of \(N\):1 denotes that the model would see \(N\)\(L_{s}\) sentences followed by 1 \(L_{t}\) sentence. The motivation for this is twofold: **1)** seeing a data mixture from the two languages could produce a BA that is better for transfer than having two independent LAs; **2)** LAs for low-resource \(L_{t}\)-s might otherwise overfit due to unlabelled data scarcity in \(L_{t}\), and thus could benefit from additional \(L_{s}\) data.

In bad-x, TA is then again trained on top of

Figure 1: bad-x adapter module at one MMT layer, showing the bad-x BA for one language pair (English-Wolf: En-Wo) and the POS TA. The same module (but different parameters) is added at each MMT layer.

 

the fixed BA, and the same BA-TA configuration is retained at inference, see Figure 1 again.

**Advantages and Limitations.**bad-x allows parameter-efficient transfer to arbitrary tasks and languages by learning modular bilingual and task representations. It trades-off some modularity of mad-x for increased performance and expressiveness when the goal is to perform a transfer for a fixed pair of languages. A disadvantage of bad-x with respect to modularity is that it no longer offers a zero-cost transfer (once all LAs are learnt) between all language pairs under consideration: it requires training of separate BAs for all pairs of interest. However, as we show further in SS3, bad-x might be preferable over mad-x in the cases when the goal is to improve a particular source-target direction, which is our targeted use-case.

## 3 Experiments and Results

**Tasks and Languages.** We evaluate band-X on three standard cross-lingual tasks which allow for experimentation with low-resource target languages: POS, DP, and NLI. For POS and DP, we sample ten low-resource languages from the Universal Dependencies (UD) 2.7 dataset Zeman et al. (2020), taking into account: **1)** the availability and the size of the corresponding Wikipedia; and **2)** typological diversity to ensure that different language families are covered.3 For NLI, we rely on the recent AmericasNLI dataset Ebrahimi et al. (2022), spanning ten low-resource languages from the Americas. For AmericasNLI languages, we use Wikipedia if available; otherwise we use the unlabelled data previously used by Ansell et al. (2022). English is the source language in all experiments for all tasks.4 All languages along with their language codes are listed in Table 3 in the Appendix.

Footnote 3: As a result, our ten languages cover eight different language families and five different writing systems.

Footnote 4: For UD target languages, we use the training split for evaluation if available, since it is larger than the test or evaluation splits.

### Experimental Setup

**MMT.** In all our experiments, we use mBERT, an MMT model pretrained on the Wikipedias of 104 languages Devlin et al. (2019).5

Footnote 5: mBERT demonstrated a slight edge in transfer performance over XLM-R for lower-resource languages in prior work Pfeiffer et al. (2020b).

**Training Setup: LAs, Bas.** To enable a fair comparison between mad-x and bad-x under the same training and inference conditions, we train our own mad-x LAs from scratch with the MLM objective on monolingual Wikipedias: training is run for 25,000 steps, with a batch size of 64 and a learning rate of \(1e-4\). We evaluate the LAs every 500 training steps and finally choose the LA that yields the lowest perplexity, as evaluated on the 5% of the Wikipedia data that acts as a validation set.

Pfeiffer et al. (2020b) empirically established that strong task performance of mad-x on low-resource languages can be achieved already after 20,000 LA training steps, and that longer training offers only modest to negligible performance gains. Driven by their findings, we train mad-x LAs for 25,000 iterations due to computational constraints, a large number of experiments, and the low-resource nature of our target languages.

bad-x BAs are trained on the Wikipedia data of both \(L_{s}\) and \(L_{t}\). The standard bad-x variant termed **Balanced bad-x** (also bad-x 1:1) is trained by alternating one batch of the \(L_{s}\) data (i.e. English) followed by one batch of the \(L_{t}\) data, for 50,000 iterations (i.e., this way we match the total number of iterations performed by training mad-x \(L_{s}\) and \(L_{t}\) LAs for 25,000 iterations each), and we adopt all the hyperparameters from mad-x LA training. We select as the final BA the one with the lowest \(L_{t}\) perplexity. Bilinguality of the bad-x BAs allows us to directly train TA on top of it and perform the inference with the same configuration.

**Multilingual Adapter (MA).** Given \(N\) target languages of interest, one could alternatively train a multilingual adapter on unlabelled text from \(L_{s}\) and all \(N\) target languages \(L_{t}\): while this is more computationally efficient than both Band-X and MAD-X6 it could, presumably, again lead to the "curse of multilinguality", as the adapter parameters would be shared across \(N\)+1 languages. On the other hand, MA has the chance to benefit from similarities between target languages (especially in the case of AmericasNLI). Concretely, we train two multilingual adapters: one for the set of UD languages and the other for the set of AmericansNLI languages. Multilingual UD adapter is trained by alternating one batch of English Wikipedia and one batch from each of 10 UD languages' Wikipedia for 50,000 iterations. Multilingual AmericasNLI 

adapter is obtained following the same procedure, only using Wikipedias of the NLI target languages.

**Training Setup: TAs.** For POS and DP, TA is trained by stacking it on top of the source (i.e. English) LA (with mad-x), the English-\(L_{t}\) BA (with bad-x) or the multilingual adapter MA and performing 15,000 steps with a batch size of 8 and a learning rate of \(5e-5\). We evaluate the TAs every 250 steps on English validation set, and select as the final TAs the ones with the best accuracy (POS) and LAS score (DP). The adapter reduction factor Pfeiffer et al. (2020a) is 2 for LAs and BAs and 16 for TAs. For AmericasNLI, we train its TA using the English MultiNI data Williams et al. (2018) following the setup of Ebrahimi et al. (2022): 5 epochs with a batch size of 32, and a learning rate of \(2e-5\). We evaluate the TA every 625 steps and choose the one with the best English validation accuracy.

**bad-x: BA Variants.** Besides Balanced bad-x, we consider other variants of bad-x BAs that differ in the data ratios between \(L_{s}\) and \(L_{t}\); we denote these variants as **bad-x 1:\(N\)**, where 1 batch of \(L_{s}\) data is followed by \(N\) batches of \(L_{t}\) data, and vice versa: **bad-x \(N\):1**. With these variants, we aim to answer the following question: given a fixed number of MLM training steps (i.e., a fixed computational budget) for BAs, is it possible to further impact/improve transfer performance? Is the optimal data sampling ratio task-dependent?

### Results and Discussion

The results for all languages and tasks with mad-x and Balanced bad-x are summarized in Table 1, with additional results in the appendix. As a general trend, we observe that the proposed Balanced bad-x variant outperforms mad-x and MAs over a majority of languages and across all three tasks: besides offering higher average results, we also report gains on 8/10 (POS; accuracy), 9/10 (DP; UAS), and 8/10 (NLI; accuracy) target languages. This confirms the positive impact of BA training, which is able to capture additional interactions of each language pair, in lieu of LA training.

**Performance across Tasks.** In particular, bad-x gains on average 1.06% in accuracy and 0.66% in \(F_{1}\) compared to mad-x on POS task. It outscores the multilingual adapter on POS even more: 3.55% in accuracy and 2.87% in F1 on average. The gains over mad-x are even more pronounced on the more complex DP task, which shares the target languages set with POS: bad-x outperforms mad-x on average with a gap of 2.62% in UAS and 2.38% in LAS scores. The gain is particularly high for Wolof, a West-African language spoken by more than five million people, with ~9% improvement over mad-x in both UAS and LAS scores. Wolof is also a language with one of the highest gains in POS. In the DP task, bad-x achieves similar gains over the multilingual adapter: 2.22% UAS and 2.82% in LAS scores on average. Multilingual adapter achieves high scores on Wolof, which re-establishes Wool as a language that highly benefits from the involvement of other languages during training. We also observe the superiority of Balanced bad-x over mad-x on NLI, now on another set of low-resource languages, with average accuracy gains of 2.4%. The highest improvement of 6.67% is observed for Wixarika.

**Performance across Languages.** Importantly, we find that improvements in all three tasks are met for target languages coming from diverse language families (e.g., for Uralic, Indo-European, NigerCongo, Turkic, Aymaran families) and with diverse typological traits. We speculate that stacking TAs on top of BAs instead of an English-specialized LA forces the model to also take into account information from the target language, which mitigates overfitting to English-only language properties. Furthermore, coupling two languages in the BA training might also allow for some information flow between the languages (e.g., some sharing at lexical level). This also might provide a positive impact on transfer performance, while this effect cannot be achieved with individual LAs as in mad-x. Multilingual adapters lag behind mad-x and bad-x as they aim to fit too many languages into a small number of adapter parameters, which demonstrates the necessity of language and especially language-pair specialization when performance for a particular source-target transfer direction is paramount.

**bad-x Variants.** Figure 2 shows the 'average-across-languages' scores for mad-x and for all tested bad-x variants (based on data sampling ratios at BA training; SS3.1). The results indicate several findings. First, all bad-x variants outperform mad-x on all three tasks on average. Second, there is no single best-performing bad-x variant for all tasks, that is, the 'winning' variant seems to be task-dependent. In particular, DP benefits the most from 5:1 sampling, while for POS and NLI the 1:2 variant outscores the others although DP 

and POS share exactly the same BA training data.

Note that, due to computational constraints, we did not extensively search for the best sampling ratios of the source and target language during BA training, thus the optimal strategy might not be covered by our experiments. However, these findings warrant further investigation in future work.

**Multiple Runs.** To validate that our results hold in the presence of different parameter initializations, we perform a comparison of mad-x and Balanced bad-x when the scores are averages of 3 runs with the same random seeds both for mad-x and bad-x. Due to computational constraints, we select only a subset of languages for this evaluation. In particular, we choose 4 UD (myv, kpv, te and wo) and 4 AmericasNLI languages (cni, gn, shp and hch) and compare mad-x and Balanced bad-x by taking the average scores obtained from 3 runs. The results are shown in Table 2, and again point to bad-x's superiority over mad-x in terms of transfer performance in all three tasks.

## 4 Conclusion

We have presented bad-x, a novel adapter-based framework for zero-shot cross-lingual transfer. bad-x targets improving transfer performance for particular fixed source-target transfer directions through the introduction and use of dedicated bilingual language-pair adapters (BAs). The effectiveness of the BAs and the bad-x framework has been demonstrated on three standard transfer tasks, across a plethora of low-resource languages. In future work, we will experiment with more efficient approaches to bilingual adapters, e.g., based on contextual parameter generation Ansell et al. (2021), and port the bad-x framework to more languages and tasks.

## Acknowledgements

\(\blacksquare\) This work has been supported by the ERC PoC Grant MultiConvAI (no. 957356) and a Huawei research donation to the University of Cambridge. Marinela Parovic is supported by Trinity College External Research Studentship. We would like to thank Alan Ansel and Jonas Pfeiffer for interesting discussions and suggestions.

\begin{table}
\begin{tabular}{l l r r r r r r r r r r} \hline \hline
**Task** & **Method** & af & bm & eu & myv & kpv & mt & hr & te & ug & wo & **avg** \\  & mad-x & **85.43*** & 41.61 & 58.90 & 66.84 & 47.63 & **69.94*** & 52.65 & 75.27 & **47.07*** & 61.78 & 60.71 \\ POS & ma & 84.49 & **42.89*** & 58.87 & 61.59 & 42.95 & 62.24 & **52.73** & 75.41 & 40.79 & 61.05 & 58.50 \\  & bad-x & 84.94 & 42.40 & **59.48*** & **68.11*** & **50.26*** & 69.40 & 52.35 & **75.63*** & 46.67 & **64.50*** & **61.37** \\  & mad-x & 54.50 & 12.17 & 32.06 & 33.64 & 23.01 & **44.16*** & 27.49 & 48.54 & **15.13** & 24.84 & 31.55 \\ DP & ma & 55.08 & **14.91*** & 31.33 & 33.36 & 17.79 & 40.32 & 26.19* & 47.91 & 13.08 & 31.17 & 31.11 \\  & bad-x & **55.75*** & 14.47 & **33.30*** & **37.74*** & **25.81*** & 42.45 & **29.19*** & **51.51*** & 15.11 & **33.93*** & **33.93** \\  & & cni & xym & bzd & gn & NAH & otO & quy & tar & snP & nCh & **avg** \\  & mad-x & 42.53 & 46.67 & 44.53 & 54.53 & 47.56 & 41.18 & **49.47** & 37.87 & 41.73 & 38.40 & 44.45 \\ NLI & ma & 42.67 & 38.30 & 44.00 & 42.53 & 44.17 & 40.64 & 43.33 & **42.40*** & 46.67 & 42.80 & 42.80 \\  & bad-x & **48.13*** & **47.33*** & **44.93** & **58.00*** & **48.24*** & **41.44** & 49.33 & 38.93 & **47.07*** & **45.07*** & **46.85** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results of multilingual adapters (ma), mad-x, and bad-x (Balanced bad-x, 1:1) on all tasks and languages. Standard evaluation measures: \(F_{1}\) for POS, LAS for DP, and accuracy for NLI. **Bold**: the best performing approach. An asterisk (*) indicates significant gains over the the other two competitor models (Student’s \(t\)-test with \(p=0.05\)).

Figure 2: The average accuracy (POS and NLI) and UAS scores of mad-x and different bad-x variants (see §3.1). Full results are available in Appendix C.

 



## References

* A. Ansell, E. Maria Ponti, J. Pfeiffer, S. Ruder, G. Glavas, I. Vulic, and A. Korhonen (2021)MAD-G: multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 4762-4781. External Links: Link Cited by: SS3.
* A. Bapna and O. Firat (2019)Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 1538-1548. External Links: Link Cited by: SS1, SS2.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020)Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8440-8451. External Links: Link, Document Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link, Document Cited by: SS1, SS2.
* A. Ebrahimi, M. Mayer, A. Oncevay, V. Chaudhary, L. Chiruzzo, A. Fan, J. Ortega, R. Ramos, A. Rios, I. Vladimir, G. A. Gimenez-Lugo, E. Mager, G. Neubig, A. Palmer, R. A. Coto Solano, N. Thang Vu, and K. Kann (2022)AmericasNLI: evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL 2022), Dublin, Ireland, pp. . External Links: Link Cited by: SS1, SS2.
* J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig (2021)Towards a unified view of parameter-efficient transfer learning. CoRRabs/2110.04366. External Links: Link, 2110.04366 Cited by: SS1, SS2.
* N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97, pp. 2790-2799. External Links: Link, Document Cited by: SS1, SS2.
* J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson (2020)XTREME: a massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, pp. 119 of _Proceedings of Machine Learning Research_, pages 4411-4421. Note: PMLR External Links: Link Cited by: SS1, SS2.
* K. K. Kim, Z. Wang, S. Mayhew, and D. Roth (2019)Cross-lingual ability of multilingual BERT: an empirical study. CoRRabs/1912.07840 External Links: Link, 1912.07840 Cited by: SS1, SS2.
* A. Leuscher, V. Ravishankar, I. Vulic, and G. Glavas (2020)From zero to hero: on the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4483-4499. External Links: Link, Document Cited by: SS1, SS2.
* J. Pfeiffer, A. Ruckle, C. Poth, A. Kamath, I. Vulic, S. Ruder, K. Cho, and I. Gurevych (2020)AdapterHub: a framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp. 46-54. External Links: Link, Document Cited by: SS1, SS2.
* J. Pfeiffer, I. Vulic, I. Gurevych, and S. Ruder (2020)RAD-X: an adapter-based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 7654-7673. External Links: Link, Document Cited by: SS1, SS2.
* J. Philip, A. Berard, M. Galfe, and L. Besacier (2020)Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4465-4470. External Links: Link, Document Cited by: SS1, SS2.
* S. Rebuffi, H. Bilen, and A. Vedaldi (2017)Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, Utrecht, Germany, pp. . External Links: Link, ISBN 978-1-469-3898-3, Link, Document Cited by: SS1, SS2.
* A. Ustun, A. Berard, L. Besacier, and M. Galle (2021)Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

650-6662, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Ustun et al. (2020) Ahmet Ustun, Arianna Bisazza, Gosse Bouma, and Gertan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2302-2315, Online. Association for Computational Linguistics.
* Vidoni et al. (2020) Marko Vidoni, Ivan Vulic, and Goran Glavas. 2020. Orthogonal language and task adapters in zero-shot cross-lingual transfer. _CoRR_, abs/2012.06460.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
* Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddharn, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498. Online. Association for Computational Linguistics.
* Zeman et al. (2020) Daniel Zeman, Joakim Nivre, et al. 2020. Universal dependencies 2.7. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.

 

Details of the Experimental Setup

**Computing Infrastructure.** All experiments were run on a single NVIDIA GeForce RTX 3090 GPU; training one bad-x BA or multilingual LA for 50,000 iterations took around 24 hours (mad-x LA for 25,000 steps took around 12 hours). Training of any TA took less than two hours. Evaluation is performed within the AdapterHub framework Pfeiffer et al. (2020).

**Hyperparameters.** All hyperparameters were taken from Pfeiffer et al. (2020), as discussed in the main paper, and no hyperparameter search was done. All reported results except those in table 2 are from a single run.

## Appendix B Languages

The list of languages in each task along with their language codes is provided in Table 3.

## Appendix C Bad-x: Full results

Full results on all languages for mad-x and all bad-x variants are given in Tables 4, 5 and 6 for POS, DP and NLI, respectively.

 



\begin{table}
\begin{tabular}{l c r r r r r r r r r r} \hline \hline
**Method** & AF & BM & EU & MYV & KPV & MT & MR & TE & UG & WO & **avg** \\ \cline{2-13} MAD-x & 66.64/54.90 & 35.19/12.17 & 54.71/32.06 & 55.18/33.64 & 43.74/23.01 & 60.74/44.16 & 46.08/27.49 & 63.77/48.54 & 33.74/15.13 & 46.04/24.84 & 50.58/31.55 \\ bad-x 1: 2: 67.83/55.42 & 37.70/15.10 & 53.88/31.84 & 58.46/38.07 & 44.20/29.25 & 61.79/43.29 & 48.71/30.53 & 68.93/52.58 & 33.03/14.94 & 51.72/30.77 & 52.62/53.35 \\ bad-x 1: 1: 68.02/55.75 & 37.20/14.47 & 55.42/33.30 & 58.61/37.44 & 43.47/25.81 & 61.87/42.45 & 48.01/29.19 & 68.69/51.51 & 35.07/15.11 & 54.82/33.93 & 53.02/33.93 \\ bad-x 2: 61.7 & 61.75/15.70 & 36.36/14.11 & 54.78/33.40 & 57.87/37.58 & 43.04/22.81 & 63.18/34.68 & 49.87/30.40 & 66.90/49.98 & 34.31/14.40 & 55.66/36.93 & 53.07/33.58 \\ bad-x 5: 1 & 68.03/56.03 & 36.56/14.40 & 53.65/13.84 & 62.03/42.22 & 45.86/24.67 & 62.68/42.28 & 49.52/30.40 & 66.65/48.54 & 35.74/14.31 & 57.08/36.78 & 53.78/34.15 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of and all Bad-x variants on POS. Scores are accuracy/F1. The last column is the average score over all languages.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Tasks** & **Languages** & & & & & & & & & & & \\ \multirow{2}{*}{POS, DP} & Afrikaans & Bambara & Basque & Erzya & Komi-Zyryan & Maltese & Marathi & Telugu & Uyghur & Wolof \\  & af & bm & eu & myv & kpv & mt & mr & te & ug & wo \\ Treebank & AfriBooms & CRB & BDT & JR & Lattice & MUDT & UFAL & MTG & UDT & WTB \\ \hline NLI & Ashääninka & Aymara & Bribri & Guarani & Nähuatl & Otomi & Quechua & Rāramuri & Shipibo-Konibo & Wixarika \\  & cnI & aYM & bzd & gn & nah & otO & quy & tar & ship & hch \\ \hline \hline \end{tabular}
\end{table}
Table 3: Lists of tasks with all of the languages.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l} \hline \hline
**Method** & car & bzd & gn & nan & oto & quy & tar & ship & hech & avg \\ \multicolumn{1}{c}{mad-x} & 42.53 & 46.67 & 44.53 & 54.53 & 47.56 & 41.18 & 49.47 & 37.87 & 41.73 & 38.40 & 44.45 \\ bad-x 1: 2: 45.60 & 52.13 & 45.47 & 56.93 & 45.53 & 45.05 & 54.13 & 39.07 & 47.20 & 45.47 & 47.66 \\ bad-x 1: 1: 48.13 & 47.33 & 44.93 & 58.00 & 48.24 & 41.44 & 49.33 & 38.93 & 47.07 & 45.07 & 46.85 \\ bad-x 2: 1: 46.27 & 50.27 & 46.13 & 51.47 & 48.10 & 40.51 & 53.20 & 37.60 & 48.13 & 43.60 & 46.53 \\ bad-x 5: 1: 43.20 & 52.13 & 45.73 & 56.27 & 46.75 & 43.18 & 55.73 & 37.47 & 50.40 & 42.53 & 47.34 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of mod-x and all bad-x variants on NLI. Scores are accuracy. The last column is the average score over all languages.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l} \hline \hline
**Method** & AF & BM & EU & MYV & KPV & MT & MR & TE & UG & WO & **avg** \\ \multicolumn{1}{c}{mad-x} & 66.64/54.50 & 35.19/12.17 & 54.71/32.06 & 55.18/33.64 & 43.74/23.01 & 60.74/44.16 & 46.08/27.49 & 63.77/48.54 & 33.74/15.13 & 46.04/24.84 & 50.58/31.55 \\ bad-x 1: 2: 67.83/55.42 & 37.70/15.10 & 53.88/31.84 & 58.46/38.07 & 44.20/29.25 & 61.79/43.29 & 48.71/30.53 & 68.93/52.58 & 33.03/14.94 & 51.72/30.77 & 52.62/53.35 \\ bad-x 1: 1: 68.02/55.75 & 37.20/14.47 & 55.42/33.30 & 58.61/37.44 & 43.47/25.81 & 61.87/42.45 & 48.01/29.19 & 68.69/51.51 & 35.07/15.11 & 54.82/33.93 & 53.02/33.93 \\ bad-x 2: 61.7 & 61.75/15.70 & 36.36/14.11 & 54.78/33.40 & 57.87/37.58 & 43.04/22.81 & 63.18/34.68 & 49.87/30.40 & 66.90/49.98 & 34.31/14.40 & 55.66/36.93 & 53.07/33.58 \\ bad-x 5: 1 & 68.03/56.03 & 36.56/14.40 & 53.65/13.84 & 62.03/42.22 & 45.86/24.67 & 62.68/42.28 & 49.52/30.40 & 66.65/48.54 & 35.74/14.31 & 57.08/36.78 & 53.78/34.15 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of mod-x and all bad-x variants on DP. Scores are UAS/LAS. The last column is the average score over all languages.