SlowGuess/NAACL_2022_OCR huggingface dataset:
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulic. 2022. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL 2022), Dublin, Ireland. Association for Computational Linguistics.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 4762–4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ankur Bapna and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548, Hong Kong, China. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir, Gustavo A. Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando A. Coto Solano, Ngoc Thang Vu, and Katharina Kann. 2022. AmericasNLI: evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL 2022), Dublin, Ireland. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. CoRR, abs/2110.04366.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2019. Cross-lingual ability of multilingual BERT: An empirical study. CoRR, abs/1912.07840.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Jerin Philip, Alexandre Berard, Matthias Galle, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465-4470, Online. Association for Computational Linguistics.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ahmet Üstün, Alexandre Berard, Laurent Besacier, and Matthias Galle. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
6650-6662, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Association for Computational Linguistics.
Marko Vidoni, Ivan Vulic, and Goran Glavaš. 2020. Orthogonal language and task adapters in zero-shot cross-lingual transfer. CoRR, abs/2012.06460.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Daniel Zeman, Joakim Nivre, et al. 2020. Universal dependencies 2.7. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
A Details of the Experimental Setup
Computing Infrastructure. All experiments were run on a single NVIDIA GeForce RTX 3090 GPU; training one BAD-X BA or multilingal LA for 50,000 iterations took around 24 hours (MAD-X LA for 25,000 steps took around 12 hours). Training of any TA took less than two hours. Evaluation is performed within the AdapterHub framework (Pfeiffer et al., 2020a).
Hyperparameters. All hyperparameters were taken from (Pfeiffer et al., 2020b), as discussed in the main paper, and no hyperparameter search was done. All reported results except those in table 2 are from a single run.
B Languages
The list of languages in each task along with their language codes is provided in Table 3.
C BAD-X: Full results
Full results on all languages for MAD-X and all BAD-X variants are given in Tables 4, 5 and 6 for POS, DP and NLI, respectively.
Tasks	Languages
POS, DP	Afrikaans	Bambara	Basque	Erzya	Komi-Zyryan	Maltese	Marathi	Telugu	Uyghur	Wolof
AF	BM	EU	MYV	KPV	MT	MR	TE	UG	WO
Treebank	AfriBooms	CRB	BDT	JR	Lattice	MUDT	UFAL	MTG	UDT	WTB
NLI	Asháninka	Aymara	Bribri	Guarani	Náhuatl	Otomí	Quechua	Raramuri	Shipibo-Konibo	Wixarika
CNI	AYM	BZD	GN	NAH	OTO	QUY	TAR	SHP	HCH
Table 3: Lists of tasks with all of the languages.
Method	AF	BM	EU	MYV	KPV	MT	MR	TE	UG	WO	avg
MAD-X	86.97/85.43	45.92/41.61	70.68/58.90	72.92/66.84	57.18/47.63	74.12/69.94	57.58/52.65	79.81/75.27	60.26/47.07	68.00/61.78	67.34/60.71
BAD-X 1:2	87.09/85.53	48.40/43.91	72.03/60.88	75.55/69.49	57.88/48.43	72.79/68.40	59.45/54.31	81.33/76.63	63.86/46.53	71.78/65.74	69.02/61.98
BAD-X 1:1	86.68/84.94	47.05/42.40	71.16/59.48	74.52/68.11	59.67/50.26	73.54/69.40	57.64/52.35	80.40/75.63	62.86/46.67	70.48/64.50	68.40/61.37
BAD-X 2:1	87.01/85.26	45.59/40.96	71.58/60.19	75.37/69.28	58.22/49.41	73.85/70.21	59.33/54.24	80.28/75.56	62.67/46.99	71.92/65.99	68.58/61.81
BAD-X 5:1	86.98/85.44	48.67/44.35	70.75/59.76	75.98/69.59	57.68/48.52	71.62/67.66	58.81/54.21	79.28/74.58	58.39/43.45	70.30/64.55	67.85/61.21
Table 4: Results of MAD-X and all BAD-X variants on POS. Scores are accuracy/F1. The last column is the average score over all languages.
Method	AF	BM	EU	MYV	KPV	MT	MR	TE	UG	WO	avg
MAD-X	66.64/54.50	35.19/12.17	54.71/32.06	55.18/33.64	43.74/23.01	60.74/44.16	46.08/27.49	63.77/48.54	33.74/15.13	46.04/24.84	50.58/31.55
BAD-X 1:2	67.83/55.42	37.70/15.10	53.88/31.84	58.46/38.07	44.20/22.95	61.79/43.29	48.71/30.53	68.93/52.58	33.03/14.94	51.72/30.77	52.62/33.55
BAD-X 1:1	68.02/55.75	37.20/14.47	55.42/33.30	58.61/37.74	44.34/25.81	61.87/42.45	48.01/29.19	68.69/51.51	35.07/15.11	54.82/33.93	53.20/33.93
BAD-X 2:1	67.81/55.70	36.35/14.11	54.78/33.40	58.78/37.58	43.04/22.81	63.18/43.68	49.88/30.40	66.90/49.98	34.31/14.40	55.66/33.69	53.07/33.58
BAD-X 5:1	68.03/56.03	36.56/14.40	53.65/31.84	62.03/42.22	45.86/24.67	62.68/42.28	49.52/30.40	66.65/48.54	35.74/14.31	57.08/36.78	53.78/34.15
Table 5: Results of MAD-X and all BAD-X variants on DP. Scores are UAS/LAS. The last column is the average score over all languages.
Method	CNI	AYM	BZD	GN	NAH	OTO	QUY	TAR	SHP	HCH	avg
MAD-X	42.53	46.67	44.53	54.53	47.56	41.18	49.47	37.87	41.73	38.40	44.45
BAD-X 1:2	45.60	52.13	45.47	56.93	45.53	45.05	54.13	39.07	47.20	45.47	47.66
BAD-X 1:1	48.13	47.33	44.93	58.00	48.24	41.44	49.33	38.93	47.07	45.07	46.85
BAD-X 2:1	46.27	50.27	46.13	51.47	48.10	40.51	53.20	37.60	48.13	43.60	46.53
BAD-X 5:1	43.20	52.13	45.73	56.27	46.75	43.18	55.73	37.47	50.40	42.53	47.34
Table 6: Results of MAD-X and all BAD-X variants on NLI. Scores are accuracy. The last column is the average score over all languages.