SlowGuess/NAACL_2021_OCR huggingface dataset:
ciation for Computational Linguistics, pages 4996- 5001, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4512-4525. Association for Computational Linguistics.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3795-3805, Minneapolis, Minnesota. Association for Computational Linguistics.
Holger Schwenk and Xian Li. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA).
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan First, and Karthik Raman. 2020. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8854-8861. AAAI Press.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. 2020. Meta-dataset: A dataset of datasets for learning to learn from
few examples. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
S. Upadhyay, M. Faruqui, G. Tur, H. Dilek, and L. Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6034–6038.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5721-5727, Hong Kong, China. Association for Computational Linguistics.
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, and Pascale Fung. 2020. Meta-transfer learning for code-switched speech recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3770-3776. Association for Computational Linguistics.
Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Borje F. Karlsson, Biqing Huang, and Chin-Yew Lin. 2020. Enhanced meta-learning for cross-lingual named entity recognition with minimal resources. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9274-9281. AAAI Press.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Tables 3 and 4 show the statistics of MTOD and TyDiQA respectively per language and split.
Lang	ISO	Train	Dev	Test
English	EN	30,521	4,181	8,621
Spanish	ES	3,617	1,983	3,043
Thai	TH	2,156	1,235	1,692
Table 3: Statistics of MTOD dataset (Schuster et al., 2019) per language and split.
Lang	ISO	Train	Dev	Test
English	EN	3,326	370	440
Arabic	AR	13,324	1,481	921
Bengali	BN	2,151	239	113
Finnish	FI	6,169	686	782
Indonesian	ID	5,131	571	565
Russian	RU	5,841	649	812
Swahili	SW	2,479	276	499
Telugu	TE	5,006	557	669
For MTOD, we fine-tune PRE on English training data. We use a batch size of 32, a dropout rate of 0.3, AdamW with a learning rate of  $4\mathrm{e} - 5$ , and  $\epsilon$  of  $1\mathrm{e} - 8$ . We train for around 2000 steps. Beyond that point more training does not reveal necessary, so we perform early stopping at that point. For MONO, using a smaller learning rate of  $4\mathrm{e} - 5$  helped achieve a good convergence for that model. For all FT experiments, we use the same learning rate of  $1\mathrm{e} - 3$ , which gave a better convergence.
For QA, we use a batch size of 4, doc stride of 128, a fixed maximum sequence length of 384, and a maximum length of questions of 30 words. We use AdamW optimizer throughout all experiments, which uses weight decay of  $1\mathrm{e} - 3$ , learning rate of  $3\mathrm{e} - 5$ , and a scheduler of 4 warm-up steps.[11] We fine-tune PRE for 2 epochs and observe no more gains in performance. For all MONO and FT experiments, we use the same learning rate of  $3\mathrm{e} - 5$ . This is the same optimizer and learning rate used for the outer loops in meta-learning as well.
For X-METRA-ADA and X-METRA, we sample 2500 tasks in total for both MTOD and QA. For each task, we randomly sample  $k = q = 6$  examples from each intent class to form the support and query sets respectively (we consider all classes not only the intersection across languages). For QA, we use only one support example per query class and 6 query examples as classes. For the inner loop, we use learn2learn pre-built optimizer.
For the outer loop, we use a standard Adam optimizer. In splitting the few-shot set, we use  $75\%$  for the meta-training and  $25\%$  for the meta-adaptation for MTOD. For QA, we use  $60\%$  of the query for meta-train and the remaining for meta-adaptation.
C Results on in-House Intent Classification Dataset
We perform an extensive evaluation including other languages for intent classification. We use an in-house dataset covering 6 target languages in addition to English. Statistics of train/dev/test splits are shown in Table 5. Table 6 shows a better performance in the favor of X-METRA with an average cross-lingual gain of  $13.5\%$  in accuracy over PRE. We notice that few-shot learning on the language of interest leads to the best performance, as indicated by higher numbers on the diagonal in the confusion matrix. Evaluation on more languages shows some complicity trends between languages from the same family. In addition to that, we notice that languages like Japanese and Korean help each other where few-shot on one helps zero-shot on the other by a margin of 15.6 and 5.6 on Korean and Japanese respectively.
Table 4: Statistics of TyDiQA-GoldP dataset per language and split. Korean is excluded due to some encoding issues.
Lang	Train	Dev	Test
English	5,438	1,814	1,814
German	1,570	526	526
French	1,082	362	362
Italian	1,082	362	362
Portuguese	1,150	386	386
Japanese	1,070	358	358
Korean	938	314	314
Table 5: Statistics of In-House multilingual intent classification Dataset.
D Full results for QA
Tables 7 and 8 show the full results for F1 and Exact Match (EM) metrics for QA respectively.
Type	Model	Test on
DE	FR	IT	PT	JA	KR
PRE	EN	19.1	30.0	30.1	26.1	14.6	5.1
X-METRA	DE	34.3	33.3	30.0	30.2	13.5	8.9
FR	19.2	34.1	29.9	29.1	5.8	9.0
IT	18.3	32.2	44.4	30.2	6.7	10.2
PT	19.1	27.7	30.1	31.4	5.8	9.0
JA	24.1	25.7	33.2	26.1	30.9	20.7
KR	24.4	25.6	34.4	25.0	20.2	30.7
Table 6: X-METRA results on an In-House multilingual intent data. Bold results highlight best results for each test language.
11 Those hyperparameters are chosen based on Hu et al. (2020).
Model	Test on
AR	BN	FI	ID	RU	SW	TE
MONO
AR	74.0 ±1.1	30.1 ±2.4	50.0 ±0.8	59.5 ±1.3	48.4 ±0.8	50.8 ±1.7	24.1 ±2.7
BN	32.2 ±2.6	38.9 ±0.8	33.9 ±1.4	36.3 ±1.5	31.8 ±1.4	37.2 ±1.8	34.7 ±4.2
FI	54.2 ±2.5	30.7 ±1.3	63.3 ±1.5	52.5 ±1.7	43.0 ±2.1	48.6 ±1.7	28.7 ±2.8
ID	58.0 ±1.8	31.8 ±0.5	48.2 ±2.0	67.1 ±1.9	45.1 ±1.8	50.3 ±1.8	29.4 ±2.7
RU	50.9 ±2.3	34.5 ±2.1	45.2 ±4.2	52.0 ±4.0	54.4 ±1.3	47.1 ±2.1	30.7 ±2.5
SW	35.8 ±1.5	27.6 ±1.5	33.6 ±2.1	37.4 ±1.9	25.7 ±1.7	60.3 ±1.2	13.2 ±2.3
TE	34.0 ±0.9	38.0 ±2.2	39.5 ±0.6	35.3 ±1.1	35.9 ±1.1	43.5 ±1.0	61.4 ±1.0
FT
AR	77.0 ±0.3	36.8 ±2.9	58.8 ±0.6	67.0 ±2.7	60.9 ±0.8	52.4 ±3.6	32.0 ±1.0
BN	60.7 ±0.4	51.0 ±2.7	59.2 ±0.6	67.1 ±1.6	59.2 ±0.3	56.2 ±0.8	43.7 ±0.9
FI	60.3 ±1.9	36.7 ±1.3	70.9 ±0.4	65.7 ±1.4	62.1 ±0.5	50.9 ±1.3	36.4 ±3.6
ID	65.7 ±1.4	37.0 ±1.1	60.8 ±0.2	77.0 ±0.4	61.1 ±0.5	56.8 ±1.0	36.7 ±0.4
RU	60.9 ±2.5	37.2 ±2.0	59.0 ±2.1	66.8 ±1.3	64.8 ±0.4	55.2 ±1.8	36.8 ±1.3
SW	57.4 ±0.5	35.2 ±1.5	56.2 ±1.0	65.4 ±1.8	58.8 ±0.8	70.2 ±1.7	33.1 ±2.8
TE	54.0 ±3.2	39.1 ±2.1	54.8 ±2.3	63.5 ±2.6	58.1 ±0.9	56.9 ±1.8	65.4 ±0.6
X-METRA
AR	78.4 ±0.6	33.0 ±0.8	58.2 ±0.2	66.4 ±1.4	59.9 ±0.1	53.2 ±3.8	31.4 ±3.0
BN	56.9 ±3.2	53.2 ±0.5	56.7 ±1.4	67.4 ±1.2	56.7 ±1.3	56.0 ±0.9	41.7 ±0.6
FI	58.9 ±0.6	33.6 ±1.1	72.8 ±0.3	61.9 ±2.0	60.7 ±0.9	46.5 ±1.2	36.6 ±1.7
ID	65.8 ±0.3	35.0 ±2.2	60.5 ±0.9	77.7 ±0.2	60.4 ±1.3	57.4 ±1.1	35.3 ±0.3
RU	60.3 ±1.6	37.2 ±0.7	59.1 ±0.3	66.8 ±0.8	66.2 ±0.1	53.7 ±0.8	33.2 ±3.1
SW	58.5 ±0.0	36.9 ±1.2	56.0 ±0.2	64.8 ±0.7	58.4 ±0.4	71.9 ±0.2	33.7 ±1.5
TE	56.0 ±3.0	38.8 ±0.1	53.6 ±1.7	61.1 ±1.9	58.6 ±0.6	55.8 ±0.2	66.4 ±0.5
X-METRA-ADA
AR	76.6 ±0.1	49.6 ±1.3	63.4 ±0.4	70.9 ±0.1	60.1 ±1.0	56.8 ±0.4	42.4 ±2.5
BN	59.4 ±0.3	57.8 ±0.6	59.2 ±0.2	63.1 ±0.2	56.5 ±0.2	56.1 ±0.3	44.1 ±0.4
FI	62.8 ±1.3	50.8 ±1.3	73.0 ±0.3	65.5 ±1.2	60.1 ±0.4	54.9 ±0.3	42.5 ±0.5
ID	66.7 ±0.3	49.9 ±0.5	62.6 ±0.7	77.3 ±0.1	58.3 ±0.9	58.1 ±0.6	42.6 ±0.4
RU	62.2 ±0.7	47.6 ±1.6	63.1 ±0.2	63.4 ±0.9	66.9 ±0.1	56.0 ±1.1	43.3 ±1.2
SW	59.1 ±0.7	49.1 ±1.1	58.1 ±0.2	62.1 ±1.0	54.6 ±0.6	70.3 ±0.2	43.2 ±0.7
TE	58.2 ±2.8	52.1 ±1.7	61.5 ±1.0	62.0 ±0.5	58.2 ±0.5	59.7 ±1.4	72.8 ±0.1
Table 7: Full F1 Results on TyDiQA-GoldP between external, pre-training, monolingual and fine-tuning baselines on one hand and X-METRA and X-METRA-ADA on the other hand.
E More Ablation
Figure 8 compares between the learning curves for language-specific and joint training with respect to slot filling for both Spanish and Thai.
F More Analysis
More Downsampling Analysis Figure 9 shows a downsampling analysis on Spanish. Due to the typological similarity between Spanish and English, even lower percentages starting from  $50\%$  of the query reach a maximal performance for both intent classification and slot filling.
BERTology Analysis We analyze the degree of contribution of M-BERT layers by freezing each pair of layers separately. Our analysis is not conclusive as the performance doesn't change significantly between layers. We then proceed to freeze all layers of M-BERT to discover that linear layers are more important in refining the cross-lingual
alignment to the target language as shown by the narrow gap between freezing vs non-freezing BERT layers in Figure 10. This can be explained by the challenge of fine-tuning M-BERT alone with many layers and higher dimensionality for such a low-resource setting.
Model	Test on
AR	BN	FI	ID	RU	SW	TE
MONO
AR	57.5 ±1.5	19.7 ±2.9	35.1 ±1.0	44.2 ±1.3	25.2 ±0.9	33.8 ±1.4	14.9 ±1.7
BN	17.1 ±1.4	24.5 ±2.9	17.5 ±0.4	20.8 ±2.0	14.4 ±0.5	20.5 ±1.4	19.9 ±5.0
FI	33.7 ±4.0	15.6 ±1.6	49.8 ±1.3	35.3 ±2.3	21.4 ±1.4	26.1 ±9.9	16.5 ±3.9
ID	39.7 ±1.4	18.6 ±1.3	32.7 ±1.9	54.9 ±0.1	23.8 ±0.6	34.4 ±1.2	16.9 ±4.9
RU	30.8 ±1.9	26.3 ±4.9	29.7 ±2.4	34.9 ±4.0	37.9 ±1.6	30.7 ±3.1	19.9 ±1.9
SW	16.0 ±1.3	16.5 ±1.5	15.6 ±1.0	21.1 ±1.3	10.5 ±1.3	48.6 ±1.2	5.3 ±1.7
TE	18.8 ±2.0	26.3 ±1.5	23.8 ±2.6	21.6 ±2.5	20.4 ±1.2	26.7 ±1.7	46.3 ±1.1
FT
AR	61.3 ±1.0	26.5 ±4.4	43.1 ±1.0	52.2 ±2.0	37.9 ±2.5	35.6 ±3.3	21.0 ±3.0
BN	42.2 ±0.9	38.0 ±4.4	44.8 ±1.2	51.5 ±2.2	36.8 ±1.6	37.2 ±1.7	27.3 ±0.2
FI	43.2 ±1.8	23.6 ±1.1	56.5 ±0.6	50.8 ±2.1	40.5 ±0.8	33.5 ±1.2	20.7 ±3.3
ID	49.4 ±1.6	23.3 ±2.4	46.4 ±0.4	63.8 ±0.5	40.5 ±0.1	38.1 ±2.1	24.1 ±0.5
RU	42.6 ±2.6	24.8 ±3.3	43.5 ±2.0	52.4 ±2.3	46.5 ±0.4	37.6 ±1.5	24.5 ±1.3
SW	38.9 ±0.6	23.0 ±1.4	40.1 ±1.4	50.0 ±1.7	38.0 ±0.8	59.0 ±3.1	23.5 ±1.4
TE	36.1 ±2.2	30.0 ±2.3	40.0 ±2.5	49.4 ±2.1	38.6 ±0.9	39.0 ±1.7	49.2 ±0.5
X-METRA
AR	63.3 ±0.8	21.2 ±1.9	42.6 ±1.0	51.8 ±1.2	34.9 ±1.1	36.0 ±3.5	20.9 ±1.7
BN	29.2 ±16.5	39.0 ±1.9	41.9 ±1.6	51.1 ±1.7	34.1 ±0.4	37.1 ±1.4	25.6 ±0.2
FI	42.0 ±1.0	20.4 ±0.7	59.1 ±1.1	46.0 ±2.7	36.8 ±1.3	30.9 ±0.6	22.5 ±0.9
ID	54.8 ±7.9	20.1 ±1.5	46.1 ±1.2	65.2 ±0.5	38.5 ±1.9	39.6 ±0.8	23.1 ±1.4
RU	42.9 ±1.3	26.5 ±1.2	43.0 ±0.6	53.0 ±0.1	48.9 ±0.4	35.3 ±1.0	21.6 ±2.4
SW	39.9 ±0.4	26.0 ±1.1	40.0 ±0.7	50.3 ±0.4	38.0 ±0.9	61.4 ±0.4	23.9 ±0.7
TE	38.0 ±3.9	28.3 ±0.0	37.0 ±2.3	47.6 ±3.4	36.3 ±0.5	36.9 ±1.2	49.7 ±0.5
X-METRA-ADA
AR	55.0 ±0.3	36.0 ±3.0	43.8 ±0.5	55.2 ±0.5	35.4 ±2.6	40.0 ±0.2	31.9 ±2.2
BN	38.4 ±0.3	41.0 ±0.8	43.5 ±0.4	46.7 ±0.1	32.4 ±0.4	37.9 ±0.4	33.8 ±0.7
FI	40.9 ±1.1	34.2 ±1.1	57.9 ±1.0	49.0 ±1.4	35.3 ±0.3	38.0 ±0.7	30.0 ±1.0
ID	45.4 ±0.4	33.9 ±1.1	47.6 ±0.4	63.4 ±0.4	36.3 ±0.9	43.4 ±0.8	31.9 ±0.2
RU	39.4 ±0.1	34.8 ±1.5	45.1 ±0.5	48.6 ±0.9	47.5 ±0.3	39.3 ±1.3	33.8 ±1.3
SW	36.7 ±0.6	36.3 ±1.4	42.5 ±0.5	45.8 ±1.4	32.4 ±0.7	59.6 ±0.5	33.8 ±1.0
TE	37.9 ±1.9	38.1 ±2.6	44.9 ±1.4	48.0 ±0.3	38.8 ±0.4	43.5 ±1.6	56.4 ±0.4
Table 8: Full EM Results on TyDiQA-GoldP between external, pre-training, monolingual and fine-tuning baselines on one hand, X-METRA and X-METRA-ADA on the other hand.
(a) Slot F1 on Spanish
(b) Slot F1 on Thai
Figure 8: Ablation Study on the role of the adaptation in X-METRA-ADA compared to X-METRA (MAML with only the meta-training stage) for different languages, language-specific vs joint training. All models are compared to their fine-tuning counterparts.
Figure 9: Downsampling Analysis for Few-shot on Spanish with Different Percentages of Query data.
Figure 10: The effect of freezing BERT layers of X-METRA-ADA during few-shot on intent classification.