SlowGuess/NAACL_2021_OCR huggingface dataset:
auxiliary tasks, as opposed to previous hierarchical MTL work. For all tasks we use the default settings of MaChAmp and set the loss weight of each UD subtask to 0.25.
Data For each language, we manually picked a matching UD treebank from version 2.6 (Nivre et al., 2020) (details in the Appendix). Whenever available, we picked an in-language treebank, otherwise we choose a related language. We used size, annotation quality, and domain as criteria.
We target a low-resource setup, and hence all our experiments assume no target-language training nor development data for the target task. For all our experiments we use the English training from the Facebook and Snips data, and their English development sets (all converted to match our guidelines, see § 2). We use strict-span F1 score for slots (where both span and label must match exactly) and accuracy for intents as main evaluation metric
as is standard for these tasks. $^{11}$  All reported results (including analysis and test data) are the average over 5 runs with different random seeds.
To choose the final model, we use the scores on the English development data. We are aware that this was recently shown to be sub-optimal in some settings (Keung et al., 2020), however there is no clear solution on how to circumvent this in a pure zero-shot cross-lingual setup (i.e. without assuming any target language target task annotation data).
We use multilingual BERT (mBERT) as contextual encoder for our experiments. We are also interested in low-resource setups. As all of our languages are included in pre-training of mBERT (except the de-st dialect), we also study XLM15 (XLM-MLM-TLM-XNLI15-1024), which in pre-training covers only 5 of the 13 xSID languages, to simulate further a real low-resource setup.
Table 3 reports the scores on 13 xSID languages, for 2 tasks (slot and intent prediction) and 2 pre
11 Ill-formed spans are automatically converted to match the BIO-scheme (first word with I is converted to B, and B-I spans with different labels are converted to all match the first label).
(b)
Figure 3: Performance increase over baseline for each auxiliary task with respect to the language distance (lang2vec) to English for mBERT (a) and XLM15 (b). It should be noted that the lines carry no meaning (i.e. we can not conclude performance based on language distance alone), and are shown to make trends visible.
trained language models. Languages are ordered by language distance, whenever available. Below we discuss the main findings per task.
Slots For slot filling, auxiliary tasks are beneficial for the majority of the languages, and the best performing multi-task model (aux-mlx) achieves  $+1.3$  for mBERT and  $+7.7$  for XLM15 average improvements over the baseline. By comparing mBERT and XLM15, there are significant performance drops for languages not seen during XLM15 pre-training, e.g., Danish (da) and Indonesian (id). This confirms that having a language in pre-training has a large impact on cross-lingual transfer for this task. For other languages involved in pre-training, both aux-mlx and aux-ad beat the baseline model. This supports our hypothesis that, after multilingual pre-training, auxiliary tasks (with token-level prediction both self-supervised and supervised) help the model learn the target language and a better latent alignment for cross-lingual slot filling.
Intents For intent classification the nmt-transfer model is very strong as it uses explicit translations, especially for languages not seen during pretraining. Using nmt as an auxiliary task does not come close, however, it should be noted that this only uses a fraction of the data and computational costs (see § 5.4). One main limitation of the nmt-transfer model is that it is dependant on a high-quality translation model, which in turn requires a large quantity of in-domain parallel data. Results on Kazakh (kk) confirm this, where the translation model is trained on out-of-domain data, because in-domain data was not available (§ 3.2).
Model	xSID	MultiAtis++
Slots	Intents	Slots	Intents
mBERT
base	61.00	72.91	71.12	87.28
nmt-transfer	44.133	88.2211	49.201	92.828
aux-mlm	62.329	67.021	69.151	83.330
aux-nmt	56.290	69.620	66.951	84.280
aux-ud	54.810	54.970	54.181	64.810
XLM15
base	32.05	54.15	22.71	70.63
nmt-transfer	41.858	85.0811	20.974	83.578
aux-mlm	39.7710	56.948	62.108	81.547
aux-nmt	23.850	43.682	0.310	42.120
aux-ud	37.109	46.821	52.956	80.307
Table 4: Results on the test data, average over all languages except English. Significance tested with almost stochastic order (Dror et al., 2019) test with Bonferroni correction (Bonferroni, 1936) as implemented by Ulmer (2021): 1,000 iterations  $p = 0.05$ . The number in superscript indicates the number of languages (/12) with significant improvements compared to the baseline.
4.2 Test Data
Our main findings are confirmed on the test data (Table 4), where we also evaluate on MultiAtis++. The nmt-transfer model perform superior on intents, whereas its performance on slots is worse. The best auxiliary setups are aux-mlm followed by aux-ud. Most significant gains with auxiliary tasks are obtained for languages not included in pre-training (XLM15). We believe there is a bug for aux-nmt with XLM15 (see also results in Appendix C), which we unfortunately could not resolve before submission time. Furthermore, we
Figure 4: F1 scores variants for each model, averaged over 12 languages (English is not included).
believe more tuning of machine translation can increase its viability as auxiliary task. In general our results on MultiAtis++ are lower compared to Xu et al. (2020), which is probably because they used a black-box translation model.
5.1 Effect of Language Distance
In Figure 3a we plot the performance increase over baseline for each auxiliary task with respect to the language distance when using mBERT. The results confirm that aux- MLM is the most promising auxiliary model, and clearly show that it is most beneficial for languages with a large distance to English. Figure 3b shows the same plot for the XLM15 models, and here the trends are quite different. First, we see that also for close languages, aux-ud as well as aux- MLM are beneficial. Second, the aux-ud model also performs better for the more distant languages.
5.2 Slot Detection Versus Classification
To evaluate whether the detection of the slots or the classification of the label is the bottleneck, we experiment with two varieties of the F1 score. For the first variant, we ignore the label and consider only whether the span is correct. We refer to this as unlabeled F1. For span detection, we allow for partial matches (but with the same label) which count towards true positives for precision and recall. We refer to this metric as loose F1.
Average scores with all three F1 scores for both pre-trained embeddings are plotted in Figure 4. One of the main findings is that nmt-transfer does very well on the loose F1 metric, which means that
Figure 5: Pearson correlations between target tasks performance (average of slots/intents) and 1) language distance as estimated by lang2vec, and 2) the auxiliary task. For nmt-transfer, the auxiliary task is the BLEU score of the machine translation, and for the baseline there is no auxiliary task.
it is poor at finding spans, instead of labeling them. For the other models the difference between strict and unlabeled F1 is smaller, and both can gain approximately  $5 - 10\%$  absolute score for both types of errors. The only other large difference is for aux-nmt with XLM15, which makes more errors in the labeling (unlabeled F1 is higher). An analysis of the per-language results show that this is mainly due to errors made in the Kazakh dataset.
5.3 Correlation Auxiliary Task Performance
In Figure 5 we plot the absolute Pearson correlations between the auxiliary task (auxiliary task performance can be found in Appendix C) and the target tasks performance as well as between the target tasks and the language distance (from lang2vec, see Table 3). Here we use the average of slots/intents as score for the target task. The results show that when using only datasets from languages included in the pre-trained language model (i.e., mBERT), both language distance and auxiliary task performance are competitive predictors, whereas if also new languages are considered (XLM15) auxiliary task performance is clearly a stronger predictor.
5.4 Computational Costs
All experiments are executed on a single v100 Nvidia GPU. To compare computational costs, Table 5 reports the average training time over all languages for each of the models. The training time for nmt-transfer is the highest, followed by aux-nmt, then come the leaner auxiliary tasks. The inference
Model	Time (minutes)
base	3
nmt-transfer	5,145
aux-mlm	220
aux-nmt	464
aux-ud	57
Table 5: Average minutes to train a model, averaged over all languages and both embeddings. For nmttransfer we include the training of the NMT model.
time of all the models for the SLU tasks is highly similar due to the similar architecture (except for nmt-transfer requiring fairSeq a-priori).
5.5 Case Study: Improving on de-st
Our lowest-resource language variety de-st is not included in either embeddings, and the performance on it is generally low. To mitigate this, we investigate whether a small amount of raw data could improve the aux-mlm model. We scraped 23,572 tweets and 6,583 comments from ask.fm manually identified by a native speaker, and used these as auxiliary data in the aux-mlm model. Although this data is difficult to obtain and contains a mix including standard German and others, it resulted in an increase from 49.9 to 56.2 in slot F1 scores and from 68.0 to 68.7 for intents, compared to using the German data in aux-mlm, thereby largely outperforming the baseline. This shows that even small amounts of data are highly beneficial in aux training, confirming results of Muller et al. (2021).
For related datasets, we refer to § 2.1; in this section we will discuss different approaches on how to tackle cross-lingual SLU. Work on cross-lingual SLU can broadly be divided into two approaches, whether it is based mainly on parallel data or multilingual representations. The first stream of research focuses on generating training data in the target language with machine translation and mapping the slot labels through attention or an external word aligner. The translation-based approach can be further improved by filtering the resulting training data (Gaspers et al., 2018; Do and Gaspers, 2019), post-fixing the annotation by humans (Castellucci et al., 2019), or by using a soft-alignment based on attention, which alleviates error propagation and outperforms annotation projection using external word aligners (Xu et al., 2020).
The second stream of research uses multilingual representations. Upadhyay et al. (2018) use bilingual word embeddings based on Smith et al. (2017) in a bidirectional Long Short-Term Memory model for zero-shot SLU. Recent work focuses on finding better multilingual representations. Schuster et al. (2019) use a multilingual machine translation encoder as word representations. Liu et al. (2019b) propose refining the alignment of bilingual word representations. The best performing variants use contextualized BERT variants (Chen et al., 2019; Xu et al., 2020), which we depart from.
We propose a third, orthogonal line of research: joint target-language auxiliary task learning. We hypothesize that jointly training on target language auxiliary tasks helps to learn properties of the target language while learning a related task simultaneously. We frame masked language modeling, Universal Dependency parsing and machine translation as new auxiliary tasks for SLU.
Some work on SLU showed that syntax in graph convolution networks is beneficial for slots (Qin et al., 2020). Contemporary work shows that high-resource English data helps target language modeling in sequential transfer setups (Phang et al., 2020). We focus on non-English target data for joint SLU in a single cross-lingual multi-task model instead.
We introduced xSID, a multilingual dataset for spoken language understanding with 13 languages from 6 language families, including an unstudied German dialect. xSID includes a wide variety of intent types and homogenized annotations. We propose non-English multi-task setups for zero-shot transfer to learn the target language: masked language modeling, neural machine translation and UD parsing. We compared the effect of these auxiliary tasks in two settings. Our results showed that masked language modeling led to the most stable performance improvements; however, when a language is not seen during pre-training, UD parsing led to an even larger performance increase. On the intents, generating target language training data using machine translation was outperforming all our proposed models, at a much higher computational cost however. Our analysis further shows that nmt-transfer struggles with span detection. Given training time and availability trade-off, MLM multitasking is a viable approach for SLU.
We would like to thank Yiping Duan, Kristian Nørgaard Jensen, Illona Flecchi, Mike Zhang, and Caroline van der Goot for their annotation efforts. We thank Dennis Ulmer for his help with significance testing. Furthermore, we thank Fabian Triefenbach, Judith Gaspers and the anonymous reviewers for the feedback. We also thank NVIDIA, Google cloud computing and the ITU High-performance Computing cluster for computing resources. This research is supported in part by the Independent Research Fund Denmark (DFF) grant 9131-00019B and 9063-00077B and an Amazon Faculty Research (ARA) Award.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. Proceedings of International Conference on Learning Representations (ICLR).
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Valentina Bellomaria, Giuseppe Castellucci, Andrea Favalli, and Raniero Romagnoli. 2019. Almawaveslu: a new dataset for SLU in Italian. In Proceedings of the Sixth Italian Conference on Computational Linguistics.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Carlo Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilita. *Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerici di Firenze*, 8:3-62.
Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multilingual intent detection and slot filling in a joint BERT-based model. arXiv:1907.02884 [cs]. ArXiv: 1907.02884.
Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2016. The IWSLT 2016 evaluation campaign. In
International Workshop on Spoken Language Translation.
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint: 1902.10909.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019. On the use of BERT for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 108-117, Hong Kong. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7059-7069.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint: 1805.10190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Quynh Do and Judith Gaspers. 2019. Cross-lingual transfer learning with data selection for large-scale spoken language understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1455–1460, Hong Kong, China. Association for Computational Linguistics.
Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance - how to properly compare deep neural models. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2773-2785. Association for Computational Linguistics.
Arash Einolghozati, Abhinav Arora, Lorena Sainz-Maza Lecanda, Anuj Kumar, and Sonal Gupta. 2021. El volumen louder por favor: Code-switching in task-oriented semantic parsing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1009-1021, Online. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Judith Gaspers, Penny Karanasou, and Rajen Chatterjee. 2018. Selecting machine-translated data for quick bootstrapping of a natural language understanding system. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 137-144, New Orleans - Louisiana. Association for Computational Linguistics.
Jonathan Godwin, Pontus Stenetorp, and Sebastian Riedel. 2016. Deep semi-supervised learning with linguistically motivated sequence labeling task hierarchies. arXiv preprint: 1612.09113.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2014. An empirical investigation of catastrophic forgetting in gradient-based neural networks. Proceedings of International Conference on Learning Representations (ICLR).
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Harald Hammarstrom and Sebastian Nordhoff. 2011. Langdoc: Bibliographic infrastructure for linguistic typology. Oslo Studies in Language, 3(2):31-43.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923-1933, Copenhagen, Denmark. Association for Computational Linguistics.
Xiaodong He, Li Deng, Dilek Hakkani-Tur, and Gokhan Tur. 2013. Multi-style adaptive training for robust cross-lingual spoken language understanding. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8342-8346. IEEE.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Kenji Imamura and Eiichiro Sumita. 2019. Recycling a pre-trained BERT encoder for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 23-31, Hong Kong. Association for Computational Linguistics.
Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020. Don't use English dev: On the zero-shot cross-lingual evaluation of contextual embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 549–554, Online. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint: 1907.11692.
Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019b. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International