SlowGuess/NAACL_2021_OCR huggingface dataset:
Rob van der Goot<sup>1</sup>, Ibrahim Sharaf<sup>2</sup>, Aizhan Imankulova<sup>3</sup>, Ahmet Üstün<sup>4</sup>, Marija Stepanović<sup>1</sup>, Alan Ramponi<sup>5,6</sup>, Siti Oryza Khairunnisa<sup>3</sup>, Mamoru Komachi<sup>3</sup>, Barbara Plank<sup>1</sup>
<sup>1</sup>IT University of Copenhagen, <sup>2</sup>Factmata, <sup>3</sup>Tokyo Metropolitan University, <sup>4</sup>University of Groningen, <sup>5</sup>University of Trento, <sup>6</sup>Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI)
{robv,bap1}@itu.dk
The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual (x) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification.
Digital conversational assistants have become an integral part of everyday life and they are available, e.g., as standalone smart home devices or in smartphones. Key steps in such task-oriented conversational systems are recognizing the intent of a user's utterance, and detecting the main arguments, also called slots. For example, for an utterance like "Add reminder to swim at 11am tomorrow", these key Natural Language Understanding (NLU), or Spoken Language Understanding (SLU) tasks are illustrated in Figure 1. As slots depend on the intent type, leading models typically adopt joint solutions (Chen et al., 2019; Qin et al., 2020).
Despite advances in neural modeling for slot and intent detection (§ 6), datasets for SLU remain limited, hampering progress toward providing SLU for many language varieties. Most avail
Add reminder to swim at 1 am tomorrow
intent: add Reminder
Figure 1: English example from xsID annotated with intents (add Reminder) and slots (todo, datetime). The full set of languages is shown in Table 2.
able datasets either support only a specific domain (like air traffic systems) (Xu et al., 2020), or are broader but limited to English and a few other languages (Schuster et al., 2019; Coucke et al., 2018). We release xSID, a new benchmark intended for SLU evaluation in low-resource scenarios. xSID contains evaluation data for 13 languages from six language families, including a very low-resource dialect. It homogenizes annotation styles of two recent datasets (Schuster et al., 2019; Coucke et al., 2018) and provides the broadest public multilingual evaluation data for modern digital assistants.
Most previous efforts to multilingual SLU typically focus on translation or multilingual embeddings transfer. In this work, we propose an orthogonal approach, and study non-English auxiliary tasks for transfer. We hypothesize that jointly training on target language auxiliary tasks helps to learn properties of the target language while learning a related task simultaneously. We expect that this helps to refine the multilingual representations for better SLU transfer to a new language. We evaluate a broad range of auxiliary tasks not studied before in such combination, exploiting raw data, syntax in Universal Dependencies (UD) and parallel data.
Our contributions i) We provide xSID, a new cross-lingual SLU evaluation dataset covering Arabic (ar), Chinese (zh), Danish (da), Dutch (nl), English (en), German (de), Indonesian (id), Italian (it), Japanese (ja), Kazakh (kk), Serbian (sr), Turkish (tr) and an Austro-Bavarian German dialect, South Tyrolean (de-st). ii) We experiment with new
<sup>1</sup>The source code, dataset and predictions are available at: https://bitbucket.org/robvanderg/xsid
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2479-2497 June 6-11, 2021. ©2021 Association for Computational Linguistics
Dataset	Source	Langs.	Lang. Fams.	Domains2	Intents	Slots	#sents
Atis	Hemphill et al. (1990)	1	1	1	24	83	5,871
Snips	Coucke et al. (2018)	1	1	5	7	39	14,484
HWU64	Xingkun Liu and Rieser (2019)	1	1	22	64	54	25,716
Almawave-SLU	Bellomaria et al. (2019)	1	1	1	7	39	8,542
CSTOP†	Einolghozati et al. (2021)	2	1	2	19	10	5,800
Leyzer*	Sowański and Janicki (2020)	3	1	20	186	86	16,257
Facebook	Schuster et al. (2019)	3	2	3	12	11	57,049
multiAtis++	Xu et al. (2020)	9	4	1	23	83	45,046
xSID	This work	13	6	7	16	33	10,000
Table 1: Existing SLU datasets. Note that xSID is intended to serve as evaluation data only (Snips+Facebook can be used as English training data). †Code-switched data (Slanglish). *Automatically generated data. Language families are counted based on highest level of Glottolog (Hammarström and Nordhoff, 2011).
non-English auxiliary tasks for joint cross-lingual transfer on slots and intents: UD parsing, machine translation (MT), and masked language modeling. iii) We compare our proposed models to strong baselines, based on multilingual pre-trained language models mBERT (Devlin et al., 2019) and xlm-mlm-tlm-xnli15-1024 (Conneau et al., 2020) (henceforth XLM15), where the former was pre-trained on 12 of our 13 languages, and XLM15 on 5 of our 13 languages, thereby simulating a low-resource scenario. We also compare to a strong machine translation model (Qin et al., 2020).
The remainder of this paper is structured as follows: we start by giving an overview of existing datasets and introduce xSID (§ 2), then we discuss our baselines and proposed extensions (§ 3). After this, we discuss the performance of these models (§ 4), and provide an analysis (§ 5) before we end with the related work on cross-lingual SLU (§ 6) and the conclusion (§ 7).
2.1 Other SLU Datasets
An overview of existing datasets is shown in Table 1. It should be noted that we started the creation of xSID at the end of 2019, when less variety was available. We choose to use the Snips (Coucke et al., 2018) and Facebook (Schuster et al., 2019) data as a starting point.
Most existing datasets are English only (all datasets in Table 1 include English), and they differ in the domains they cover. For example, Atis (Hemphill et al., 1990) is focused on airline-related queries, CSTOP (Einolghozati et al., 2021)
contains queries about whether and devices, and other datasets cover multiple domains.
Extensions of Atis to new languages are a main direction. These include translations to Chinese (He et al., 2013), Italian (Bellomaria et al., 2019), Hindi and Turkish (Upadhyay et al., 2018) and very recently, the MultiAtis++ corpus (Xu et al., 2020) with 9 languages in 4 language families. To the best of our knowledge, this is the broadest publicly available SLU corpus to date in terms of the number of languages, yet the data itself is less varied. Almost simultaneously, Schuster et al. (2019) provide a dataset for three new topics (alarm, reminder, weather) in three languages (English, Spanish and Thai). English utterances for a given intent were first solicited from the crowd, translated into two languages (Spanish and Thai), and manually annotated for slots. We follow these approaches, but depart from the Snips (Coucke et al., 2018) and Facebook (Schuster et al., 2019) datasets to create a more varied resource covering 13 languages, while homogenizing the annotations.
xSID is a cross-lingual SLU evaluation dataset covering 13 languages from six language families with English training data. In what follows, we provide details on the creation of xSID (§ 2.2), including homogenization of annotation guidelines and English source training data (§ 2.3). For data statement and guidelines, we refer the reader to Section E, F and G in the Appendix.
2.2 xSID
As a starting point, we extract 400 random English utterances from the Snips data (Coucke et al., 2018) as well as 400 from the Facebook data (Schuster et al., 2019), which for both consist of 250 utterances from the test-split and 150 from the dev-split. We maintain the splits from the original data in
2The notion of domain is ill-defined within the scope of this task. We report the numbers from the paper, and, for Snips, we have identified the following: alarm, reminder, weather, restaurant, creative works.
Lang.	Language Family	Annotation
ar	Afro-Asiatic	Silly Movie 2.0 Silly Movie 2.0 Silly Movie 2.0 i biografen
da	Indo-European	Jeg vilGPCne se spilletiderne for Silly Movie 2.0 im Kino siehen
de	Indo-European	Ich wurdeGPCne den Vorstellungsbeginn für Silly Movie 2.0 im Kino siehen
de-st	Indo-European	I mecht es Programm fir Silly Movie 2.0 in Film Haus sechn
en	Indo-European	I'd like to see the showtimes for Silly Movie 2.0 at the movie house
id	Austronesian	Saya ingin melihat jam tayang untuk Silly Movie 2.0 di gedung bioskop
it	Indo-European	Mi piacerebbe vedere gli orari degli spettacoli per Silly Movie 2.0 al cinema
ja	Japonic	映画館の Silly Movie 2.0の上映時間を見せて。
kk	Turkic	MEN Silly Movie 2.0 SaçanapЛamacьнин Кинотацда керсетлим укытbing Кергим келдì
nl	Indo-European	Ik wil graag de speeltijden van Silly Movie 2.0 in het filmhuis zien
sr	Indo-European	Želela bih da vidim raspored pri Kazakhstanja za Silly Movie 2.0 u bioskopu
tr	Turkic	Silly Movie 2.0'in sinema salonundaki seanslarıngörmek istiyorum
zh	Sino-Tibetan	我想看 Silly Movie 2.0在影院的放映
Table 2: Examples of annotation for all languages in our dataset with intent: SearchScreeningEvent, and two slots: movie_name and object_location_type. Includes information on language families from Glottolog (Hammarström and Nordhoff, 2011).
xSID (i.e. sentences in xSID test are from Snips test or Facebook test). We then translate this sample into all of our target languages. It should be noted that some duplicates occur in the random sample of the Facebook data. Since these instances naturally occur more often, we decided to retain them to give a higher weight to common queries in the final evaluation.<sup>3</sup>
xSID includes Arabic (ar), Chinese (zh), Danish (da), Dutch (nl), English (en), German (de), Indonesian (id), Italian (it), Japanese (ja), Kazakh (kk), Serbian (sr), Turkish (tr) and an Austro-Bavarian German dialect, South Tyrolean (de-st).<sup>4</sup> We have 13 evaluation languages with 800 sentences per language<sup>5</sup> resulting in a final dataset of 10,000 sentences. The language selection is based on availability of translators/annotators (most of them are co-authors of this paper, i.e. highly-educated with a background in NLP). We favor this setup over crowd-sourcing, i.e. quality and breadth in annotation and languages, and because for some languages crowd-sourcing is not an option.<sup>6</sup> For more information on the data and annotators we refer to the dataset statement in Appendix E.
The first step of the dataset creation was the translation. For this, the goal was to provide a fluent translation which was as close as possible to the original meaning. Because the data consists of simple, short utterances, we consider our annotator pool to be adequate for this task (even though they are not professional translators). The intents could easily be transferred from the English data, but the slots needed to be re-annotated, which was done by the same annotators.
Unfortunately, we were unable to retrieve annotation guidelines from the earlier efforts. Hence, as a first step of and as part of training, we derived annotation guidelines by jointly re-annotating dev and test portions of the English parts of the two data sources. These guidelines were revised multiple times in the process to derive the final guidelines for the whole dataset. Ultimately, the data collection process proceeded in two steps: translation of the data from English, and slot annotation in the target language. The aim of the guidelines was to generalize labels to make them more broadly applicable to other intent subtypes, and remove within-corpus annotation variation (see Appendix G for details). We calculated inter-annotator agreement for the guidelines; three annotators native in Dutch annotated 100 samples, and reached a Fleiss Kappa (Fleiss, 1971) score of 0.924, which is very high agreement. Common mistakes included annotation of question words, inclusion of locations in reminders, and the inclusion of function words in the spans. We updated the guidelines after the agreement study.
This decision has been made after discussion with a real-world digital assistant team.
4The dialect is spoken by roughly 450,000 speakers in an Alpine province in Northern Italy. It has no official ISO language code nor a normed writing form.
5Except for Japanese where we only have the Facebook data.
<sup>6</sup>We did not have access to native speakers of Thai and Spanish part of the Facebook data (Schuster et al., 2019), which is why there are not included (yet).
After these target phase annotation rounds, we finalized the guidelines, which are provided in the Appendix G and form the basis for the provided data. Table 2 provides an example annotation for all 13 languages for the example sentence "I'd like to see the showtimes for Silly Movie 2.0 at the movie house". These example translations illustrate not only the differences in scripts, but also differences in word order and length of spans, confirming the distances between the languages.
2.3 English Training Data
Because of our revised guidelines for the Facebook data and mismatches in granularity of labels between the Snips and Facebook data, we homogenize the original training data for both sources and include it in our release. For the Facebook data, this includes rule-based fixing of spans and recognition of the REFERENCE and RECURRING_TIME labels. For the Snips data, we convert a variety of labels that describe a location to the LOCATION label which is used in the Facebook data, and labels describing a point or range in time to DATETIME. After this process, we simply concatenate both resulting datasets, and shuffle them before training. The resulting training data has 43,605 sentences.
Our main hypothesis is that we can improve zero-shot transfer with target-language auxiliary tasks. We hypothesize that this will help the multilingual pre-trained base model to learn peculiarities about the target language, while it is learning the target task as well. To this end, we use three (sets of) tasks with a varying degree of complexity and availability: 1) Masked Language Modeling (MLM): which is in spirit similar to pre-training on another domain (Gururangan et al., 2020), however, we learn this jointly with the target task to avoid catastrophic forgetting (McCloskey and Cohen, 1989); 2) Neural Machine Translation (NMT): where we learn English SLU as well as translation from English to the target language; and 3) Universal Dependency (UD) parsing: to insert linguistic knowledge into the shared parameter space to learn from syntax as auxiliary task besides learning the SLU task.
In the following subsections, we first describe the implementation of our baseline model, and the machine translation-based model, and then de
Figure 2: Overview of the baseline model.
scribe the implementation of all auxiliary tasks (and the data used to train them). Auxiliary tasks are sorted by dataset availability (MLM  $\succ$  NMT  $\succ$  UD), where the first type can be used with any raw text, the second one needs parallel data - which is readily available for many languages as a byproduct of multilingual data sources - and the last one requires explicit human annotation. For South Tyrolean, a German dialect, no labeled target data of any sort is available; we use the German task data instead. We provide more details of data sources and sizes in Appendix B.
All our models are implemented in MaChAmp v0.2 (van der Goot et al., 2021), an AllenNLP-based (Gardner et al., 2018) multi-task learning toolkit. It uses contextual embeddings, and fine-tunes them during training. In the multi-task setup, the encoding is shared, and each task has its own decoder. For slot prediction, a greedy decoding with a softmax layer is used, for intents it uses a linear classification layer over the [CLS] token (see Figure 2). The data for each task is split in batches, and the batches are then shuffled. We use the default hyperparameters of MaChAmp for all experiments which were optimized on a wide variety of tasks (van der Goot et al., 2021). The following models are extensions of this baseline. In the NMT-transfer model ( $\S$  3.2), the training data is translated before passing it into the model. For the auxiliary models ( $\S$  3.3, 3.4 and 3.5), we simply add another decoder next to the intent and slot decoders. The losses are summed, and typically weighted (multiplied) by a factor which is given in
We also tried to use a CRF layer for slots which consistently led to lower performance.
<sup>9</sup>Hyperparameter settings used in experiments are reported in Appendix A.
7For more details on this procedure, we refer to scripts/0.fixOrigAnnotation.py in the repo.
corresponding subsections. We enable the proportional sampling option of MaChAmp (multinomial sampling  $\alpha = 0.5$ ) in all multi-task experiments, to avoid overfitting to the auxiliary task.
3.2 Neural Machine Translation with Attention (nmt-transfer)
For comparison, we trained a NMT model to translate the NLU training data into the target language, and map the annotations using attention. As opposed to most previous work using this method (Xu et al., 2020; He et al., 2013; Schuster et al., 2019), we opt for an open-source implementation and provide the scripts to rerun the experiments. More specifically, we use the Fairseq toolkit (Ott et al., 2019) implementation of the Transformer-based model (Vaswani et al., 2017) with default hyperparameters. Sentences were encoded using byte-pair encoding (BPE) (Sennrich et al., 2016), with a shared vocabulary of 32,000 tokens. At inference time, we set the beam size to 4, and extracted alignment scores to target tokens calculated from the attention weights matrix. These scores are used to align annotation labels to target language outputs; we map the label of each token to the highest scoring alignment target token. We convert the output to valid BIO tags: we use the label of the B for the whole span, and an I following an O is converted to a B.
Data To ensure that our machine translation data is suitable for the target domain, we choose to use a combination of transcribed spoken parallel data. For languages included in the IWSLT 2016 Ted talks dataset (Cettolo et al., 2016), we use the train and development data included, and enlarge the training data with the training split from Opensubtitles $^{10}$  (Lison and Tiedemann, 2016), and Tatoeba (Tiedemann, 2012). For languages absent in IWSLT2016, we used the Opensubtitles data for training and Tatoeba as development set. For Kazakh, the Opensubtitles data only contains 2,000 sentences, so we concatenated out-of-domain data from the WMT2019 data (Barrault et al., 2019), consisting of English-Kazakh crawled corpora. We adapt the BertBasic tokenizer (which splits punctuation, it does not perform subword tokenization) to match the Facebook and Snips dataset tokenization and use this to pre-tokenize the data.
3.3 Masked Language Modeling (aux-mlm)
Previous work has shown that continuing to train a language model with an MLM objective on raw data close to the target domain leads to performance improvements (Gururangan et al., 2020). However, in our setup, task-specific training data and target data are from different languages. Therefore, in order to learn to combine the language and the task in a cross-lingual way, we train the model jointly with MLM and task-specific classification objective on target and training languages respectively. We apply the original BERT masking strategy and we do not include next sentence prediction following Liu et al. (2019a). For computational efficiency, we limit the number of input sentences to 100,000 and use a loss weight of 0.01 for MLM training.
Data For our masked language modeling objective, we use the target language machine translation data described above.
3.4 Machine Translation (aux-nmt)
To jointly learn to transfer linguistic knowledge from English to the target language together with the target task, we implement a NMT decoder based on the shared encoder. We use a sequence-to-sequence model (Sutskever et al., 2014) with a recurrent neural network decoder, which suits the auto-regressive nature of the machine translation tasks (Cho et al., 2014), and an attention mechanism to avoid compressing the whole source sentence into a fixed-length vector (Bahdanau et al., 2015). We found that fine-tuning the shared encoder achieves good performance on our machine translation datasets (Conneau and Lample, 2019; Clinchant et al., 2019), alleviating the need for freezing its parameters during training in order to avoid catastrophic forgetting (Imamura and Sumita, 2019; Goodfellow et al., 2014). Similar to MLM, we use 100,000 sentences, and a weight of 0.01.
Data For this auxiliary task, we use the same data as for NMT-TRANSFER, described in detail above.
3.5 Universal Dependencies (aux-ud)
Using syntax in hierarchical multi-task learning has previously shown to be beneficial (Hashimoto et al., 2017; Godwin et al., 2016). We here use full Universal Dependency (UD) parsing, i.e., part-of-speech (POS) tagging, lemmatization, morphological tagging and dependency parsing as joint
10http://www.opensubtitles.org/
mBERT lang2vec	en —	de-st —	de 0.18	da 0.18	nl 0.19	it 0.22	sr 0.23	id 0.24	ar 0.30	zh 0.33	kk 0.37	tr 0.38	ja* 0.41	Avg.
Slots
base	97.6	48.5	33.0	73.9	80.4	75.0	67.4	71.1	45.8	72.9	48.5	55.7	59.9	61.0
nmt-transfer	0.0	50.9	34.5	60.8	63.7	51.0	41.3	54.2	48.2	27.9	0.2	52.0	45.0	44.1
aux-mlm	97.3	53.0	34.6	75.9	82.2	78.0	63.8	69.5	48.1	69.4	51.3	58.4	63.5	62.3
aux-nmt	0.0	44.5	33.3	71.4	76.9	71.9	58.5	62.9	38.7	70.3	38.2	50.2	58.7	56.3
aux-ud	97.5	47.6	29.1	73.7	73.3	61.8	56.8	61.1	42.6	64.9	45.2	53.8	47.6	54.8
Intents
base	99.7	67.8	74.2	87.5	72.3	81.7	75.7	80.7	63.1	83.3	60.1	74.7	53.9	72.9
nmt-transfer	0.0	89.9	97.5	97.3	98.3	96.8	92.5	98.1	89.2	97.3	24.6	98.3	78.8	88.2
aux-mlm	99.5	66.3	75.4	80.7	73.5	80.1	65.4	72.2	59.7	78.3	47.6	62.2	42.9	67.0
aux-nmt	0.0	63.0	72.8	86.6	70.8	78.7	71.9	75.5	56.4	80.9	56.3	68.9	53.6	69.6
aux-ud	99.5	62.6	58.2	67.7	60.1	62.9	59.9	65.0	45.7	70.4	24.6	43.9	38.5	55.0
XLM15	en	de-st	de	da	nl	it	sr	id	ar	zh	kk	tr	ja*
Slots
base	97.0	39.4	33.3	26.3	30.9	27.3	15.9	14.9	49.1	57.6	10.9	45.5	33.4	32.1
nmt-transfer	0.0	43.0	34.7	59.5	61.9	49.4	39.7	53.0	47.2	28.4	0.0	50.5	34.7	41.8
aux-mlm	97.2	44.0	35.9	45.8	49.5	40.7	18.7	24.6	48.9	64.8	13.6	60.5	30.4	39.8
aux-nmt	0.0	32.3	32.3	26.3	28.0	24.1	12.4	13.7	38.0	29.4	7.2	33.1	16.6	24.4
aux-ud	97.0	46.0	34.6	36.3	45.4	45.3	22.0	21.6	45.1	52.5	13.1	50.1	33.2	37.1
Intents
base	99.7	61.3	78.5	56.3	45.4	48.0	41.4	36.4	67.5	78.8	29.9	67.3	39.1	54.1
nmt-transfer	0.0	80.6	97.6	97.6	97.7	96.6	92.4	96.7	88.4	97.1	16.7	97.9	61.7	85.1
aux-mlm	99.6	63.9	86.3	62.8	59.9	53.0	31.4	42.1	64.0	86.5	25.7	63.3	44.4	56.9
aux-nmt	0.0	52.0	52.0	60.7	44.7	44.7	40.3	43.1	54.3	54.1	21.0	53.2	25.2	45.4
aux-ud	99.5	62.6	72.2	47.2	42.3	52.5	33.6	31.8	45.7	57.1	30.3	51.1	35.5	46.8
Table 3: Results on slot labeling (in strict F1) and intent classification (in accuracy) on the development split of all 13 languages. Average over 5 seeds, standard deviations can be found in Appendix D. Sorted by language distance to en (de-st excluded), which is the cosine distance between the syntax, phonology and inventory vectors of lang2vec (Littell et al., 2017). Bold languages are included during pre-training. The last column (Avg.) is the average of all cross-lingual experiments, i.e. without English. *For Japanese we have 50% less evaluation data.