SlowGuess/NAACL_2021_OCR huggingface dataset:
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1297-1303, Hong Kong, China. Association for Computational Linguistics.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier.
Benjamin Muller, Antonis Anastasopoulos, Benoit Sagot, and Djamé Seddah. 2021. When being unseen from mbert is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. English intermediate-task training improves zero-shot crosslingual transfer too. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 557-575, Suzhou, China. Association for Computational Linguistics.
Libo Qin, Minheng Ni, Yue Zhang, Wanxiang Che, Yangming Li, and Ting Liu. 2020. Multi-domain spoken language understanding using domain- and task-aware parameterization. arXiv:2004.14871 [cs]. ArXiv: 2004.14871.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3795-3805, Minneapolis, Minnesota. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of International Conference on Learning Representations (ICLR).
Marcin Sowański and Artur Janicki. 2020. Leyzer: A dataset for multilingual virtual assistants. In International Conference on Text, Speech, and Dialogue, pages 477-486. Springer.
Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102-107, Avignon, France. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Dennis Ulmer. 2021. deep-significance: Easy and better significance testing for deep neural networks. https://github.com/Kaleidophon/deep-significance.
Shyam Upadhyay, Manaal Faruqui, Gokhan Tur, Hakkani-Tur Dilek, and Larry Heck. 2018. (Almost) zero-shot cross-lingual spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6034–6038. IEEE.
Rob van der Goot, Ahmet Üstun, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176-197, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Pawel Swietojanski Xingkun Liu, Arash Eshghi and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In Proceedings of the Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS), Ortigua, Siracusa (SR), Italy. Springer.
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for crosslingual NLU. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5052-5063, Online. Association for Computational Linguistics.
Parameter	Value
Optimizer	Adam
β1, β2	0.9, 0.99
Dropout	0.3
Epochs	20
Batch size	32
Learning rate	0.001
Weight decay	0.01
LR scheduler	slanted triangular
Decay factor	0.38
Cut frac	0.2
Table 6: Hyperparameter setting used in the experiments
A Hyperparameter settings
Hyperparameters follow the default setting of MaChAmp 0.2 (van der Goot et al., 2021) with 20 epochs, as reported in Table 6.
B Auxiliary Datasets
Table 7 reports the data sources for the treebanks and the dataset sizes in number of words and sentences for both the treebanks and the parallel data.
C Scores on auxiliary tasks
Even though it was not our goal to improve the auxiliary tasks, performance on these can still be relevant to analyze whether there is any correlation to performance on the xSID tasks. In Table 8, we report the full results for all tasks. These are the scores the correlations of Figure 5 are based on.
D Standard Deviations
Standard deviations of our main results (Table 3) are shown in Table 9.
E xSID Data Statement
Following (Bender and Friedman, 2018), the following outlines the data statement for xSID:
A. CURRATION RATIONALE Collection of utterances intended to be used for digital assistants, generated by crowd-workers. We selected a random sample from two much larger sets (Coucke et al., 2018; Schuster et al., 2019) which we translated and annotated for slots and intents for the cross-lingual study of SLU.
B. LANGUAGE VARIETY The English data was created by native English speakers and all translations are translated by native speakers. We translated to the following languages according to the iso 639-3 codes: 'deu', 'jpn', 'tur', 'nld', 'ita', 'dan', 'arb', 'kaz', 'srp', 'eng', 'ind', 'cmn'. Southtyrolean does not have an iso 693-3 language code.
C. SPEAKER DEMOGRAPHIC The original data is generated by crowd-workers and their demographics are unknown.
D. ANNOTATOR DEMOGRAPHIC Translators and annotators are the same people. Their age ranges from 20 to 57, with the majority being below 30, almost all annotators have a background in NLP (except for Chinese, and one inter-annotator for Dutch). Most annotators are currently doing a PhD, whereas there is one postdoc and two faculty.
E. SPEECH SITUATION The original data is generated in June 2017 (Coucke et al., 2018) and probably in 2019 (Schuster et al., 2019). The crowd workers were tasked to type sentences as how they would ask them in spoken form to a digital assistant given a topic (intent).
F. TEXT CHARACTERISTICS The genre of the data is determined by the set of supported intents:
I. PROVENANCE APPENDIX The original datasets have been released with the following licenses:
Schuster et al. (2019): CC-BY-SA license.
Coucke et al. (2018): CC0 1.0 Universal
We use the CC-BY-SA license for our redistribution of the data.
Lang.	Universal Dependencies	Parallel data
Treebank	#sents	#words	#sents	#words
ar	UD_Arabic-PADT	6,075	191,869	22,666,885	122,580,047
da	UD_Danish-DDT	4,383	80,378	11,021,827	71,415,893
de	UD_German-GSD	13,814	259,194	14,325,270	99,354,451
en	UD_English-EWT	12,543	204,585	0	0
es	UD_Spanish-GSD	14,187	375,149	61,434,251	415,369,072
fr	UD_French-GSD	14,449	344,970	41,921,465	280,924,433
hi	UD_Hindi-HDTB	13,304	281,057	93,016	620,929
id	UD_Indonesian-GSD	4,477	97,531	5,370,460	30,758,822
it	UD_Italian-ISDT	13,121	257,616	26,344,624	180,169,211
ja	UD_Japanese-GSD	7,027	167,482	1,883,365	12,891,698
kk	UD_Kazakh-KTB	1,047	9,872	595,060	10,058,764
nl	UD_Dutch-LassySmall	5,787	75,080	28,835,007	196,968,670
pt	UD_Portuguese-GSD	9,664	238,714	33,375,963	218,626,646
sr	UD_Serbian-SET	3,328	74,259	22,319,620	133,297,245
th	UD_Thai-PUD	1,000	22,322	3,281,533	4,332,396
tr	UD_Turkish-IMST	3,664	36,822	45,788,547	229,132,015
zh	UD_Chinese-GSD	3,997	98,616	9,475,118	89,458,907
Table 7: Dataset sizes for auxiliary tasks, for our auxiliary setting we constrained the model to only use 10,000 sentences of the treebanks and 100,000 of the parallel data
Lang	ud(avg.)	mlm	aux-nmt	nmt-transfer	ud(avg.)	mlm	aux-nmt	nmt-transfer
mBERT	XLM15
ar	88.12	1.81	11.97	16.78	88.51	3.21	1.01	16.78
da	94.10	3.17	13.86	56.24	89.35	3.00	9.80	56.24
de	93.00	1.87	14.22	25.93	92.83	2.63	1.19	25.93
en	95.24	4.17	0.00	0.00	94.90	5.18	0.00	0.00
id	91.31	2.32	20.89	27.96	88.02	1.13	4.02	27.96
it	96.83	4.03	8.98	44.73	95.46	2.52	7.17	44.73
ja	97.97	2.91	4.53	10.08	97.12	2.79	1.53	10.08
kk	70.55	1.48	2.67	0.00	52.08	2.65	0.09	0.00
nl	93.84	3.81	11.66	53.43	90.14	2.59	9.50	53.43
sr	94.18	3.54	9.78	35.50	89.81	4.14	7.25	35.50
tr	83.04	1.71	9.31	14.45	79.41	2.87	0.87	14.45
zh	94.80	1.52	11.52	20.31	93.00	2.01	0.12	20.31
Table 8: Results on auxiliary tasks: for UD, we use the average over UPOS accuracy, lemma accuracy, morphological feature accuracy (all features as 1 label) and dependency LAS, for masked language modeling (mlx) we use perplexity, and the last two columns (nmt) are bleu scores.
mBERT lang2vec	en —	de-st —	de 0.18	da 0.18	nl 0.19	it 0.22	sr 0.23	id 0.24	ar 0.30	zh 0.33	kk 0.37	tr 0.38	ja* 0.41	Avg.
Slots
base	0.2	4.8	3.1	1.7	1.0	1.0	2.4	2.2	1.3	2.3	4.1	1.4	6.5	2.6
nmt-transfer	0.0	2.0	0.7	1.3	1.7	1.4	1.6	1.4	1.2	1.1	0.3	1.7	2.8	1.4
aux-mlm	0.4	1.8	0.9	1.9	1.3	0.9	2.0	2.3	3.0	4.0	2.0	3.1	3.1	2.2
aux-nmt	0.0	1.9	1.3	1.3	1.1	1.3	3.2	1.9	2.1	2.3	1.9	2.6	4.9	2.1
aux-ud	0.2	1.4	1.0	0.7	1.7	4.0	1.1	2.9	0.3	1.8	1.4	2.8	3.5	1.9
Intents
base	0.0	4.5	3.5	2.8	2.0	2.0	2.5	3.4	2.4	1.8	3.1	1.1	3.8	2.8
nmt-transfer	0.0	2.9	0.6	1.0	0.5	1.1	2.4	0.6	0.5	0.8	6.6	0.4	4.2	1.8
aux-mlm	0.2	2.6	2.6	2.0	2.6	4.2	4.6	7.2	3.4	2.3	4.5	5.9	8.9	4.2
aux-nmt	0.0	4.2	2.7	2.0	3.0	4.2	2.7	1.6	2.5	3.6	3.3	3.6	5.7	3.3
aux-ud	0.2	3.2	3.2	4.0	1.8	3.9	4.4	3.8	5.2	3.5	1.2	3.5	4.1	3.5
XLM15	en	de-st	de	da	nl	it	sr	id	ar	zh	kk	tr	ja*
Slots
base	0.3	2.3	1.5	1.7	1.7	2.0	3.7	0.9	1.2	4.0	1.1	2.9	3.4	2.2
nmt-transfer	0.0	4.3	1.0	2.0	0.9	1.0	1.6	1.8	1.1	2.7	0.0	1.5	19.5	3.1
aux-mlm	0.4	3.1	0.4	4.0	3.6	1.4	1.2	3.1	1.4	2.7	1.3	2.0	8.5	2.7
aux-nmt	0.0	4.1	4.1	2.2	2.0	1.6	2.4	0.8	3.0	17.5	3.9	1.9	5.2	4.1
aux-ud	0.3	2.4	0.5	3.5	3.7	1.8	3.9	4.1	1.6	1.8	3.0	2.2	4.9	2.8
Intents
base	0.2	5.3	2.0	6.1	6.0	6.7	4.5	2.8	3.1	5.4	10.0	3.2	7.0	5.2
nmt-transfer	0.0	3.9	1.2	0.7	1.0	0.8	2.4	1.0	0.4	1.0	6.4	0.5	8.3	2.3
aux-mlm	0.1	2.1	2.0	4.6	6.0	2.6	7.9	8.3	4.2	1.3	6.0	7.0	4.6	4.7
aux-nmt	0.0	5.1	5.1	2.0	1.6	3.1	4.8	3.0	5.3	13.9	10.1	3.7	11.5	5.8
aux-ud	0.3	1.6	2.7	3.9	4.0	4.6	4.4	5.0	4.4	4.1	4.6	5.8	3.2	4.0
Table 9: Standard deviation matching all results from our main results (Table 3)
F Translation Guidelines
We aim to provide a fluent translation which is as similar (in meaning) as possible to the original. In some cases translations naturally have more distance, i.e. '7 pm' might translate to '7 in the evening' for languages in which there is no equivalent for 'pm'. The goal is to obtain sentences as they could possibly be used in the target language. Some general guidelines:
- In general, named entities are not translated, with the exception of place names, like cities and countries. So names of playlists, persons etc. stay the same, and things mentioned between quotes as well. In languages where names are often transcribed differently (i.e. Serbian), this is done during annotation.
- In case of grammatical mistakes, they are kept (if possible) in the target translation.
- We keep capitalization and punctuation as in
the original data (if they exist in the target language).
- Abbreviations not common in fluent discourse are expanded (e.g., Wed  $\mapsto$  "mercoledi", meds  $\mapsto$  medicin), also words that do not exist in the target language are paraphrased: 'whatis the high tomorrow'  $\mapsto$  'whatis the maximum temperature tomorrow'.
- Some things can not be translated directly. For example, the phrase 'play me X' does not exist in many languages. E.g. 'me' might not be translated.
- Possessive determiners (e.g. “my”) should be preserved and translated whenever possible.
- For languages in which words are not separated by whitespace (i.e. Japanese and Chinese), we ask the translator to include whitespaces at word boundaries to simplify the annotation of the slots.
Figure 6: Example showing the annotation of a Dutch instance in Brat
G Annotation Guidelines
This section describes our annotation guidelines. The aim of these guidelines was to make annotations homogeneous across earlier efforts for which guidelines were not available. Two major changes compared to the original annotations include: i) to generalize labels to make them more broadly applicable to other intent subtypes (an example is the recurrent datetime event from the Facebook data (Schuster et al., 2019), which was originally only applied to reminders and not to alarms, as discussed below); ii) we drop annotations of nouns as slots which are directly inferrable from the intent label (e.g. the 'reminder/noun' label was only applicable to nouns, but it was sometimes expressed as a verb and hence annotations were missing; as they are already annotated in the sentence-level intent slots, we drop such obvious slots).
For the annotation, we use Brat (Stenetorp et al., 2012), and provided the annotators with the gold English annotation (see Figure 6). English annotation was conducted by three annotators who discussed and resolved any initial disagreements. For annotation of the other languages, annotators were instructed to follow the English annotation when possible to maintain consistency.
Because no annotation guidelines were released with the original data, we provide guidelines for our re-annotation of the slots below. Examples are shown in Figure 7.
Spans We exclude function words in the beginning of an NP or VP, like 'for', 'from' in the examples above. An exception is when it is in the middle of a span as contiguous slots are preferred, like in example 2. This is different from previous releases of the data (Schuster et al., 2019), where datetime included 'for', 'at', 'to' and 'on'. We decided to drop them to make the annotations more homogeneous across slot labels, while capturing the core ('head spans') of the slots.
When two words of the same type occur sequentially, we annotate them as one span. This happens both for datetime (example 2, [5 to 6 am]) as well as reference (example 6, [all my]). Furthermore, we keep the annotation on the word-level to simplify processing. If only a part of a word belongs to a label, we annotate the whole word with that label.
Slot labels After our adaptations of the original labels, we annotate the following labels:
- datetime: Indicating a date or a time. Only concrete times are annotated (not, 'until deleted', 'what time' or 'when'), and times relative to other events are included (e.g. 'after work', 'later'). Non-concrete times, like 'until deleted' (example 9) are excluded.
- recurringDatetime: a recurring event, can be used for alarms and reminders. This category prioritizes over datetime. Example: 'make alarm for [weekdays at 7 am]', if at least one recurringDatetime exists in an instance, all datetimes should be annotated as recurringDatetime (even if they are in different spans, see example 9).
- location: describes a location; can be a proper noun (like 'New York') or a nominal or adjective referring to a location ('my area', 'out (outside)'). If a location is part of a reminder item, it is annotated as reminder/to do instead.
- reference: modifies the scope of an alarm or reminder, usually 'my' or 'all' used in front of the word 'reminder(s)' or 'alarm(s)'. Multiple sequential references are annotated as one span ('cancel [all my] alarms').
- reminder/todo: the item that should be reminded, the word 'to' should be excluded. In special cases, we also apply this for alarms (see example 8).
- weather/attribute: A property that describes an aspect of the weather; e.g. 'cold', 'rain', 'temperature', 'severe'. Also includes weather-related items like 'coat' and 'umbrella' if used in relation to the weather.
Figure 7: Example annotation for the slots. Slot-spans are indicated by square brackets and their label is shown directly behind the span.
Figure 8: Example of sentence-final verb in German. Green: reminderTodo, Orange: datetime
Erinnere
anzurufen
7.1 Language-specific exceptions
German/Dutch sentence-final verbs and split phrases German and Dutch allow for sentence-final verbs. For example, reminder todo slots might end up being split in the German translation, as another phrase or slot might be in between. See for an example Figure 8. We annotate the resulting non-continuous span for 'call mum' as two separate reminderTodo entities.
Handling of difficulties in Arabic compound morphology We encountered three special morphological cases during translating and annotating Arabic samples, in which a word can be a compound of two segments:
1. Verbs can be prefixed by a preposition (e.g. to, for, by):
$$
\frac {1}{2} \times 1 = \frac {1}{2} \times 3 = \frac {1}{2}
$$
Remind me to buy milk
In this case, we decided to annotate the whole compound word  $\mathfrak{sl}$  (EN: to buy milk) as a reminder.
2. Nouns can be suffixed with a possessive determiner (e.g. my, our):
$$
\sum_ {i = 1} ^ {n} a _ {i}
$$
Turn off my alarm
In this case we decided to annotated the whole compound word  $\underline{\underline{S}}\dot{\underline{v}}\dot{\underline{u}}$  (EN: "my alarm") as
reference-part. This label was only used for Arabic, and is converted to reference during evaluation.
3. Singular and dual nouns can be written as one word without the need to modify it with the numbers one or two:
$$
\text {t w o s t a r s} \mapsto \dot {\cup} \dot {\cup}
$$
$$
\text {o n e s t a r} \mapsto \ddot {a} _ {\infty}
$$
Serbian orthographic transcription of foreign named entities The rules of standard Serbian spelling dictate that named entities from other languages be transcribed orthographically into Serbian. Since Serbian orthography is near-perfectly phonemic, this means that the transcription of named entities will be highly dependent on the source pronunciation (although not solely determined by it). As a result, different transcription rules apply to not only different types of named entities but also different source languages.
When translating to Serbian, we recognize three groups of named entities that require different rules for translation and annotation:
1. Common named entities, such as well-known place names, names of historical and public figures, as well as names of popular literary, visual, and musical works. Such named entities are simply translated into their established
Serbian equivalents.
2. Less common named entities, such as lesser-known place names or names of public figures. When translating these entities, we follow the pronunciation and transcription rules specific to the source language, which means that we have to look up the origin and native pronunciation of all unfamiliar entities. However, since many of these are unlikely to be found in Serbian texts or corpora, we keep the source transcription (in English) in square brackets for future reference. We annotate the source and its Serbian translation as two separate spans with the same label.
3. Names of songs, playlists, video games, and lesser-known films and TV programs. These were neither translated nor transcribed, but left as they are in the source text, as such named entities are not commonly transcribed and have no official translation in Serbian.