SlowGuess/NAACL_2022_OCR huggingface dataset:
Hyperparameters and Optimization. For training TOD-XLMR (§3.1), we select the effective batch size of 8. In target-language-specific intermediate training (§3.2), we fix the maximum sequence length to 256 subword tokens; for RS objectives, we limit the context and response to 128 tokens each. We train for 30 epochs in batches of size 16 for MLM/TLM, and 32 for RS. We search for the optimal learning rate among the following values:  $\{10^{-4}, 10^{-5}, 10^{-6}\}$ . We apply early stopping based on development set performance (patience: 3 epochs for MLM/TLM, 10 epochs for RS). In downstream fine-tuning, we train in batches of 6 (DST) and 24 instances (RR) with the initial learning rate fixed to  $5 \cdot 10^{-5}$ . We also apply early stopping (patience: 10 epochs) based on the development set performance, training maximally for 300 epochs in zero-shot setups, and for 15 epochs in target-language few-shot training. In all experiments, we use Adam (Kingma and Ba, 2015) as the optimization algorithm.
We now present and discuss the downstream cross-lingual transfer results on  $\mathrm{MULTI}^2\mathrm{WOZ}$  for DST and RR in two different transfer setups: zero-shot transfer and few-shot transfer.
5.1 Zero-Shot Transfer
Dialog State Tracking. Table 3 summarizes zero-shot cross-lingual transfer performance for DST. First, we note that the transfer performance of all models for all four target languages is extremely low, drastically lower than the reference English DST performance of TOD-XLMR, which stands at  $47.9\%$ . These massive performance drops, stemming from cross-lingual transfer are in line with findings from concurrent work (Ding et al., 2021; Zuo et al., 2021) and suggest that reliable cross-lingual transfer for DST is much more difficult to
Model	DE	AR	ZH	RU	Avg.
w/o intermediate specialization
XLM-R	1.41	1.15	1.35	1.40	1.33
TOD-XLMR	1.74	1.53	1.75	2.16	1.80
with conversational target-lang. specialization
MLM on Mono-CC	3.57	2.71	3.34	5.17	3.70
Bi-CC	3.66	2.17	2.73	3.73	3.07
Multi-CC	3.65	2.35	2.06	5.39	3.36
TLM on OS	7.80	2.43	3.95	6.03	5.05
TLM + RS-X on OS	7.84	3.12	4.14	6.13	5.31
TLM + RS-Mono on OS	7.67	2.85	4.47	6.57	5.39
Table 3: Performance of multilingual conversational models in zero-shot cross-lingual transfer for Dialog State Tracking (DST) on MULTI<sup>2</sup>WOZ, with joint goal accuracy (\%) as the evaluation metric. Reference English DST performance of TOD-XLMR:  $47.86\%$ .
achieve than for most other language understanding tasks (Hu et al., 2020; Ponti et al., 2020).
Despite low performance across the board, we do note a few emerging and consistent patterns. First, TOD-XLMR slightly but consistently outperforms the vanilla XLM-R, indicating that conversational English pretraining brings marginal gains. All of our proposed models from §3.2 (the lower part of Table 3) substantially outperform TOD-XLMR, proving that intermediate conversational specialization for the target language brings gains, irrespective of the training objective.
Expectedly, TLM and RS training on parallel OS data brings substantially larger gains than MLMing on flat monolingual target-language corpora (Mono-CC) or simple concatenations of corpora from two (Bi-CC) or more languages (Multi-CC). German and Arabic seem to benefit slightly more from the cross-lingual Response Selection training (RS-X), whereas for Chinese and Russian we obtain better results with the monolingual (target language) RS training (RS-Mono).
Response Retrieval. The results of zero-shot transfer for RR are summarized in Table 4. Compared to DST results, for the sake of brevity, we show the performance of only the stronger baseline (TOD-XLMR) and the best-performing variants with intermediate conversational target-language training (one for each objective type): MLM on Mono-CC, TLM on OS, and TLM + RS-Mono on OS. Similar to DST, TOD-XLMR exhibits a near-zero cross-lingual transfer performance for RR as well, across all target languages. In sharp contrast to DST results, however, conversational specializa
Model	DE	AR	ZH	RU	Avg.
w/o intermediate specialization
TOD-XLMR	3.3	2.9	1.9	2.7	2.7
with conversational target-lang. specialization
MLM on Mono-CC	22.9	25.5	24.5	33.4	26.6
TLM on OS	44.4	30.3	34.1	39.3	37.0
TLM + RS-Mono on OS	44.3	30.9	34.8	39.6	37.4
Table 4: Performance of multilingual conversational models in zero-shot cross-lingual transfer for Response Retrieval (RR) on  $\mathbf{MULTI}^2\mathbf{WOZ}$  with  $\mathrm{R}_{100}@1$  (\%) as the evaluation metric. Reference English RR performance of TOD-XLMR:  $64.75\%$
tion for the target language – with any of the three specialization objectives – massively improves the zero-shot cross-lingual transfer for RR. The gains are especially large for the models that employ the parallel OpenSubtitles corpus in intermediate specialization, with the monolingual (target language) Response Selection objective slightly improving over TLM training alone.
Given the parallel nature of MULTI $^2$ WOZ, we can directly compare transfer performance of both DST and RR across the four target languages. In both tasks, the best-performing models exhibit stronger performance (i.e., smaller performance drops compared to the English performance) for German and Russian than for Arabic and Chinese. This aligns well with the linguistic proximity of the target languages to English as the source language.
5.2 Few-Shot Transfer and Sample Efficiency
Next, we present the results of few-shot transfer experiments, where we additionally fine-tune the task-specific TOD model on a limited number of target-language dialogs from the development portion of MULTI $^2$ WOZ, after first fine-tuning it on the complete English training set from MultiWOZ (see §4). Few-shot cross-lingual transfer results, averaged across all four target languages, are summarized in Figure 1. The figure shows the performance for different sizes of the target-language training data (i.e., number of target-language shots, that is, percentage of the target-language development portion from MULTI $^2$ WOZ). Detailed per-language few-shot results are given in Table 5, for brevity only for TOD-XLMR and the best target-language-specialized model (TLM+RS-Mono on OS). We provide full per-language results for all specialized models from Figure 1 in the Appendix. The few-shot results unambiguously show that
the intermediate conversational specialization for the target language(s) drastically improves the target-language sample efficiency in the downstream few-shot transfer. The baseline TOD-XLMR – not exposed to any type of conversational pretraining for the target language(s) – exhibits substantially lower performance than all three models (MLM on Mono-CC, TLM on OS, and TLM+RS-Mono on OS) that underwent conversational intermediate training on respective target languages. This is evident even in the few-shot setups where the three models are fine-tuned on merely  $1\%$  (10 dialogs) or  $5\%$  (50 dialogs) of the MULTI²WOZ development data (after prior fine-tuning on the complete English task data from MultiWOZ).
As expected, the larger the number of task-specific (DST or RR) training instances in the target languages (50% and 100% setups), the closer the performance of the baseline TOD-XLMR gets to the best-performing target-language-specialized model – this is because the size of the in-language training data for the concrete task (DST or RR) becomes sufficient to compensate for the lack of conversational target-language intermediate training that the specialized models have been exposed to. The sample efficiency of the conversational target-language specialization is more pronounced for RR than for DST. This seems to be in line with the zero-shot transfer results (see Tables 3 and 4), where the specialized models displayed much larger cross-lingual transfer gains over TOD-XLMR on RR than on DST. We hypothesize that this is due to the intermediate specialization objectives (especially RS) being better aligned with the task-specific training objective of RR than that of DST.
TOD Datasets. Research in task-oriented dialog has been, for a long time, limited by the existence of only monolingual English datasets. While earlier datasets focused on a single domain (Henderson et al., 2014a,b; Wen et al., 2017), the focus shifted towards the more realistic multi-domain task-oriented dialogs with the creation of the MultiWOZ dataset (Budzianowski et al., 2018), which has been refined and improved in several iterations (Eric et al., 2020; Zang et al., 2020; Han et al., 2021). Due to the particularly high costs of creating TOD datasets (in comparison with other language understanding tasks) (Razumovskaia et al., 2021), only a handful of monolingual TOD datasets
Figure 1: Few-shot cross-lingual transfer results for Dialog State Tracking (left figure) and Response Retrieval (right figure), averaged across all four target languages (detailed per-language results available in the Appendix). Results shown for different sizes of the training data in the target-language (i.e., different number of shots):  $1\%$ ,  $5\%$ ,  $10\%$ ,  $50\%$  and  $100\%$  of the MULTI $^2$ WOZ development sets (of respective target languages).
Lang	Model	DST	RR
1%	5%	10%	50%	100%	1%	5%	10%	50%	100%
DE	TOD-XLMR	7.68	19.26	28.08	33.17	34.10	10.25	32.47	35.56	45.39	49.46
TLM+RS-Mono on OS	15.88	24.14	28.38	32.57	35.45	46.08	48.94	49.98	53.43	55.72
AR	TOD-XLMR	1.48	1.57	6.18	15.62	17.63	6.36	18.72	23.57	36.04	42.69
TLM+RS-Mono on OS	4.42	6.79	8.27	14.39	21.48	33.45	37.09	38.01	41.89	47.15
ZH	TOD-XLMR	8.63	12.55	16.40	23.45	25.49	15.69	31.10	33.22	41.97	48.14
TLM+RS-Mono on OS	11.63	14.90	17.97	22.81	28.84	38.45	43.71	45.27	48.50	51.81
RU	TOD-XLMR	4.34	21.89	30.01	37.58	37.61	8.90	31.31	34.51	43.33	47.45
TLM+RS-Mono on OS	13.74	17.44	18.63	24.33	29.15	41.97	45.44	46.02	49.90	53.16
Table 5: Per-language few-shot transfer performance (sample efficiency results) on DST and RR for the baseline TOD-XLMR and the best specialized model (TLM+RS-Mono on OS).
in languages other than English (Zhu et al., 2020) or bilingual TOD datasets have been created (Gunasekara et al., 2020; Lin et al., 2021). Mrksic et al. (2017b) were the first to translate 600 dialogs from the single-domain WOZ 2.0 (Mrksic et al., 2017a) to Italian and German. Concurrent work (Ding et al., 2021; Zuo et al., 2021), which we discuss in detail in §2.2 and compare thoroughly against our MULTI $^2$ WOZ, introduces the first multilingual multi-domain TOD datasets, created by translating portions of MultiWOZ to several languages.
Language Specialization and Cross-lingual
Transfer. Multilingual transformer-based models (e.g., mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020)) are pretrained on large general-purpose and massively multilingual corpora (over 100 languages). While this makes them versatile and widely applicable, it does lead to suboptimal representations for individual languages, a phenomenon commonly referred to as the "curse of multilinguality" (Conneau et al., 2020). There-
fore, one line of research focused on adapting (i.e., specializing) those models to particular languages (Lauscher et al., 2020; Pfeiffer et al., 2020). For example, Pfeiffer et al. (2020) propose a more computationally efficient approach for extending the model capacity for individual languages: this is done by augmenting the multilingual PrLM with language-specific adapter modules. Glavaš et al. (2020) perform language adaptation through additional intermediate masked language modeling in the target languages with filtered text corpora, demonstrating substantial gains in downstream zero-shot cross-lingual transfer for hate speech and abusive language detection tasks. In a similar vein, Moghe et al. (2021) carry out intermediate fin-tuning of multilingual PrLMs on parallel conversational datasets and demonstrate its effectiveness in zero-shot cross-lingual transfer for the DST task.
Lauscher et al. (2020) show that few-shot transfer, in which one additionally fine-tunes the PrLM on a few labeled task-specific target-language in-
stances leads to large improvements for many task- and-language combinations, and that labelling a few target-language examples is more viable than further LM-specialization for languages of interest under strict zero-shot conditions. This finding is also corroborated in our work for two TOD tasks.
To ensure full reproducibility of our results and further fuel research on multilingual TOD, we release the parameters of TOD-XLMR within the Huggingface repository as the first publicly available multilingual PrLM specialized for TOD. We also release our code and data and provide the annotation guidelines for manual post-editing and quality control utilized during the creation of MULTI<sup>2</sup>WOZ in the Appendix. This makes our approach completely transparent and fully reproducible. All resources developed as part of this work are publicly available at: https://github.com/umanlp/Multi2WOZ.
Task-oriented dialog (TOD) has predominantly focused on English, primarily due to the lack of robust TOD datasets in other languages (Razumovskaia et al., 2021), preventing systematic investigations of cross-lingual transfer methodologies in this crucial NLP application area. To address this gap, in this work, we have presented MULTI $^2$ WOZ – a robust multilingual multi-domain TOD dataset. MULTI $^2$ WOZ encompasses gold-standard dialogs in four languages (German, Arabic, Chinese, and Russian) that are directly comparable with development and test portions of the English MultiWOZ dataset, thus allowing for the most reliable and comparable estimates of cross-lingual transfer performance for TOD to date. Further, we presented a framework for multilingual conversational specialization of pretrained language models that facilitates cross-lingual transfer for downstream TOD tasks. Our experiments on MULTI $^2$ WOZ for two prominent TOD tasks – Dialog State Tracking and Response Retrieval – reveal that the cross-lingual transfer performance benefits from both (i) intermediate conversational specialization for the target language and (ii) few-shot cross-lingual transfer for the concrete downstream TOD task. Crucially, we show that our novel conversational specialization
for the target language leads to exceptional sample efficiency in downstream few-shot transfer.
In hope to steer and inspire future research on multilingual and cross-lingual TOD, we make MULTI $^2$ WOZ publicly available and will extend the resource to further languages from yet uncovered language families (e.g., Turkish).
The work of Goran Glavaš has been supported by the Multi2ConvAI project of MWK BadenWürttemberg. Simone Paolo Ponzetto has been supported by the JOIN-T 2 project of the Deutsche Forschungsgemeinschaft (DFG). Chia-Chien Hung has been supported by JOIN-T 2 (DFG) and Multi2ConvAI (MWK BW). The work of Anne Lauscher was funded by Multi2ConvAI and by the European Research Council (grant agreement No. 949944, INTEGRATOR). The work of Ivan Vulić has been supported by the ERC PoC Grant MultiConvAI (no. 957356) and a Huawei research donation to the University of Cambridge.
In this work, we have presented MULTI $^2$ WOZ, a robust multilingual multi-domain TOD dataset, and focused on the multilingual conversational specialization of pretrained language models. Although the scope of this manuscript does not allow for an in-depth discussion of the potential ethical issues associated with conversational artificial intelligence in general, we would still like to highlight the ethical sensitivity of this area of NLP research and emphasize some of the potential harms of conversational AI applications, which propagate to our work. For instance, issues may arise from unfair stereotypical biases encoded in general purpose (Lauscher et al., 2021) as well as in conversational (Barikeri et al., 2021) pretrained language models and from exclusion of the larger spectrum of (gender) identities (Lauscher et al., 2022). Furthermore, (pre)training as well as fine-tuning of large-scale PrLMs can be hazardous to the environment (Strubell et al., 2019): in this context, the task-agnostic intermediate conversational specialization for the target languages that we introduce, which allows for highly sample-efficient fine-tuning for various TOD tasks can be seen as a step in the positive direction, towards the reduction of the carbon footprint of neural dialog models.
$^{7}$ https://huggingface.co/umanlp/TOD-XLMR
Soumya Barikeri, Anne Lauscher, Ivan Vulic, and Goran Glavš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1941-1955, Online. Association for Computational Linguistics.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bosheng Ding, Junjie Hu, Lidong Bing, Sharifah Aljunied Mahani, Shafiq R. Joty, Luo Si, and Chunyan Miao. 2021. Globalwoz: Globalizing multiwoz to develop multilingual task-oriented dialogue systems. CoRR, abs/2110.07679.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association.
Goran Glavaš, Mladen Karan, and Ivan Vulić. 2020. XHate-999: Analyzing and detecting abusive language across domains and languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6350–6365, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, et al. 2020. Overview of the ninth dialog system technology challenge: Dstc9. arXiv preprint arXiv:2011.06486.
Ting Han, Ximing Liu, Ryuichi Takanabu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. 2021. Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 206-218. Springer.
Matthew Henderson, Inigo Casanueva, Nikola Mrksic, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulic. 2020. ConveRT: Efficient and accurate conversational representations from transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2161-2174, Online. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014a. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014b. The third dialog state tracking challenge. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 324-329.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5392-5404, Florence, Italy. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Chia-Chien Hung, Anne Lauscher, Simone Paolo Ponzetto, and Goran Glavas. 2022. DS-TOD: efficient domain specialization for task oriented dialog. Accepted for publication in Findings of the Association for Computational Linguistics: ACL 2022.
Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender. arXiv preprint arXiv:2202.11923.
Anne Lauscher, Tobias Lueken, and Goran Glavš. 2021. Sustainable modular debiasing of language models. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 4782-4797, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018, Online. Association for Computational Linguistics.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung. 2021. Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling. arXiv preprint arXiv:2106.02787.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portoroz, Slovenia. European Language Resources Association (ELRA).
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for